Backend Developer
Learning Insights Pipeline
Realtime learning analytics for education
2026 — Present·3 engineers
0M+
events ingested / day
<0s
dashboard lag
0k
students served
Overview
A realtime analytics pipeline that ingests learner behavior events — lesson views, quiz attempts, agent conversations — and turns them into live dashboards for teachers and curriculum teams: funnels, retention, struggle detection.
Challenges
- Ingesting spiky event traffic (evening study peaks) without dropping data.
- Dashboards had to reflect activity within seconds, not nightly batches.
- Aggregations across 20k students strained naive queries.
Solutions
- Buffered ingestion through RabbitMQ with idempotent consumers and dead-letter recovery.
- Maintained incremental aggregates in PostgreSQL updated per event batch, with Redis serving hot dashboard reads.
- Partitioned event tables by week and pre-computed funnel/retention rollups.
Results
- Sustained 1M+ events/day with zero-loss ingestion during peak spikes.
- Dashboard freshness under 2 seconds end-to-end.
- Struggle-detection alerts adopted by teachers across 20k students.
Architecture
Tech stack
- Node.js
- RabbitMQ
- PostgreSQL
- Redis
- Docker
- AWS
Process
- 01
Event schema design
1 weekDefined event taxonomy and aggregate models.
- 02
Pipeline build
3 monthsIngestion, consumers, rollups, dashboard APIs.
- 03
Scale tuning
ContinuousPartitioning, batching, cache strategy.