Full-stack Developer
MarketLens
Automated market reporting platform
2025 — 2026·4 engineers
0M+
data points / day
2h → 5min
report generation time
0+
automated reports / month
Overview
A platform that ingests market analytics data from multiple external sources and generates business-ready reports with indicators, charts, tables, and narrative insights — fully automated from ingestion to export, deployed on AWS ECS.
Challenges
- Aggregating millions of daily data points without blocking user-facing APIs.
- Report generation was a multi-hour manual process prone to inconsistent numbers.
- Deployments needed to be repeatable across dev, staging, and production.
Solutions
- Designed RabbitMQ worker pipelines for aggregation, analysis, report rendering, retries, and failed-task recovery.
- Optimized PostgreSQL batch operations, transaction boundaries, and indexing for reporting queries; cached hot aggregates in Redis.
- Automated Docker-based deployments to AWS ECS with GitHub Actions.
Results
- Report generation went from ~2 hours of manual work to under 5 minutes, fully automated.
- Pipeline sustains 2M+ data points per day; 300+ reports generated monthly.
- Processing status, retry history, and export state exposed through internal admin APIs.
Architecture
Tech stack
- Node.js
- PostgreSQL
- Redis
- RabbitMQ
- Next.js
- AWS ECS
- GitHub Actions
- Docker
Process
- 01
Pipeline design
2 weeksDefined job boundaries for collection, analysis, and rendering.
- 02
Implementation
10 monthsAPIs, workers, data models, report views.
- 03
Operations
ContinuousECS deployment automation and monitoring.