The problem
Each team had spun up its own custom compiler — different conventions, different validation rules, different storage paths. Every new client meant a 3-day engineering onboarding. The compilers diverged faster than they could be maintained, and bugs in one didn't get fixed in the others.
I built one Proteus service to replace them. Heterogeneous in, structured out, all driven by a JSONPath-style configuration that ops could edit without redeploying.
Architecture
FastAPI microservice with a Factory Pattern
- Modular pipeline stages: parse → transform → validate → sanitize → write
- Each stage looked up at runtime from a registry — adding a new transformer is a class, not a service
- Reusable across clients with different schemas because the pipeline itself is data, not code
JSONPath-driven configuration
- Source-side selectors written in JSONPath, target-side schemas declared inline
- Config-level sanitization hooks (
sanitizers.py) for currency conversion, normalization, and enrichment lookups - Configs and validation schemas live in MongoDB with in-memory container caching — hot reloads without redeployment
Dual-queue SQS ingestion + DLQ
- Primary queue for the standard compilation flow
- Re-compilation queue for replaying missed or failed data after a config fix
- Dead Letter Queue with 15-day retention — safe recovery and bulk replay of failed events without losing the input
Autoscaling on AWS ECS
Tasks scaled dynamically on three signals — queue depth, incoming message rate, and container I/O utilization. The third one mattered most: queue-depth-only scaling overshoots when a small number of large messages stall a worker. Adding I/O as a scaling signal was what kept the p99 stable when message-size distribution shifted.
Multi-sink writes
- Processed outputs land in MongoDB, BigQuery, or S3 — sink chosen per-config
- Full backup copy to S3 regardless of primary sink
- Write-ahead logs retained for 48 hours, enabling recovery during downstream storage failures
What it ran in production
- 100 GB+ data daily on average; 200 GB/day peak
- 100+ concurrent parallel processing tasks
- 99.95% availability across the 28-month tenure
- New-client onboarding cut from 3 days → under 4 hours by standardizing ingestion + compilation + validation + storage in one service
Why the design held up
1. Config as data, not code
Everything client-specific lives in MongoDB — schema, JSONPath selectors, sanitizers, sink choice. Adding a client is a config commit, not a deploy. That's the only reason 3 days → 4 hours was achievable; if onboarding had stayed in code we'd have been bottlenecked on review and deploy cycles.
2. Re-compilation queue is a separate queue
Mixing replay traffic with live ingestion in one queue is how you turn a config fix into a production incident — replays compete with real-time data for worker capacity. Splitting the queue means replays drain at their own rate without affecting current SLAs.
3. WAL + S3 backup before the primary sink
Downstream storage will fail. Writing the WAL and S3 backup before the primary sink means a MongoDB or BigQuery outage doesn't lose the message — when the downstream comes back, the WAL replays. 48 hours was chosen as the window where a human is reliably going to have noticed an outage.
Stack
- Python, FastAPI
- AWS SQS (primary + recompile + DLQ), ECS with autoscaling
- MongoDB (configs + outputs), BigQuery (analytics outputs), S3 (backup + outputs)
- Docker, Jenkins CI/CD