The problem
Inventory writes were arriving from multiple upstream services concurrently — webhook callbacks, batch ingests, manufacturer onboardings. Without coordination, two writes for the same SKU could interleave through downstream Elasticsearch and specsheet steps, leaving the system in a partially updated state.
Architecture
The dashed red arrow is the compensating-transaction path. If a downstream step (specsheet attach, ES index, event emit) fails, prior side effects are reverted before the SKU lock releases — never just abandoned.
Design
Saga with compensating transactions
Each ingestion is a saga of discrete steps: validate → write to DynamoDB → index in Elasticsearch → attach specsheet → emit downstream events. If any step fails, prior steps are compensated — not just rolled back, compensated — because some of them have already published events to other services.
Per-SKU (per-aggregate) locks, not table locks
The contended resource is the SKU, not the table. Locking the table would serialize all 35 writes-per-second through a single mutex. Per-SKU locks let independent writes run in parallel and only serialize when the same SKU is being modified — which is the actual race we care about.
Locks live in Redis with a TTL ceiling so a crashed worker can't hold a SKU forever. The saga acquires the lock as its first step and releases on completion or compensation.
What it ran in 12 months
All numbers below are organic ingestion only — they exclude the initial seed that bootstrapped the table.
- 3.91M product writes across 198 active days
- ~7,800 distinct manufacturers (up from a handful at seed time)
- Sustained throughput: ~35 writes/sec average; 100+ writes/sec peak
- Largest single-day burst: 407K writes during the first major customer onboarding
- Second-largest single-day burst: 216K writes
- Onboarded 2,500+ new manufacturers in a single month — over 30× the prior month's volume
Steady-state growth
In organic steady state the system absorbed 25–30% month-over-month volume growth without any throughput re-tuning — a clean validation that the saga + per-SKU lock design holds under compounding load.
The dramatic events were the customer-onboarding burst days, not the steady curve: the Nov-style 407K-write day was an order of magnitude above sustained throughput, and the system absorbed it with no duplicate-row incidents and no manual intervention. That's the proof point — not the curve, the bursts.
Why this scales linearly
Saga + per-SKU lock are the unit of horizontal scaling. Adding workers doesn't change the per-SKU contention, so latency and cost scale linearly with throughput, not super-linearly. The biggest customer-onboarding burst ran clean on the same primitives the steady curve runs on — no special-cased path, no scramble.
What I'd add next
- p50 / p99 ingestion latency on the API endpoints
- Lock contention rate (% of writes that waited)
- Saga rollback rate (% of ingestions that hit a compensating step)
- Cost per write before vs. after the saga + lock implementation
Stack
- Django REST Framework
- DynamoDB (us-east-2) for the inventory table
- Redis for distributed locks
- Elasticsearch for the search-side projection
- CloudWatch + Sentry for observability