Architecting for Chaos: Scaling Heroku APIs, Dynos, and Redis to Handle 100M+ Data & Real-Time Mobile Traffic

1. The Dual-Front Battle: Massive Data Meets Massive Traffic

When a platform processes hundreds of millions of records each day while serving thousands of real-time mobile and web API calls, the infrastructure must not only work; it must survive failure and scale through it.

The platform had two non-negotiable missions: ingest and process 100M+ lead records from CRMs, partner APIs, and business datasets; and serve thousands of concurrent clients across U.S. ZIP codes in real time.

• 30-second web request timeouts.
• 2 GB memory cap per dyno.
• Redis object-size limits.
• Limited horizontal scaling before costs increased materially.

2. Designing for Chaos: The Split Architecture

The implementation separated the platform into two planes of control:

• Web dynos handled API requests from React web and Flutter mobile applications.
• Worker dynos handled ingestion, enrichment, and distribution in the background.

This ensured that ingestion jobs could not block customer-facing APIs.

Data Sources (CRMs / APIs / Files)
        ↓
FastAPI Ingestion Layer
        ↓
Redis Queue (RQ) — job orchestration
        ↓
Worker Dynos (10 concurrent)
        ↓
PostgreSQL (partitioned tables)
        ↓
APIs to React + Flutter Apps

3. Redis: The Beating Heart of the System

Redis was not used only as a cache. It became the central nervous system of ingestion and API health.

• RQ handled background ingestion and transformation jobs.
• Priority queues ensured SLA-sensitive data moved first.
• Job metadata enabled real-time progress visibility for operators and mobile users.
• Failed jobs retried up to five times with exponential backoff.
• Datasets above 500 MB were split into 200 MB chunks to avoid SSL and memory failures.

Redis ensured that work was not lost when dynos restarted mid-execution.

4. Horizontal Scaling with Heroku Dynos

• 10 worker dynos managed ingestion and data enrichment.
• 3 web dynos supported API traffic and app interaction.
• Autoscaling rules used Redis queue depth and response latency.

Each dyno ran lightweight, modular services for failure containment. When one dyno fell, another resumed work from Redis job metadata; when traffic spiked, new dynos balanced the queues.

5. Keeping Mobile and Web Clients Real-Time

• WebSockets handled bi-directional updates for leads, route changes, and status synchronization.
• Streaming APIs delivered large lead lists progressively rather than freezing the UI.
• Geo-partitioned queries fetched only an agent’s assigned ZIP codes.
• A Redis cache served hot data in under 100 ms, avoiding repetitive database lookups.

6. Surviving Heroku’s Limits

Operational controls addressed recurring dyno restarts, memory leaks and OOM kills, the 30-second HTTP timeout, and Redis SSL errors caused by oversized payloads.

• Long-running work moved to background jobs.
• Chunked loading kept requests lightweight.
• Heartbeat and auto-recovery scripts detected failed workers.
• Circuit breakers reduced cascading failures during Redis outages.

7. Observability and Enterprise Lessons

The team implemented live queue dashboards, p95/p99 API latency monitoring, alerting for retries, connection failures and dyno restarts, plus operational kill-switches to pause, retry, or drain queues safely in production.

1. Split ingestion and API workloads early.
2. Let queues manage chaos; Redis becomes the control tower.
3. Architect for graceful failure, not perfect uptime.
4. Build observability into the platform from day one.

Originally published by Prestanda Consulting on Medium. View the original Medium publication.