Let’s Handle 1 Million Requests per Second, It’s Scarier Than You Think!

Summary

TL;DR: The creator builds, benchmarks, and scales a Node.js API to > 1 million HTTP requests per second, exposing the engineering trade‑offs, costs, and the need for low‑level optimisations (C++, Redis clustering, UUIDs).

Verdict: WATCH – the deep dive into real‑world high‑scale engineering, cost analysis, and step‑by‑step walkthrough offers valuable insights for anyone interested in performance engineering or cloud architecture.

Key Takeaways

• Scale target: Simulate one of the world’s busiest routes (~1 M RPS) using AWS EC2, hundreds of CPU cores, and terabytes/minute of traffic.
• CPU utilisation basics: Core utilisation = (active time / total time) × 100; CPU utilisation can be summed across cores (may exceed 100 %).
• Threading demo: A single‑threaded Node loop uses ~100 % of one core; spawning 12 threads on a 12‑core box pushes utilisation to ~1200 % (method 1).
• Framework overhead: Express ≈ 14 k RPS, Fastify ≈ 66 k RPS, custom “Cpeak” ≈ 73 k RPS on a 12‑core Mac Studio.
• Cluster mode with PM2: Running 12 Node instances on the same machine raised the simple route from ~8 k RPS to ~42 k RPS.
• Network bottleneck: On a 128‑core, 50 Gbps AWS instance the simple route hit 6 M RPS, but a 30 KB payload capped at ~100 k RPS due to NIC limits.
• Database limits: PostgreSQL on a high‑end instance delivered only ~35 k writes / s; raising IOPS and storage size improved it to ~66 k /s but costs surged past $15 k / month.
• Redis as cache: Moving writes to an in‑memory Redis store lifted throughput to ~100 k RPS (writes) and ~300 k RPS (reads) while cutting DB costs dramatically.
• Redis clustering: Deploying 30 Redis nodes (15 masters + 15 replicas) distributes load; each node still caps ~100‑200 k RPS, but combined they achieve the 1 M RPS goal.
• UUID optimisation: Switching from sequential IDs to crypto.randomUUID() (122‑bit) eliminates lock contention; birthday‑paradox analysis shows a collision is astronomically unlikely (≈ 86 k years at 1 M RPS).

Insights

1. Network speed can dominate even when CPU is idle – a 50 Gbps NIC limited the payload‑heavy route to ~100 k RPS.
2. Framework choice matters: a zero‑dependency custom HTTP server can be 5× faster than Express for high‑throughput APIs.
3. Cost escalates faster than performance: Doubling IOPS or adding more DB instances can raise monthly spend by thousands while only modestly improving RPS.
4. In‑memory stores plus background sync are a pragmatic compromise between latency and durability for massive write workloads.
5. UUID collision risk is negligible at web‑scale; using a 122‑bit random identifier removes the need for coordinated sequence generation.

Key Topics

• High‑scale HTTP benchmarking (autocannon)
• CPU core and thread utilisation
• Node.js vs C++ for ultra‑low latency
• Express, Fastify, and custom “Cpeak” framework comparison
• PM2 process clustering
• AWS EC2 instance selection & cost modeling
• PostgreSQL write/read bottlenecks & IOPS tuning
• Redis as an in‑memory database and clustering strategy
• UUID generation & birthday‑paradox analysis
• Real‑world cost vs performance trade‑offs

Key Moments

• 0:45 – Introduction & motivation (aiming for 1 M RPS).
• 2:10 – CPU utilisation formulas and single‑thread demo.
• 4:00 – Multi‑threading demo (12 threads → 1200 % utilisation).
• 6:30 – Autocannon basics and first benchmark (≈ 18 k RPS).
• 9:15 – Framework comparison: Express vs Fastify vs Cpeak.
• 12:40 – PM2 clustering on local machine (≈ 42 k RPS).
• 15:20 – Launching 128‑core AWS instances; cost discussion.
• 18:05 – Network bottleneck discovery (≈ 6 GB/s limit).
• 22:30 – PostgreSQL write benchmark (≈ 35 k RPS) and IOPS upgrade.
• 27:45 – Switching writes to Redis (≈ 100 k RPS).
• 31:10 – Redis clustering setup (30 nodes) and achieving 1 M RPS.
• 35:00 – UUID optimisation & birthday‑paradox explanation.

Notable Quotes

"A simple mistake in a high‑stake environment can cost a company millions of dollars in seconds."

Best For

Backend engineers, site‑reliability engineers, and cloud architects who need to understand the practical limits and cost implications of scaling APIs to millions of requests per second.

Action Items

• Profile your own API to identify CPU, network, or DB bottlenecks.
• Replace heavy frameworks with minimal‑overhead servers for high‑throughput endpoints.
• Use PM2 (or similar) to run multiple Node instances and fully utilise all cores.
• Offload hot‑path writes to Redis and schedule asynchronous batch syncs to your durable database.
• Consider Redis clustering when a single node cannot meet your RPS target.
• Evaluate the cost‑benefit of scaling compute vs. optimizing data paths (e.g., in‑memory caches, UUIDs).