At Way.com I built and hardened an API gateway handling 5M+ requests a day, using Resilience4j, Redis, and Spring Cloud. A gateway is a strange piece of infrastructure to design for: it's the thinnest layer in the whole system, and also the one where a mistake has the widest blast radius. Most of what I learned building it only shows up once real traffic hits it.
The first lesson was that a gateway needs its own resilience posture, separate from the services behind it. The naive version wires one circuit breaker for "the gateway" as a whole. That's wrong — it means one slow downstream service trips a breaker that then blocks traffic to every other, perfectly healthy, downstream service behind the same gateway. The fix is a circuit breaker per downstream dependency, so a single degraded service degrades only the requests that actually depend on it.
The second was bulkheading thread pools per downstream, not sharing one pool across all of them. Without that isolation, a slow dependency doesn't just serve its own requests slowly — it occupies threads that requests to healthy dependencies are waiting on too, and a local slowdown turns into a gateway-wide one. This is the kind of lesson that's easy to state and easy to skip, because it costs nothing until the day it does, and by then it's an incident, not a design review comment.
Caching at the gateway layer with Redis looks like a free win and mostly is, with one sharp edge: cache stampedes. When a popular key expires under sustained load, every request that would have hit the cache now hits the origin at once, and a system that looked fine a second earlier gets hit with a coordinated spike it wasn't sized for. The fix is straightforward once you've seen it happen — jittered TTLs so keys don't all expire in the same instant, and only caching idempotent reads in the first place — but it's rarely the first thing anyone reaches for before it happens once.
The last lesson isn't a pattern, it's a prerequisite: observability has to exist before you need it, not get added after an incident. At 5M+ requests a day, you cannot debug a problem by reading logs live — you need per-route error-rate and latency dashboards you can glance at during an incident, not build during one. Every resilience pattern above is only useful if you can actually see it working, or failing, in real time.
None of these are exotic ideas. What makes them hard is that each one is invisible right up until the specific failure mode it protects against actually happens — and a gateway sitting in front of 5M+ requests a day eventually sees all of them. If you're building or hardening a gateway of your own and want a second set of eyes on where it's still exposed, that's a good use of a consultancy session.