I built and hardened an API gateway handling 5M+ requests a day at 99.95% availability, using Resilience4j, Redis and Spring Cloud Gateway. A gateway is a strange piece of infrastructure to design for: it is the thinnest layer in the whole system and also the one where a mistake has the widest blast radius. Most of what I learned building it only showed up once real traffic hit it.
A gateway needs its own resilience posture
The first lesson was that the gateway's resilience is a separate problem from the resilience of the services behind it. The naive version wires one circuit breaker for “the gateway” as a whole. That is wrong, and wrong in a way that is worse than having no breaker: one slow downstream service trips a breaker that then blocks traffic to every other, perfectly healthy, downstream service behind the same gateway.
The fix is a circuit breaker per downstream dependency, so a single degraded service degrades only the requests that actually depend on it. Resilience4j is built around named instances for exactly this reason — the registry is keyed, and the intended unit is one breaker per dependency rather than one per application.
Bulkhead the thread pools too, per downstream
The second lesson was bulkheading concurrency per downstream rather than sharing one pool across all of them. Without that isolation, a slow dependency does not just serve its own requests slowly — it occupies the threads that requests to healthy dependencies are waiting on, and a local slowdown becomes a gateway-wide one.
This is the bulkhead pattern, and Resilience4j implements it two ways — a semaphore bound on concurrent calls, or a genuinely separate thread pool. The semaphore version is cheaper and sufficient when the caller is already asynchronous; the thread-pool version is what you want when a blocking call could otherwise pin a shared worker.
It is the kind of lesson that is easy to state and easy to skip, because it costs nothing until the day it does, and by then it is an incident rather than a design-review comment.
Caching at the gateway, and the stampede
Caching at the gateway layer with Redis looks like a free win and mostly is, with one sharp edge: cache stampedes. When a popular key expires under sustained load, every request that would have hit the cache now hits the origin at once, and a system that looked fine a second earlier takes a coordinated spike it was not sized for.
The fixes are straightforward once you have seen it happen. Jitter the TTLs so keys do not all expire in the same instant — SET takes the expiry inline, so there is no reason for every write to use the same round number. Only cache idempotent reads in the first place. And be deliberate about which variant of cache-aside you are implementing, because the failure behaviour on a cache miss under load is the part that decides whether the pattern helps or hurts.
None of that is the first thing anyone reaches for before it happens once.
Observability is a prerequisite, not a follow-up
The last lesson is not a pattern, it is an ordering constraint: observability has to exist before you need it, not get added after an incident. At 5M+ requests a day you cannot debug a problem by reading logs live — you need per-route error-rate and latency dashboards you can glance at during an incident, not build during one.
Two specifics that matter more than the choice of tool. Percentiles, not averages: an average latency across a fleet hides precisely the tail you are chasing, and Prometheus' guidance on histograms is worth reading before you trust a quantile you did not compute carefully. And the metrics have to be per route and per downstream, because the whole point of per-dependency breakers and bulkheads is that failures are now localised — a gateway-wide error rate averages away the signal your resilience design was built to produce.
Every resilience pattern above is only useful if you can actually see it working, or failing, in real time.
What makes these hard
None of these are exotic ideas. What makes them hard is that each one is invisible right up until the specific failure mode it protects against actually happens — and a gateway sitting in front of 5M+ requests a day eventually sees all of them.
If you are building or hardening a gateway of your own and want a second set of eyes on where it is still exposed, that is a good use of a consultancy session.