Nothing achieves literal 100% uptime — anyone who promises you that is selling something. What is achievable, and what this is about, is getting close enough that a downstream failure degrades your system instead of cascading through it, and doing that with observability good enough that you know within seconds which state you are in, not minutes.
This is commerce-platform work: a Resilience4j resiliency framework in the core API server that took uptime from 99.5% to 99.95%, a custom Azure API gateway that brought P99 latency down 35%, and monitoring dashboards on Azure Data Explorer, Prometheus, Thanos and Grafana that took mean time to detect from 15 minutes to under 3. Four layers — a circuit breaker, an autoscaler that reacts to the breaker's state rather than just CPU, a metrics pipeline that survives a single Prometheus instance falling over, and platform telemetry the application cannot see.
The three circuit breaker states, and the knob that matters for each
Resilience4j's circuit breaker is a state machine: closed (calls flow normally, failures are counted), open (calls fail fast without touching the downstream at all), half-open (a small number of trial calls decide whether to close again or re-open). The concept is not the part people get wrong — it is the sliding window and threshold configuration that decides how twitchy or how slow to react the breaker actually is:
resilience4j.circuitbreaker:
instances:
catalogService:
registerHealthIndicator: true
slidingWindowType: TIME_BASED
slidingWindowSize: 30 # seconds
minimumNumberOfCalls: 10 # don't trip on 2 failures out of 2 calls
failureRateThreshold: 50 # % of calls that must fail to trip
waitDurationInOpenState: 15s # how long before trying half-open
permittedNumberOfCallsInHalfOpenState: 5
slowCallDurationThreshold: 2s # a "slow" call counts toward the failure rate too
slowCallRateThreshold: 50
minimumNumberOfCalls is the setting most configs skip, and it is the one that stops a low-traffic service tripping its breaker on two unlucky failures out of two total calls. slowCallDurationThreshold matters as much as outright failures: a dependency that is alive but degraded to five-second responses is functionally down for anything with a real SLA, and a breaker that only counts hard failures will happily keep feeding traffic into it.
Worth reading the pattern description alongside the library docs, because it is explicit about the thing the config alone does not tell you: a breaker is not a retry, and putting a retry inside a breaker with a short window is how you trip it on your own retries.
Pair the breaker with a bulkhead — cap concurrent calls to that dependency independently of the breaker's state — and a fallback that returns something genuinely useful, a cached last-known-good value rather than a bare exception, for anything on the user-facing path.
Wiring breaker state into autoscaling, not just CPU
A HorizontalPodAutoscaler scaling on CPU alone misses the signal you actually want. Pods burning CPU while their breakers are open, uselessly failing fast, are not a problem more replicas solve. The problem is upstream request volume outpacing capacity while the breaker is closed and dependencies are healthy.
Resilience4j already exports breaker-state and call-count metrics through Micrometer; scrape those with Prometheus and expose them to the HPA through its custom metrics API:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: core-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: core-api
minReplicas: 3
maxReplicas: 20
metrics:
- type: Pods
pods:
metric:
name: resilience4j_circuitbreaker_calls_permitted_rate
target:
type: AverageValue
averageValue: "50"
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 75
Scaling on permitted-call throughput rather than raw CPU means the HPA reacts to what is actually saturating — request volume the breaker is letting through — instead of a proxy that can lag or mislead under GC pressure or JIT warm-up. Two operational cautions from the HPA docs worth internalising: custom metrics require an adapter serving the custom metrics API, and the stabilisation window exists because a metric that oscillates will otherwise produce scaling that oscillates with it.
Memory sizing deserves its own honesty. Do not set a pod's memory limit from a load test's average — set it from the real p99 under a sustained failure scenario, with the breaker open, fallback logic executing and retries queued, because that is when memory pressure peaks rather than during the happy path a load test usually emphasises. An undersized limit under exactly the failure condition you built the breaker to survive is how a resiliency pattern gets undone by an OOMKill.
Why Prometheus alone is not enough, and what Thanos buys
A single Prometheus instance is both your metrics store and a single point of failure for your metrics store. It falls over and you are blind exactly when you need visibility most. Its own storage documentation is upfront that local storage is not clustered or replicated, and points at remote-write integrations for durability — which is the gap Thanos fills.
Thanos sits alongside Prometheus rather than replacing it. A Sidecar ships each instance's blocks to object storage for durable long-term retention, and a Querier gives one query interface across every cluster's Prometheus instances plus that long-term store. That second part is what matters once you run more than one cluster or region, because “what is the breaker state across all regions right now” stops being answerable from any single Prometheus.
The practical payoff, and the reason MTTD moved from 15 minutes to under 3: a Grafana dashboard built against the Querier keeps working when an individual Prometheus restarts, which happens routinely — config reloads, pod evictions, node maintenance. And you get real historical trend queries, breaker trip frequency over the last 90 days rather than the last 15, without provisioning one Prometheus with an unreasonably large local disk.
Azure Monitor: the platform signal Resilience4j cannot see
Resilience4j's metrics are entirely application-level. They know your code's view of “did this call succeed” and nothing about the platform underneath it. Azure Monitor on AKS fills that gap: node-level CPU pressure, control-plane health, network throttling, managed-database connection-pool saturation — all of which can cause exactly the symptoms a circuit breaker reacts to, timeouts and slow calls, without the downstream service being unhealthy at all.
The move that pays off is correlating the two rather than treating them as separate dashboards nobody cross-references. Pull breaker-state metrics into the same Grafana view as the node and network metrics, so “the catalog breaker just opened” and “the underlying node hit CPU throttling three seconds earlier” appear on one timeline. That is the difference between debugging an application incident and correctly identifying an infrastructure one — and between “restart the pod” and “cordon the node,” which are not the same fix.
What “close to 100%” actually looks like
Not a single breaker config and not a single autoscaling rule, but a chain where each layer covers what the one below it cannot: the breaker stops a slow dependency cascading into your own service's threads and memory, the HPA reacts to the breaker's real signal instead of a lagging CPU proxy, Thanos keeps visibility into all of it alive across restarts and clusters, and platform telemetry tells you when the problem was never the dependency graph at all.
Skip any one layer and the others compensate imperfectly. Run all four and most failures degrade gracefully instead of cascading — which is the difference between 99.5% and 99.95%, and, days into an on-call rotation, the definition of resiliency that actually matters. If you want the discipline for deciding which of these numbers to hold yourself to, Google's chapter on service level objectives is the right place to start: an availability target you picked because it sounded good is not an SLO, and the difference shows up the first time you have to decide whether to page someone.