Nothing achieves literal 100% uptime — anyone who promises you that is selling something. What's actually achievable, and what this is about, is getting close enough that a downstream failure degrades your system instead of cascading through it, and doing that with observability good enough that you know within seconds which state you're in, not minutes. That's a circuit breaker, an autoscaler that reacts to the breaker's state instead of just CPU, and a metrics pipeline that survives a single Prometheus instance falling over.

The three circuit breaker states, and the config knob that actually matters for each

Resilience4j's circuit breaker is a state machine: closed (calls flow normally, failures are being counted), open (calls fail fast without touching the downstream at all), half-open (a small number of trial calls decide whether to close again or re-open). The part people get wrong isn't the concept — it's the sliding window and threshold configuration that decides how twitchy or how slow-to-react the breaker actually is:

resilience4j.circuitbreaker:
  instances:
    pricingEngine:
      registerHealthIndicator: true
      slidingWindowType: TIME_BASED
      slidingWindowSize: 30          # seconds
      minimumNumberOfCalls: 10       # don't trip on 2 failures out of 2 calls
      failureRateThreshold: 50       # % of calls that must fail to trip
      waitDurationInOpenState: 15s   # how long before trying half-open
      permittedNumberOfCallsInHalfOpenState: 5
      slowCallDurationThreshold: 2s  # a "slow" call counts toward the failure rate too
      slowCallRateThreshold: 50

minimumNumberOfCalls is the setting most configs skip, and it's the one that prevents a low-traffic service from tripping its breaker off two unlucky failures out of two total calls. slowCallDurationThreshold matters just as much as outright failures — a downstream dependency that's alive but degraded to 5-second response times is functionally down for anything with a real SLA, and a breaker that only counts hard failures will happily keep sending traffic into that slow response time indefinitely.

Pair the breaker with a bulkhead (cap concurrent calls to that dependency independently of the breaker's own state) and a fallback that returns something genuinely useful — a cached last-known-good price, not a bare exception — for anything on the user-facing path.

Wiring circuit breaker state into Kubernetes autoscaling, not just CPU

A default HPA scaling on CPU alone misses the actual signal you want: pods burning CPU while their breakers are open, uselessly failing fast, aren't the problem you need more replicas for. The problem is upstream request volume outpacing capacity while the breaker is closed and dependencies are healthy. Resilience4j already exports breaker-state and call-count metrics via Micrometer; scrape those with Prometheus and expose them to the HPA through the Prometheus Adapter's custom metrics API:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: pricing-engine
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pricing-engine
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: resilience4j_circuitbreaker_calls_permitted_rate
        target:
          type: AverageValue
          averageValue: "50"
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75

Scaling on permitted-call throughput rather than raw CPU means the HPA reacts to the thing that's actually saturating — request volume the breaker is letting through — instead of a proxy for it that can lag or mislead under GC pressure or JIT warm-up.

Memory sizing deserves its own honesty: don't set a pod's memory limit from a load test's average — set it from the real p99 under a sustained failure scenario (breaker open, fallback logic executing, retries queued), because that's when memory pressure actually peaks, not during the happy path your load test probably emphasized. Undersized limits under exactly the failure condition you built the breaker to survive is how a resiliency pattern gets undone by an OOMKill.

Why Prometheus alone isn't enough, and what Thanos actually buys you

A single Prometheus instance is both your metrics store and a single point of failure for your metrics store — it falls over, you're blind exactly when you need visibility most (during an incident, under load). Thanos sits alongside Prometheus rather than replacing it: a Sidecar ships each Prometheus instance's blocks to object storage (S3/Azure Blob/GCS) for durable long-term retention, and a Querier gives you one query interface across every cluster's Prometheus instances plus that long-term store — which is the part that actually matters once you're running more than one cluster or more than one region, since "what's the breaker state across all three regions right now" stops being answerable from any single Prometheus.

The practical payoff: a Grafana dashboard built against the Thanos Querier keeps working when an individual Prometheus instance restarts (which happens routinely — config reloads, pod evictions, node maintenance), and you get real historical trend queries (breaker trip frequency over the last 90 days, not the last 15) without provisioning one Prometheus instance with an unreasonably large local disk.

Azure Monitor: the platform signal Resilience4j can't see

Resilience4j's metrics are entirely application-level — they know your code's view of "did this call succeed," nothing about the platform underneath it. Azure Monitor fills the gap: node-level CPU pressure, AKS control-plane health, network-layer throttling, managed-database connection pool saturation — signals that can cause exactly the symptoms a circuit breaker reacts to (timeouts, slow calls) without the actual downstream service being unhealthy at all.

The move that pays off is correlating the two, not treating them as separate dashboards nobody cross-references: pull Resilience4j breaker-state metrics into the same Grafana view as Azure Monitor's node/network metrics, so "the pricing-engine breaker just opened" and "the underlying node hit CPU throttling three seconds earlier" show up on the same timeline. That's the difference between debugging an application incident and correctly identifying an infrastructure one — and between "restart the pod" and "cordon the node," which are not the same fix.

What "close to 100%" actually looks like in practice

Not a single circuit breaker config, and not a single autoscaling rule — a chain where each layer covers what the layer below it can't: the breaker stops a slow dependency from cascading into your own service's threads and memory, the HPA reacts to the breaker's actual signal instead of a lagging CPU proxy, Thanos keeps your visibility into all of that alive across restarts and clusters, and Azure Monitor (or your cloud's equivalent) tells you when the problem was never the application's dependency graph at all, but the node underneath it. Skip any one layer and the others compensate imperfectly; run all four together and most failures degrade gracefully instead of cascading — which, days into an on-call rotation, is the actual definition of resiliency that matters.