The framing that changed how I ship was giving up on the idea that a deployment is an event. It is a dial. You turn it from 0% to 100% over some period, and at every position you get to ask whether to keep turning. Everything people call progressive delivery falls out of taking that literally.

At Udaan I built a canary deployment framework with A/B testing and feature flags on top of that idea. Deployment incidents fell 70% and release velocity doubled. The mechanism was the easy part; the gate was where the actual engineering was.

Three different dials, routinely confused

Canary releases, feature flags and A/B tests get talked about interchangeably and they answer different questions.

  • A canary release routes a small slice of production traffic to a new version and asks: is this build healthy?
  • A feature flag decouples release from deploy inside one version and asks: should this behaviour be on, for whom?
  • An A/B test splits users across variants and asks: which one produces the better outcome?

They compose, and the composition is the point. The canary tells you the build will not fall over; the flag lets you turn a specific behaviour off without a rollback; the A/B test tells you whether the behaviour was worth shipping at all. Conflating them gives you the worst of each — a rollback as the only lever for a product question, or an experiment that cannot be stopped without a redeploy.

It is worth being honest about the cost. Every flag is a branch, and branches that never get deleted become permanent untested combinations. Flags need an owner and an expiry the day they are created, or the flag system quietly becomes the least-tested part of the codebase.

The rollout ladder

Canary rollout progression with promotion gates Traffic to a new version steps up through 1 percent, 5 percent, 25 percent, 50 percent and 100 percent. Between every step sits an automated gate that compares the canary's error rate and latency against the stable version, promoting on pass and rolling back to zero percent on fail. 100% 0% time, one gate between every step 1% 5% 25% 50% 100% gate gate gate gate
Each gate compares the canary's error rate and tail latency against the stable version over a fixed observation window. Pass promotes to the next step; fail returns traffic to 0%.

The mechanism underneath is deliberately unremarkable. Kubernetes' own rolling update already replaces pods incrementally with a maxSurge/maxUnavailable budget, which gets you a crude canary for free — the new version serves a fraction of traffic simply because it is a fraction of the replicas. What it does not give you is a decision point: a rolling update proceeds unless a readiness probe fails, and a build that comes up healthy and then returns 500s on a specific code path passes that bar comfortably. If you would rather adopt than build, Argo Rollouts is the controller-shaped version of what we assembled.

The gate is the whole product

An automated promotion gate needs three things, and the third is the one that gets skipped.

A comparison, not a threshold. “Canary error rate below 1%” is a bad gate, because the correct threshold differs per service and drifts. “Canary error rate not materially worse than stable, measured over the same window” travels everywhere, and it automatically accounts for the outage that is not your build's fault.

Tail latency, not the mean. The mean hides exactly the regression you are hunting. This is where it matters whether your metrics pipeline can actually answer the question — a naive average across pods is not a percentile, and Prometheus' own guidance on histograms and summaries spells out what histogram_quantile does and does not promise before you gate a release on it.

An observation window long enough for the failure to appear. Cache warm-up, JIT compilation, connection-pool growth and scheduled jobs all mean a pod's first sixty seconds are unrepresentative. A gate that samples too early promotes broken builds confidently, which is worse than no gate at all, because the team stops reading the alerts.

We drove the gate off dashboards built on Azure Data Explorer, Prometheus, Thanos and Grafana. Thanos mattered more than it sounds: gating on a comparison means querying across instances and retaining enough history to know what “normal” is, and a single Prometheus with local retention cannot answer either question during the incident when you need it. The same investment took mean time to detect from 15 minutes to under 3, which is the number I would point at if I had to justify observability spend to a finance team — a gate is only as fast as the signal feeding it.

Flags as the fast lever, rollback as the slow one

The reason incidents fell 70% is not that fewer bad builds were written. It is that a bad build stopped being an event that required a deployment to resolve. At 1% of traffic, a regression affects a fraction of requests and the remedy is a configuration change, not a pipeline run — and the difference between a thirty-second flag flip and a nine-minute rollback deploy is the difference between a blip and an incident review.

This is also why blue/green and canary are not competitors. Blue/green gives you an instant, total switch and an instant, total revert, at the cost of running two full environments and learning nothing incremental. Canary gives you graded exposure and real signal at low blast radius. Which you want depends on whether your risk is “this might be catastrophic” or “this might be subtly worse”.

Why velocity doubles rather than falling

The counter-intuitive outcome is that adding gates to the pipeline made releases faster. Release velocity doubled, and the mechanism is not mysterious: the cost of a release is dominated by the fear surrounding it. Batch a fortnight of changes together because deploying is scary, and each release is now large, hard to attribute when it breaks, and correspondingly scarier. Make exposure gradual and reversible in seconds, and the rational batch size collapses toward one change.

That relationship — smaller batches, faster recovery, and higher throughput arriving together rather than trading off — is the one consistently reported in DORA's research on delivery performance, and it matched what we measured. If you want the discipline behind choosing what to gate on in the first place, Google's SRE material on alerting from SLOs is the best starting point I know: a promotion gate is an alert you have agreed to act on automatically, so it deserves the same rigour about burn rate and window length as anything that pages a human.

Progressive delivery is not really a deployment technique. It is the decision to stop treating “is this safe?” as a question you answer once, before shipping, with a review.