Zero-Downtime Deployments with Docker & Kubernetes

Kubernetes performs rolling updates by default, which leads many teams to believe they have zero-downtime deployments. Then they watch error rates during a deploy and find a burst of failed requests every time.

The default is a rolling update, not a graceful one. Getting from one to the other is a handful of specific configurations, each addressing a specific way requests get dropped.

Health Checks Must Mean Different Things

The most common cause of deploy errors is using one health check for two different questions.

Readiness answers whether this instance should receive traffic right now. Liveness answers whether this instance is broken and should be restarted. Conflating them causes real damage: if your readiness check fails because a database is briefly unavailable, that is correct — stop sending traffic. If your liveness check fails for the same reason, Kubernetes restarts every instance simultaneously during a database blip, converting a brief degradation into a full outage.

Readiness should verify that the process can actually serve a request, including its critical dependencies. Liveness should verify only that the process is not wedged, and should not check dependencies at all.

A startup check is worth adding separately for applications with slow initialization. Without it, you have to set liveness timeouts generous enough to accommodate the slowest start, which means genuine hangs take much longer to detect.

Graceful Shutdown Is Where Requests Actually Get Dropped

This is the subtle one, and it is where most remaining deploy errors come from.

When a pod is terminated, two things happen concurrently and independently: the container receives a termination signal, and the pod is removed from the service endpoints. There is no ordering guarantee between them. Propagating the endpoint removal through the cluster takes a moment, and during that moment traffic is still being routed to a pod that has already begun shutting down.

The fix is to make the container wait before it starts shutting down. A pre-stop hook that sleeps for a few seconds gives the endpoint removal time to propagate. The pod continues serving during that window, receives no new traffic by the end of it, and only then begins its shutdown.

Then the application itself must shut down properly: stop accepting new connections, finish the requests already in flight, close database connections and flush buffers, and exit. An application that exits immediately on receiving the signal drops every request it was handling.

Make sure the signal actually reaches your application. If your container starts the process through a shell, the shell is process one and may not forward signals. The process gets killed after the grace period instead of shutting down cleanly, and you will see this as a low, persistent rate of deploy errors that never quite goes away.

Set the termination grace period longer than your pre-stop wait plus your longest reasonable request. If it expires, the process is killed regardless of what it was doing.

Configure the Rollout Deliberately

The default rolling update allows some pods to be unavailable while new ones start. For a service under real traffic, set maximum unavailable to zero and allow surge instead. New pods start and become ready before old pods are removed, so total capacity never drops below the current level.

This requires headroom in the cluster to run extra pods briefly, which is a reasonable cost.

Add a disruption budget as well. It governs voluntary disruptions — node drains, cluster upgrades, autoscaler decisions — which are not deployments but will take down your service just as effectively if they evict everything at once.

Database Migrations Are the Actual Hard Part

Application deployments are a solved problem. Schema changes are not, because during a rolling update the old and new versions of your application run simultaneously against one database. Any migration that breaks the old version causes errors for as long as the rollout takes.

The discipline is expand and contract, and it means every breaking change becomes a sequence of non-breaking ones.

To add a required column: add it as nullable with a default, deploy code that writes it, backfill existing rows, then make it required in a later release.

To rename a column: add the new one, deploy code that writes both and reads the new one with a fallback, backfill, deploy code that uses only the new one, then drop the old one in a subsequent release.

To remove a column: deploy code that stops referencing it first, then drop it later.

The rule that makes this manageable: any given deployment must work against both the previous schema and the next one. If it does not, you do not have zero-downtime deployments regardless of how your rollout is configured.

Watch for migrations that lock. Adding an index without the concurrent option locks the table against writes for the duration, which on a large table is an outage in the middle of a deploy that otherwise looked fine.

Build Images That Deploy Fast and Start Fast

Image size affects deploy speed directly, because every node that has not seen the image must pull it before a pod can start.

Multi-stage builds keep build tooling out of the runtime image. Order your layers so that dependency installation comes before copying source code — dependencies change rarely and source changes constantly, so this ordering means most builds reuse the cached dependency layer.

Always deploy by immutable tag, never by a floating latest tag. Floating tags make it impossible to know what is actually running, and they make rollback ambiguous at exactly the moment you need certainty.

Progressive Delivery for Risky Changes

A rolling update replaces all instances. If the new version has a problem that health checks do not catch — a subtle logic error, a performance regression under real traffic — every user is affected before anyone notices.

Canary deployments route a small percentage of traffic to the new version and compare error rates and latency against the old one before proceeding. Automated analysis of those metrics with an automatic rollback on regression is the mechanism that catches what health checks cannot.

Blue-green runs both versions at full capacity and switches traffic at once, which makes rollback instant but costs double capacity during the transition and does not limit exposure the way a canary does.

Feature flags are the most flexible option and worth mentioning here because they change the calculus: they decouple deploying code from releasing behavior. Deploy the code dark, enable it for a small group, expand gradually, and turn it off without a deployment if something is wrong. For risky changes this is usually the best available tool.

Rollback Must Be Tested

Every team assumes rollback works. Many discover during an incident that it does not, usually because a migration was not backward compatible.

Practice it. Deploy a change to staging and roll it back. Confirm that the previous version starts, serves traffic, and works against the current schema. A rollback path that has never been exercised is an assumption, not a capability.

Keep enough revision history to roll back more than one release, and make sure rollback is a single documented command rather than a sequence someone has to reconstruct under pressure.

Watch the Right Signals During a Deploy

Infrastructure metrics tell you whether pods are running. They do not tell you whether the deployment is going well.

Watch error rate, latency percentiles, and a business metric — orders, signups, whatever indicates the product is functioning — segmented by version where possible. An increase in the new version's error rate is your signal to stop, and it will appear before anything shows up in CPU or memory.

Set an explicit abort condition before you start, and honor it. The failure pattern is watching a rising error rate while telling yourself it will settle. It rarely does.

The Short Version

Zero-downtime deployment is not one feature. It is readiness and liveness meaning different things, a pre-stop delay that outlasts endpoint propagation, an application that shuts down gracefully and actually receives the signal, a rollout that adds before it removes, migrations that are backward compatible by construction, and a rollback path you have actually tested.

Miss any one and you will see errors on every deploy. Get all of them and deployments become routine enough that you stop scheduling them for the middle of the night.