Deployment Strategies & Progressive Delivery

Deploy vs Release

18 min Lesson 1 of 28

Deploy vs Release

One of the most powerful ideas to come out of big-tech engineering culture in the last decade is deceptively simple: deploying code and releasing a feature are two completely separate acts, and conflating them is the root cause of a surprising number of outages, botched launches, and midnight rollbacks.

At Google, Meta, Netflix, and Amazon, new code ships to production servers dozens to hundreds of times per day. But users only see a feature when the company consciously decides to show it. That gap — between the binary landing on a machine and a human being able to use it — is the space where modern progressive delivery lives.

What "Deploy" Actually Means

A deployment is the mechanical act of getting new code onto production infrastructure. The artifact — a Docker image, a compiled binary, a Lambda ZIP — travels from your CI system to your runtime environment. The process is owned by the engineering team, driven by automation, and ideally invisible to users.

Key properties of a well-run deployment:

Fully automated — no human clicks a button in production.
Idempotent — running it twice leaves the system in the same state.
Auditable — every deploy is tagged with a commit SHA, a build ID, the deploying principal, and a timestamp.
Reversible — the system can reach the previous known-good state within a bounded time window (often < 5 minutes for stateless services).

What "Release" Actually Means

A release is a business decision: the moment a feature becomes visible to some or all users. Release is owned by a product manager, a go-to-market team, or an SRE gating on error budget. It involves timing, marketing, legal review, support readiness, and A/B population selection — none of which have anything to do with Git.

Because release is a business act, it should be controllable at runtime, not baked into the binary. The mechanism that makes this possible is the feature flag (also called a feature toggle or feature gate). The code ships with the new behavior behind a flag; the flag is off by default; product flips it on when the moment is right.

The key insight: when deploy and release are coupled, every deployment is a risk event. Decoupling them means deployments become boring and releases become a business-controlled dial — not a fire drill.

Why Coupling Deploy and Release Is Dangerous

In a coupled system, the instant new code lands in production, every user sees the new behavior. This creates three interrelated problems:

Blast radius is 100%. If the new code has a latent bug, every request hits it simultaneously. Your on-call is fielding alerts before the deploy pipeline has even finished printing "success."
Rollback is the only recovery path. Since you cannot turn off the feature without reverting the code, a rollback undoes all the engineering work — and rollbacks themselves carry risk (database schema changes, in-flight requests, cache invalidation).
Release timing is held hostage by engineering. If product wants to launch at 9 AM on a Monday for maximum marketing impact, engineering must deploy at exactly that moment — introducing change to a system at peak traffic, with the full team watching.

The midnight deploy trap: many teams schedule deployments at 2 AM to reduce user impact. This is a symptom of coupled deploy/release. It forces engineers to work nights, reduces the available response team during the deploy, and still carries the same rollback risk — just with fewer people awake to handle it. Fix the coupling, not the clock.

The Mechanics of Decoupling

The practical implementation uses a feature flag evaluated at request time. The flag check happens in application code and reads its value from a centralized flag store — a purpose-built system like LaunchDarkly, Flagsmith, Unleash, or a home-built Redis-backed service. The flag store is read on every request (with aggressive caching); changing a flag value propagates to all instances within seconds without a new deploy.

Deploy puts code on servers (engineering); Release exposes the feature to users (product). The gap between them is controlled by a feature flag.

A Minimal Feature Flag in Practice

You do not need a commercial flag platform to start. The pattern below uses environment variables for illustration, then shows the step up to a proper flag store.

# ── Simplest possible flag: env var read at startup ───────────────────────────
# app/config.py (Python / FastAPI example)
import os

FEATURE_NEW_CHECKOUT = os.getenv("FEATURE_NEW_CHECKOUT", "false").lower() == "true"

# In your route handler:
# if FEATURE_NEW_CHECKOUT:
#     return new_checkout_handler(request)
# return legacy_checkout_handler(request)

# Problem: changing this flag requires a redeploy — still coupled!
# Solution: move the flag value to a runtime-readable store.

# ── Flag stored in Redis, read at request time ────────────────────────────────
# redis-cli
SET feature:new_checkout false
# later, product decides to launch:
SET feature:new_checkout true
# TTL not needed — this is intentionally persistent

# ── Application reads flag per-request (Python pseudo-code) ───────────────────
# import redis
# r = redis.Redis(host="redis-flags.internal", decode_responses=True)
#
# def is_enabled(flag_name: str, default: bool = False) -> bool:
#     val = r.get(f"feature:{flag_name}")
#     if val is None:
#         return default
#     return val.lower() == "true"
#
# if is_enabled("new_checkout"):
#     return new_checkout_handler(request)

Cache your flag reads. Reading Redis on every HTTP request adds latency. Use a local in-process cache with a short TTL (typically 5–30 seconds). This means flag changes propagate within 30 seconds across all instances — acceptable for most use cases and far faster than a deploy.

Gitops Integration: Flags as Configuration

Mature teams store their flag definitions in Git alongside their service manifests. A Kubernetes ConfigMap or a Helm values file holds the flag defaults; the flag store is seeded from it at deploy time. This gives you a complete audit trail of every flag state change through your normal PR review process, not a separate web UI that nobody remembers to check.

# ── Helm values.yaml — flag defaults baked into the chart ─────────────────────
featureFlags:
  new_checkout: false
  redesigned_search: false
  new_pricing_engine: true     # launched last quarter, now default-on

# ── Kubernetes ConfigMap rendered by Helm ─────────────────────────────────────
apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: checkout
data:
  new_checkout: "false"
  redesigned_search: "false"
  new_pricing_engine: "true"

# ── Application reads ConfigMap via mounted volume ───────────────────────────
# ConfigMap is mounted at /etc/flags/
# The app reads /etc/flags/new_checkout on startup
# Kubernetes does NOT hot-reload mounted ConfigMaps by default;
# use a sidecar or a proper flag SDK for live updates.

Production Failure Modes

Even with clean decoupling, the following failure patterns appear repeatedly in production:

Flag explosion. Teams accumulate hundreds of stale flags that were never cleaned up after a successful launch. Each one is a conditional branch that must be tested, and a bug magnet. Enforce a TTL policy: flags older than 90 days with no state changes get auto-deleted or raise an alert.
The flag-store outage. If your flag store goes down, what happens? If the application fails closed (returns an error), a flag-store outage becomes a total service outage. Always define a sensible default and fail open to the safe path. For a new, unvalidated feature, the safe path is off.
Testing the dark code. Code that ships behind a flag but is never tested in that off state can accumulate dependency rot. Run your integration test suite with flags both on and off in CI — most flag SDKs provide a test helper that overrides the store.
Schema coupling. A flag can gate UI and application logic, but not database schema changes. If your new feature requires a new column, you must add that column in a prior deploy (with a default) before you can flag-gate the feature. This is the Expand-Contract pattern, covered in a later lesson.

The Netflix / Amazon rule of thumb: any feature that has been at 100% traffic for 30+ days with no issues should have its flag removed and the dead code branch deleted. Shipping new behavior is the goal; permanent flag spaghetti is the antipattern.

Summary

Decoupling deploy from release is not a DevOps technique — it is a philosophy that changes who owns risk and when. Engineering owns deploy: it must be automated, fast, and reversible. Product owns release: it must be intentional, measured, and reversible without an engineering incident. Feature flags are the mechanism that makes both possible simultaneously. Every subsequent topic in this tutorial — rolling deployments, canary releases, A/B experiments — is an elaboration of this single foundational idea.