Artifact Management & Release Engineering

Release Trains & Cadence

18 min Lesson 8 of 28

Release Trains & Cadence

Continuous deployment sends every merged commit to production automatically. That model works well for teams with mature test coverage, strong observability, and small, independently deployable services. But many large organizations — platform teams, regulated industries, teams with many downstream consumers — need a different model: the release train. Understanding when and why to choose a train over continuous deployment, and how to run one correctly, is core big-tech engineering knowledge.

What Is a Release Train?

A release train is a time-boxed, scheduled release cycle. The train departs on a fixed schedule (weekly, bi-weekly, monthly, quarterly). Features that are ready board the train; features that are not ready wait for the next one. The key insight is that the train does not wait for features — features wait for the train. This inverts the relationship between a feature and its ship date: dates are fixed, scope is variable.

The model was codified in scaled agile frameworks but its mechanics are purely operational. Google ships Chrome on a four-week train. Kubernetes releases every four months. Ubuntu LTS ships every two years. Each is a train with a different cadence calibrated to its audience and risk tolerance.

Release train cadence: Feature B missed the v1.4 cut and waits for v1.5. Dates are immovable.

Trains vs. Continuous Deployment

Both models can coexist in the same organization. The right choice depends on risk profile, consumer coupling, and team maturity.

Continuous deployment — every commit to main deploys automatically after tests pass. Best for: consumer-facing web services with feature flags, small blast radius per deploy, high test confidence, no downstream consumers pinned to a specific version.
Release trains — best for: platform libraries with many consumers, embedded firmware, mobile apps where app store review adds latency, regulated environments requiring audit trails per release, and large monorepos where coordinating many teams into a single ship event is operationally simpler than per-team deploys.

The hybrid model: Most big-tech companies run continuous deployment for their own backend services and release trains for externally-consumed SDKs, CLIs, and mobile apps. Your microservice might deploy 50 times a day while the Go SDK for your API ships on a monthly train. Both are valid — match the model to the coupling and risk tolerance.

Cutting a Release Branch

At the train's scheduled cut time, a release branch is created from the current state of main. From that moment on, main keeps moving (the next train is already loading) while the release branch is stabilized in isolation. Only bug fixes cherry-picked from main may merge into the release branch — no new features allowed after cut.

# Cut the release branch at the scheduled moment
# Convention: release/MAJOR.MINOR
git checkout main
git pull origin main
git checkout -b release/1.5
git push -u origin release/1.5

# Tag the branch point as release candidate
git tag v1.5.0-rc.1
git push origin v1.5.0-rc.1

# Cherry-pick a qualifying fix from main into the release branch
git checkout release/1.5
git cherry-pick <commit-sha> --no-ff
git push origin release/1.5

# After stabilization: tag the final release
git tag v1.5.0
git push origin v1.5.0

# Merge release fixes back to main (use --no-ff to preserve history)
git checkout main
git merge --no-ff release/1.5 -m "chore: merge release/1.5 fixes back to main"
git push origin main

The branch protection rules for a release branch should be stricter than main: require two reviewers (including the release engineer), require all CI checks to pass, block force-pushes absolutely, and optionally require a signed tag before merge. These constraints are what makes a release branch a real quality gate rather than theater.

Automate the cut: At companies like Google and Meta, release branch creation is automated by a release bot that fires on a calendar trigger, cuts the branch, creates an RC tag, opens a tracking issue, pings the on-call release engineer in Slack, and posts a diff summary. Human involvement starts at stabilization, not at branch creation. Aim for that level of automation — manual cuts forget steps under pressure.

Code Freeze Windows

A code freeze is a period during which merges to a branch (or to main for continuous deployment teams) are restricted to a predefined set of changes — typically only severity-1 bug fixes and security patches. Freezes serve two purposes: reducing risk immediately before a ship event and giving QA a stable target to test against.

Two common freeze patterns:

Pre-ship freeze on the release branch — the most common pattern for trains. After the branch is cut, the release engineer applies a FROZEN label or branch protection rule. Any proposed cherry-pick requires explicit sign-off. Duration: 24–72 hours depending on test cycle length.
Org-wide main freeze — used during major holidays, compliance audits, or after a severity-1 incident. All merges to main are blocked for all teams. Requires executive sign-off to override. GitHub branch protection can enforce this with a required status check that always fails during the freeze window.

# Enforce a freeze via GitHub branch protection (GitHub CLI)
# Add a "freeze" required status check that your CI reports as failed during windows.
# During freeze: the freeze-check job exits 1, blocking all merges.

# freeze-check.yml — CI job injected into every PR during freeze
name: Freeze Check
on: [pull_request]
jobs:
  freeze:
    runs-on: ubuntu-latest
    steps:
      - name: Check freeze status
        env:
          FREEZE: ${{ vars.RELEASE_FREEZE }}   # repo/org variable toggled by release bot
        run: |
          if [ "$FREEZE" = "true" ]; then
            echo "::error::Repository is in a code freeze. Only release engineers may merge."
            exit 1
          fi
          echo "No active freeze. Merge allowed."

Production pitfall — freeze bypass pressure: In every organization, someone will push hard to merge "just this one critical thing" during a freeze. The correct answer is: define the override process in advance (who has authority, what documentation is required) and stick to it. Ad-hoc bypass decisions made under pressure are a leading cause of ship-day incidents. If your freeze is routinely bypassed, either your cadence is wrong or your feature flag usage is insufficient.

Cadence Design: Choosing the Right Train Frequency

There is no universally correct cadence. Too slow and you accumulate large, risky batches and slow feedback. Too fast and you defeat the purpose (you are just doing continuous deployment with extra steps). Key inputs to the decision:

Consumer upgrade lag — if your enterprise customers patch quarterly, a weekly train produces releases that pile up undeployed, adding no value. Align to the slowest sane consumer update cycle.
Test cycle duration — if a full regression suite takes 6 hours, a weekly train gives you meaningful stabilization time; a daily train does not.
Coordination cost — each train requires release engineer time for cut, stabilization, and promotion. Cadence must not exceed the team's operational bandwidth.
Regulatory checkpoints — some certifications (SOC 2, FDA, PCI-DSS) require release artifacts to be documented, signed, and audited. Monthly or quarterly trains make audit packaging tractable; daily trains produce an unmanageable volume of artifacts.

Decision matrix: when to choose continuous deployment vs. a release train.

Running the Train: Roles and Rituals

A well-run train has a release engineer (RE) — a rotating role, not a permanent job — who owns the train from cut to ship. Their responsibilities: cut the branch on time, triage cherry-pick requests (approve only qualifying fixes), coordinate with QA, manage the freeze, sign the release artifacts, and write the post-ship summary. The RE has unilateral authority to reject a cherry-pick regardless of feature urgency.

Standard rituals around each train:

T-minus-1 day: RE announces cut time in team channel; feature owners confirm their work is merged or explicitly deferred to the next train.
T-0: Branch cut, RC tag pushed, freeze begins on release branch, automated smoke tests fire.
Stabilization window (24-72 hrs): Only cherry-picked fixes land; CI must be green continuously.
Ship day: Final tag pushed; artifacts promoted from staging to production via the promotion pipeline; release notes published.
T+1 day: Post-ship health review — error rates, latency, rollback triggers.

Track everything in a release issue: Open a GitHub/Jira issue for each train at cut time. Link every cherry-pick PR to it. Record the RC tags, ship time, and post-ship health metrics. After six months you will have a log of every release decision — invaluable for audit, retrospectives, and training new release engineers.

Release trains and continuous deployment are not opposites — they are tools. The discipline is knowing which one fits the problem. As you mature your platform, you will likely run both simultaneously, with trains governing the externally-visible surface and continuous deployment running everything behind it. The shared foundation for both is the same: automated tests, clean branching hygiene, and an unambiguous definition of "done."