The ten lessons of this tutorial each addressed a distinct security layer — shift-left culture, threat modeling, SAST, SCA, DAST, container and IaC scanning, SBOMs, signing, and secrets management. This capstone lesson assembles them into a single, production-grade pipeline design, explains how the gates interlock, and shows how attestation chains connect every step into an auditable provenance record that survives a supply-chain investigation.
The model here reflects how companies like Google, GitHub, and Shopify actually structure DevSecOps: security is not a scan that blocks a PR — it is a chain of signed claims about software that accumulates from the first commit to the running workload. If any link in that chain is missing or invalid, the deployment gate refuses the artifact.
The Five-Gate Model
A mature secure delivery pipeline organizes controls into five ordered gates. Each gate either produces a signed attestation (a cryptographic claim about the artifact) or consumes attestations from previous gates. Nothing advances without the gate's attestation being present and valid.
Gate 1 — Code Integrity: Gitleaks secrets scan + Semgrep/CodeQL SAST + dependency license check. Runs on every PR commit in parallel. Produces a code-scan attestation signed with the CI OIDC identity.
Gate 2 — Dependency Trust: Trivy/Grype SCA against OSV + CVE databases, Syft SBOM generation attached to the PR. Blocks on any critical CVE with a fix available. Produces a sbom attestation and a vuln-scan attestation.
Gate 3 — Build Integrity: Reproducible build in an ephemeral, network-isolated runner. IaC scan (Checkov/tfsec) on Terraform/Helm changes. Container image built and signed with Cosign (keyless, Sigstore). SLSA provenance attestation generated by the builder. Image pushed to registry only after all attestations are attached.
Gate 4 — Pre-Deploy Verification: Kyverno or OPA/Gatekeeper policy admission checks the image for all required attestations before allowing it into any Kubernetes namespace. DAST (OWASP ZAP API scan) runs against the staging environment after deploy. Gate passes only when DAST finds no high/critical findings.
Gate 5 — Runtime Continuous Assurance: Falco runtime security rules watch for anomalous process behavior. Vault dynamic secrets rotate credentials before each deployment. Dependency-Track monitors the production SBOM for newly published CVEs and pages on-call when a critical emerges.
The five-gate secure delivery pipeline: gates 1-2 run on every PR, gate 3 builds and signs on merge, gate 4 verifies all attestations before deploying to Kubernetes, gate 5 watches the runtime continuously.
Wiring It Together: The GitHub Actions Workflow
The following skeleton shows how all five gates map to a real GitHub Actions workflow. Jobs within the same stage run in parallel. The build-and-sign job uses needs: to enforce gate ordering — it will not run unless gates 1 and 2 both pass.
Why pass the digest, not the tag? Tags are mutable — :latest can point to a different image tomorrow. A digest (sha256:abc…) is an immutable reference to a specific image layer hash. Every step in the pipeline after the build job references the digest, not the tag, so there is no window for a tag-swap attack (where an attacker replaces the registry image between your signing step and your deploy step).
The Kyverno Admission Policy: Closing the Loop
Gate 4 is only meaningful if Kubernetes enforces it. Without an admission controller, a developer could push a manually built image that bypassed every scan and deploy it directly with kubectl. A Kyverno ClusterPolicy requires that every Pod image has a valid Cosign signature from your CI OIDC identity before the API server accepts the Pod spec.
# kyverno/require-image-signature.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-images
annotations:
policies.kyverno.io/description: >
All images must be signed by the GitHub Actions OIDC identity
using keyless Cosign (Sigstore). Unsigned images are rejected
at admission time — they never start.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: check-image-signature
match:
any:
- resources:
kinds: [Pod]
namespaces: [production, staging]
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- entries:
- keyless:
subject: "https://github.com/myorg/myapp/.github/workflows/secure-pipeline.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
rekor:
url: https://rekor.sigstore.dev
attestations:
- predicateType: https://cyclonedx.org/bom
conditions:
- all:
- key: "{{ metadata.component.version }}"
operator: NotEquals
value: ""
Failure Modes and Escape Hatches
Production pipelines must handle emergencies without completely abandoning security. Three common failure modes and their mitigations:
Supply-chain scanner outage: If the Trivy database server is unreachable, fail open (allow the build) but page security on-call and set a 24-hour deadline for a manual scan. Do not let a vendor outage become a P0 deploy blocker — but do not silently skip the scan either. Record the skip as a signed exception attestation.
Zero-day in a transitive dependency with no fix: The correct response is not to suppress the finding. Use a .trivyignore file with an expiry date and a Jira ticket reference. Kyverno can be configured to accept an image with a vuln-exception attestation that includes the ticket ID and expiry, signed by a security engineer's key.
Emergency hotfix to production: Define a "break-glass" procedure: a GitHub environment with required reviewers (two senior engineers + one security engineer), a mandatory post-incident review, and an automated Jira ticket. The Kyverno policy can be configured to allow images signed by a break-glass key with a 2-hour TTL. Every break-glass event is logged to an immutable audit trail (CloudTrail / Rekor).
Start with audit mode, then enforce. When rolling out Kyverno admission policies to an existing cluster, set validationFailureAction: Audit first. This logs policy violations without blocking deployments. Run in audit mode for two weeks, fix all violations, then flip to Enforce. Flipping directly to Enforce on a cluster with unsigned legacy images will cause an immediate outage.
Attestation pinning is not enough alone. Cosign signatures and SLSA provenance prove the image was built by your CI system. They do not prove the image is free of vulnerabilities that were published after the build. This is why Gate 5 (Dependency-Track + SBOM continuous monitoring) is mandatory — an image that was clean on build day can become critical-vulnerable the next morning when a new CVE is published against one of its packages. The SBOM gives you the inventory to know instantly which running workloads are affected.
Measuring the Pipeline: Security KPIs
A pipeline you cannot measure is a pipeline you cannot improve. The four KPIs that matter at big-tech scale are: Mean Time to Detect (MTTD) — how long from a vulnerability being published until your Dependency-Track alert fires; Mean Time to Remediate (MTTR) — how long from alert to a fixed image running in production; Gate Escape Rate — the percentage of CVEs that reached production without being caught at gates 1-4 (target: zero critical/high escapes); and Pipeline Coverage — the percentage of production images that have a complete, valid attestation chain (target: 100%). Track these in a Grafana dashboard fed from Dependency-Track's API and your audit log pipeline.