Advanced Docker & Container Security

Image Signing & Provenance

18 min Lesson 5 of 28

Image Signing & Provenance

Scanning an image for CVEs tells you what vulnerabilities exist now, but it cannot tell you who built the image, which source code produced it, or whether it was tampered with in transit. Image signing and provenance close that gap. They answer the question every production system should ask before pulling an image: "Can I prove this artifact is exactly what my CI pipeline produced, and nothing else?"

Why the Supply Chain Attack Surface Is Real

The 2020 SolarWinds breach and the codecov incident showed that attackers do not always compromise the running application — they compromise the build pipeline and poison the artifact before it ever reaches production. Container images are a high-value target: a single poisoned base image can propagate to thousands of derivative images across hundreds of teams.

A robust image-signing workflow establishes a cryptographic chain of custody from source code commit to running pod. Every link in that chain — the source repo, the build system, the registry, and the runtime — must be verifiable.

Key terms: Signing attaches a cryptographic signature to an image digest so that anyone with the public key can verify it was produced by the expected party. Provenance is a signed attestation that records how the image was built — which repo, which commit SHA, which workflow, which builder — so you can audit the full chain of custody.

Sigstore & Cosign

Sigstore is a Linux Foundation project (backed by Google, Red Hat, and Chainguard) that makes signing keyless and auditable. Its core tool is cosign. The architecture has three components:

Fulcio — a certificate authority that issues short-lived signing certificates bound to OIDC identities (a GitHub Actions workflow, a GCP service account, etc.).
Rekor — an append-only transparency log that records every signing event, so you can verify a signature was issued at a specific time and detect if it was back-dated.
cosign — the CLI that orchestrates signing, verification, and attestation against those services.

End-to-end supply chain: source commit triggers a CI build that signs the image via Fulcio and Rekor, attaches a SLSA provenance attestation, and pushes to the registry — where the runtime verifies everything before admission.

Keyless Signing in GitHub Actions

With keyless signing, there is no long-lived private key to rotate or leak. The CI workflow exchanges its OIDC token for a short-lived certificate from Fulcio, signs the image digest, and the certificate (and signing event) are recorded permanently in Rekor. Any observer can later query Rekor to confirm the signature was issued by the legitimate GitHub Actions workflow for your repository.

# .github/workflows/build-sign.yml
name: Build, Push & Sign

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write   # Required for keyless signing (OIDC)

jobs:
  build-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push (capture digest)
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      # Sign by digest, not by tag — tags are mutable; digests are not
      - name: Sign image (keyless)
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

      # Generate and attach a SLSA provenance attestation
      - name: Attest provenance
        run: |
          cosign attest --yes \
            --predicate <(echo \'{"buildType":"https://github.com/actions","builder":{"id":"${{ github.workflow_ref }}"},"invocation":{"configSource":{"uri":"${{ github.repositoryUrl }}","digest":{"sha1":"${{ github.sha }}"}}}}') \
            --type slsaprovenance \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

Always sign by digest (@sha256:...), never by tag. Tags like :latest can be re-pointed to a different image at any time. Signing a digest creates an unforgeable binding between the cryptographic content hash and the signature — it is immutable by design.

Verifying Signatures

Verification confirms three things simultaneously: the signature is cryptographically valid, the certificate was issued by Fulcio from the expected OIDC identity, and the event is recorded in Rekor. If any of these checks fail, cosign exits non-zero.

# Verify a keyless signature — check the certificate identity and issuer
cosign verify \
  --certificate-identity "https://github.com/myorg/myrepo/.github/workflows/build-sign.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo@sha256:abc123...

# Verify and print the full Rekor transparency log entry
cosign verify \
  --certificate-identity "https://github.com/myorg/myrepo/.github/workflows/build-sign.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo@sha256:abc123... \
  | jq .

# Verify a SLSA provenance attestation
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity "https://github.com/myorg/myrepo/.github/workflows/build-sign.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo@sha256:abc123... \
  | jq .payload | base64 -d | jq .

Enforcing Signature Verification at Admission — Policy Controllers

Signing is useless if nothing checks the signature before the pod starts. In production Kubernetes, you enforce signature verification at the admission layer so that unsigned or wrongly-signed images are rejected before they ever schedule. The two main options are:

Sigstore Policy Controller (from the Sigstore project) — a Kubernetes admission webhook that checks cosign signatures and attestations against a ClusterImagePolicy CRD.
Kyverno — a general-purpose Kubernetes policy engine with first-class cosign support. Widely adopted at big-tech companies that also need non-image policy (e.g. require labels, block privilege escalation).

# Kyverno ClusterPolicy: only allow images signed by your CI workflow
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [production, staging]
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/myrepo*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/myrepo/.github/workflows/build-sign.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Enforce mode will block all unsigned images, including your own if the workflow is misconfigured. Roll out in Audit mode first: violations are logged but not blocked. Promote to Enforce only after validating that every workload in the targeted namespace passes. Many teams have caused outages by enabling enforce mode before migrating all images.

Content Trust with Docker Notary (Legacy Context)

Before Sigstore, Docker shipped Docker Content Trust (DCT) backed by Notary v1. You may encounter it in older pipelines and enterprise registries. DCT is enabled with DOCKER_CONTENT_TRUST=1 and uses a separate key infrastructure. It works but has significant operational overhead (root key ceremony, delegation keys, key rotation) compared to keyless Sigstore. New greenfield projects should use cosign. If you maintain a Notary-based pipeline, consider migrating — Notary v2 (now rebranded as ORAS Notation) aligns more closely with OCI standards, and cosign remains the dominant ecosystem choice.

SLSA Levels — A Maturity Framework

SLSA (Supply-chain Levels for Software Artifacts, pronounced "salsa") is a framework from Google that defines four levels of supply-chain integrity. Most teams target SLSA Level 2 or 3 for production images:

Level 1 — build scripts exist and provenance is generated (no verification required).
Level 2 — hosted CI build (e.g. GitHub Actions), provenance is signed by the build service. Covers the vast majority of real-world deployments.
Level 3 — hardened build platform: ephemeral build environments, no persistent credentials in the build, two-party review required. Google and GitHub internally operate at this level.
Level 4 — hermetic, reproducible builds with two-person approval. Extremely rare outside of operating system and hardware firmware projects.

The cosign attest --type slsaprovenance command in the workflow above produces a SLSA Level 2 provenance record. To reach Level 3, you additionally need the build to run in an isolated, ephemeral environment with no network access during compilation, and the builder itself must be attested. GitHub Actions with actions/attest-build-provenance can generate Level 3 provenance automatically.

Production Failure Modes

Teams adopting image signing at scale regularly hit these failure modes:

Signing after tagging a different digest — the build step outputs a digest, but if the image is re-tagged or re-pushed before signing, the digest changes and the signature covers the wrong artifact. Always capture the digest from the build step output and sign that explicit reference.
Rekor rate limits — the public Rekor instance (rekor.sigstore.dev) has rate limits. Large enterprises running hundreds of CI builds per hour should deploy a private Rekor instance or use a paid SaaS offering (Chainguard Enforce, GitHub Artifact Attestations).
Clock skew invalidating certificates — Fulcio issues certificates valid for 10 minutes. If your runner clock is skewed by more than a few minutes, the certificate will be considered invalid at verification time. Ensure NTP is synchronized on all runners.
Multi-arch images and manifest lists — sign the manifest list digest (the multi-arch index), not the platform-specific digests. cosign sign --yes IMAGE:TAG resolves the manifest list automatically; verify with the same digest.

Summary

Image signing with cosign and Sigstore provides a keyless, auditable, and operationally lightweight way to establish cryptographic chain of custody from source commit to running container. Sign by digest in CI after every build, attach a SLSA provenance attestation, and enforce verification at the Kubernetes admission layer with Kyverno or the Sigstore Policy Controller. These practices, combined with the image scanning from the previous lesson, give you defense-in-depth against supply chain attacks — the most dangerous threat vector in modern container operations.