Continuous Integration Fundamentals

Secrets & Security in CI

22 min Lesson 8 of 28

Secrets & Security in CI

A CI pipeline runs with access to your production cloud accounts, container registries, signing keys, and deployment credentials. That makes it one of the highest-value targets in your entire engineering estate. The Solar Winds supply-chain breach, the Codecov credential exfiltration, and dozens of public post-mortems share a common thread: an attacker who reached a build system walked straight into production. This lesson covers how top-tier teams protect secrets and harden CI pipelines so a compromised runner cannot become a company-wide incident.

Why Hardcoded Secrets Are a Category of Vulnerability

The naive approach — pasting a token into a workflow file or a .env committed to the repository — is so common that automated scanners (GitHub's Secret Scanning, TruffleHog, Gitleaks) make a full-time job of finding them. Even if you delete the secret from the file, git history preserves it forever unless you rewrite history and rotate the credential. At big-tech companies the policy is absolute: no secret ever touches source control, not even in a private repo, not even encrypted, not even base64-encoded. Obfuscation is not security.

Production pitfall — the fork PR attack: On GitHub Actions, a workflow triggered by pull_request from a fork runs in the context of the fork — it cannot access repository secrets by default. But if you use pull_request_target (which runs in the context of the base repo) and check out the PR's code, you have just given a contributor full read access to every secret in your org. This is one of the most common CI privilege escalation vectors.

Secret Stores: Where Secrets Actually Live

Every major CI platform ships a native secret store. These encrypt secrets at rest, inject them into jobs as environment variables or files, and — critically — mask their values in log output so they cannot be exfiltrated through log aggregation pipelines.

GitHub Actions secrets are the baseline. Repository secrets are scoped to one repo; environment secrets add an approval gate (require a reviewer before deploying to production); organisation secrets can be shared across repos with explicit allow-lists.

# .github/workflows/deploy.yml

name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production        # gates on environment protection rules

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
          aws-region: us-east-1

      - name: Log in to ECR
        run: |
          aws ecr get-login-password --region us-east-1 \
            | docker login --username AWS --password-stdin \
              123456789012.dkr.ecr.us-east-1.amazonaws.com

      - name: Deploy
        env:
          # Secret injected from GitHub Secrets — never printed in logs
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
        run: ./scripts/deploy.sh

For teams that need centralised secret management across multiple CI platforms, cloud providers, and runtime environments, a dedicated secret manager is the production standard: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Secrets are fetched at job runtime with a short-lived token; they are never stored in the CI platform itself.

# Fetching a secret from HashiCorp Vault inside a GitHub Actions job
# Requires: hashicorp/vault-action, Vault JWT auth method configured

      - name: Import secrets from Vault
        uses: hashicorp/vault-action@v3
        with:
          url: https://vault.internal.company.com
          method: jwt
          role: github-actions
          secrets: |
            secret/data/production/db   password | DB_PASSWORD ;
            secret/data/production/api  key      | API_KEY

      - name: Use secrets
        run: |
          echo "DB_PASSWORD is available as an env var (masked in logs)"
          ./migrate --db-password "$DB_PASSWORD"

Pro practice — secret rotation: Treat CI secrets like passwords: rotate them on a schedule (quarterly minimum) and immediately after any team member with access leaves. Automation makes this zero-effort: Vault supports automatic rotation for AWS IAM keys, database credentials, and PKI certificates. AWS Secrets Manager can auto-rotate RDS passwords on a configurable schedule with a Lambda function.

Least-Privilege CI Credentials

Every CI credential should be scoped to the minimum permissions needed for that specific job — nothing more. This is the principle of least privilege applied to pipelines. In practice:

Separate credentials per job stage: The job that runs tests needs read access to the package registry. The job that pushes a Docker image needs write access to ECR. The job that deploys needs AssumeRole on the deploy role only. These should be three different credentials, not one all-powerful token.
Scope to branches and environments: Credentials that can write to production should only be available in jobs triggered from main and only in the production environment (with approval gates).
Short-lived tokens over long-lived API keys: An API key that never expires is a liability that compounds over time. OIDC tokens (next section) expire in minutes. AWS STS AssumeRole tokens expire in 1-12 hours. Prefer them.

OIDC: The Right Way to Authenticate CI to Cloud

OpenID Connect (OIDC) eliminates long-lived credentials entirely. Instead of storing an AWS access key in GitHub Secrets, you configure AWS to trust GitHub's identity provider. When a job runs, GitHub issues the runner a signed JWT (an OIDC token) containing verified claims: the repository name, the branch, the environment, the workflow name. AWS exchanges this JWT for a temporary STS token. No secret is stored anywhere — the pipeline proves its identity cryptographically.

OIDC token exchange — the runner proves identity with a signed JWT; AWS issues temporary credentials scoped to the IAM role. No static secret is stored.

Setting up OIDC between GitHub Actions and AWS requires three one-time steps:

# 1. Create the GitHub OIDC provider in AWS (Terraform example)
resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  # GitHub's OIDC thumbprint (verify at https://token.actions.githubusercontent.com)
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

# 2. Create a least-privilege IAM role with a trust policy scoped to your repo
resource "aws_iam_role" "github_actions_deploy" {
  name = "github-actions-deploy"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.github.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
        }
        StringLike = {
          # Restrict to main branch of this specific repo only
          "token.actions.githubusercontent.com:sub" =
            "repo:myorg/myrepo:ref:refs/heads/main"
        }
      }
    }]
  })
}

# 3. Attach a least-privilege policy — only ECR push + ECS deploy, nothing else
resource "aws_iam_role_policy" "deploy_policy" {
  name = "deploy-policy"
  role = aws_iam_role.github_actions_deploy.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:PutImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload"
        ]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["ecs:UpdateService", "ecs:DescribeServices"]
        Resource = "arn:aws:ecs:us-east-1:123456789012:service/prod/*"
      }
    ]
  })
}

Key idea — sub-claim scoping: The sub claim in the OIDC JWT encodes the exact context: repo:org/repo:ref:refs/heads/main for pushes to main, repo:org/repo:environment:production for environment-scoped jobs. Always scope your IAM trust policy to the most restrictive sub pattern possible. A wildcard like repo:myorg/* means any repo in your org can assume the deploy role — a critical misconfiguration.

Masked Variables and Log Hygiene

CI platforms automatically mask secret values in log output: if the value of a secret appears in stdout or stderr, the platform replaces it with ***. This is the last line of defence, not the first. Several failure modes bypass it:

Encoding tricks: base64 or URL-encoding a secret before printing it will not be masked. Attackers who can execute arbitrary steps in your pipeline know this.
Splitting across lines: Some CI platforms only mask exact string matches. A secret split across multiple print statements may not be caught.
Third-party actions: An action with access to GITHUB_TOKEN or any injected env var can exfiltrate it over the network, bypassing log masking entirely. Always pin third-party actions to a full commit SHA, not a tag — tags are mutable and can be hijacked.

# Bad: using a mutable tag — the action could be replaced under your feet
- uses: some-org/some-action@v3

# Good: pin to a specific commit SHA — immutable
- uses: some-org/some-action@a81bbbf8298c0fa03ea29cdc473d45769f953675

# Also good: use Dependabot to keep pinned SHA dependencies up to date
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

Detecting Secret Leaks Before They Hit the Repo

The cheapest fix is catching a leaked secret before it is ever committed. Two complementary tools:

Pre-commit hooks with Gitleaks or detect-secrets: Scans staged changes before a commit is made. Runs in milliseconds. Catches secrets locally before they touch the remote.
CI secret scanning step: Gitleaks as a CI job step scans every pull request diff. GitHub's built-in Secret Scanning runs on every push to a repository and can block pushes via push protection.

# .github/workflows/security.yml — Gitleaks secret scan on every PR

name: Secret Scan

on:
  pull_request:

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # Full history — scan all commits in the PR

      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

Pro practice — pre-commit hooks at scale: Enforce pre-commit hooks across a large engineering org using a tool like pre-commit (the Python framework) with a shared config in a central repo. Require gitleaks and detect-secrets as hooks. Make the hook install mandatory in your developer onboarding script. This shifts detection left by days — before the PR, before code review, before any CI even runs.

Common Failure Modes at Production Scale

Secrets in build arguments: docker build --build-arg API_KEY=$API_KEY bakes the value into the image layer history. Anyone who pulls the image can run docker history --no-trunc and read it. Use multi-stage builds and mount secrets via BuildKit: RUN --mount=type=secret,id=api_key ...
Overly broad OIDC trust: Forgetting to scope the sub claim means any branch or PR can assume the production deploy role. Lock it to ref:refs/heads/main and use environment protection rules.
Logging sensitive env vars: A debugging step like env | sort will print every environment variable — including injected secrets — to the log. Remove debug steps before merging; better yet, never print the entire environment.
Shared runner pools: On GitHub Actions public runners, each job gets a fresh ephemeral VM. On self-hosted runners, a malicious job can leave files or environment modifications that bleed into the next job on the same runner. Use ephemeral self-hosted runners (the runner registers, runs one job, then terminates) or isolate runners per environment.