Secrets Management & PKI

Secrets in Kubernetes & CI

18 min Lesson 6 of 28

Secrets in Kubernetes & CI

Native Kubernetes Secrets are base64-encoded, not encrypted. Any engineer with kubectl get secret access can read every password, API key, and TLS certificate in the namespace. At Google, Meta, and Netflix, the Kubernetes secrets store is treated as a distribution bus, not a vault — the real source of truth lives in HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager. The two tools that bridge that gap are External Secrets Operator (ESO) and the Secrets Store CSI Driver. For CI pipelines, the modern answer to credential-free secrets is OIDC keyless authentication. This lesson covers all three, plus the failure modes that leak secrets into build logs and environment dumps.

External Secrets Operator (ESO)

ESO runs as a controller inside the cluster. You create an ExternalSecret CRD that points at an entry in your external secret store; ESO fetches it on a configurable interval and materialises it as a standard Kubernetes Secret object. Applications read the native Secret — they do not need any SDK or sidecar. The external store remains the single source of truth; ESO is a read-only sync engine.

The two-layer model: a SecretStore (or ClusterSecretStore) holds the provider config and credentials, while an ExternalSecret declares which keys to sync. Separating them means one platform team manages store credentials while application teams author their own ExternalSecret objects.

# Install ESO via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace \
  --set installCRDs=true

---
# ClusterSecretStore: platform team owns this (one per cluster region)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

---
# ExternalSecret: application team authors this per service
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payments-db-creds
  namespace: payments
spec:
  refreshInterval: 1h          # re-sync every hour; rotate without pod restart
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payments-db-secret   # the K8s Secret ESO will create/update
    creationPolicy: Owner      # ESO owns the Secret; deleting ExternalSecret deletes Secret
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@pg.internal:5432/payments"
  data:
    - secretKey: username
      remoteRef:
        key: prod/payments/db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

Why creationPolicy: Owner matters: When the ExternalSecret is deleted, ESO garbage-collects the derived Kubernetes Secret automatically. Without this, orphaned Secrets persist in the namespace indefinitely, accumulating stale credentials that failed audits and confuse operators. Always set Owner in production; use Merge only when you intentionally combine multiple ExternalSecrets into one Secret.

Secrets Store CSI Driver

Where ESO materialises a Kubernetes Secret (which lands in etcd), the CSI driver takes a different approach: it mounts the secret directly into the pod filesystem as a file, bypassing etcd entirely. The secret never touches the Kubernetes control plane at rest. This satisfies stricter compliance requirements (PCI DSS Level 1, FedRAMP High) where even encrypted etcd storage is unacceptable.

The driver consists of a DaemonSet on each node and provider-specific plugins. The AWS provider uses IRSA (IAM Roles for Service Accounts); the Azure provider uses Managed Identity; the Vault provider uses the Vault agent injector pattern but without the sidecar.

# Install CSI driver + AWS provider
helm repo add secrets-store-csi-driver \
  https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store \
  secrets-store-csi-driver/secrets-store-csi-driver \
  -n kube-system \
  --set syncSecret.enabled=true \
  --set enableSecretRotation=true \
  --set rotationPollInterval=3600s

helm repo add aws-secrets-manager \
  https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system aws-provider \
  aws-secrets-manager/secrets-store-csi-driver-provider-aws

---
# SecretProviderClass: maps AWS secret to a mount path
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payments-db-spc
  namespace: payments
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/payments/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db_username
          - path: password
            objectAlias: db_password
  secretObjects:                        # optional: also sync to a K8s Secret
    - secretName: payments-db-csi
      type: Opaque
      data:
        - objectName: db_username
          key: username
        - objectName: db_password
          key: password

---
# Pod using the CSI mount
apiVersion: v1
kind: Pod
metadata:
  name: payments-api
  namespace: payments
spec:
  serviceAccountName: payments-sa      # must have IRSA annotation
  containers:
    - name: api
      image: payments-api:v2.3.1
      volumeMounts:
        - name: secrets-vol
          mountPath: "/mnt/secrets"
          readOnly: true
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: payments-db-csi
              key: username
  volumes:
    - name: secrets-vol
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: payments-db-cpc

ESO writes secrets into etcd as native Kubernetes Secrets; the CSI driver bypasses etcd entirely, mounting secrets directly into pod memory via tmpfs.

OIDC Keyless CI Authentication

The traditional CI secret problem: your pipeline needs an AWS or Vault token, so you store it in GitHub/GitLab CI variables — now that static credential is a long-lived secret that can be stolen from log output, environment dumps, or a compromised runner. OIDC keyless authentication eliminates static CI credentials entirely.

The mechanism: when a GitHub Actions job runs, GitHub's OIDC provider issues a short-lived JWT (audience-scoped, signed by GitHub) attesting which repository, branch, and workflow triggered the job. AWS (or Vault, GCP, Azure) is configured to trust GitHub's OIDC provider and exchange that JWT for a cloud credential valid for the duration of the job — typically 15 minutes.

# Step 1: Create the OIDC Trust in AWS (Terraform)
resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"]
}

resource "aws_iam_role" "github_deploy" {
  name = "github-deploy-payments"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.github.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          # Restrict to a specific repo AND the main branch — critical
          "token.actions.githubusercontent.com:sub" = "repo:my-org/payments:ref:refs/heads/main"
        }
      }
    }]
  })
}

---
# Step 2: GitHub Actions workflow — no secrets at all
name: Deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write   # REQUIRED: allows the runner to request an OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy-payments
          aws-region: us-east-1
          # No access-key-id or secret-access-key needed

      - name: Push image to ECR
        run: |
          aws ecr get-login-password | docker login --username AWS \
            --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
          docker build -t payments-api .
          docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/payments-api:$GITHUB_SHA

      - name: Run database migration via SSM (never log credentials)
        run: |
          aws ssm send-command \
            --document-name "AWS-RunShellScript" \
            --targets "Key=tag:Role,Values=payments-db" \
            --parameters 'commands=["cd /app && php artisan migrate --force"]' \
            --output-s3-bucket-name my-ssm-logs

Pro practice — sub claim scoping: The sub claim in the IAM trust policy is your primary blast-radius control. repo:my-org/payments:ref:refs/heads/main restricts the role to main-branch pushes only. A pull request from a fork cannot assume this role. In large organisations, create a separate role per environment (staging vs production) with different sub conditions — never let a staging pipeline assume a production deployment role.

Avoiding Env Leaks in CI and Kubernetes

Even when secrets are delivered correctly, they commonly leak through operational mistakes. These are the failure modes seen in real incident postmortems:

Log injection via set -x: Bash debug mode prints every command with its expanded arguments. A script that runs curl -H "Authorization: Bearer $TOKEN" ... with set -x active will print the token verbatim. Never use set -x in CI scripts that handle secrets; use set -e (fail fast) instead.
Environment dumps: env, printenv, kubectl exec -- env, and framework debug pages (Django DEBUG=True, Laravel APP_DEBUG=true) will print every environment variable. Disable debug endpoints in production and never run env in a CI log step.
Docker build-time ARGs: docker build --build-arg DB_PASSWORD=secret embeds the value in the image layer history, readable by anyone with docker history myimage. Use multi-stage builds and pass secrets at runtime; for build-time needs, use Docker BuildKit secret mounts (--secret id=db,src=.env).
Kubernetes Secret in YAML committed to Git: A base64-encoded secret in a committed YAML file is effectively plaintext. Use SOPS or Sealed Secrets to encrypt before committing, or better, do not commit secret values at all — store them in the external store and reference via ESO.
Resource quota dumps and describe output: kubectl describe pod shows env entries including values from valueFrom.secretKeyRef — those values appear masked in recent Kubernetes versions, but older clusters expose them. Audit who has describe RBAC on the namespace.

# Safe secret passing pattern in CI (GitHub Actions)
- name: Build with BuildKit secret (never in image layers)
  run: |
    docker buildx build \
      --secret id=npmrc,src=$HOME/.npmrc \
      --tag my-app:$GITHUB_SHA \
      .
  # In Dockerfile:
  # RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
  #     npm install

# Mask additional values that appear in logs
- name: Fetch and mask deploy token
  run: |
    DEPLOY_TOKEN=$(aws secretsmanager get-secret-value \
      --secret-id prod/deploy-token \
      --query SecretString --output text)
    echo "::add-mask::$DEPLOY_TOKEN"   # GitHub masks this value in all future log lines
    echo "DEPLOY_TOKEN=$DEPLOY_TOKEN" >> $GITHUB_ENV

Production pitfall — GITHUB_ENV exposure: Writing a secret to $GITHUB_ENV makes it available as an environment variable to subsequent steps, but it also appears in the Set up job log output in some runner versions. For secrets that must be passed between steps, prefer writing them to a temporary file with 600 permissions, or use GitHub Actions output variables with masking. Never write a raw secret value to a log step or an artifact upload.

ESO vs CSI Driver: When to Use Which

Both tools solve the same problem but suit different compliance postures. ESO is the right default for the majority of workloads: it is simpler to operate, integrates with any Kubernetes tooling that reads native Secrets, and supports rotation without pod restarts. Use the CSI driver when compliance mandates that secrets must never touch etcd — PCI DSS environments, government workloads, or scenarios where you have no control-plane encryption at rest.

In practice, large platforms run both: ESO for application secrets (database credentials, API keys) and the CSI driver for TLS private keys and HSM-backed material where the cryptographic boundary requirement is stricter.

Summary

Kubernetes native Secrets are a distribution mechanism, not a trust boundary. External Secrets Operator synchronises secrets from Vault or cloud secret stores into native Secrets via a pull model, keeping the external store as the source of truth. The CSI driver delivers secrets as tmpfs mounts, bypassing etcd for high-compliance environments. OIDC keyless CI authentication removes static credentials from CI pipelines entirely, replacing them with short-lived, scoped tokens tied to a specific repository and branch. Env leak prevention is an operational discipline — the tooling is necessary but not sufficient without log hygiene, Docker layer discipline, and RBAC scoping on describe access.