Kubernetes Workloads & Configuration

Secrets

18 min Lesson 2 of 32

Secrets

Every production workload eventually needs a credential: a database password, an API key, a TLS certificate, an OAuth client secret. The worst mistake an engineer can make is encoding that credential in the container image or checking it into a Git repository. Kubernetes Secrets are the platform-native answer to that problem — but they come with important nuances around encoding, encryption, and access control that every professional must understand before going to production.

What a Secret Is (and Is Not)

A Secret is a Kubernetes object stored in etcd — the cluster's key-value store — alongside every other resource. By default, Secret values are base64-encoded, not encrypted. Base64 is not a security boundary; it is just a transport encoding that ensures binary-safe storage. Anyone with kubectl get secret access to the namespace can decode the value in one command. This is the single most important thing to internalise about Kubernetes Secrets: the API object by itself provides isolation, not confidentiality.

True confidentiality requires Encryption at Rest (encrypting the etcd data on disk via an EncryptionConfiguration API resource) combined with strict RBAC that limits get/list/watch on Secret objects to only the workloads and humans who need them.

Secret Types

Kubernetes recognises several built-in types, each validated and handled slightly differently:

Opaque — the default. An arbitrary key-value bag. Use this for passwords, API tokens, connection strings, and any custom credential.
kubernetes.io/dockerconfigjson — image pull credentials for private registries. Referenced via imagePullSecrets in the Pod spec. The API enforces a valid .dockerconfigjson structure.
kubernetes.io/tls — a TLS certificate/key pair. Keys must be named tls.crt and tls.key. Used by Ingress controllers and cert-manager.
kubernetes.io/service-account-token — auto-created by the control plane for each ServiceAccount. Mounted automatically into Pods unless automountServiceAccountToken: false is set.
kubernetes.io/basic-auth, kubernetes.io/ssh-auth — specialised types with enforced key names for basic-auth credentials and SSH private keys.

Secret types and their consumers. The dashed arrow represents External Secrets Operator syncing from an external store into a native Kubernetes Secret.

Creating Secrets

The safest way to create a Secret is imperatively from a file, so the plaintext never touches your shell history. Manifests that embed base64 values are acceptable only when stored in a secret-management tool (Sealed Secrets, Vault, or an external secrets operator), never as plain YAML in Git.

# --- From literal values (never use --from-literal in production; shell history logs it) ---
kubectl create secret generic db-creds \
  --from-literal=password=supersecret \
  --namespace=payments

# --- Better: from files, so the value never hits the shell ---
echo -n 'supersecret' > /tmp/db-password.txt   # -n strips the trailing newline
kubectl create secret generic db-creds \
  --from-file=password=/tmp/db-password.txt \
  --namespace=payments
shred -u /tmp/db-password.txt                  # destroy the temp file

# --- Create a TLS secret from an existing cert/key pair ---
kubectl create secret tls api-tls \
  --cert=server.crt \
  --key=server.key \
  --namespace=payments

# --- Create an image pull secret for a private registry ---
kubectl create secret docker-registry ghcr-creds \
  --docker-server=ghcr.io \
  --docker-username=myorg \
  --docker-password=$GHCR_TOKEN \
  --namespace=payments

# --- Inspect (base64 will be shown; decode manually) ---
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 --decode

Shell history is a credential store. Any secret value passed via --from-literal is written to your shell history file and may appear in audit logs, CI logs, or pair-programming sessions. Always write credentials to a file first (echo -n to avoid the trailing newline that breaks base64 round-trips), reference the file, then destroy it. On production break-glass scripts, set HISTFILE=/dev/null for the session.

Mounting Secrets into Pods

There are two ways to surface a Secret inside a container: as environment variables or as volume-mounted files. The volume mount is almost always preferable in production because environment variables are visible in process listings (/proc/<pid>/environ), logged by many frameworks on startup, and inherited by child processes. Kubernetes mounts Secret volumes on a tmpfs filesystem (in-memory, never written to disk), which is a meaningful security improvement over env vars.

# --- Manifest: both mount patterns side-by-side ---
apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: payments
spec:
  serviceAccountName: payments-app          # narrow SA, not default
  automountServiceAccountToken: false        # opt-out unless the app needs kube API access
  volumes:
    - name: db-secret-vol
      secret:
        secretName: db-creds
        defaultMode: 0400                    # read-only, owner only
        items:
          - key: password
            path: db-password               # mounted at /etc/secrets/db-password
  containers:
    - name: app
      image: ghcr.io/myorg/payments:v3.1.0
      # pattern 1: env var (acceptable for non-sensitive config; avoid for passwords)
      env:
        - name: DB_HOST
          value: "postgres.payments.svc.cluster.local"
        - name: DB_PASSWORD                  # SECRET — prefer volume for this
          valueFrom:
            secretKeyRef:
              name: db-creds
              key: password
      # pattern 2: volume mount (preferred for credentials)
      volumeMounts:
        - name: db-secret-vol
          mountPath: /etc/secrets
          readOnly: true
      # The app reads /etc/secrets/db-password at startup instead of an env var

Automatic Secret rotation: When a Secret is updated, Kubernetes propagates the new value to volume mounts within the kubelet sync period (default ~60 s for a mounted volume). Environment variables are not updated — the Pod must restart. For zero-downtime rotation of credentials, always prefer volume mounts and write your application to re-read the file on a signal or timer.

Encryption at Rest

By default, etcd stores all Secret data in plaintext (the base64 is decoded before persistence). To encrypt at rest, apply an EncryptionConfiguration to the API server. Managed Kubernetes services (EKS, GKE, AKS) all support this via a one-line cluster setting (e.g., --secrets-encryption-key-arn in EKS with an AWS KMS key).

In a self-managed cluster, the EncryptionConfiguration manifest is referenced by the API server flag --encryption-provider-config. The aescbc or aesgcm providers use a locally-managed key; the kms provider (strongly recommended) delegates to an external KMS so the key never lives on the cluster node at all.

Verify your encryption is actually active — checking the cluster setting is not enough. After enabling, run: kubectl get secret db-creds -o json | etcdctl get ... and confirm the raw etcd value starts with k8s:enc:aescbc (or your chosen provider). Many teams enable the setting but forget to re-encrypt existing Secrets with kubectl get secrets --all-namespaces -o json | kubectl replace -f -.

Production-Grade Secrets: Sealed Secrets and External Secrets Operator

Plain Kubernetes Secret manifests cannot be safely committed to Git because they contain base64 values any developer can decode. Two open-source projects solve this problem at scale:

Sealed Secrets (Bitnami) — A controller that holds a cluster-side RSA private key. You encrypt a Secret manifest into a SealedSecret CRD using the corresponding public key (kubeseal CLI). The sealed YAML is safe to commit to Git — only the cluster's controller can decrypt it. The controller reconciles SealedSecret objects into real Kubernetes Secrets inside the cluster. This fits a GitOps model perfectly: everything is in Git, nothing is plaintext.
External Secrets Operator (ESO) — A controller that syncs secrets from an external store (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, and many others) into Kubernetes Secrets. You define an ExternalSecret CRD that names the external store path and the target Secret name. ESO handles rotation — when the external secret is updated, ESO re-syncs within the configured refreshInterval. This is the preferred pattern at big-tech scale because it keeps the single source of truth outside the cluster, enables cross-cluster sharing, and integrates with existing secret management policies.

# --- Sealed Secrets: seal a Secret manifest before committing to Git ---

# 1. Fetch the cluster public key (once, store in repo)
kubeseal --fetch-cert \
  --controller-namespace=sealed-secrets \
  --controller-name=sealed-secrets > pub-cert.pem

# 2. Seal an existing Secret manifest
kubectl create secret generic db-creds \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml \
  | kubeseal --cert=pub-cert.pem --format=yaml > sealed-db-creds.yaml

# 3. Commit sealed-db-creds.yaml to Git — it is safe (asymmetrically encrypted)
git add sealed-db-creds.yaml && git commit -m "chore: add sealed DB credentials"

# --- External Secrets Operator: sync from AWS Secrets Manager ---
# ExternalSecret CRD (commit this to Git — no sensitive data)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager       # ClusterSecretStore pointing at AWS SM
    kind: ClusterSecretStore
  target:
    name: db-creds                 # the Kubernetes Secret ESO will create/update
    creationPolicy: Owner
  data:
    - secretKey: password          # key in the resulting K8s Secret
      remoteRef:
        key: payments/db           # AWS Secrets Manager secret name
        property: password         # JSON property inside the SM secret

Choosing between them: Use Sealed Secrets when your entire secret lifecycle lives inside one Kubernetes cluster and you already use GitOps (ArgoCD/Flux). Use External Secrets Operator when you have an existing secret store (Vault, AWS SM), need to share secrets across clusters, or require fine-grained rotation policies managed by a security team outside of Kubernetes. Most mature organisations use ESO.

RBAC for Secrets — the Principle of Least Privilege

Because Secrets are first-class API objects, standard Kubernetes RBAC controls who can read them. The single most impactful security practice is to ensure no application or human has list or watch on Secrets in namespaces they do not own. A list on Secrets returns all values — it is equivalent to a data breach. Scope every ServiceAccount Role to get on only the specific Secret names the application needs, never a wildcard.

Audit Secret access regularly: kubectl auth can-i get secrets --as=system:serviceaccount:payments:payments-app -n payments and cross-reference with your API server audit log for unexpected list/watch calls on the secrets resource.