Secrets Management & PKI

Cloud-Native Secret Stores

18 min Lesson 5 of 28

Cloud-Native Secret Stores

When you run workloads on AWS, you do not need a self-managed Vault cluster for every secret. AWS provides two purpose-built managed services — AWS Secrets Manager and AWS Systems Manager Parameter Store (SSM) — plus a key management backbone called KMS that underpins encryption for both. Understanding when to use each service, how KMS envelope encryption actually works under the hood, and where production teams get burned is the core of this lesson.

AWS Secrets Manager vs. SSM Parameter Store

Both services store sensitive configuration, but they target different use cases and have meaningfully different cost models and operational characteristics.

AWS Secrets Manager is the right choice when you need:

Automatic rotation — Secrets Manager has native integration with RDS, Redshift, DocumentDB, and a Lambda-based extensible rotation framework. It handles the credential swap atomically: creates the new credential, updates the database, updates the secret, and retires the old credential — all without application downtime.
Cross-account secret sharing — resource-based policies on a secret let you share it with other AWS accounts in your organization, which is the standard pattern for centralized secrets management in multi-account architectures.
Per-secret audit trail — every read, write, and rotation event is a CloudTrail API call tagged to the specific secret ARN. This granularity satisfies SOC 2 and PCI DSS audit requirements.

SSM Parameter Store is the right choice when you need:

Hierarchical configuration storage — parameters live under paths like /prod/myapp/db/password, and IAM policies can grant access to entire path prefixes. This is powerful for applications that read all their configuration from SSM at startup.
Cost-sensitive workloads — Standard parameters are free. SecureString parameters (KMS-encrypted) are also free at the standard tier. Secrets Manager charges $0.40 per secret per month plus $0.05 per 10,000 API calls. At scale with thousands of microservices each reading multiple secrets, the cost difference is significant.
Simple key/value config that does not require rotation — feature flags, environment names, non-rotating API keys, and structured config that does not need the full Secrets Manager lifecycle.

The practical rule at big tech: use Secrets Manager for database credentials, API keys, and OAuth client secrets that must rotate. Use SSM Parameter Store for all other application configuration, including non-sensitive values and configuration that rotates on a human schedule (manual rotation). Never store secrets as plaintext SSM Standard String parameters — always use SecureString.

KMS Envelope Encryption: How Your Data Is Actually Protected

Both Secrets Manager and SSM SecureString use AWS KMS to encrypt data at rest. Understanding the envelope encryption pattern is essential — it is the same pattern used by every major cloud provider and is what makes cloud-native secret stores both secure and performant at scale.

The naive approach to encryption would be: send your secret to KMS, KMS encrypts it with the CMK (Customer Master Key), you store the ciphertext. The problem: KMS has a 4 KB payload limit and is a network call — calling KMS directly for every read would be both expensive and slow at high request rates.

Envelope encryption solves this elegantly:

AWS generates a short-lived Data Encryption Key (DEK) using the KMS CMK. The DEK is returned in two forms: plaintext (used immediately for encryption) and ciphertext (the DEK encrypted by the CMK, stored alongside the data).
Your data (the secret value) is encrypted locally using the plaintext DEK with AES-256-GCM. This is fast and has no payload limit.
The plaintext DEK is discarded immediately after use. Only the encrypted DEK and the encrypted data are persisted.
At decryption time, the encrypted DEK is sent to KMS. KMS uses the CMK to decrypt it and returns the plaintext DEK. Your data is then decrypted locally.

The result: the CMK never directly touches your plaintext data and never leaves KMS hardware. All KMS operations are logged to CloudTrail. Revoking access to the CMK immediately makes all protected data permanently inaccessible — even to AWS employees.

KMS envelope encryption: data is encrypted locally with a DEK; only the DEK travels to KMS. Every KMS operation is logged to CloudTrail.

Working with AWS Secrets Manager

The following examples show the full lifecycle: creating a secret, retrieving it in code, and configuring automatic rotation for an RDS database credential.

# Create a secret with a KMS Customer Managed Key (CMK)
aws secretsmanager create-secret \
  --name "prod/myapp/db" \
  --description "Production RDS credentials for myapp" \
  --kms-key-id "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123" \
  --secret-string '{"username":"myapp","password":"s3cret-r0tates"}'

# Retrieve a secret value at runtime (returns JSON)
aws secretsmanager get-secret-value \
  --secret-id "prod/myapp/db" \
  --query SecretString --output text

# Enable automatic rotation every 30 days (RDS MySQL single-user rotation Lambda)
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db" \
  --rotation-lambda-arn "arn:aws:lambda:us-east-1:123456789012:function:SecretsManagerRDSMySQLRotationSingleUser" \
  --rotation-rules AutomaticallyAfterDays=30

# Retrieve in a Python application (best practice: cache in memory, refresh on AWSSDK exception)
# import boto3, json
# client = boto3.client('secretsmanager', region_name='us-east-1')
# secret = json.loads(client.get_secret_value(SecretId='prod/myapp/db')['SecretString'])
# conn = psycopg2.connect(host=DB_HOST, user=secret['username'], password=secret['password'])

Never call GetSecretValue on every database query. Retrieve the secret once at application startup, cache it in memory, and implement a retry-on-authentication-failure pattern: if the database rejects credentials, call GetSecretValue again (rotation may have occurred) and reconnect. This pattern works correctly with Secrets Manager rotation without any application downtime.

Working with SSM Parameter Store

SSM Parameter Store is excellent for hierarchical application configuration. The path-based structure aligns well with IAM least-privilege: a microservice's task role can be granted ssm:GetParametersByPath on /prod/myapp/* and nothing else.

# Store a SecureString parameter encrypted with a CMK
aws ssm put-parameter \
  --name "/prod/myapp/stripe/secret-key" \
  --type "SecureString" \
  --key-id "alias/prod-app-secrets" \
  --value "sk_live_XXXXXXXXXXXX" \
  --overwrite

# Retrieve a single parameter (--with-decryption is required for SecureString)
aws ssm get-parameter \
  --name "/prod/myapp/stripe/secret-key" \
  --with-decryption \
  --query "Parameter.Value" --output text

# Retrieve ALL parameters under a path prefix (use in container entrypoint to build env)
aws ssm get-parameters-by-path \
  --path "/prod/myapp" \
  --recursive \
  --with-decryption \
  --query "Parameters[*].{Name:Name,Value:Value}"

# Terraform: reference SSM parameter in infra code
# data "aws_ssm_parameter" "stripe_key" {
#   name = "/prod/myapp/stripe/secret-key"
# }
# Then reference: data.aws_ssm_parameter.stripe_key.value

IAM Policy Design for Least-Privilege Access

The most critical security control for both services is the IAM policy attached to the compute role (ECS task role, EC2 instance profile, Lambda execution role). Access must be scoped to specific resource ARNs — never Resource: "*" on secret-read actions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadSpecificSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/*"
      ]
    },
    {
      "Sid": "AllowKMSDecrypt",
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        }
      }
    }
  ]
}

The kms:ViaService condition in the KMS statement is important: it restricts the kms:Decrypt permission so the role can only decrypt KMS data when the request originates through Secrets Manager. The role cannot use the same KMS key to decrypt arbitrary ciphertext directly — tightly constraining the blast radius of a compromised role.

The most common production misconfiguration is granting secretsmanager:GetSecretValue with Resource: "*". Any compromised container in the account can then exfiltrate every secret. Always scope to the exact ARN prefix for the service. Use aws secretsmanager list-secrets + CloudTrail anomaly detection (GuardDuty findings: UnauthorizedAccess:IAMUser/AnomalousBehavior) to detect unauthorized enumeration.

Secrets Manager in Kubernetes and ECS

On Amazon EKS, the Secrets Store CSI Driver with the AWS provider mounts Secrets Manager or SSM Parameter Store values directly as Kubernetes volume files (or syncs them into native Kubernetes Secrets). The pod's service account IRSA role controls access. On Amazon ECS, the ECS Secrets integration injects Secrets Manager values as environment variables at container startup without any application code changes — you reference the secret ARN in the task definition.

Both patterns eliminate hardcoded credentials at the infrastructure layer. However, environment variable injection (the ECS default) has a nuance: the secret is read once at container launch and does not update if the secret rotates. For rotation-sensitive workloads on ECS, use the CSI Driver pattern (on EKS) or build refresh logic into the application.

AWS Secrets Manager caches secrets in the AWS SDK (the Java SDK and Python SDK both have a SecretCache helper). At high request rates — a service making tens of thousands of database connections per second — caching eliminates Secrets Manager API throttling (the default quota is 10,000 API calls per second per region, shared across all principals in the account). Build caching into every high-throughput consumer from day one.