Python for DevOps Automation

Working with JSON, YAML & Config

18 min Lesson 3 of 28

Working with JSON, YAML & Config

DevOps scripts live and die by their ability to read, validate, and emit structured data. Kubernetes manifests, Terraform variables, CI pipeline definitions, feature flags, and Ansible inventories are all YAML or JSON. Knowing how to handle these formats correctly — including the production failure modes — separates a reliable ops engineer from someone whose script silently corrupts a production config.

JSON: The Universal API Language

Python's json module is part of the standard library and covers 95 % of real-world JSON work. The key operations are json.loads() (string to dict), json.dumps() (dict to string), json.load() (file handle to dict), and json.dump() (dict to file handle).

import json
from pathlib import Path

# --- Parsing JSON from a string (common: API response body) ---
raw = '{"service": "api-gateway", "replicas": 3, "healthy": true}'
cfg = json.loads(raw)
print(cfg["replicas"])   # 3  (int, not string)

# --- Reading a JSON file ---
config_path = Path("/etc/myapp/config.json")
with config_path.open() as fh:
    config = json.load(fh)

# --- Emitting JSON (pretty-printed, stable key order) ---
output = json.dumps(config, indent=2, sort_keys=True)
print(output)

# --- Writing a JSON file atomically ---
tmp = config_path.with_suffix(".json.tmp")
with tmp.open("w") as fh:
    json.dump(config, fh, indent=2)
tmp.replace(config_path)   # atomic rename — never leaves a half-written file

Never write config files with a plain open+write. If your script crashes mid-write, you leave a truncated file that silently breaks the service on next restart. Always write to a temp file in the same directory, then rename() (or Python's Path.replace()). On Linux, same-filesystem renames are atomic at the kernel level.

YAML: The Config Format of the Cloud-Native Stack

YAML is not in the standard library. The production-grade choice is PyYAML (import name yaml). Always use yaml.safe_load() — never yaml.load() without an explicit Loader argument, because the default loader can execute arbitrary Python code embedded in a YAML file, which is a critical remote-code-execution vector.

# pip install pyyaml
import yaml
from pathlib import Path

# --- Reading a Kubernetes manifest ---
manifest_text = Path("deployment.yaml").read_text()
manifest = yaml.safe_load(manifest_text)

name = manifest["metadata"]["name"]
image = manifest["spec"]["template"]["spec"]["containers"][0]["image"]
print(f"Deploying {name} with image {image}")

# --- Reading a file with multiple YAML documents (---separator) ---
with open("multi-doc.yaml") as fh:
    docs = list(yaml.safe_load_all(fh))   # returns a generator — consume it

# --- Emitting YAML ---
data = {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {"name": "api-gateway", "namespace": "production"},
    "spec": {"replicas": 3},
}
print(yaml.dump(data, default_flow_style=False, sort_keys=False))

Preserve key order in YAML output. Pass sort_keys=False to yaml.dump(). Kubernetes and Helm do not require a specific order, but humans reading diffs expect apiVersion to come before spec. Sorted keys make diffs noisy and code reviews harder.

The YAML Norway Problem and Other Gotchas

YAML has several notorious parsing surprises that have caused real production outages. The most famous is the Norway Problem: YAML 1.1 (which PyYAML still uses by default) parses bare NO as boolean False. Country codes in Ansible inventories or environment lists can silently become False. Always quote strings that look like booleans or null: "NO", "yes", "null", "true", "on".

Other common traps: a bare integer key (123: value) becomes a Python int key, not a string — breaking downstream dict["123"] lookups. Octal literals (0777) are parsed as integers. And YAML timestamps (2024-01-15) become Python datetime.date objects.

Structured config data flow: load from file, parse to Python dict, validate schema, then consume — failing fast on bad input.

Validating Config with jsonschema and Pydantic

Parsing a YAML file without validating it means your script will fail later with a confusing KeyError or TypeError deep inside business logic. Validate at the boundary — immediately after loading, before any processing. Two options dominate at big tech:

jsonschema — validates any dict against a JSON Schema definition; works for both JSON and YAML data; minimal dependency.
Pydantic v2 — defines models as Python classes; gives you typed attributes, default values, and rich error messages; preferred for complex configs and when you also need IDE autocompletion.

# pip install pydantic
from pydantic import BaseModel, Field, ValidationError
from typing import List
import yaml, sys

class ContainerSpec(BaseModel):
    image: str
    port: int = Field(gt=0, lt=65536)
    env_vars: List[str] = []

class DeployConfig(BaseModel):
    service: str
    namespace: str = "default"
    replicas: int = Field(ge=1, le=100)
    container: ContainerSpec

raw = yaml.safe_load(open("deploy.yaml"))

try:
    cfg = DeployConfig(**raw)
except ValidationError as exc:
    print("Config validation failed:", file=sys.stderr)
    for err in exc.errors():
        print(f"  {err['loc']}: {err['msg']}", file=sys.stderr)
    sys.exit(1)

# Now cfg.container.image is a guaranteed str — safe to use
print(f"Deploying {cfg.service} x{cfg.replicas} in {cfg.namespace}")

Fail fast, fail loudly. A config validation error caught at startup saves hours of debugging a cascading failure at 3 AM. Pydantic prints the full path to the bad field (e.g. container > port) and a human-readable message. Ship that message to your logging system before exiting.

Environment-Based Config: The Twelve-Factor Way

Production services should not read secrets from YAML files checked into git. The Twelve-Factor App pattern stores credentials, database URLs, and API keys in environment variables and reads config files only for non-secret, version-controllable settings. Python's os.environ and the python-dotenv package handle this cleanly.

import os
from dotenv import load_dotenv   # pip install python-dotenv

# Loads .env file into os.environ (no-op if file absent — safe in production)
load_dotenv()

DB_URL   = os.environ["DATABASE_URL"]          # KeyError if missing — intentional
API_KEY  = os.environ.get("API_KEY", "")       # Optional with default
LOG_LVL  = os.environ.get("LOG_LEVEL", "INFO").upper()

# Combine: base config from YAML + secrets from env
import yaml
with open("base_config.yaml") as fh:
    config = yaml.safe_load(fh)

config["db"]["url"] = DB_URL   # Override YAML placeholder with real secret
config["api_key"]   = API_KEY

Use os.environ["KEY"] (not .get()) for required secrets. If a required variable is missing, you want a loud KeyError at startup, not a silent empty string that causes a mysterious auth failure 200 requests later. Reserve .get("KEY", default) for genuinely optional settings.

TOML: The Python Ecosystem's Own Config Format

Since Python 3.11, tomllib is in the standard library (read-only). It is the format of pyproject.toml and is increasingly used for tool configs. If you need to write TOML, install tomli-w.

The full decision framework: use JSON when talking to APIs or machines; use YAML for Kubernetes, Ansible, and CI pipelines (humans + machines); use TOML for Python project metadata and tool configs; use env vars for secrets and runtime overrides.