Platform Engineering & Developer Experience

Internal Developer Platforms

18 min Lesson 2 of 28

Internal Developer Platforms

An Internal Developer Platform (IDP) is the curated set of tools, workflows, and self-service abstractions that a platform team exposes to product engineers. It is not a portal, not a wiki, and not a monolithic tool — it is an opinionated integration layer that sits between raw infrastructure (Kubernetes, Terraform, Vault, Datadog) and the cognitive surface area a developer should ever need to touch. Google's internal Borg/Monarch stack, Netflix's Paved Road, and Spotify's Backstage-based platform are canonical examples. Every serious large-scale engineering organisation has converged on this pattern because the alternative — every team managing its own infra surface — does not scale.

Anatomy of an IDP

A production-grade IDP is composed of three interlocking layers:

Golden Paths — opinionated, pre-built software delivery pipelines and runtime configurations. A golden path for a Python microservice bundles: a project scaffold (Cookiecutter template), a CI pipeline (.github/workflows or Tekton TriggerTemplate), a Kubernetes Deployment/HPA/PDB, service-mesh mTLS, RBAC policy, and Datadog APM injection. The developer clones one template and gets production-ready infra on day one.
Self-Service Catalogue — the mechanism by which teams request and configure platform resources (databases, queues, object storage, DNS entries, feature-flag namespaces) without filing tickets. Backed by Backstage scaffolder actions, Crossplane Compositions, or GitOps PRs auto-merged via policy engines.
Paved Roads — the runtime guardrails: pre-approved base container images, enforced resource quotas, OPA/Kyverno admission policies, validated Helm chart versions, and the observability pipeline that every workload inherits automatically. A team that stays on the paved road never has to configure Fluentd, TLS certificates, or PodDisruptionBudgets manually.

IDP layer model: developers interact only with the platform surface; raw infra is abstracted away.

Golden Paths in Practice

A golden path is not enforced by policy — teams can deviate — but it is the path of least resistance. Spotify found that 80 % of their microservices followed the golden path within six months of introduction, not because of mandates, but because starting from a compliant template was simply faster than configuring Kubernetes networking from scratch.

The canonical implementation today uses Backstage Software Templates (scaffolder). A template YAML defines user inputs, the source repository to clone, and a sequence of actions (fetch:template, publish:github, catalog:register, trigger:ci):

# backstage/templates/python-microservice/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: python-microservice
  title: Python Microservice (Golden Path)
  description: Production-ready FastAPI service with CI, k8s, mTLS, and observability.
  tags: [python, microservice, recommended]
spec:
  owner: platform-team
  type: service
  parameters:
    - title: Service Details
      required: [name, owner, environment]
      properties:
        name:
          type: string
          pattern: '^[a-z][a-z0-9-]{2,40}$'
        owner:
          type: string
          ui:field: OwnerPicker
        environment:
          type: string
          enum: [prod, staging, sandbox]
  steps:
    - id: fetch-template
      name: Fetch skeleton
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}
    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: github.com?owner=acme&repo=${{ parameters.name }}
        defaultBranch: main
        repoVisibility: internal
    - id: register
      name: Register in catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml
    - id: trigger-ci
      name: Trigger bootstrap pipeline
      action: http:backstage:request
      input:
        method: POST
        path: /api/proxy/tekton/trigger
        body:
          serviceName: ${{ parameters.name }}

Maintain a template registry in your Backstage catalog. Mark templates as recommended, deprecated, or experimental. Run automated conformance checks (Open Policy Agent + GitHub Actions) on the generated output so that a template update immediately catches drift in existing services via scheduled policy scans.

Self-Service Infrastructure

Self-service means developers provision what they need — a PostgreSQL RDS instance, an S3 bucket, a Redis cluster — without opening a ticket to an ops team. The leading implementation pattern is Crossplane Compositions: Kubernetes custom resources that wrap cloud provider resources and enforce platform standards (encryption at rest, backup schedules, private subnets only). The developer applies a YAML object; Crossplane reconciles it into actual AWS/GCP resources.

# Developer applies this (simple, opinionated)
apiVersion: platform.acme.io/v1alpha1
kind: PostgresDatabase
metadata:
  name: orders-db
  namespace: orders-team
  labels:
    app: orders-service
    cost-center: commerce
spec:
  size: medium          # maps to db.t3.large; enforced by Composition
  environment: prod
  owner: orders-team
  backupRetentionDays: 30
---
# Platform Composition enforces: private subnet, encrypted, backup window, tagging
# Developers never write aws_db_instance Terraform — the Composition handles it.
# Status surface:
#   kubectl get postgresdb orders-db -o jsonpath='{.status.atProvider.endpoint}'

An alternative for teams already deep in Terraform is Terraform Cloud workspaces triggered via API. The developer merges a PR into a infra-requests repo; a GitHub Actions workflow calls the TFC API to apply a workspace scoped to that team's AWS account. The platform team owns the module; the developer provides only variable values.

Self-service only scales if the underlying modules are heavily tested (Terratest, Checkov, tfsec) and opinionated. An ungoverned self-service portal where developers write raw Terraform is worse than a ticket system — you accumulate snowflake configurations at the speed of every team's autonomy.

Paved Roads and Guardrails

The paved road is what a service gets for free by staying on the golden path. Concretely: a Kubernetes admission webhook (Kyverno) prevents deployment of non-approved base images; a Linkerd or Istio mesh injects mTLS sidecars automatically; resource requests and limits are validated; OPA denies any pod running as root or without a readiness probe. Observability is not opt-in — every service in the mesh emits RED metrics and distributed traces to Datadog or Tempo without a single line of instrumentation code.

The most common IDP failure mode is paved roads without escape hatches. When a team has a legitimate edge case (a GPU workload, a latency-sensitive service that cannot afford sidecar overhead, a legacy app that cannot change its base image), a rigid platform creates friction and drives shadow IT. Always document the escape hatch: the process to get a policy exception, who approves it, and how the deviation is tracked in the catalog so it can be revisited.

The Internal Feedback Loop

An IDP that nobody adopts is infrastructure theatre. Treat the platform as a product with real users (internal developers). Instrument your golden paths: track time-to-first-deploy, number of template instantiations, policy violation rates, and self-service request lead time. We cover Developer Experience Metrics in depth in lesson 6; the key point here is that the IDP must expose these metrics to its own platform team on a live dashboard. Google's DORA metrics program originated exactly this way — measuring the impact of internal platform improvements on deployment frequency and change failure rate.

Run a quarterly Platform Demo Day: product teams demo how they used the IDP to ship faster. Nothing drives adoption like peer visibility. Pair it with a public roadmap in Backstage itself — use a TechDocs page that lists upcoming golden paths so teams know what to wait for versus what to build themselves.