Asynchronous Processing & Messaging

Publish/Subscribe: Topics & Fan-Out

18 min Lesson 3 of 10

Publish/Subscribe: Topics & Fan-Out

In the previous lesson we explored point-to-point message queues, where one producer sends a message and exactly one consumer processes it. Publish/Subscribe (Pub/Sub) breaks that 1-to-1 constraint: a single published event can be delivered to many independent subscribers simultaneously. This fan-out capability is fundamental to decoupled, event-driven architectures at companies like Uber, Spotify, and LinkedIn.

Core idea: Publishers and subscribers never know about each other. A publisher sends a message to a named topic (or channel); the broker copies and routes that message to every active subscriber of that topic. Adding or removing a subscriber requires zero changes to the publisher.

Topics: The Unit of Routing

A topic is a named logical channel. Publishers write to a topic; subscribers declare interest in one or more topics. The broker handles discovery and delivery. You can think of a topic as a radio frequency: the broadcaster transmits on 99.7 FM without knowing who is listening — cars, kitchens, and phones all receive the same signal independently.

In a large system you design a topic taxonomy deliberately. A common pattern is a hierarchical name: orders.placed, orders.shipped, orders.cancelled. Some brokers (like MQTT) support wildcards — a subscriber to orders.* receives all three. Others (like Kafka) require an exact topic name, so you subscribe to each explicitly.

Fan-Out: One Event, Many Consumers

Fan-out is what makes Pub/Sub powerful. Consider a user placing an order on an e-commerce platform. That single orders.placed event might need to:

Trigger the inventory service to reserve stock
Trigger the email service to send a confirmation
Trigger the analytics pipeline to record a conversion
Trigger the fraud detection service to score the transaction
Trigger the loyalty service to award points

In a point-to-point queue you would need five separate queues, five separate sends, and the order service would need to know about every downstream consumer. With Pub/Sub, the order service publishes once to orders.placed — the broker fans out to all five subscribers. Adding a sixth consumer (say, a warehouse picking system) requires no change to the order service at all.

Pub/Sub fan-out: the order service publishes once; the broker delivers a copy to every subscriber independently.

Push vs Pull Delivery

Brokers deliver messages to subscribers in one of two modes:

Push: The broker calls the subscriber (e.g. sends an HTTP webhook or pushes to an open connection). The subscriber is passive. Google Pub/Sub and AWS SNS operate this way by default. Fast delivery, but if the subscriber is temporarily down, the broker must buffer and retry.
Pull: The subscriber polls the broker for new messages. Kafka and AWS SQS use pull. The subscriber controls its own consumption rate, making pull naturally resilient to slow consumers — they simply poll slower. Pull is the dominant model for high-throughput systems.

Design guidance: Prefer pull for high-throughput, batch-oriented consumers (analytics pipelines, ML feature stores). Use push for low-latency notifications where the subscriber count is small and SLA is strict (webhooks, real-time dashboards).

Consumer Groups: Competing vs Fan-Out

A nuance that trips up many engineers: Pub/Sub and point-to-point queues can coexist in the same broker through consumer groups (Kafka's term; other systems use "subscriptions" or "queues").

Consider an analytics service running five worker instances for parallelism. You want:

Fan-out across services: Both Analytics AND Email AND Fraud all receive every event. (These are separate subscriptions / consumer groups.)
Competing consumers within a service: The five Analytics workers share the workload — each event is processed by exactly one worker. (These workers share one consumer group.)

Kafka achieves this cleanly: each consumer group gets an independent copy of the topic's data; within a group, partitions are divided among consumers so each message goes to only one of them. At 1 million events per second, you might run 50 partitions with 10 consumer instances per group — each instance processing 20,000 events/sec.

Each consumer group receives an independent copy of the topic (fan-out). Within a group, partitions are split across workers (competing consumers).

Exactly-Once, At-Least-Once, and Fan-Out Semantics

Delivery guarantees interact with fan-out in important ways. Most production Pub/Sub systems default to at-least-once delivery: the broker will retry if it does not receive an acknowledgment, which can result in duplicate messages. Because the broker fans out to every subscriber independently, a duplicate that reaches the Analytics group does not affect the Email group — failures are isolated per subscriber. This is a key advantage over synchronous, in-process fan-out, where a slow email sender would block the analytics pipeline.

Pitfall — confusing fan-out with broadcast: Pub/Sub is not a broadcast to all clients in real time. It delivers to registered subscribers only. For true real-time broadcast to thousands of browser clients you need WebSockets or Server-Sent Events on top of a Pub/Sub backbone — the broker fans out to a small fleet of WebSocket servers, each of which then pushes to its connected browser clients.

Topic Design: Coarse vs Fine-Grained

One of the most consequential early decisions in a Pub/Sub system is how granular your topics should be. Two extremes:

One fat topic (events) — every event type flows through one topic. Simple to operate; subscribers must filter by event type themselves, which wastes CPU and adds coupling.
Many fine topics (orders.placed, orders.shipped, ...) — each subscriber only receives what it subscribes to. Clean; but topic proliferation (thousands of topics) can stress broker metadata and monitoring tooling.

A good heuristic: create a topic per aggregate and lifecycle event. For an order domain, orders.placed, orders.shipped, orders.cancelled, orders.refunded is appropriate. Avoid creating a separate topic per customer — that leads to millions of topics and is an anti-pattern in every major broker.

Real-world numbers: Spotify runs roughly 800 topics across their internal Pub/Sub infrastructure. LinkedIn Kafka carries over 7 trillion messages per day across ~100,000 topics. Topic granularity is a design choice — start coarser than you think you need, and split only when a team genuinely needs independent scaling or access control.

Choosing a Pub/Sub System

The choice of broker shapes latency, durability, ordering, and cost:

Apache Kafka — highest throughput (millions of msgs/sec), durable log, replay capability. Steep operational overhead. Best for streaming data pipelines and event sourcing. (Covered in depth in Lesson 4.)
Google Cloud Pub/Sub — fully managed, global, at-least-once, push + pull. Good default for GCP-hosted services.
AWS SNS + SQS — SNS is the fan-out layer; SQS is the durable queue for each subscriber. Classic pattern: SNS topic → multiple SQS queues, one per subscriber group.
Redis Pub/Sub — ultra-low latency (sub-millisecond), in-memory only. No persistence, no delivery guarantee if a subscriber is offline. Fine for ephemeral real-time notifications; unsuitable for critical event pipelines.
NATS / NATS JetStream — lightweight, cloud-native, excellent for microservices. JetStream adds persistence and at-least-once delivery.

The Pub/Sub pattern is a cornerstone of loosely coupled, horizontally scalable systems. Mastering topics, fan-out, and consumer groups lets you replace brittle point-to-point HTTP calls between services with a durable, observable, independently scalable message backbone.