Resilience, Messaging & Observability

Streaming Basics with Kafka

18 min Lesson 7 of 12

Streaming Basics with Kafka

The previous lessons in this tutorial covered resilience patterns — circuit breakers, retries, bulkheads — that protect a service from downstream failures. Those patterns still assumed a synchronous request/response model. Apache Kafka introduces a fundamentally different model: event streaming. Instead of a caller waiting for a reply, a producer appends an immutable event to a durable log, and one or more consumers read from that log independently and at their own pace.

This lesson introduces Kafka's core abstractions and shows how to produce and consume messages from a Spring Boot 3 application using Spring Cloud Stream — the framework-level abstraction that lets you switch message brokers without rewriting business logic.

Why Kafka?

Traditional message queues (RabbitMQ, ActiveMQ) delete a message once it has been successfully consumed. Kafka is different: it is a distributed, replicated commit log. Messages are retained for a configurable period (hours, days, weeks) regardless of whether anyone has read them. This gives you several properties that are difficult to achieve with queues:

Replay: a new consumer can start from the beginning of the log and reconstruct state.
Fan-out at scale: thousands of independent consumer groups can read the same topic without any coordination.
Exactly-once semantics (with careful configuration): transactions and idempotent producers let you guarantee that an event is processed exactly once even under failure.
Ordering guarantees: messages within a single partition are strictly ordered.

Kafka is not a queue replacement in all cases. If you need complex routing, per-message TTL, or priority queues, RabbitMQ may still be the right tool. Kafka excels at high-throughput, durable, replayable event streams — audit logs, domain event buses, real-time pipelines.

Core Kafka Concepts

Before writing any code, make sure the vocabulary is clear:

Topic: a named, append-only log. Think of it as a category for your events (e.g., order-placed, payment-processed).
Partition: a topic is split into one or more partitions for parallelism. Each partition is an independent ordered log.
Offset: the sequential position of a message within a partition. Kafka never removes messages by default; consumers track their own offset.
Producer: writes records to a topic. It chooses which partition to write to (round-robin, hash of key, or custom).
Consumer group: one or more consumers sharing a group ID. Kafka assigns partitions to group members so each partition is consumed by exactly one member at a time. Multiple groups can read the same topic independently.
Broker: a Kafka server. A cluster has multiple brokers for replication and load distribution.

Spring Cloud Stream Overview

Spring Cloud Stream (SCS) wraps the Kafka client behind a binder abstraction. Your application code works with functional beans — Supplier, Function, and Consumer from java.util.function. SCS maps these to Kafka topics via configuration, letting you swap the binder (to RabbitMQ, for example) without touching business logic.

Add the Kafka binder to pom.xml:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

Spring Cloud BOM (managed via the parent or dependencyManagement) keeps the version aligned with your Spring Boot version. For Spring Boot 3.x use Spring Cloud 2023.x.

Defining a Producer with Supplier

A Supplier<T> bean is polled by the framework on a schedule and its return value is sent as a message. This is useful for polling-based sources (reading from a database, generating heartbeat events). For event-driven production, use StreamBridge instead (shown after the consumer example).

import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
import java.time.Instant;
import java.util.function.Supplier;

@Component
public class OrderEventSource {

    // Called every second by default; return value is sent to the bound output topic
    @Bean
    public Supplier<OrderPlacedEvent> orderSupplier() {
        return () -> new OrderPlacedEvent("ORD-" + System.nanoTime(), Instant.now());
    }
}

Bind it to a Kafka topic in application.yml:

spring:
  cloud:
    stream:
      bindings:
        orderSupplier-out-0:          # <beanName>-out-<index>
          destination: order-placed   # Kafka topic name
          contentType: application/json
      kafka:
        binder:
          brokers: localhost:9092

Naming convention: SCS builds binding names automatically as <functionName>-in-0 (input) and <functionName>-out-0 (output). The 0 is the index for multi-input/output functions. Always double-check the binding name or set it explicitly via spring.cloud.function.definition.

Defining a Consumer with java.util.function.Consumer

A Consumer<T> bean receives messages from its bound input topic:

import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
import java.util.function.Consumer;

@Component
public class OrderEventHandler {

    @Bean
    public Consumer<OrderPlacedEvent> processOrder() {
        return event -> {
            // business logic here — e.g., trigger inventory check
            System.out.println("Processing order: " + event.orderId());
        };
    }
}

spring:
  cloud:
    stream:
      bindings:
        processOrder-in-0:
          destination: order-placed
          group: inventory-service    # consumer group ID
          contentType: application/json

Setting group is critical in production. Without it, SCS creates an anonymous group on every restart, so your service always starts reading from the latest offset — missed events from any downtime are permanently lost.

Imperative Production with StreamBridge

When you need to send a message in response to an HTTP request or some other trigger (not on a schedule), inject StreamBridge:

import org.springframework.cloud.stream.function.StreamBridge;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final StreamBridge streamBridge;

    public OrderService(StreamBridge streamBridge) {
        this.streamBridge = streamBridge;
    }

    public void placeOrder(Order order) {
        // ... save to DB ...
        OrderPlacedEvent event = new OrderPlacedEvent(order.getId(), order.getCreatedAt());
        streamBridge.send("order-placed", event);   // topic name or binding name
    }
}

Message Keys and Partitioning

Kafka guarantees ordering only within a partition. If you need all events for the same order to be processed in order, they must land on the same partition. You achieve this by setting a message key. SCS exposes this via the KafkaHeaders.MESSAGE_KEY header:

import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;

Message<OrderPlacedEvent> message = MessageBuilder
    .withPayload(event)
    .setHeader(KafkaHeaders.MESSAGE_KEY, event.orderId().getBytes())
    .build();

streamBridge.send("order-placed", message);

Kafka hashes the key to decide the partition, so all events sharing the same orderId land on the same partition and are processed in the order they arrived.

Error Handling and Dead-Letter Topics

If a consumer throws an exception, SCS retries by default (configurable via maxAttempts). After exhausting retries the message can be sent to a dead-letter topic (DLT) instead of being silently dropped:

spring:
  cloud:
    stream:
      kafka:
        bindings:
          processOrder-in-0:
            consumer:
              enable-dlq: true          # send failed messages to order-placed.DLT
              dlq-name: order-placed.DLT
              max-attempts: 3
              back-off-initial-interval: 1000   # ms
              back-off-max-interval: 10000
              back-off-multiplier: 2.0

Never silently swallow consumer exceptions. A catch block that logs and returns successfully will advance the offset, permanently losing the event. Instead, let the exception propagate so SCS can apply its retry and DLT logic. Monitor your DLT topics — unprocessed events there are a sign of a bug or data contract mismatch.

Security Considerations

Production Kafka clusters should enforce authentication and encryption. The most common approach is SASL/SCRAM over TLS:

spring:
  kafka:
    security:
      protocol: SASL_SSL
    properties:
      sasl.mechanism: SCRAM-SHA-512
      sasl.jaas.config: >
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="${KAFKA_USER}"
        password="${KAFKA_PASS}";
    ssl:
      trust-store-location: classpath:kafka-truststore.jks
      trust-store-password: ${KAFKA_TRUSTSTORE_PASS}

Always load credentials from environment variables or a secrets manager (Vault, AWS Secrets Manager). A plaintext password in application.yml committed to source control is as dangerous as an exposed database password.

Local Development with Docker Compose

Spin up a minimal Kafka cluster for local development with a single docker-compose.yml:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on: [zookeeper]
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Alternatively, use the newer KRaft mode (Kafka without Zookeeper) available from Confluent's cp-kafka 7.4+ images with a single container.

Summary

Kafka's persistent, replayable log model makes it the foundation of event-driven microservice architectures. Spring Cloud Stream lets you produce and consume Kafka messages through idiomatic Spring beans — Supplier, Consumer, and StreamBridge — with topology wired entirely through configuration. The patterns you need to master immediately are: always set a consumer group, use message keys for ordering guarantees, and configure a dead-letter topic so no event is silently lost. The next lesson covers distributed tracing, which becomes essential once events flow asynchronously across services.