Resilience, Messaging & Observability

Asynchronous Messaging

18 min Lesson 5 of 12

Asynchronous Messaging

Every distributed system eventually hits the same wall: if Service A calls Service B directly over HTTP and B is slow or down, A is also slow or down. The coupling is total. Asynchronous messaging breaks that coupling by placing a message broker between the two services. A produces a message and moves on; B consumes it whenever it is ready. Neither knows the other's address, uptime, or current load.

This lesson covers the concepts, Spring Boot wiring, and the operational trade-offs you need to make deliberate design decisions — not just copy-paste working code.

What a Message Broker Does

A broker is a server (RabbitMQ, Apache Kafka, Amazon SQS, etc.) that accepts messages from producers and holds them until consumers pull or receive them. The broker provides:

Durability — messages survive a consumer restart (with persistence enabled).
Back-pressure — if the consumer is slow, messages queue rather than causing the producer to fail.
Fan-out — one published message can be delivered to multiple independent consumers.
Temporal decoupling — producer and consumer do not need to be running at the same time.

Key mental model: HTTP is a phone call — both parties must be available simultaneously. Messaging is a postal system — the sender drops the letter, the recipient reads it later. Each model is correct for different problems.

Core Concepts: Exchanges, Queues, and Topics

RabbitMQ uses an exchange → binding → queue model. The producer publishes to an exchange (not a queue). The exchange routes the message to one or more queues based on a routing key and binding rules. Consumer applications subscribe to queues. Common exchange types:

Direct — routes to queues whose binding key exactly matches the routing key.
Topic — routing keys are dot-separated patterns; * matches one word, # matches zero or more.
Fanout — ignores routing keys and broadcasts to every bound queue.

Kafka uses a different vocabulary — topics and partitions — but the decoupling principle is identical. We focus on RabbitMQ via Spring AMQP here; Kafka gets its own lesson.

Adding Spring AMQP to a Spring Boot 3 Project

Add the starter to pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

Configure the broker connection in application.yml:

spring:
  rabbitmq:
    host: localhost
    port: 5672
    username: guest
    password: guest
    listener:
      simple:
        acknowledge-mode: manual   # we ack explicitly after processing
        prefetch: 5                # max unacknowledged messages per consumer

Use acknowledge-mode: manual in production. With auto-ack the broker removes the message the moment it is delivered, even if your code throws an exception. Manual ack lets you acknowledge after successful processing, so unprocessed messages are re-queued automatically.

Declaring Infrastructure as Beans

Spring AMQP can declare the exchange, queue, and binding automatically when your application starts. Define them as beans in a configuration class:

import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RabbitConfig {

    public static final String ORDER_EXCHANGE   = "orders.exchange";
    public static final String ORDER_QUEUE      = "orders.created";
    public static final String ROUTING_KEY      = "order.created";

    @Bean
    TopicExchange orderExchange() {
        return new TopicExchange(ORDER_EXCHANGE, true, false);
        // durable=true: survives broker restart
        // autoDelete=false: not deleted when last consumer disconnects
    }

    @Bean
    Queue orderQueue() {
        return QueueBuilder.durable(ORDER_QUEUE)
                .withArgument("x-dead-letter-exchange", "orders.dlx") // dead-letter exchange
                .build();
    }

    @Bean
    Binding orderBinding(Queue orderQueue, TopicExchange orderExchange) {
        return BindingBuilder
                .bind(orderQueue)
                .to(orderExchange)
                .with(ROUTING_KEY);
    }
}

Dead-letter exchange (DLX): When a message is rejected or expires, the broker routes it to the DLX instead of dropping it silently. Always configure a DLX for production queues so failed messages are inspectable rather than lost.

Producing a Message

RabbitTemplate is the thread-safe central component for sending. Inject it and publish:

import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final RabbitTemplate rabbitTemplate;

    public OrderService(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }

    public void placeOrder(Order order) {
        // persist order to DB first ...
        rabbitTemplate.convertAndSend(
            RabbitConfig.ORDER_EXCHANGE,
            RabbitConfig.ROUTING_KEY,
            order              // Jackson serialises to JSON automatically
        );
        // method returns immediately — no waiting for a consumer
    }
}

By default convertAndSend serialises using Java serialisation. Override this by declaring a MessageConverter bean:

import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RabbitConfig {
    // ... exchange / queue / binding beans above ...

    @Bean
    Jackson2JsonMessageConverter jsonMessageConverter() {
        return new Jackson2JsonMessageConverter();
    }
}

Consuming a Message

Annotate a method with @RabbitListener and Spring creates a listener container that polls the queue on a background thread pool:

import com.rabbitmq.client.Channel;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.amqp.support.AmqpHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Service;
import java.io.IOException;

@Service
public class InventoryService {

    @RabbitListener(queues = RabbitConfig.ORDER_QUEUE)
    public void handleOrder(Order order,
                            Channel channel,
                            @Header(AmqpHeaders.DELIVERY_TAG) long deliveryTag)
            throws IOException {

        try {
            reserveStock(order);
            channel.basicAck(deliveryTag, false);   // ack: remove from queue
        } catch (InsufficientStockException e) {
            // permanent failure: send to DLX, do NOT requeue
            channel.basicReject(deliveryTag, false);
        } catch (Exception e) {
            // transient failure: put back in queue for retry
            channel.basicNack(deliveryTag, false, true);
        }
    }

    private void reserveStock(Order order) { /* ... */ }
}

Idempotency: The Consumer's Most Important Contract

Because messages can be delivered more than once (network glitch, consumer crash before ack), your consumer must be idempotent: processing the same message twice must have the same effect as processing it once. Common strategies:

Store a processed-message ID table; skip if already seen.
Use database unique constraints so a duplicate insert fails harmlessly.
Design operations to be naturally idempotent (setting a value to X is safe to repeat; incrementing a counter is not).

Never assume at-most-once delivery in a distributed system. The AMQP spec guarantees at-least-once delivery. Design every consumer for idempotency from day one, or you will spend hours debugging phantom double-charges and duplicate records in production.

Security Considerations

A message broker is a trust boundary. Harden it:

TLS in transit — use spring.rabbitmq.ssl.enabled=true with mutual TLS between services and broker.
Per-service credentials — each microservice gets its own RabbitMQ user with least-privilege permissions (read from its queue, write to its exchange only).
Message-level integrity — if messages cross trust boundaries, sign the payload (e.g. HMAC-SHA256) and verify before processing. A rogue producer could inject fraudulent messages otherwise.
No PII in routing keys — routing keys can appear in broker logs; keep them structural, not data-bearing.

Choosing Between Synchronous HTTP and Async Messaging

Neither pattern dominates universally. Use this heuristic:

Use HTTP when the caller needs an immediate answer (e.g. query a product price before showing it to the user).
Use messaging when the caller only needs to know the work was accepted, not that it was finished (e.g. place an order, send a notification, trigger a background report).

Async messaging increases resilience and throughput but adds operational complexity: you need a running broker, dead-letter handling, consumer monitoring, and idempotency logic. That cost is worth paying once your system genuinely needs it.

Summary

Asynchronous messaging decouples services in time and space: the producer calls convertAndSend and continues; the consumer processes at its own pace. Spring AMQP wraps RabbitMQ with RabbitTemplate for sending and @RabbitListener for receiving. Use a Jackson2JsonMessageConverter for interoperability, manual acks for safety, a dead-letter exchange for failure visibility, and design every consumer to be idempotent. In the next lesson you will see how these patterns combine to build fully event-driven microservices.