Capacity & Scaling Data Infrastructure

Capacity & Scaling Data Infrastructure

Sizing Redis and Kafka correctly — and knowing when and how to scale them — is one of the highest-leverage skills a production engineer can have. Get it wrong and you face silent data loss, cascading latency spikes, or a six-hour cluster migration at 2 AM. This lesson teaches the sizing models, scaling patterns, and upgrade strategies used at companies running hundreds of terabytes and millions of messages per second.

Redis: Capacity Sizing

Redis is an in-memory store, so every byte counts. The sizing formula is straightforward:

• Working set size: estimate the total byte footprint of all live keys using the OBJECT ENCODING command and Redis DEBUG JMAP (or the redis-memory-analyzer tool). Do not rely on INFO memory alone — used_memory_rss always exceeds used_memory due to fragmentation.
• Fragmentation ratio: target mem_fragmentation_ratio between 1.0 and 1.5. Ratios above 2.0 on Redis 6+ trigger active defragmentation (activedefrag yes) automatically, but you still need headroom.
• Headroom rule: provision at 60–70 % peak utilisation. A fully loaded Redis node cannot accept a BGSAVE fork (needs ~100 % of used_memory for CoW pages) without swapping, which destroys latency.

For eviction sizing, set maxmemory to 75 % of available RAM and choose an eviction policy before you need it. The big-tech default for cache workloads is allkeys-lru; for session stores use volatile-lru so only TTL-bearing keys are eligible. Never leave maxmemory at 0 on a shared host — OOM Killer will terminate the process with no warning.

Redis: Scaling Patterns

There are three axes of scale for Redis:

1. Vertical scaling: move to a larger instance. Straightforward but limited. Above ~256 GB RAM, fork-based BGSAVE becomes painfully slow; consider switching to RDB-disabled persistence + Replica AOF.
2. Read scaling via replicas: route read-only commands to replicas with client-side read preference or a proxy (Envoy, Twemproxy). Replicas add replication lag; never route reads that require linearisability to a replica.
3. Horizontal sharding via Redis Cluster: 16,384 hash slots distributed across N primaries. Minimum viable cluster is 3 primaries + 3 replicas. Target ≤200 GB of working set per primary shard at the 70 % headroom rule. Resharding is online but CPU-expensive — schedule during low-traffic windows and watch cluster_stats_messages_sent for backpressure.

Kafka: Capacity Sizing

Kafka sizing has three independent dimensions: throughput, storage, and partitions.

• Throughput: a single Kafka broker on modern hardware handles 200–600 MB/s of aggregate write throughput. Measure actual producer byte-rate with kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec. The bottleneck is almost always the NIC or disk write bandwidth, not CPU.
• Storage: total bytes = message_rate × avg_message_size × retention_hours × replication_factor. Add 20 % for index files. Use kafka-log-dirs.sh to measure actual disk usage per topic/partition.
• Partition count: start with max(target_throughput / per_partition_throughput, consumer_parallelism). A single partition saturates at roughly 10–100 MB/s depending on consumer count and compression. Over-partitioning has real costs: more open file descriptors, longer leader election, higher ZooKeeper/KRaft metadata load. Google and LinkedIn historically cap partitions-per-broker at 4,000; above that, broker restarts become painful.

Kafka: Scaling Patterns

Kafka scales horizontally by adding brokers and rebalancing partition leadership. The process:

1. Add new brokers to the cluster (they join with empty logs).
2. Run the partition reassignment tool (kafka-reassign-partitions.sh) to generate and execute a reassignment plan that spreads replicas across the enlarged broker set.
3. Monitor replication progress with kafka-reassign-partitions.sh --verify. Network throttle the replication to avoid starving producers: set leader.replication.throttled.rate and follower.replication.throttled.rate (bytes/sec) on affected topics before starting.

Upgrade Strategies

Both Redis and Kafka have well-defined rolling upgrade paths. The cardinal rule is: never skip major versions.

Redis rolling upgrade: upgrade replicas first, then failover to a replica (using FAILOVER command in Redis 6.2+ or CLUSTER FAILOVER), then upgrade the former primary. For Sentinel setups, trigger a manual failover with SENTINEL FAILOVER <master-name> after upgrading all replicas.

Kafka rolling upgrade: upgrade one broker at a time. Set inter.broker.protocol.version and log.message.format.version to the current version before starting so the cluster stays protocol-compatible during the transition. Only bump those values once all brokers are on the new version. This is the same pattern used for KRaft migrations: run in mixed mode (ZooKeeper + KRaft controllers) until all brokers are migrated, then cut over.

Capacity Planning in Practice

Treat your data infrastructure capacity exactly as you treat Kubernetes node capacity: model it, alert on it, and plan upgrades before you hit 70 % utilisation. Key metrics to track in your observability stack (you already have Prometheus/Grafana from earlier in the course):

• Redis: redis_memory_used_bytes / redis_memory_max_bytes, redis_evicted_keys_total, redis_connected_clients
• Kafka: kafka_server_brokertopicmetrics_bytesinpersec, kafka_log_log_size, kafka_controller_kafkacontroller_activecontrollercount, consumer group lag via kafka_consumer_group_lag