Backend Engineer: Architecting High-Throughput Caching & Message Queues with Redis, Celery, Kafka & NATS JetStream
Choosing the right asynchronous messaging and caching substrate is one of the highest-leverage decisions in backend systems engineering. Using Redis Lists as a durable financial event log risks message loss during failover, while standing up a multi-broker Apache Kafka cluster just to send password-reset emails adds unnecessary operational complexity. As a Backend Engineer and Cloud Engineer, I design right-sized caching and messaging topologies across Redis 7, Celery, Kafka, and NATS JetStream.
1. Preventing Redis Cache Stampedes with Probabilistic Early Expiration & Singleflight Locks
When a heavily accessed cache key (such as a tenant's pricing catalog or homepage feed) expires at `TTL = 0`, hundreds of concurrent requests simultaneously miss the cache and hammer PostgreSQL—causing a cascading database outage. We prevent cache stampedes using a distributed Redis singleflight lease or probabilistic early recomputation:
import json
import time
from typing import Any, Callable
def get_or_recompute_singleflight(
redis_client: Any,
cache_key: str,
ttl_seconds: int,
compute_fn: Callable[[], dict],
) -> dict:
raw = redis_client.get(cache_key)
if raw is not None:
return json.loads(raw)
lock_key = f"lock:{cache_key}"
# Only one worker acquires the recomputation lock (NX + 5s lease)
if redis_client.set(lock_key, "1", nx=True, ex=5):
try:
fresh_data = compute_fn()
redis_client.setex(cache_key, ttl_seconds, json.dumps(fresh_data))
return fresh_data
finally:
redis_client.delete(lock_key)
# Concurrent readers wait briefly for the leader to populate the cache
for _ in range(20):
time.sleep(0.05)
retry_raw = redis_client.get(cache_key)
if retry_raw is not None:
return json.loads(retry_raw)
return compute_fn()
2. Matching Workload Semantics: Celery (`acks_late`) vs. Kafka vs. NATS JetStream
- Celery + Redis / RabbitMQ: Ideal for heterogeneous background jobs (PDF generation, webhook retries) when configured with
task_acks_late = True,task_reject_on_worker_lost = True, and explicit database idempotency keys. - Apache Kafka / Redpanda: Purpose-built for replayable, partitioned event logs and multi-consumer stream processing (Event-Driven Kafka & Pub/Sub Microservices).
- NATS JetStream: Ultra-low-latency Go binary broker with subject-based wildcard routing and built-in key-value stores, ideal for edge and microservice RPC backplanes (Backend Engineer Distributed Systems Guide).
Need a Backend Engineer to optimize your Redis caching, Celery workers, or Kafka/NATS message queues? Book a 1-Day Engineering Sprint.