Backend Engineer: Architecting High-Throughput Caching & Message Queues with Redis, Celery, Kafka & NATS JetStream

Choosing the right asynchronous messaging and caching substrate is one of the highest-leverage decisions in backend systems engineering. Using Redis Lists as a durable financial event log risks message loss during failover, while standing up a multi-broker Apache Kafka cluster just to send password-reset emails adds unnecessary operational complexity. As a Backend Engineer and Cloud Engineer, I design right-sized caching and messaging topologies across Redis 7, Celery, Kafka, and NATS JetStream.

1. Preventing Redis Cache Stampedes with Probabilistic Early Expiration & Singleflight Locks

When a heavily accessed cache key (such as a tenant's pricing catalog or homepage feed) expires at `TTL = 0`, hundreds of concurrent requests simultaneously miss the cache and hammer PostgreSQL—causing a cascading database outage. We prevent cache stampedes using a distributed Redis singleflight lease or probabilistic early recomputation:

import json
import time
from typing import Any, Callable

def get_or_recompute_singleflight(
    redis_client: Any,
    cache_key: str,
    ttl_seconds: int,
    compute_fn: Callable[[], dict],
) -> dict:
    raw = redis_client.get(cache_key)
    if raw is not None:
        return json.loads(raw)

    lock_key = f"lock:{cache_key}"
    # Only one worker acquires the recomputation lock (NX + 5s lease)
    if redis_client.set(lock_key, "1", nx=True, ex=5):
        try:
            fresh_data = compute_fn()
            redis_client.setex(cache_key, ttl_seconds, json.dumps(fresh_data))
            return fresh_data
        finally:
            redis_client.delete(lock_key)

    # Concurrent readers wait briefly for the leader to populate the cache
    for _ in range(20):
        time.sleep(0.05)
        retry_raw = redis_client.get(cache_key)
        if retry_raw is not None:
            return json.loads(retry_raw)
    return compute_fn()

2. Matching Workload Semantics: Celery (`acks_late`) vs. Kafka vs. NATS JetStream

  • Celery + Redis / RabbitMQ: Ideal for heterogeneous background jobs (PDF generation, webhook retries) when configured with task_acks_late = True, task_reject_on_worker_lost = True, and explicit database idempotency keys.
  • Apache Kafka / Redpanda: Purpose-built for replayable, partitioned event logs and multi-consumer stream processing (Event-Driven Kafka & Pub/Sub Microservices).
  • NATS JetStream: Ultra-low-latency Go binary broker with subject-based wildcard routing and built-in key-value stores, ideal for edge and microservice RPC backplanes (Backend Engineer Distributed Systems Guide).

Need a Backend Engineer to optimize your Redis caching, Celery workers, or Kafka/NATS message queues? Book a 1-Day Engineering Sprint.


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs