TypeScript & Node.js Backend Engineer: Singleflight Cache Stampede Coalescing, Stale-While-Revalidate & Non-Blocking Upstream Timeouts

Two of the fastest ways a high-traffic Node.js & TypeScript API brings down production are: (1) a hot cache key (such as a global leaderboard or pricing catalog) expiring and triggering 20+ concurrent identical database queries simultaneously (a cache stampede / thundering herd), and (2) an unresponsive upstream service without strict request timeouts or concurrency caps starving the Node.js connection pool until /healthz fails Kubernetes/ALB liveness probes.

22 Concurrent API Requests Stale Cache Entry Valid Immediate 2ms Response (SWR) Singleflight In-Flight Map 1 Shared Promise per Cache Key 1 Single Postgres Query Zero Pool Saturation
Singleflight request coalescing + Stale-While-Revalidate (SWR): 22 simultaneous expired-key requests collapse into one shared Promise while stale values serve in <2ms.

1. Eliminating Cache Stampedes in TypeScript with SingleflightCache<T> & SWR

When a 300-second Redis or in-memory cache key expires under burst traffic, checking if (!cached) { value = await loadFromDb(); } fires N redundant database queries in parallel. By storing the in-flight Promise<T> inside a map keyed by cache key and decoupling soft TTL (when background refresh starts while still serving stale data) from hard TTL (when callers wait for the coalesced promise), PostgreSQL load drops by >95%:

interface CacheRecord<T> {
  value: T;
  freshUntil: number;
  staleUntil: number;
}

export class SingleflightCache<T> {
  private readonly store = new Map<string, CacheRecord<T>>();
  private readonly inflight = new Map<string, Promise<T>>();

  constructor(
    private readonly freshTtlMs: number = 300_000,
    private readonly staleGraceMs: number = 120_000
  ) {}

  async getOrLoad(key: string, loader: () => Promise<T>): Promise<T> {
    const now = Date.now();
    const hit = this.store.get(key);

    // 1. Fresh hit: return immediately
    if (hit && now < hit.freshUntil) {
      return hit.value;
    }

    // 2. Stale-While-Revalidate: serve stale snapshot & trigger singleflight background refresh
    if (hit && now < hit.staleUntil) {
      void this.refreshOnce(key, loader).catch(() => undefined);
      return hit.value;
    }

    // 3. Cold miss or hard expiry: coalesce all concurrent callers onto one Promise
    return this.refreshOnce(key, loader);
  }

  private refreshOnce(key: string, loader: () => Promise<T>): Promise<T> {
    const existing = this.inflight.get(key);
    if (existing) {
      return existing;
    }

    const job = (async () => {
      try {
        const value = await loader();
        // Add up to 10% positive jitter so batch keys never expire on the same millisecond
        const jitter = Math.floor(Math.random() * this.freshTtlMs * 0.1);
        const now = Date.now();
        this.store.set(key, {
          value,
          freshUntil: now + this.freshTtlMs + jitter,
          staleUntil: now + this.freshTtlMs + jitter + this.staleGraceMs,
        });
        return value;
      } finally {
        this.inflight.delete(key);
      }
    })();

    this.inflight.set(key, job);
    return job;
  }
}

2. Keeping /healthz Fast When Upstream Inference Hosts Degrade

In Node.js Express, Fastify, or NestJS services, two hidden failure modes take down downstream pods alongside an upstream dependency outage:

  • Missing explicit request timeouts: Node's default fetch() / http calls can hang for minutes waiting on dead TCP sockets or deprecated DNS hosts, tying up upstream sockets and memory. Always attach AbortSignal.timeout(ms).
  • Synchronous CPU work or blocking probes inside /healthz: If /healthz awaits heavy downstream dependencies or shares an unbounded retry loop, the Load Balancer marks healthy pods dead and triggers cascading restarts. Keep /healthz event-loop fast and protect upstream calls behind a lightweight Circuit Breaker:
export class UpstreamCircuitBreaker {
  private failures = 0;
  private openUntil = 0;

  constructor(
    private readonly failureThreshold = 5,
    private readonly cooldownMs = 15_000,
    private readonly timeoutMs = 450
  ) {}

  async callInference<T>(url: string, body: unknown, fallback: () => T): Promise<T> {
    if (Date.now() < this.openUntil) {
      return fallback();
    }

    try {
      const res = await fetch(url, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
        signal: AbortSignal.timeout(this.timeoutMs),
      });

      if (!res.ok) {
        throw new Error(`Upstream HTTP ${res.status}`);
      }

      this.failures = 0;
      return (await res.json()) as T;
    } catch (err) {
      this.failures += 1;
      if (this.failures >= this.failureThreshold) {
        this.openUntil = Date.now() + this.cooldownMs;
      }
      throw err;
    }
  }
}

3. Related Backend, Caching & Full Stack Architecture Guides

Hire a Senior TypeScript, Node.js & Backend Engineer

Facing p99 latency spikes, cache stampedes, PostgreSQL pool exhaustion, or failing health probes? Book a fixed-price 1-Day Backend Incident & Performance Sprint for $100 USD (Credit/Debit Card Checkout) or Rp 1.000.000 (QRIS / Indonesian Bank Transfer) at /en-us/1-day-service/, or hire on Upwork.


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs