TypeScript & Node.js Backend Engineer: Singleflight Cache Stampede Coalescing, Stale-While-Revalidate & Non-Blocking Upstream Timeouts
Two of the fastest ways a high-traffic Node.js & TypeScript API brings down production are: (1) a hot cache key (such as a global leaderboard or pricing catalog) expiring and triggering 20+ concurrent identical database queries simultaneously (a cache stampede / thundering herd), and (2) an unresponsive upstream service without strict request timeouts or concurrency caps starving the Node.js connection pool until /healthz fails Kubernetes/ALB liveness probes.
1. Eliminating Cache Stampedes in TypeScript with SingleflightCache<T> & SWR
When a 300-second Redis or in-memory cache key expires under burst traffic, checking if (!cached) { value = await loadFromDb(); } fires N redundant database queries in parallel. By storing the in-flight Promise<T> inside a map keyed by cache key and decoupling soft TTL (when background refresh starts while still serving stale data) from hard TTL (when callers wait for the coalesced promise), PostgreSQL load drops by >95%:
interface CacheRecord<T> {
value: T;
freshUntil: number;
staleUntil: number;
}
export class SingleflightCache<T> {
private readonly store = new Map<string, CacheRecord<T>>();
private readonly inflight = new Map<string, Promise<T>>();
constructor(
private readonly freshTtlMs: number = 300_000,
private readonly staleGraceMs: number = 120_000
) {}
async getOrLoad(key: string, loader: () => Promise<T>): Promise<T> {
const now = Date.now();
const hit = this.store.get(key);
// 1. Fresh hit: return immediately
if (hit && now < hit.freshUntil) {
return hit.value;
}
// 2. Stale-While-Revalidate: serve stale snapshot & trigger singleflight background refresh
if (hit && now < hit.staleUntil) {
void this.refreshOnce(key, loader).catch(() => undefined);
return hit.value;
}
// 3. Cold miss or hard expiry: coalesce all concurrent callers onto one Promise
return this.refreshOnce(key, loader);
}
private refreshOnce(key: string, loader: () => Promise<T>): Promise<T> {
const existing = this.inflight.get(key);
if (existing) {
return existing;
}
const job = (async () => {
try {
const value = await loader();
// Add up to 10% positive jitter so batch keys never expire on the same millisecond
const jitter = Math.floor(Math.random() * this.freshTtlMs * 0.1);
const now = Date.now();
this.store.set(key, {
value,
freshUntil: now + this.freshTtlMs + jitter,
staleUntil: now + this.freshTtlMs + jitter + this.staleGraceMs,
});
return value;
} finally {
this.inflight.delete(key);
}
})();
this.inflight.set(key, job);
return job;
}
}
2. Keeping /healthz Fast When Upstream Inference Hosts Degrade
In Node.js Express, Fastify, or NestJS services, two hidden failure modes take down downstream pods alongside an upstream dependency outage:
- Missing explicit request timeouts: Node's default
fetch()/httpcalls can hang for minutes waiting on dead TCP sockets or deprecated DNS hosts, tying up upstream sockets and memory. Always attachAbortSignal.timeout(ms). - Synchronous CPU work or blocking probes inside
/healthz: If/healthzawaits heavy downstream dependencies or shares an unbounded retry loop, the Load Balancer marks healthy pods dead and triggers cascading restarts. Keep/healthzevent-loop fast and protect upstream calls behind a lightweight Circuit Breaker:
export class UpstreamCircuitBreaker {
private failures = 0;
private openUntil = 0;
constructor(
private readonly failureThreshold = 5,
private readonly cooldownMs = 15_000,
private readonly timeoutMs = 450
) {}
async callInference<T>(url: string, body: unknown, fallback: () => T): Promise<T> {
if (Date.now() < this.openUntil) {
return fallback();
}
try {
const res = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(this.timeoutMs),
});
if (!res.ok) {
throw new Error(`Upstream HTTP ${res.status}`);
}
this.failures = 0;
return (await res.json()) as T;
} catch (err) {
this.failures += 1;
if (this.failures >= this.failureThreshold) {
this.openUntil = Date.now() + this.cooldownMs;
}
throw err;
}
}
}
3. Related Backend, Caching & Full Stack Architecture Guides
- Downstream (CDN / RFC 7234) vs Upstream (Redis) Caching Architecture
- Senior Backend Pattern — Transactional Outbox, Idempotency Keys & PostgreSQL Locks
- Backend Engineer Pillar Guide — Python, Go, TypeScript & Distributed Systems
- Full Stack Engineer Pillar Guide — React, Next.js 15, TypeScript & Real-Time Systems
Hire a Senior TypeScript, Node.js & Backend Engineer
Facing p99 latency spikes, cache stampedes, PostgreSQL pool exhaustion, or failing health probes? Book a fixed-price 1-Day Backend Incident & Performance Sprint for $100 USD (Credit/Debit Card Checkout) or Rp 1.000.000 (QRIS / Indonesian Bank Transfer) at /en-us/1-day-service/, or hire on Upwork.