Backend Engineer: Designing Bounded-Backpressure Parallel Ingestion Engines with Python asyncio Queues
When ingesting millions of documents, product catalog records, or vector embeddings into PostgreSQL or Elasticsearch, a common anti-pattern is calling asyncio.gather(*[process(item) for item in million_items]). Spawning a million unbounded coroutines simultaneously exhausts container RAM (OOMKilled), overwhelms downstream connection pools, and loses all in-flight work if a single task raises an unhandled exception. As a Backend Engineer, I design bounded-backpressure producer-consumer ingestion engines using asyncio.Queue(maxsize=...), structured concurrency (asyncio.TaskGroup), and micro-batched PostgreSQL COPY / upsert sinks.
1. Bounded Producer-Consumer Pipeline with Micro-Batch Flushing
By setting a strict maxsize on asyncio.Queue, upstream producers automatically pause via await queue.put(item) whenever downstream database or embedding workers experience latency spikes—enforcing natural end-to-end backpressure with a constant memory footprint:
import asyncio
from typing import AsyncIterable, Any
SENTINEL = object()
async def stream_producer(
source: AsyncIterable[dict[str, Any]],
queue: asyncio.Queue[ Any ],
num_workers: int,
) -> None:
async for record in source:
await queue.put(record) # Blocks automatically when queue is full (backpressure)
for _ in range(num_workers):
await queue.put(SENTINEL)
async def batch_consumer_worker(
worker_id: int,
queue: asyncio.Queue[Any],
db_pool: Any,
batch_size: int = 250,
) -> int:
buffer: list[dict[str, Any]] = []
processed = 0
while True:
item = await queue.get()
try:
if item is SENTINEL:
if buffer:
await flush_batch_to_postgres(db_pool, buffer)
processed += len(buffer)
return processed
buffer.append(item)
if len(buffer) >= batch_size:
await flush_batch_to_postgres(db_pool, buffer)
processed += len(buffer)
buffer.clear()
finally:
queue.task_done()
async def flush_batch_to_postgres(db_pool: Any, batch: list[dict[str, Any]]) -> None:
async with db_pool.acquire() as conn:
await conn.executemany(
'''
INSERT INTO catalog_items (sku, tenant_id, title, price_cents, payload)
VALUES ($1, $2, $3, $4, $5::jsonb)
ON CONFLICT (tenant_id, sku) DO UPDATE
SET title = EXCLUDED.title,
price_cents = EXCLUDED.price_cents,
payload = EXCLUDED.payload
''',
[(r["sku"], r["tenant_id"], r["title"], r["price_cents"], r["payload"]) for r in batch],
)
2. Orchestrating Workers Safely with Python 3.11+ asyncio.TaskGroup
Pairing our bounded queue with async with asyncio.TaskGroup() as tg: guarantees structured concurrency: if any worker encounters an unrecoverable database error, the task group cancels sibling tasks cleanly rather than leaving orphaned background coroutines hanging. For CPU-bound parsing steps alongside async I/O, combine this pipeline with our Python GIL, Threading & Multiprocessing architecture and Backend Engineer high-concurrency PostgreSQL guide.
Need a Backend Engineer to scale your Python asyncio data ingestion and bulk indexing pipelines? Book a 1-Day Engineering Sprint.