Backend Engineer: Designing Bounded-Backpressure Parallel Ingestion Engines with Python asyncio Queues

When ingesting millions of documents, product catalog records, or vector embeddings into PostgreSQL or Elasticsearch, a common anti-pattern is calling asyncio.gather(*[process(item) for item in million_items]). Spawning a million unbounded coroutines simultaneously exhausts container RAM (OOMKilled), overwhelms downstream connection pools, and loses all in-flight work if a single task raises an unhandled exception. As a Backend Engineer, I design bounded-backpressure producer-consumer ingestion engines using asyncio.Queue(maxsize=...), structured concurrency (asyncio.TaskGroup), and micro-batched PostgreSQL COPY / upsert sinks.

1. Bounded Producer-Consumer Pipeline with Micro-Batch Flushing

By setting a strict maxsize on asyncio.Queue, upstream producers automatically pause via await queue.put(item) whenever downstream database or embedding workers experience latency spikes—enforcing natural end-to-end backpressure with a constant memory footprint:

import asyncio
from typing import AsyncIterable, Any

SENTINEL = object()

async def stream_producer(
    source: AsyncIterable[dict[str, Any]],
    queue: asyncio.Queue[ Any ],
    num_workers: int,
) -> None:
    async for record in source:
        await queue.put(record)  # Blocks automatically when queue is full (backpressure)
    for _ in range(num_workers):
        await queue.put(SENTINEL)

async def batch_consumer_worker(
    worker_id: int,
    queue: asyncio.Queue[Any],
    db_pool: Any,
    batch_size: int = 250,
) -> int:
    buffer: list[dict[str, Any]] = []
    processed = 0
    while True:
        item = await queue.get()
        try:
            if item is SENTINEL:
                if buffer:
                    await flush_batch_to_postgres(db_pool, buffer)
                    processed += len(buffer)
                return processed
            buffer.append(item)
            if len(buffer) >= batch_size:
                await flush_batch_to_postgres(db_pool, buffer)
                processed += len(buffer)
                buffer.clear()
        finally:
            queue.task_done()

async def flush_batch_to_postgres(db_pool: Any, batch: list[dict[str, Any]]) -> None:
    async with db_pool.acquire() as conn:
        await conn.executemany(
            '''
            INSERT INTO catalog_items (sku, tenant_id, title, price_cents, payload)
            VALUES ($1, $2, $3, $4, $5::jsonb)
            ON CONFLICT (tenant_id, sku) DO UPDATE
              SET title = EXCLUDED.title,
                  price_cents = EXCLUDED.price_cents,
                  payload = EXCLUDED.payload
            ''',
            [(r["sku"], r["tenant_id"], r["title"], r["price_cents"], r["payload"]) for r in batch],
        )

2. Orchestrating Workers Safely with Python 3.11+ asyncio.TaskGroup

Pairing our bounded queue with async with asyncio.TaskGroup() as tg: guarantees structured concurrency: if any worker encounters an unrecoverable database error, the task group cancels sibling tasks cleanly rather than leaving orphaned background coroutines hanging. For CPU-bound parsing steps alongside async I/O, combine this pipeline with our Python GIL, Threading & Multiprocessing architecture and Backend Engineer high-concurrency PostgreSQL guide.

Need a Backend Engineer to scale your Python asyncio data ingestion and bulk indexing pipelines? Book a 1-Day Engineering Sprint.


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs