Backend & AI Engineer Case Study: Building a Low-Latency Event Ingestion, Semantic Scoring & Web Push Alert Engine

In competitive marketplaces and operational monitoring systems—such as our production On-Time Upwork Job Feed platform—discovering high-fit opportunities or critical incidents minutes before competitors requires an event-driven ingestion and classification pipeline. As a Senior Backend Engineer and AI Application Engineer, here is how I architected a sub-second polling, deduplication, semantic relevance scoring, and VAPID Web Push delivery engine using Python (Django, Celery, Redis) and React PWA clients.

1. Idempotent Feed Ingestion & Content-Hash Deduplication

External RSS/API feeds frequently republish items with updated query parameters or timestamps. To prevent duplicate alerts, every ingested item is normalized and fingerprinted with a deterministic SHA-256 key stored in a Redis `SET NX EX` filter before hitting PostgreSQL:

import hashlib
import redis

def is_new_feed_item(r: redis.Redis, external_id: str, title: str, ttl_seconds: int = 604800) -> bool:
    digest = hashlib.sha256(f"{external_id}::{title.strip().lower()}".encode("utf-8")).hexdigest()
    redis_key = f"feed:seen:{digest}"
    # Atomic SET if Not eXists with 7-day expiration
    return bool(r.set(redis_key, "1", nx=True, ex=ttl_seconds))

2. Semantic Skill-Match Scoring & Instant VAPID Push Fan-Out

3. Related Backend & AI Engineering Guides


⚡ Hire Aldian Fazrihady — Senior Cloud, GCP, Backend, AI Application & Full Stack Engineer

Need production cloud architecture, GCP/AWS infrastructure, high-throughput backend systems, AI agents, or full stack web engineering? Book a 1-Day Programming Sprint for $100 USD (International Card / Apple Pay) or Rp 1,000,000 (Indonesian QRIS / Bank Virtual Account), or hire directly on Upwork:

Book $100 / Rp 1M 1-Day Sprint → Hire on Upwork (100% Job Success) →


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs