Backend & AI Engineer Case Study: Building a Low-Latency Event Ingestion, Semantic Scoring & Web Push Alert Engine
In competitive marketplaces and operational monitoring systems—such as our production On-Time Upwork Job Feed platform—discovering high-fit opportunities or critical incidents minutes before competitors requires an event-driven ingestion and classification pipeline. As a Senior Backend Engineer and AI Application Engineer, here is how I architected a sub-second polling, deduplication, semantic relevance scoring, and VAPID Web Push delivery engine using Python (Django, Celery, Redis) and React PWA clients.
1. Idempotent Feed Ingestion & Content-Hash Deduplication
External RSS/API feeds frequently republish items with updated query parameters or timestamps. To prevent duplicate alerts, every ingested item is normalized and fingerprinted with a deterministic SHA-256 key stored in a Redis `SET NX EX` filter before hitting PostgreSQL:
import hashlib
import redis
def is_new_feed_item(r: redis.Redis, external_id: str, title: str, ttl_seconds: int = 604800) -> bool:
digest = hashlib.sha256(f"{external_id}::{title.strip().lower()}".encode("utf-8")).hexdigest()
redis_key = f"feed:seen:{digest}"
# Atomic SET if Not eXists with 7-day expiration
return bool(r.set(redis_key, "1", nx=True, ex=ttl_seconds))
2. Semantic Skill-Match Scoring & Instant VAPID Push Fan-Out
- Two-Stage Relevance Filter: Fast lexical keyword filtering discards obvious mismatches in < 1ms, while dense embedding cosine similarity scores technical stack fit across Cloud, GCP, Backend, AI, and Full Stack criteria.
- Real-Time WebSocket + VAPID Web Push Delivery: High-scoring events trigger an immediate Django Channels WebSocket broadcast to active tabs and an encrypted RFC 8292 Web Push notification to mobile devices.
3. Related Backend & AI Engineering Guides
- Hire a Senior Backend Engineer: Python, Django, FastAPI, Celery & Redis
- Hire a Senior AI Application Engineer: Semantic Classification & LLM Pipelines
⚡ Hire Aldian Fazrihady — Senior Cloud, GCP, Backend, AI Application & Full Stack Engineer
Need production cloud architecture, GCP/AWS infrastructure, high-throughput backend systems, AI agents, or full stack web engineering? Book a 1-Day Programming Sprint for $100 USD (International Card / Apple Pay) or Rp 1,000,000 (Indonesian QRIS / Bank Virtual Account), or hire directly on Upwork:
Book $100 / Rp 1M 1-Day Sprint → Hire on Upwork (100% Job Success) →