AI Application Engineer: Designing Episodic & Semantic Long-Term Agent Memory with Mem0, Zep, LangMem & pgvector
Stuffing an entire 100-turn conversation transcript into an LLM's context window on every turn degrades reasoning accuracy ("lost-in-the-middle" attention decay), inflates time-to-first-token (TTFT), and multiplies token costs. Conversely, naive sliding-window truncation causes the AI agent to forget critical user constraints, architectural preferences, and historical decisions made days earlier. As an AI Application Engineer and Backend Engineer, I architect three-tier long-term agent memory systems combining working session state, episodic temporal timelines, and semantic fact graphs backed by PostgreSQL pgvector.
1. Three-Tier Cognitive Memory Hierarchy for Production Agents
- Working Memory (Short-Term Buffer): The last 6–10 raw conversational turns plus active tool-call outputs stored in Redis or Google ADK
Session.state. - Episodic Memory (Temporal Event Log): Timestamped session summaries and interaction episodes indexed by time decay and cosine similarity so agents recall what happened last Tuesday.
- Semantic Memory (Consolidated Entity & Fact Graph): Background extraction workers distill durable user facts (e.g.,
prefers_cloud="GCP",primary_db="PostgreSQL 16") and resolve contradictions when preferences change over time.
2. Time-Decayed Hybrid Memory Retrieval in PostgreSQL pgvector
Pure cosine similarity often retrieves outdated facts from six months ago over slightly less similar facts updated yesterday. We combine pgvector HNSW cosine similarity with an exponential recency decay score directly in SQL:
-- Retrieve top-K agent memories ranked by semantic similarity + temporal recency + importance
SELECT
memory_id,
fact_text,
memory_type,
importance_weight,
updated_at,
(
0.65 * (1.0 - (embedding <=> $1::vector)) +
0.20 * EXP(-0.015 * EXTRACT(EPOCH FROM (NOW() - updated_at)) / 3600.0) +
0.15 * importance_weight
) AS composite_memory_score
FROM agent_long_term_memories
WHERE tenant_id = $2::uuid
AND user_id = $3::uuid
AND superseded_by_id IS NULL
ORDER BY embedding <=> $1::vector ASC
LIMIT 25;
3. Asynchronous Memory Consolidation & Conflict Resolution
Rather than blocking the user-facing response stream to extract memories, our Google ADK & Vertex AI agent runtime emits completed turn events to an async background worker. When a newly extracted fact contradicts an existing memory row, the worker sets superseded_by_id on the old row rather than overwriting it destructively—preserving full temporal auditability alongside our PostgreSQL RLS & pgvector isolation policies.
Need an AI Application Engineer to build persistent, low-latency long-term memory for your AI agents? Book a 1-Day Engineering Sprint.