prom-memory: building episodic memory for an AI system that actually remembers

Why episodic memory matters

AI assistants are stateless by default. Every session starts from zero. The system prompt provides identity and rules, the context window provides the working set, and when the session ends, everything evaporates.

Memory is not retrieval. Retrieval finds documents. Memory knows what happened, what you decided, what broke, and what you learned from it.

The memory layer is an episodic memory service: structured fact storage with semantic search, organized by a taxonomy that distinguishes decisions from gotchas from milestones from patterns. It doesn’t store documents. It stores typed facts about what happened during sessions, with enough metadata to retrieve them by meaning, filter them by context, and let them decay when they stop being useful.

The stack

The memory service runs as a lightweight API backed by a single-file embedded database. The storage engine has three layers: structured fact storage, full-text search, and vector similarity search via locally-generated embeddings.

graph TD
    A[AI Session] -->|API Calls| B[Memory Service]
    B --> C[Fact Validation + Metadata]
    C --> D[Embedded Database]
    D --> E[Structured Columns]
    D --> F[Full-Text Index]
    D --> G[Vector Embeddings]
    H[Search Query] --> B
    B --> I{Hybrid Search}
    I --> F
    I --> G
    I --> J[Result Fusion + Ranking]
    J --> K[Heat-Weighted Results]

Why an embedded database? Because it’s the right tool for this scale. The memory layer stores thousands of facts, not millions. A single database file is trivially backed up, trivially migrated, and runs on hardware from a GPU dev kit to a mini PC. No connection pooling, no cluster management, no ops burden. The database is a file. The backup is a file copy.

Why not a full database server? The memory service doesn’t need multi-user access, complex transactions, or any of the features that justify server-class database complexity. The single-writer, single-reader pattern of one AI system writing its own memories is a perfect fit for an embedded engine.

Structured forgetting: the CogniLayer fact types

Not all memories are the same. A decision has different retrieval patterns than a gotcha. A milestone matters for narrative context; a pattern matters for problem solving. The CogniLayer taxonomy assigns a type to every fact, and the type determines how the fact is stored, searched, and eventually compressed or archived.

Each fact also carries metadata: timestamp, session identifier, context scope, related entities, and a heat score. The heat score is the mechanism for structured forgetting.

Heat decay: memories that cool over time

Every fact enters the memory layer with a heat score. The score decays over time unless the fact is retrieved, referenced, or linked to other facts. High-heat facts surface in session primers and search results. Low-heat facts fade into the archive layer, still retrievable but no longer proactively surfaced.

graph LR
    A[Fact Created] -->|Heat: 1.0| B[Active Memory]
    B -->|Time decay| C[Cooling]
    C -->|Retrieved| B
    C -->|Not retrieved| D[Low Heat]
    D -->|Below threshold| E[Archive Layer]
    E -->|Explicit recall| B
    B -->|Sticky tag| F[Permanent: No Decay]

The decay function is configurable per fact type. Decisions decay slowly because they remain relevant across many sessions. Gotchas decay slowly because the failure mode they document tends to recur. Environment facts decay quickly because system state changes. Metrics decay at a medium rate because trend data has a shelf life.

Three decay categories:

• Sticky: never decays. Foundation-layer memories. Used for architectural decisions, core gotchas, and user preferences that define the system’s behavior.
• Standard: decays on a schedule. The default. Most facts live here.
• Ephemeral: decays rapidly. Session-specific context, temporary state, notes that are only relevant to the current work block.

The goal is not to remember everything. The goal is to forget the right things at the right time, and never forget the things that matter.

This maps directly to the Middle-Out compression algorithm: sticky facts are the foundation band (never compress), standard facts are the middle band (compress when they cluster), and ephemeral facts are the active band (keep until they’re integrated, then discard).

Hybrid search: full-text + vector embeddings

The memory layer uses two search strategies simultaneously and fuses the results.

Full-text search handles exact keyword matches, boolean queries, and phrase matching. When you search for a specific project name or error message, full-text finds every fact that contains those terms. It’s fast, deterministic, and handles the cases where the user knows exactly what they’re looking for.

Vector search handles semantic similarity. Embeddings are generated on write using a local model running on GPU hardware. When you search for a concept like “tool call authorization,” vector search finds facts about authorization frameworks even if they never use that exact phrase. It handles the cases where the user knows the concept but not the exact terms stored in memory.

The fusion is weighted: full-text matches get a precision bonus (exact matches are usually what you want), vector matches get a recall bonus (finding related facts the user didn’t think to search for). Heat scores weight the final ranking, so a hot decision outranks a cold environment note from three months ago.

From storage to intelligence

The first version had four operations: add facts, search facts, archive facts, review what’s pending. Functional but limited. The system accumulated memory but couldn’t do anything with it beyond retrieval.

The second version added capabilities that turn stored facts into connected knowledge:

Linking and traversal turn flat fact storage into a knowledge graph. A decision about architecture can be linked to the gotcha that motivated it, the milestone where it shipped, and the pattern it established. Traversal follows these links, giving the system a way to understand not just “what happened” but “why it happened and what it led to.”

Consolidation is the manual compression path. When a set of facts about the same topic has accumulated over many sessions, consolidation merges them into a single summary fact. The original facts are preserved (history is inviolable) but the summary becomes the primary retrieval target. This feeds the Middle-Out compression pipeline.

Drift detection compares what the system currently believes against what it decided in the past. If a decision was made months ago and the current session is contradicting it without acknowledging the change, drift detection catches it. This is the self-consistency mechanism: the system can check whether it’s being coherent across time, not just within a single session.

How memory enters a session

At the start of every session, the memory system assembles a compact payload from the most relevant facts: recent decisions, active gotchas, high-heat patterns, and any context flagged as “surface next session.”

graph LR
    A[Session Start] --> B[Memory Primer]
    B --> C[Recent Decisions]
    B --> D[Active Gotchas]
    B --> E[High-Heat Patterns]
    B --> F[Flagged Context]
    C --> G[Compact Payload]
    D --> G
    E --> G
    F --> G
    G --> H[Injected into Session Context]

The primer is budget-constrained. It doesn’t dump every fact into the context window. It selects the highest-value facts within a token budget, prioritized by heat, recency, and type. Decisions and gotchas get priority over metrics and environment facts. The result is a focused memory injection that gives the session relevant history without burning half the context window on old notes.

The primer is also context-aware. Different operational contexts (work vs personal vs project-specific) only surface facts relevant to that context. The memory wall between contexts is enforced at the retrieval layer, not just the storage layer.

What I learned building this

Memory volume is not the problem; memory quality is. Early sessions stored everything. Every observation, every minor decision, every configuration value. The search results became noisy. The heat decay system was the first major fix: facts that aren’t retrieved cool down and stop polluting results. The fact type taxonomy was the second: being able to filter by decision vs environment reduced noise dramatically.

The session primer is the most important component. Not search. Not storage. The primer. Because the primer determines what the system knows at the start of every session, it’s the highest-leverage point in the entire memory architecture. A bad primer means a bad session. I spent more time tuning the primer’s selection algorithm than any other component.

Cross-session drift is real. Without drift detection, the system makes contradictory decisions across sessions without noticing. An early session decides on an architectural rule. A later session proposes violating it because the simpler path looks appealing. Drift detection catches this: “You decided X previously. Are you intentionally reversing that?” The system becomes self-correcting across time.

The system that doesn’t check its own consistency will eventually contradict itself. At production scale, “eventually” happens every week.

Backup is a file copy. This is the quiet advantage of an embedded database. The entire memory store, including embeddings, full-text index, and all structured data, is a single file. Backup is: copy the file. Restore is: copy it back. No export/import pipeline, no schema migration, no dump utility. The simplest backup strategy that could possibly work, and it works.

Cold-start timeouts kill agent workflows. The memory service loads the embedding model on first request. If the model hasn’t been loaded recently and the first request comes from an autonomous agent with a short timeout, the request fails. The agent moves on without memory context. This caused silent quality degradation in autonomous sessions until I added a health check that pre-loads the model at startup.

Open threads

Automated compression. The consolidation capability is the manual path. The automated path is a scheduled job that identifies clusters of related facts in the middle band (not recent, not foundational) and generates summary facts. The algorithm is designed around the Middle-Out compression model. Implementation is next.

Embedding model upgrades. The current model is pinned to a specific runtime version due to hardware constraints. As the compute environment evolves, the embedding model can move to newer runtimes, which unlocks better models and faster inference.

Shared memory across agents. Currently, the memory service serves one primary consumer and one autonomous agent fleet. The next step is making it a shared service for all personas and agents in the system, with per-consumer access control and context scoping. The memory wall enforcement that currently operates at the application level needs to move into the service’s authentication layer.