A Survey on the Memory Mechanism of Large Language Model based Agents

core: LLM-based agents are distinguished from base LLMs by a memory module that records prior observations and lets the agent self-evolve across trials. The survey organizes memory along three axes: sources (inside-trial / cross-trial / external), forms (textual / parametric), and operations (writing / management / reading).

Five panes: sources, forms, operations, evaluation, and the paper's own limits.

primer — what you need before you start

the task. The Memory Survey organizes 1000+ agent-memory papers into a 3-axis taxonomy: where the memory comes from, what form it takes, and how the agent uses it. Memory is what separates a "single-shot LLM" from a "long-lived agent".

how it was done before. Two prior patterns: (1) every-trial-fresh agents — same plan, same failure; (2) hand-coded external state — K/V cache, vector DB, etc., but no agent-side bookkeeping. The paper frames the missing piece: a memory module the agent can read and write.

glossary — terms you will see.
agent — the LLM + a controller loop; reads context, emits actions.
environment — the outside world the agent acts on (browser, REPL, files, API).
observation — text the environment emits back to the agent (search result, error, page).
action — a structured call the agent emits (tool call, click, code line).
inside-trial memory — working state of the current run; chain-of-thought is one form.
cross-trial memory — persistent across runs; Reflexion-style buffers.
external memory — RAG corpus, external DB, a notes file the agent writes to.
episodic memory — store of past events the agent can replay.

symbol table — what the letters mean.
$M_t$ — memory state at step t (may include text + metadata).
$r$ — a recall operation: query the memory for relevant items.
$w$ — a write operation: append or update the memory.
$\theta$ — parametric memory (model weights).
$\text{eval}(M)$ — how well the memory contributes to downstream task success.
$\mathcal{D}_{\text{tasks}}$ — the set of tasks the agent is being benchmarked on.

tone legend — what the [brackets] mean.
[obs] observation   [ops] memory operation
[src] memory source   [form] memory form
[eq] equation   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Six panes cover, in pedagogical order: memory sources; memory forms; memory operations; direct vs indirect evaluation; the paper's own limits. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: three categories of memory content. Inside-trial — historical steps within the current run, the most relevant for the next action. Cross-trial — patterns learned from previous runs of the same task, enabling self-evolution. External knowledge — pre-existing or retrieved knowledge outside the agent-environment loop.

The taxonomy maps 30+ memory-augmented agents onto these three categories. Most agents use inside-trial information; only some combine it with cross-trial or external knowledge to enable self-evolution across runs.

inside-trial cross-trial external

point: textual memory lives in the prompt context — natural language, easy to inspect, easy to share across agents, but bounded by the context window. Parametric memory is encoded in model weights via fine-tuning, RL, or LoRA adapters — dense, unbounded by context length, but expensive to update and hard to inspect.

The choice between textual and parametric is a cost / inspectability trade-off. Textual wins on cost, sharing, and transparency. Parametric wins on density and scale but is opaque and update-expensive.

textual parametric

point: three operations in order. Writing at step t projects raw observations into informative stored contents. Management summarizes, merges, and forgets — including reflection-based compression that keeps only the most useful entries. Reading at step t+1 retrieves relevant context for the next action.

Reading is usually implemented by computing similarity between the memory pool and the next context. Management is realized by a reflection process (e.g., Generative Agents, TPTU) that compresses and selects what to keep — usually thresholded by importance score.

writing management reading

point: two paradigms. Direct evaluation judges memory itself — subjective (human / GPT-4 ratings) plus objective (retrieval accuracy, perplexity, MAUVE, ROUGE). Indirect evaluation measures downstream task quality: conversation (MT-Bench / MT-Bench++), multi-source QA, long-context reasoning (needle-in-haystack, multi-hop), action success rate, code pass@k.

Indirect is more practical — agents are usually deployed for their downstream task, not for memory quality per se. But indirect conflates memory quality with model capability, making it hard to attribute wins.

direct indirect

point: stored memory can be a vector for hallucination — the agent cites fabricated "experiences" as ground truth. Social bias propagates from the source data into the memory pool. Retrieval mechanisms are not yet explainable — it is hard to surface "why this memory?" to the user. Adversarial attacks on memory writes (prompt injection) and privacy leaks from stored personal information are open security concerns.

Multi-step planning that uses memory efficiently is an open problem — current agents do not separate planning from memory. Memory sharing across multiple agents in the same environment is the open frontier for multi-agent systems.

limit open direction