Reflexion: Language Agents with Verbal Reinforcement Learning

core: Reflexion reinforces language agents with verbal self-reflection and an episodic memory buffer; no weight updates, faster learning from trial-and-error.

primer — what you need before you start

the task. Reflexion targets agents that fail repeatedly — multi-hop QA (HotpotQA), fact verification (FEVER), text games (ALFWorld), web shopping (WebShop), code generation (HumanEval, MBPP). The agent tries, fails, reflects in plain language, then tries again. No fine-tuning; just text-to-text feedback.

how it was done before. Two prior recipes: (1) RL/fine-tuning on failure traces — slow, expensive, needs lots of compute; (2) prompt-only retry — same plan, same failure. Reflexion splits the difference: verbal reflection is fast like prompting but information-bearing like training.

glossary — terms you will see.
trial — one full agent run from start to finish (action / step count limited by task).
reflection — short free-form paragraph the agent writes to itself after a failed trial. Names what went wrong, proposes a fix.
episodic memory — sliding buffer of past reflections; the next trial's prompt includes the most recent ones.
Self-Reflection — the paper's coined term for this kind of verbal feedback.
Refl-on-time / Refl-on-Act — variants of when to reflect: after every step vs only after a wrong action.
HotpotQA — multi-hop question answering.
ALFWorld — text adventure game benchmark.
HumanEval / MBPP — Python coding benchmarks; pass@1 is the metric.

symbol table — what the letters mean.
$s_t$ — the agent's state at trial t (prompt contents + memory buffer).
$r_t$ — the reflection written after trial t.
$r_1, r_2, ..., r_{t-1}$ — episodic memory buffer contents at trial t.
$a_t$ — the action / output of trial t.
$\text{obs}_t$ — environment observation after trial t (or verifier verdict for code).
$r_t = \\text{Reflect}(s_t, a_t, \\text{obs}_t)$ — verbal reflection is generated from current state + action + observation.
$\\text{Memory}_t = (r_1, r_2, ..., r_{t-1})$ — sliding buffer fed into trial t's prompt.

tone legend — what the [brackets] mean.
[obs] observation   [react] action step
[reflection] the agent's self-critique   [memory] episodic buffer content
[verifier] external verdict (correct / incorrect)   [take] key takeaway
[sys] system / meta   [limit] paper-reported limitation

Six panes cover, in pedagogical order: the problem (agents repeat mistakes); the mechanism (verbal self-reflection + episodic buffer); worked trials on HotpotQA, ALFWorld, WebShop; HumanEval pass@1 lift; paper-reported limits (no weight updates, vague reflections fail). Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: Standard, CoT-only, and ReAct-only all finish with Rene Artois. Reflexion alone sees Trial 2 — its memory carries the lesson that the show title was the wrong search target. Below: one HotPotQA question, four prompting styles. The question, the INCORRECT / CORRECT verdicts, and Reflexion's self-reflection are verbatim; the intermediate Search / Obs lines are schematic ("first 5 sentences") to fit the tty pane.

question · INCORRECT / CORRECT verdict · Self-Reflection text are verbatim. Intermediate Search / Obs are schematic ("first 5 sentences (schematic): ...") so they fit the tty pane.

Trial 1 Trial 2 Self-Reflection Memory FAIL SUCCESS

point: at trial the agent sees the prompt and emits actions . On failure it generates a self-reflection ; that text is appended to memory . Trial then sees alongside the original prompt.

action obs / fail self-reflection memory slot

point: the task is to examine the mug with the desklamp. Trial 1 walks drawers 1-6 (closed), then desk 1, picks up the mug, walks back to desk 1 and again to desk 2, and tries to use desklamp 1 twice — nothing happens, the desklamp and the mug were never co-located. The self-reflection names the mistake: look for the desklamp first, both objects are on desk 1. Trial 2 does exactly that and turns on the desklamp with the mug in hand.

task description, the desk-1 obs ("creditcard 3, a desklamp 1, ... a mug 1 ..."), the desk-2 obs ("alarmclock 1, a bowl 1, a mug 3 ..."), the Status lines, and the self-reflection text are all from the original case. Use desklamp 1 / take mug 1 from desk 1 are ALFWorld verbs.

agent action env obs self-reflection Memory FAIL SUCCESS

point: The agent writes an implementation, the unit tests fail on an edge case (empty array). The Self-Reflection prompt drives a self-reflection that names the missing guard; trial 2 adds it. Memory here is a sliding window of last 1 — the only domain where the window is smaller than 3.

task signature (minSubArraySum docstring + two examples) and the Self-Reflection prompt are verbatim. Trial 1 / Trial 2 code bodies and the test feedback are schematic; the original work reports Pass@1 and the ablation table, not a per-trial trace for this problem.

code test feedback self-reflection Memory (last 1) FAIL PASS

point: HumanEval Pass@1 climbs from GPT-4 80.1 to 91.0 (PY) and 60.0 to 68.0 (RS); Leetcode Hard doubles 7.5 to 15.0. But MBPP Python actually drops 80.1 to 77.1 because false-positive test suites let the agent stop early. WebShop terminates after four trials without improvement.

Reflexion GPT-4 baseline Prev SOTA / ablation paper-report numbers

The four cards are paper-report results, not computed from the printout above. WebShop is terminated by the authors after four trials; the agent does not generate useful reflections there.