WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

core: WikiSkill separates the agent workspace into three layers — an immutable Raw Layer of execution traces, a persistent Wiki Layer of accumulated patterns and audit logs, and a Skills Layer of active procedural instructions — and drives an iterative loop where the Inference Agent rolls out with current skills (no wiki access), the Wiki Maintainer consolidates sampled traces into patterns, the Skill Proposer reads the wiki and proposes skill updates, and a gating step accepts only proposals that improve validation. Skills can be rolled back; the wiki never is.

Five panes follow: the three-layer workspace; one full iteration through the case study (Qwen-3.6-27B on ALFWorld, Iter 0-1-4 verbatim); main results across five models; cross-model transfer plus ablation; and the paper's own limits. Traces are schematic; numbers marked paper-report come from the paper tables.

primer — what you need before you start

the task. WikiSkill is a framework for co-evolving an LLM agent's procedural skills and a persistent knowledge base (wiki). The workspace is split into three filesystem layers: an immutable Raw Layer where every iteration's execution traces are stored, a Wiki Layer where patterns of success and failure are accumulated across iterations, and a Skills Layer where the active procedural instructions live. The evolution loop tightens these two representations against each other: the wiki gives the Skill Proposer long-horizon context, and validation gating decides which skill proposals are kept.

how it was done before. Three prior families: EvoSkill keeps a flat feedback history of past proposal outcomes; Trace2Skill runs parallel success / error analysts on training rollouts and merges their patches; SkillOpt runs a six-stage ReflACT pipeline (Rollout, Reflect, Aggregate, Select, Update, Evaluate). None of them maintains a separate, evolving knowledge representation. WikiSkill adds one — a wiki that compounds across iterations while skills are validated and rolled back.

glossary — terms you will see.
agent — an LLM-based system with environment tools and an active skill set.
skill — a filesystem directory with SKILL.md (full content) and PURPOSE.md (provenance back-link to wiki).
wiki — persistent structured knowledge: patterns/, logs.md, impact.md, index.md.
Raw Layer — immutable execution traces from training rollouts.
Wiki Layer — structured patterns + audit logs; compounds across iterations.
Skills Layer — active procedural instructions; gated by validation.
iteration — one cycle: inference -> maintain -> propose -> gate.
validation gating — accept proposal only if validation score strictly improves.
rollback — skill set reverts to the last accepted state; wiki is not rolled back.
provenance — PURPOSE.md back-link from a skill to the wiki patterns that inspired it.
skill retrieval — when many skills exist, picking which ones to load for a given task.
skill triggering — deciding when (or whether) to invoke a loaded skill during execution.
benchmarks — LiveMath (math competition reasoning), SealQA (web-search factual QA), SpreadSheet (Python code on spreadsheet tasks), OfficeQA (long-context document QA), ALFWorld (text-based embodied household tasks).

symbol table — what the letters mean.
$\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{val}}, \mathcal{D}_{\text{test}}$ — train / validation / test task splits.
$S_k$ — the active skill set at iteration $k$.
$W_k$ — the persistent wiki at iteration $k$.
$\mathcal{R}_{\text{best}}$ — the best validation score seen so far.
$\mathcal{T}_{\text{train},k}$ — set of training rollouts at iteration $k$.
$\mathcal{T}_{\text{sample},k} \subset \mathcal{T}_{\text{train},k}$ — sampled subset for the Wiki Maintainer (up to 8 traces, 5 fail + 3 pass).
$P_k$ — the proposal (skill modification) at iteration $k$.
$a_k \in \{\text{Accepted}, \text{Rejected}\}$ — the gating decision.
$\mathcal{M}_{\text{WM}}$ — the Wiki Maintainer agent.
$\mathcal{M}_{\text{P}}$ — the Skill Proposer agent (ReAct-style).
$\pi$ — the Inference Agent policy.

the gating rule. $S_k = S'_k \text{ if } \mathcal{R}(\mathcal{T}_{\text{val},k}) > \mathcal{R}_{\text{best}}; \text{ else } S_k = S_{k-1}$. Wiki updates unconditionally; only the skill set is gated.

tone legend — what the [brackets] mean.
[obs] observation   [arch] architecture / layer
[loop] iteration step   [pipe] pipeline trace
[gating] accept / reject decision   [res] paper-reported result row
[abl] ablation row   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Five panes cover, in pedagogical order: workspace layout, one iteration through a real case study, main results, cross-model transfer plus ablation, and the paper's own limits. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: the workspace is a filesystem with three directories. raw/ holds immutable execution traces from every training rollout. wiki/ holds four file types: patterns/*.md (one file per recurring failure or success pattern), logs.md (chronological evolution log written by the Wiki Maintainer), impact.md (proposal diffs and accept/reject outcomes written by the harness), and index.md (a catalog of the current pattern pages). skills/ holds the active procedural instructions — each skill is a directory with SKILL.md (full content injected into the agent prompt) and PURPOSE.md (provenance back-link to the wiki patterns that inspired the skill).

layer / role description structural / layout key takeaway (brackets)

point: one iteration runs four components in order. The Inference Agent rolls out training tasks using the current skills and is denied wiki access. The Wiki Maintainer samples up to eight training traces (five failing for root-cause analysis, three passing for strategy extraction), creates or patches patterns/*.md, updates index.md, and appends to logs.md. The Skill Proposer runs as a ReAct-style agent, reads the wiki index and skill-impact.md, uses read_file to inspect specific pattern pages and raw traces on demand, and produces an atomic proposal P_k that creates or patches a single skill. The gating step validates the candidate on the validation split and accepts only if the score strictly improves. Skills can be rolled back; the wiki never is. The case study trace (Qwen-3.6-27B on ALFWorld, Iter 0 / 1 / 4) shows the loop in action: rejected proposal at Iter 0, accepted break-repetition-loop with rule "Never Return an Item to Its Origin Location" at Iter 1, refined rule "Each Operation Type ONCE Per Item" at Iter 4.

iteration step pipeline trace accept / reject decision design property

point: across five models (Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-4-31B, Gemini-3.5-Flash) and five benchmarks (LiveMath, SealQA, SpreadSheet, OfficeQA, ALFWorld), WikiSkill wins average accuracy over the strongest prior skill-evolution method on every model: +3.3 / +5.1 / +10.0 / +5.8 / +12.0 absolute percentage points. Within the Qwen family the gain grows with model scale: 4B +12.3, 9B +17.5, 27B +23.9 over the no-skill baseline. Smaller model with skills beats larger model without: Qwen-3.5-9B with WikiSkill reaches 47.4% average accuracy versus Qwen-3.6-27B no-skill at 39.4%. Prior methods regress on specific pairs — Gemma-4-31B EvoSkill on LiveMath drops from 33.9 to 29.8, Gemini-3.5-Flash SkillOpt on SealQA drops from 29.4 to 28.2 — WikiSkill does not. All numbers are paper-reported, averaged over three independent evolution runs.

result row headline result (brackets)

point: two sheets, one argument. Cross-model transfer: skills evolved by a larger model often help a smaller model more than self-evolved skills do — Qwen-3.6-27B skills lift Qwen-3.5-9B on SpreadSheet from 24.3 (no skill) / 33.6 (self-evolved) to 50.5; the same 27B skills lift Gemma-4-31B on LiveMath from 33.9 / 56.7 to 73.7. Negative transfer exists: Qwen-3.5-4B skills drop Gemini-3.5-Flash on SpreadSheet from 50.5 to 18.1 because the smaller model encodes low-level workarounds that constrain the stronger model. Ablation: the persistent wiki is the cause. Proposer wiki access = Avg +15.0 (48.7 to 63.7); Inference Agent wiki access during rollouts HURTS = Avg -2.8 (63.7 to 60.9) because the agent may obtain task-solving knowledge directly from the wiki rather than from the skills, making the resulting trajectories less informative for skill development.

transfer result ablation row design reason key takeaway (brackets)

point: four paper-acknowledged limits, stated in the paper's own Limitations section. (1) Full-injection only — all skill content goes into the system prompt; no eval of skill retrieval or triggering, which becomes important as skill count grows. (2) Strict validation gating — only proposals with strict improvement are accepted; neutral proposals (preserve performance, enable future gains) are excluded. (3) No wiki pruning — patterns, logs, and impact entries accumulate forever; long-horizon runs may produce an unwieldy wiki. (4) Not long-horizon — benchmark scope is bounded (ALFWorld tasks cap at ~134 tasks × 18 actions); online skill adaptation within a single long rollout is open future work. The paper evaluates skill quality (do the procedures work when injected?), not skill retrieval (which skills to load when there are 100s?).

paper-reported limit scope / future work key takeaway (brackets)