Continual learning is the missing half of harness RSI

1. Overview

• Agents improve in two ways that people often mix.
• They can change how they run: prompts, tools, workflow, harness code.
• They can also keep better state across related work in the same environment.
• Recent harness work covers the first path well.
• The second path, continual learning of durable environment state, is still thin.
• A strong runtime with weak memory still forgets the world it just explored.
• This post separates the two loops and argues that RSI needs both.

2. Prior research

• Lilian Weng's Harness Engineering for Self-Improvement (Jul 2026) is a clear map of the harness path.
• She treats the harness as the layer around a base model: planning, tools, context, artifacts, and checks.
• She surveys workflow loops, files as memory, sub-agents, context engineering, workflow search, self-improving harnesses, and evolutionary search.
• Near-term RSI in that line of work is not weight rewrite. It is better machinery for getting better answers.
• Meta-Harness, ADAS, AFlow, STOP, Self-Harness, Darwin Gödel Machine, ACE, and MCE mostly optimize how one rollout runs or how the harness itself evolves.
• Weng names continual learning, self-play, synthetic data, and test-time training as part of the wider RSI story, then leaves that thread for later.
• That is the opening this post takes: prior research owns Loop A; Loop B still needs its own treatment.

3. Two different improvement loops

• Loop A (harness RSI): change prompts, tools, workflow, permissions, or harness code so future rollouts work better.
• Loop B (continual learning): keep durable knowledge so later work in the same environment scores higher than early work.
• Loop A asks: how should the agent run?
• Loop B asks: what should the agent remember?
• Prior harness research mostly attacks Loop A.
• Loop B is still often a side effect of long context, not a designed learning path.
• RSI needs both.

4. What "improve with experience" must mean

• Online improvement needs shared structure across related work, not only a longer transcript.
• The structure is usually latent: conventions, maps, habits, layouts, dynamics.
• A system that only reacts to the current prompt cannot use that structure later.
• Raw reward is a weak learning signal. Task difficulty varies. Strong models can score high with no learning.
• Compare stateful performance to a reset or fresh-start baseline. That gap is the learning signal.
• Concept drift is part of the problem. Stale memory must be updated or dropped.
• Success looks like reuse: fewer rediscovery steps, better late-run score, fewer repeated mistakes.

5. Why more memory often fails

• Dedicated memory modules often fail to beat simple in-context learning.

• Common failure modes:

  • store too much

  • store the wrong thing

  • store fluent slogans instead of actionable state

  • fail to reuse what was stored

• Agents overfit recent observations and underuse older, still-relevant structure.

• More retrieval does not fix this by itself.

• The bottleneck is storage policy: what enters memory, what stays, and how later steps use it.

• Context-engineering research makes the same point for trajectories: dumping everything into context is not a policy.

6. What a good Loop B surface looks like

• Keep durable state outside the growing chat blob, often as files or short structured notes.
• Write for the next actor, not for the current transcript.
• Prefer entities, parameters, maps, encodings, and counts over strategy slogans.
• Merge conflicts. Drop weak one-offs.
• Make the memory inspectable. If humans and agents cannot read it, they cannot debug it.
• Separate mechanism (how memory is managed) from content (what is stored).
• Do not require weight updates to count as continual learning. Non-parametric state is enough for a large class of agent work.

7. Examples from Continual Learning Bench

• Continual Learning Bench is one place that forces Loop B into the open. It is an example, not the whole field.
• Each task is a sequence of instances in one shared environment with discoverable latent structure.
• Examples of that structure: schema conventions, spectrum occupancy, opponent habits, codebase layout, demand patterns.
• The bench uses reward plus gain so high base skill does not hide missing online learning.
• Thin memory systems there (for example, end-of-instance reflected notes) show the same pattern as the wider literature: the storage policy matters more than the presence of a memory slot.
• Use the bench to test ideas. Do not treat any one task list as the definition of continual learning.

8. What stays open

• Horizon: how long must a run be before durable state pays off?
• Drift: when should memory clear, mark stale, or merge?
• Wrong memory: how do we stop systems from writing confident false structure?
• Reward hacking of memory: notes that help the scorer but not the true latent object.
• Transfer: does a memory policy learned in one domain help in another?
• Joint path: when should Loop A (edit the harness) and Loop B (edit the notebook) run together?
• Humans: where should oversight sit when the agent updates its own long-lived state?

9. What to measure next

• Same-model comparisons of memory methods against in-context learning and against a fresh-start baseline.
• Learning curves: early vs late work, plus gain vs reset.
• Memory audits: does stored state match the true latent structure, or only correlate with score?
• Ablations that change storage policy only. Keep the rest of the harness fixed.
• Report model tier separately from memory method. A stronger model is not the same claim as a better learner.

Sources used for this draft

• Lilian Weng, Harness Engineering for Self-Improvement, Lil'Log, Jul 2026 (prior research overview).
• Asawa et al., Continual Learning Bench, arXiv:2606.05661 (example benchmark and findings).
• Related harness / context work as summarized in Weng (ACE, MCE, Meta-Harness, Self-Harness, and others).