Agent Learning Bench: score the learning, not the job

Most agent benchmarks score a finished job. Agent Learning Bench measures whether repeated work gets better, cheaper, and more robust across a long task horizon without prescribing how the system learns.

Manoj BajajSeptember 10, 2026Updated September 10, 2026
harness-engineeringagents

Most agent benchmarks score a finished job. They do not score whether experience makes the next hundred jobs better, cheaper, or easier to recover.

Why learning needs its own benchmark

Repeat work is normal work. The next job is rarely an exact copy, but it often happens in a world the worker has already seen: another question against the same database, another issue in the same repository, another support case under the same policy, or another workflow in the same product.

  • A Smartsheet survey found that more than 40 percent of workers spent at least one quarter of their week on manual, repetitive tasks.
  • McKinsey found that about 60 percent of occupations contain at least 30 percent of activities that could be automated with demonstrated technology. This does not mean whole jobs repeat verbatim. It means recurring work is spread through most jobs.
  • Agents deployed into these workflows will see related tasks for days or months. A system that starts from zero each time can repeat the same search, the same tool misuse, and the same mistake.
  • Google Research describes the same problem for persistent agents: without a way to use earlier successes and failures, they repeat strategic errors and discard useful insights.

Static benchmarks are good at measuring capability. They tell us whether a model and harness can solve a task now. They usually do not tell us what happens after the system has accumulated useful and misleading experience.

  • Did reward improve because of earlier jobs?
  • Did the system need fewer actions, queries, tokens, or dollars?
  • How many examples did it need before behavior changed?
  • Did new work erase an older skill?
  • Did knowledge transfer to an unseen tail of related jobs?
  • Could the system recover when a small part of the environment changed?

ARC Prize describes intelligence as skill acquisition efficiency: how efficiently a system turns priors and experience into new skills. That framing is useful beyond abstract puzzles. For production agents, we need to measure the same conversion over realistic work and over a much longer horizon.

Measure the learning system, not the learning mechanism

This benchmark should not define learning as one implementation pattern. The system can change during a run or between batches of work. It can learn online or offline.

Experience might alter:

  • the model weights;
  • a harness rule or tool policy;
  • stored examples, summaries, or procedures;
  • a retrieved history of successes and failures;
  • a world model of the environment;
  • code written by the agent or by an optimizer outside the agent.

The benchmark does not need to decide which path is "real" learning. It evaluates the end-to-end learning system and asks whether earlier experience changes later behavior. If two systems use different mechanisms, their curves should still be comparable.

Benchmark design. A shared world produces ordered jobs. Any learning system can process them. The benchmark records reward and resource use for every job, then reports learning curves.

Design tasks that contain something worth learning

A long list of unrelated benchmark questions is not a learning task. Earlier work must reveal structure that can help with later work, while later jobs must still require fresh reasoning.

A useful task has these properties:

  • A shared world. Jobs happen in the same database, codebase, website, policy set, game, or body of documents. The world contains stable structure that is not fully explained at the start.
  • Related but non-identical jobs. Repeating the exact prompt tests recall. A learning benchmark needs new jobs that reuse schema knowledge, navigation patterns, conventions, strategies, or causal rules.
  • A long horizon. Ten jobs can reward a transcript that still fits comfortably in context. Tens or hundreds of jobs expose compaction, retrieval, stale knowledge, update cost, and accumulated mistakes.
  • Clear feedback. The system needs evidence from outcomes, tests, errors, or graders. Feedback may arrive after each job or in offline batches, as long as the protocol records when the system could use it.
  • Enough headroom. If the first job is already easy, there is little learning signal. The task needs room for quality to rise or resource use to fall.
  • A holdout tail. The final part of the sequence should contain unseen jobs from the same world. Success there shows reuse, not a copied answer.
  • Retests and small changes. Earlier skills should be checked again after new work. A limited environment change should test whether useful knowledge survives and whether one stale fact can be repaired.
  • Per-job telemetry. Record reward, actions, queries, tokens, API cost, wall time, errors, and any task-specific resource. A final average hides the shape of learning.
  • Order that does not fake improvement. Later jobs should not simply be easier. Use balanced schedules, multiple orders, or repeated trials when task difficulty varies.

Reward must follow the use case. On the same database question-answering task, correct answers, SQL queries, API cost, and latency can tell different stories. A cheap wrong answer is not a win, but an expensive correct answer may also fail the product requirement. The benchmark should name the primary reward before the run and still publish the other curves.

What to measure

The next diagrams are illustrative. They show the shape each metric should detect, not results from a specific run.

Outcome improvement

The first curve asks whether task performance improves as experience accumulates. Reward can be correctness, successful completion, error reduction, or another task-specific outcome.

  • Compare the learning system with its own starting capability or reset behavior.
  • Report the curve across jobs, not only the last point or cumulative total.
  • Treat a widening gap as evidence that experience is helping.

Illustrative outcome curve. A learning system improves across jobs while a system with little or no learning stays flat.

Learning efficiency

Learning efficiency asks whether each successful job needs less work. A strong curve falls sharply after early exploration. A nearly flat curve means experience has not reduced the cost of success.

  • The resource can be tool calls, environment actions, SQL queries, files opened, tokens, dollars, wall time, or a weighted combination chosen for the product.
  • Measure cost per successful job so that a system cannot look efficient by failing early.
  • Keep the outcome curve next to the cost curve. Lower cost only matters when quality holds.

Illustrative learning-efficiency curve. Cost per successful job falls sharply for a strong learner and stays almost flat for a weak learner.

Sample efficiency

Sample efficiency measures how much experience the system needs before it reaches useful performance. Two systems may end at the same quality while one needs far fewer jobs to get there.

  • Choose a target quality that matters for the task.
  • Count jobs, feedback events, or another unit of experience required to cross it.
  • Report early behavior because production teams pay for the learning period too.

Illustrative sample-efficiency curve. One system reaches the target after a few experiences while another needs many more.

Retention

Retention asks whether new learning damages older capability. To see catastrophic forgetting, retest earlier skills while the system continues to work on later jobs.

  • Keep some earlier checks active at later checkpoints.
  • Separate temporary interference from a persistent loss.
  • A system that learns the newest task but loses an older one has moved knowledge, not accumulated it.

Illustrative retention curve. One system preserves reward on earlier skills while another forgets them after new work begins.

Holdout transfer

Holdout transfer asks whether knowledge from a long experience prefix helps on unseen jobs in the same world. The tail should reuse latent structure without repeating earlier answers.

  • Mark the boundary between the experience prefix and the holdout tail.
  • Compare tail quality and resource use with a system that did not receive the prefix.
  • A sharp collapse at the boundary suggests memorization or overfitting rather than reusable knowledge.

Illustrative holdout curve. Reusable knowledge survives on the unseen task tail while prefix memorization collapses.

Robustness and repair

Robustness asks whether useful knowledge survives a small environment change. Repair asks whether the system can update the stale part without discarding everything else.

  • Change one bounded feature, such as a column name, page layout, policy threshold, or opponent behavior.
  • Measure the immediate drop, the number of jobs needed to recover, and the final recovered quality.
  • A robust system may stumble once. It should not rebuild its understanding from zero.

Illustrative robustness curve. A robust system dips after a small change and recovers. A brittle system stays broken.

Prior work that shaped this benchmark

Two recent benchmarks provide much of the starting point, but neither closes the measurement problem for long-running work.

ARC-AGI

ARC-AGI made learning efficiency central to the definition of intelligence. ARC-AGI-3 extends this into interactive environments where an agent must explore, infer goals, build a model of the world, and act efficiently relative to people.

What we take from it:

  • Skill at one static task is not enough. Experience and the cost of acquiring skill matter.
  • Interactive tasks can measure exploration and action efficiency.
  • Human baselines make inefficient brute force visible.

What it leaves open:

  • Its abstract game environments are designed for novel skill acquisition, not repeated production work.
  • A game episode does not model months of related jobs, delayed updates, growing system state, or business-specific costs.
  • Action efficiency is one resource curve. Agent products also care about tokens, API spend, latency, errors, and the cost of maintaining the learning system.

Continual Learning Bench

Continual Learning Bench is the closest prior benchmark for agents that improve across ordered, related tasks. It introduced expert-validated shared environments and a gain metric that compares a stateful system with its stateless version.

What we take from it:

  • Tasks need hidden, reusable structure and enough headroom to learn.
  • Reward should be recorded for each instance in an ordered schedule.
  • A reset comparison helps isolate learning from base capability.
  • Its result that full-context in-context learning beat the evaluated dedicated memory systems gives us a useful simple system to test.

What it leaves open:

  • The paper says its task sequences are on the order of tens of instances and shorter than the longest deployment horizons practitioners may care about.
  • Its initial experiments focus on context retention, compaction, retrieval, and structured notes. A general learning benchmark should also admit offline updates, harness changes, world models, and weight updates.
  • Aggregate reward, gain, and total run cost can still hide when costs rise, when learning starts, or when older knowledge fails. The per-job curves need equal status.

The goal is not to replace either benchmark. ARC-AGI gives us the learning-efficiency lens. Continual Learning Bench gives us realistic shared environments and a clean learning-gain comparison. Agent Learning Bench extends those ideas toward long, repeated work and mechanism-neutral measurement.

A small study: full-context learning on Formula 1

We used in-context learning as one example learning system because Continual Learning Bench found that the simplest full-context system outperformed the dedicated memory systems it evaluated. The setup is intentionally plain: keep the full trajectory instead of starting a fresh chat for every job.

The task uses one Formula 1 SQLite database and a fixed order of analyst questions. The schema stays the same. The model and agent harness stay the same. We record binary correctness, SQL queries, API cost, and wall time.

On a short run of ten jobs, keeping context wins on all three reported axes:

  • Correct answers rise from 5 of 10 to 7 of 10.
  • Mean SQL queries fall from 2.9 to 1.4 per job.
  • API cost falls from about $0.21 to $0.13 because the repeated prefix benefits from prompt caching.

On the full run of 174 jobs, the result splits:

  • Correct answers rise from 65 of 174 to 74 of 174.
  • Mean SQL queries fall from 4.8 to 1.1 per job, a strong sign that the system keeps and uses schema knowledge.
  • API cost rises from $4.55 to $9.92.
  • Wall time rises from 68 minutes to 132 minutes.

ICL relative to a fresh chat. It improves short-run hits, queries, and cost. Over 174 jobs it still cuts queries, but API cost more than doubles.

The late prompt is almost fully cached, but the cached context still grows. Early blocks of ten jobs cost about $0.11. Late blocks cost about $0.85 to $0.92 while SQL queries remain low.

Over 170 ICL jobs, SQL queries stay near one per job while API cost per ten jobs rises from about $0.11 to about $0.90.

This is why a learning benchmark needs several curves:

  • If reward means correct answers, full-context learning produces a modest gain.
  • If reward means SQL queries, it produces a large gain.
  • If the product constraint is API spend or wall time over a long day of work, the unbounded transcript loses.
  • If we looked only at the first ten jobs, we would reach the wrong production conclusion.

This study is not a general verdict on in-context learning. It is one model, one task, and one example learning system. It shows that learning can be real while the delivery mechanism remains unsuitable for a long horizon. A better system on this task would keep the query reduction without carrying every past token into every future job.

Close

An agent benchmark should score the finished job and the effect of experience on later jobs. The task must contain reusable structure, enough time for learning and forgetting to appear, a holdout tail, bounded changes, and telemetry for both outcomes and resources.

The implementation can learn online or offline. It can update a harness, stored experience, a world model, code, or model weights. The benchmark does not need to choose. It needs to show, job by job, whether the system improved, how quickly it improved, what that improvement cost, whether it lasted, and whether it transferred.

Sources