The Harness Decides When the Work Is Done
What harness optimization research teaches us about completion checks, verified repository memory, and improving coding agents through controlled experiments.
On this page 8 sections
An autonomous coding agent can investigate the right bug, write a plausible fix, and explain its reasoning clearly. The process can exit successfully while the pull request targets the wrong branch, the test results belong to an earlier commit, or the coverage report never reaches the engineer.
The model has stopped. The engineering task still has unfinished work.
In From Copilot to Delegate, I described the sandbox as the boundary that makes autonomous engineering work manageable: isolated workspaces, scoped access, bounded execution, and reviewable outputs. That raises a further question: what should the system verify before declaring a job complete?
Joel Niklaus’s article, Don’t Train the Model, Evolve the Harness, makes this question concrete. It studies how much agent performance can improve when the model stays fixed and an automated loop changes the software around it.
For the engineering sandbox I am designing, the useful lesson is to make completion an explicit runtime contract—and make improvements to that runtime measurable.
What the experiment actually showed
The experiment used a fixed DeepSeek-V4-Pro model on Harvey’s Legal Agent Benchmark. Tasks involved documents and concrete deliverables such as memos, drafts, and issue lists. A separate model judged those deliverables against task-specific rubrics.
A proposer examined execution histories and proposed changes to the harness. Candidates were evaluated on a 24-task development set, with three trials per task. A separate 100-task test set was held out from the optimization loop.
The article reports these held-out results:
| Metric | Original harness | Optimized harness |
|---|---|---|
| Rubric criteria passed | 63.4% | 80.1% |
| Tasks passing every criterion | 0% | 5.0% |
The first measure improved by 16.7 percentage points on held-out tasks. The roughly 20-point improvement discussed in the article refers to the development set.
Both measures matter. Passing more criteria shows useful progress, while the 5% whole-task result shows how much remained unresolved. This was a legal benchmark with an LLM judge and a small development set; it does not establish an expected improvement for coding agents.
What transfers is the engineering method: inspect concrete failures, change one mechanism, and test whether the improvement survives beyond the examples that motivated it.
The largest gains came from ordinary execution problems
Some of the benchmark’s failures were surprisingly familiar. The agent produced a document but saved it under the wrong name or in a directory the evaluator did not read. Sequential writes replaced earlier content. A long-context run produced an answer about the wrong matter. Tool calls failed or the agent repeated unproductive actions.
The harness could address several of these failures directly. It could place an existing deliverable where it belonged, detect an incomplete artifact, check whether the output concerned the requested matter, or recover from a protocol error.
The article reports that five of its top six harnesses were code-based changes rather than prompt-only changes. Two prompt playbooks also entered the accepted lineage, so the result is not evidence that prompts are unimportant. It does suggest that prompts should not carry responsibilities the runtime can enforce reliably.
For a coding agent, the equivalent failures are easy to recognize: missing reports, incorrect PR targets, stale validation results, and repeated tool failures. These are useful starting points because each gives us something observable to check.
Give each workflow a completion contract
A zero exit code is useful evidence about a process. A completed engineering outcome needs additional evidence.
Before execution starts, the harness should record the requested scope, acceptance criteria, source revisions, target branches, required deliverables, and validation expectations. The workflow determines which checks apply.
| Workflow | Evidence the harness should check |
|---|---|
| Bug fix or enhancement | Changes address the agreed scope; required PRs exist and target the intended branches; validation identifies the tested commit |
| Coverage improvement | A parseable report exists, identifies the tested revision, and records the measured result against any requested target |
| Code review | A findings artifact identifies the inspected scope and revision, including a legitimate “no findings” outcome |
| Security remediation | Findings, remediation evidence, and remaining issues are recorded for the requested scope |
Some checks are mechanical. The harness can inspect a PR’s target branch, parse a report, or compare a test run’s commit with the delivered commit. Whether a change satisfies product intent may still require acceptance tests or human review.
That distinction should remain visible in the job result. “Agent finished,” “deliverable verified,” and “CI passed” describe different events. A draft PR awaiting CI can be a valid delivery milestone without being reported as a fully verified change.
If a required check fails, the harness can return the exact failure to the agent for a bounded repair attempt. It should preserve the original scope and execution budget. After the repair limit, it should report the unresolved condition and retain useful work.
The completion check belongs before workspace cleanup, while the evidence and artifacts are still available.
The upper loop delivers a task. The lower loop improves the machinery used by later tasks. Failed checks can trigger bounded repair; harness changes pass through a separate evaluation and release process.
Better context still matters
Completion checks cannot rescue every weak plan. An agent working across dozens of repositories also needs help finding the right code and operating instructions.
My planned repository-intelligence layer addresses that problem with versioned repository profiles, build and test commands, dependency relationships, and verified lessons from earlier work. A bounded context pack supplies the relevant subset before execution, with current source and code-graph evidence available for validation.
The harness research makes me more careful about what this memory is allowed to claim.
A command found in CI is evidence that the command is declared there. A successful test run is evidence that it worked at a particular revision, in a particular environment. A merged PR establishes a delivery outcome; it does not independently prove every root-cause explanation in the agent’s summary.
The same care applies to freshness. The context pack should record the source commits it describes, and the worktrees should start from those commits. If the branch moves before execution, the system should revalidate the context or continue from the recorded revision.
Otherwise, the agent begins with a briefing about one version of the repository and edits another.
There are two different things to learn
Repository learning and harness learning need separate records and promotion rules.
Repository learning might capture that a service’s integration tests require PostgreSQL, or that a consumer depends on a particular event contract. Its authority comes from current repository evidence, observed execution, or an authorized review.
Harness learning might establish that checking report existence before cleanup catches incomplete coverage jobs, or that a smaller context pack reduces discovery time without missing required repositories. Its authority comes from comparative evaluation.
A useful repository fact should not silently modify the runner’s behavior. A promising runner change should not become production policy because it helped one job.
For harness changes, I would keep a versioned record of the hypothesis, parent commit, changed mechanism, evaluation results, and rollback reference. The proposer can suggest changes to prompts, retrieval, tool wrappers, or completion checks. The evaluator and release policy remain outside that change set.
This gives the system a way to improve while keeping the reason for each change inspectable.
Start with a controlled experiment
The background section connects harness optimization to a broader family of systems that propose changes and retain those that pass a verifier. We can adopt that pattern before building an autonomous optimizer.
Start with a failure that can be reproduced. For example, several otherwise successful coverage jobs may have failed to deliver their reports. Propose one mechanism: validate and register the report before the workspace is removed.
Then test the cases it is intended to fix and cases it must preserve. A code review with no findings should still complete correctly. A bug-fix job should not acquire an unrelated coverage requirement.
For deterministic changes, saved artifacts can provide a cheap first test. A new report validator can run against previous outputs without repeating the entire agent session. But a change to retrieval or repair behavior needs fresh agent runs, because replay cannot show how the agent would respond.
Comparisons should record the model configuration, harness and skill versions, context-pack policy, repository revisions, evaluator version, and resource budget. Repeated trials help distinguish an improvement from run-to-run variation. Held-out tasks help detect overfitting, and historical tasks must use knowledge available before their eventual fixes.
The article’s one-point promotion margin was a choice for its experiment. An engineering workload needs a margin justified by its own observed variability.
Required scope and validation checks should be hard gates. Among candidates that pass them, compare verified completion rate, latency, cost, and human correction effort.
Reliability changes the measurement
The article also describes provider errors, interrupted proposer sessions, and a scoring bug that briefly promoted an inferior harness. The candidate’s score used new weights while the incumbent’s stored score reflected older weights.
That is a useful reminder to retain raw measurements and calculate both sides of a comparison under the same metric definition.
For an engineering harness, failures should identify the stage involved: discovery, workspace setup, model access, tool execution, implementation, validation, or delivery. A provider outage and a fix that fails its regression test call for different remedies.
Retries need the same care. Retrying a transient model request may be reasonable. Repeating a PR-creation action after losing its response can create duplicate work unless the harness checks whether the action already succeeded.
Prompt changes also need evaluation across the models that will use them. The article found that some tailored playbooks did not transfer well across model families. Shared execution guarantees can remain common while prompt profiles are versioned and evaluated per model and workflow.
The next change I would build
I would begin with the boundary between the agent process ending and the job becoming successful.
At that boundary, the runner should inspect the workflow’s completion contract, validate the required evidence, and record unresolved conditions. It should allow bounded repair where appropriate and preserve partial work before cleanup.
Alongside that, I would build the smallest useful repository context pack and a representative evaluation set. Those pieces make it possible to test whether better context actually improves delivered outcomes.
Automated proposal generation can come later. The first requirement is a trustworthy way to tell whether a change helped.
The harness is where an engineering system turns a request into an execution plan, constrains the tools, preserves evidence, and accepts the result. Giving those responsibilities explicit contracts makes the next improvement easier to find—and harder to mistake for progress.
Source and further reading: Joel Niklaus, Don’t Train the Model, Evolve the Harness, including its experimental protocol, results, and measurement details. The author also links the harness implementation and the Meta-Harness paper. Performance figures above are reported by that article; the coding-agent architecture and proposed implementation steps are my interpretation.