NowHarness
The interesting part of an agent is not the model, it is everything around it: tool selection, how much of the page the model reads, whether claimed work actually happened, and what stops unsafe writes. NowHarness builds those pieces one at a time and measures each against WorkArena. The evaluation layer came before the agent, a failure taxonomy decided what to build next, and every candidate feature was kept or reverted based on seed-matched re-runs. The adopted configuration raised task success from 53.3% to 83.3% on six held-out task types while cutting output tokens 61% and median latency 46%. Results live on a public dashboard.
The problem
Agent leaderboards mostly rank models, but enterprise reliability lives in the harness around the model: which tool gets selected, what the model has to read, whether the work it claims actually happened, and what stops a write it should not make. I wanted to build that layer piece by piece and prove, with statistics, which pieces earn their keep.
Approach
- 1.Built the evaluation layer before the agent, so every reported number traces to one of ~860 logged episodes. Two ground rules: an HTTP 200 is tool success, not task success, and for anything that writes, the model proposes and deterministic policy disposes.
- 2.Started with a deliberately basic agent, labeled its failures into a taxonomy, and let the taxonomy decide what to build next.
- 3.Ran ten experiment versions, changing one thing at a time with seed-matched re-runs. Two features were adopted: deterministic verification (28.6% cheaper at flat success) and structured tools (every previously-0% filter task taken off zero, Fisher p = 0.0016).
- 4.Rejected five features with their mechanisms explained, including a +6-point compression 'win' killed by seed-matched confirmation. The bar that killed the false win is the same bar the adopted features cleared.
- 5.Validated on six held-out task types that influenced zero engineering decisions, then published every run to a live dashboard.
Result
On the held-out suite the adopted harness raised task success from 53.3% ± 13.9 to 83.3% ± 11.8 while cutting output tokens 61% and median latency 46% (5 seeds per arm). On holdout, the agent discovered untaught ORDERBY sorting through the structured tool, going 0/5 to 5/5.