Autonomous Research Agents

Autonomous research scaffolds regularly publish numbers their experiments never produced. The usual explanation is that the model fabricates. In the scaffold we audited, the mechanism turned out to be mundane and mechanical: experiment output was truncated to 1,000 characters before any agent could read it, and the crash marker was appended after the program's own output — so it fell off the end of the same slice.

The writing agent could not see the real numbers. The failure detector could not see the crash. A run that raised NameError on every attempt was scored a perfect 1.0 by the reward model and written up with a full results table to two decimal places. None of it was measured.

That reframes the whole thing. It is an information-flow defect, not a model tendency — which means it can be closed with plumbing rather than prompting.

Unsupported claim detection went up 25–35% against a live agent scaffold, benchmarked against three published arXiv baselines and four reported research metrics. Repeated undetected failures — the same crash sailing through run after run — dropped by over 98%, while false positive flags stayed under 10%, because a false flag costs the agent a rewrite for nothing.

The pipeline delivered 15 reproducible agent-generated papers at over 90% causal chain integrity, where a chain is the full link from task → command → log → value. Each link is marked resolved or not, so a broken chain gets reported rather than papered over. That distinction is the point: a number that merely matches something in a log is not the same as a number causally attributable to a recorded run.

Experiments declare their results through an API injected straight into the namespace — nothing to import, nothing for the agent to get wrong:

record_result("exp1.K2.test_acc", test_acc, unit="ratio")

You pass the variable. A number typed into the call is rejected — the gate re-parses the source, finds the call site, and fails any value that is a literal rather than the result of a computation. Constant folding does not launder a typed number either.

Around that sits a static tier that rejects broken code before it costs any compute, and a runtime tier that runs the experiment in a fresh process with an empty namespace and hashes the source the child actually executed against the source that was submitted. That hash is what turns "this value came from this code" from an assumption into something checkable.