How to Validate an Agent Loop
Validation is not just running a command. A good loop records the expected evidence, the actual evidence, whether the hypothesis passed, what failed, and what should change before the next run.
Will a checker reject a completion claim when the visible aggregate metric passes but the required evidence was removed?
What we tested
We wrote a maker artifact that skipped the original failing test and proposed merge after aggregate CI became green. A separate checker evaluated test preservation, merge authority, and root-cause evidence with Pass, Partial, and Stop decisions.
Observed result
The checker returned Stop for the skipped test and automatic merge, and Partial for a parser change that lacked a before-and-after reproduction. Green CI was treated as a claim, not as permission to advance the lifecycle.
What this does not prove
The local checker is deliberately simple and does not establish semantic program correctness. The cited Proof-or-Stop work is a recent preprint with its own stated limits, including one model family, a bounded task sample, and self-hosted evidence.
Why it matters
Without a validation log, repeated agent work becomes hard to audit. You may know that something happened, but not whether the loop improved, took a shortcut, or should be scheduled. A lightweight log keeps the loop understandable.
Validation log decision
- 01Did expected evidence appear?Pass or partial
- 02Did the loop take a shortcut?Fail and tighten boundaries
- 03Did cost or iterations surprise you?Adjust budget or stop rule
- 04Is the next run predictable?Only then consider scheduling
Practical checklist
- Define expected evidence before running
- Capture actual evidence after running
- Mark the result as pass, fail, or partial
- Record what changed and what stayed unclear
- Decide the next step: stop, adjust, rerun, or escalate
- Keep the log short enough that humans will actually read it
Example
Expected: failing test passes and no tests are deleted. Actual: test passed, build passed, one unrelated file changed. Result: partial. Feedback: tighten boundary and rerun only after reverting the unrelated change.