Data Cleaning Loop

/goal Produce a clean dataset and a reproducible cleaning script, documenting every transformation.
 
Task type: Data Cleaning
Target tool: Generic Agent
 
Work toward this goal until all validation checks pass or the stop rule is reached.
 
Design hypothesis:
This data cleaning loop can produce a safer result if the scope stays narrow, validation is explicit, and a checker can reject shortcut work.
 
Smallest useful run:
Run one bounded pass on the newest relevant signal before expanding scope or adding a schedule.
 
Loop cycle:
1. Discovery — Read the latest signal for this template before acting: CI output, issue detail, review comment, dataset report, or content brief.
2. Handoff — Hand the work to one agent in an isolated branch, worktree, or clearly scoped session. Keep final approval with a human.
3. Verification — Use an independent review pass to confirm the result, inspect the diff or artifact, and reject shortcut work.
4. Persistence — Save a short run note with the signal reviewed, actions taken, validation result, and next recommended step.
5. Scheduling — Run manually until the loop is reliable; only then consider a scheduled or event-triggered run.
 
Context:
Never overwrite the raw data. Label any imputed values. Keep the script reproducible.
 
Validation:
Missing values are reported
Schema is documented
Cleaning script is reproducible
Final dataset passes validation checks
 
Validation evidence:
Record the original signal, checks run, final result, changed files or artifacts, and any checker rejection.
 
Independent checker:
Use an independent review pass to confirm the result, inspect the diff or artifact, and reject shortcut work.
 
Boundaries:
Do not silently delete rows
Do not overwrite raw data
Do not invent missing values without labeling imputation
 
Stop rule:
Stop when the cleaned dataset passes validation and the script reproduces it.
Maximum iterations: 4
 
Budget:
Example only: stop before exceeding the agreed per-run token budget.
 
Human approval:
Required before merge, deploy, delete, purchase, or external communication.
 
Fallback:
If labels are ambiguous, stop and ask for a human decision rather than guessing.
 
Loop Validation Log:
- Hypothesis: This data cleaning loop can produce a safer result if the scope stays narrow, validation is explicit, and a checker can reject shortcut work.
- Smallest useful run: Run one bounded pass on the newest relevant signal before expanding scope or adding a schedule.
- Expected evidence: Record the original signal, checks run, final result, changed files or artifacts, and any checker rejection.
- Actual evidence: [fill in after the run]
- Passed? [yes / no / partial]
- Feedback: After the run, note what the loop learned, what failed, and what should change before the next pass.
- Next step: [stop / adjust the loop / run the next pass]
 
Do not delete tests, bypass checks, or modify unrelated files just to satisfy the validation condition. If blocked, stop and summarize the blocker, attempted fixes, and recommended next action.