Skill Routing Evaluation Loop

/goal Evaluate whether a set of example tasks routes to the intended methodology skills with an acceptable accuracy and clear fallback behavior.
 
Task type: Custom
Target tool: Generic Agent
 
Work toward this goal until all validation checks pass or the stop rule is reached.
 
Design hypothesis:
This skill routing evaluation loop can produce a safer result if the scope stays narrow, validation is explicit, and a checker can reject shortcut work.
 
Smallest useful run:
Run one bounded pass on the newest relevant signal before expanding scope or adding a schedule.
 
Loop cycle:
1. Discovery — Read the latest signal for this template before acting: CI output, issue detail, review comment, dataset report, or content brief.
2. Handoff — Hand the work to one agent in an isolated branch, worktree, or clearly scoped session. Keep final approval with a human.
3. Verification — Use an independent review pass to confirm the result, inspect the diff or artifact, and reject shortcut work.
4. Persistence — Save a short run note with the signal reviewed, actions taken, validation result, and next recommended step.
5. Scheduling — Run manually until the loop is reliable; only then consider a scheduled or event-triggered run.
 
Context:
Use a small labeled test set before trusting automatic routing. Include ambiguous tasks and expected none/fallback cases.
 
Validation:
At least 10 labeled routing cases are tested
Expected and actual skill choices are compared
Ambiguous cases have a fallback
Accuracy and failure reasons are summarized
 
Validation evidence:
Record the original signal, checks run, final result, changed files or artifacts, and any checker rejection.
 
Independent checker:
Use an independent review pass to confirm the result, inspect the diff or artifact, and reject shortcut work.
 
Boundaries:
Do not route high-stakes work without human override
Do not hide ambiguous matches
Do not optimize only for accuracy while ignoring unsafe matches
Do not add a skill to routing until its checklist is clear
 
Stop rule:
Stop when the routing test passes the chosen threshold or after 3 prompt revisions.
Maximum iterations: 3
 
Budget:
Example only: stop before exceeding the agreed per-run token budget.
 
Human approval:
Required before merge, deploy, delete, purchase, or external communication.
 
Fallback:
If routing is unreliable, require manual skill selection and improve skill descriptions before retrying.
 
Loop Validation Log:
- Hypothesis: This skill routing evaluation loop can produce a safer result if the scope stays narrow, validation is explicit, and a checker can reject shortcut work.
- Smallest useful run: Run one bounded pass on the newest relevant signal before expanding scope or adding a schedule.
- Expected evidence: Record the original signal, checks run, final result, changed files or artifacts, and any checker rejection.
- Actual evidence: [fill in after the run]
- Passed? [yes / no / partial]
- Feedback: After the run, note what the loop learned, what failed, and what should change before the next pass.
- Next step: [stop / adjust the loop / run the next pass]
 
Do not delete tests, bypass checks, or modify unrelated files just to satisfy the validation condition. If blocked, stop and summarize the blocker, attempted fixes, and recommended next action.