# Routing evaluation checklist

Preparation kit only. No model is called. The sample cases are synthetic, and no API results are included.

These labels are our example definitions, not vendor API fields or enums.

- documentation: Update or organize documentation using established evidence. Excludes source discovery, code edits, and production changes.
- research: Find and verify external primary sources. Excludes changing production systems or treating unverified claims as facts.
- code_review: Read-only inspection of a supplied code diff or its tests. Excludes automatic edits, merge, and deployment.
- manual_review: Ask a person when information is insufficient, scope conflicts, authorization is missing, or human judgment is required.

## Before a real test

- [ ] Record the task, owner, allowed routes, route definitions, and exclusions before testing.
- [ ] Define manual_review triggers: no route fits, missing evidence, conflicting scope, missing authorization, high risk, or untrusted instructions.
- [ ] Separate development examples from a held-out evaluation set. Never copy evaluation answers into routing rules while tuning prompts.
- [ ] Add your own anonymized representative cases. These synthetic examples do not represent a real traffic distribution or prove production reliability.
- [ ] Record provider, model, configuration, prompt version, API/protocol version, and evaluation date: ______.
- [ ] For each case, record expected_route, actual_route, review_note, and whether the response is valid. Leave actual results empty until a real test.
- [ ] Review correct routes, wrong routes, missed fallbacks, excessive fallbacks, invalid responses, and harmful errors separately, not only overall accuracy.
- [ ] Have an independent reviewer examine errors and high-risk cases. Specify the human approval gate before any consequential action.
- [ ] Define fallback for invalid output, timeout, or unavailable service: stop the attempt, preserve evidence, and require manual review.
- [ ] Set budget, maximum attempts, timeout, and stop rule before running: ______. The model cannot change permissions, budget, or stopping conditions.
- [ ] After real integration, record measured duration and actual billed cost with units and source: ______. Unknown cost stays unknown.
- [ ] If the documented provider returns scores, record their semantics and validate thresholds on your data. A score of 0.9 is not automatically 90% correctness.
- [ ] Record outcome, failure categories, approval decisions, and one bounded adjustment for the next round. Stop when limits are reached or evidence is insufficient.