Eval scorecard
Scores per specialist and dataset, as committed in agents/python/evals/baselines/.
Read this first. These numbers are not a live measurement. Every case was answered from a replay fixture recorded on gpt-4.1-2025-04-14 (102 fixtures) between 2026-08-21 and 2026-08-26, and the scorer is deterministic: groundedness is a database check, correctness is a tool or route match, and completeness is a keyword check, not a judge model. A score tells you the recorded answers still pass the current code and prompts. It says nothing about how a different model, or the same model today, would do. Each dataset has five or six cases, so one flipped case moves a score by 0.1 or more, and the score is a property of the recording session as much as of the system (see agents/python/evals/fixtures/README.md).
Quality baselines
| Agent | Dataset | Cases | Groundedness | Correctness | Completeness | Overall |
|---|---|---|---|---|---|---|
| inventory-fulfillment | inventory_fulfillment.json | 5 | 0.80 | 1.00 | 0.50 | 0.82 |
| orchestrator | orchestrator_routing.json | 6 | 0.50 | 1.00 | 1.00 | 0.80 |
| order-management | order_management.json | 5 | 0.40 | 0.70 | 0.35 | 0.51 |
| pricing-promotions | pricing_promotions.json | 5 | 0.60 | 0.80 | 0.60 | 0.68 |
| product-discovery | product_discovery.json | 6 | 1.00 | 0.92 | 1.00 | 0.97 |
| review-sentiment | review_sentiment.json | 5 | 1.00 | 1.00 | 1.00 | 1.00 |
The baselines record no date or commit of their own. Git history on the baseline files gives the commit that last changed each one.
Cost and latency
None of the committed baselines carries token, cost or latency fields, so none are shown. Replay latency would only measure reading a JSON file.
Prompt and tool-schema size
Tokens per request before the user’s message, counted with o200k_base (evals/prompt_tokens.py; CI fails on more than 10 percent growth). This is the one number here that is measured from the code on every run, not replayed.
| Agent | Role | Instructions | Tools | Tool count | Total |
|---|---|---|---|---|---|
| inventory-fulfillment | admin | 3,081 | 894 | 10 | 3,975 |
| inventory-fulfillment | customer | 3,076 | 894 | 10 | 3,970 |
| inventory-fulfillment | seller | 3,087 | 894 | 10 | 3,981 |
| orchestrator | admin | 2,971 | 126 | 1 | 3,097 |
| orchestrator | customer | 2,967 | 126 | 1 | 3,093 |
| orchestrator | seller | 3,063 | 126 | 1 | 3,189 |
| order-management | admin | 3,843 | 1,666 | 18 | 5,509 |
| order-management | customer | 3,846 | 1,666 | 18 | 5,512 |
| order-management | seller | 3,847 | 1,666 | 18 | 5,513 |
| pricing-promotions | admin | 3,159 | 891 | 11 | 4,050 |
| pricing-promotions | customer | 3,153 | 891 | 11 | 4,044 |
| pricing-promotions | seller | 3,159 | 891 | 11 | 4,050 |
| product-discovery | admin | 3,092 | 1,342 | 13 | 4,434 |
| product-discovery | customer | 3,086 | 1,342 | 13 | 4,428 |
| product-discovery | seller | 3,107 | 1,342 | 13 | 4,449 |
| review-sentiment | admin | 3,129 | 1,213 | 13 | 4,342 |
| review-sentiment | customer | 3,125 | 1,213 | 13 | 4,338 |
| review-sentiment | seller | 3,131 | 1,213 | 13 | 4,344 |
Regenerating
uv run --project agents/python python -m scripts.build_eval_scorecard
Run it after a baseline changes (--update-baseline, see agents/python/evals/README.md). CI runs the same command with --check and fails when this page no longer matches the baselines.
Source: docs/eval-scorecard.md — this page is generated from the repository.