Eval scorecard

Scores per specialist and dataset, as committed in agents/python/evals/baselines/.

Read this first. These numbers are not a live measurement. Every case was answered from a replay fixture recorded on gpt-4.1-2025-04-14 (102 fixtures) between 2026-08-21 and 2026-08-26, and the scorer is deterministic: groundedness is a database check, correctness is a tool or route match, and completeness is a keyword check, not a judge model. A score tells you the recorded answers still pass the current code and prompts. It says nothing about how a different model, or the same model today, would do. Each dataset has five or six cases, so one flipped case moves a score by 0.1 or more, and the score is a property of the recording session as much as of the system (see agents/python/evals/fixtures/README.md).

Quality baselines

Agent Dataset Cases Groundedness Correctness Completeness Overall
inventory-fulfillment inventory_fulfillment.json 5 0.80 1.00 0.50 0.82
orchestrator orchestrator_routing.json 6 0.50 1.00 1.00 0.80
order-management order_management.json 5 0.40 0.70 0.35 0.51
pricing-promotions pricing_promotions.json 5 0.60 0.80 0.60 0.68
product-discovery product_discovery.json 6 1.00 0.92 1.00 0.97
review-sentiment review_sentiment.json 5 1.00 1.00 1.00 1.00

The baselines record no date or commit of their own. Git history on the baseline files gives the commit that last changed each one.

Cost and latency

None of the committed baselines carries token, cost or latency fields, so none are shown. Replay latency would only measure reading a JSON file.

Prompt and tool-schema size

Tokens per request before the user’s message, counted with o200k_base (evals/prompt_tokens.py; CI fails on more than 10 percent growth). This is the one number here that is measured from the code on every run, not replayed.

Agent Role Instructions Tools Tool count Total
inventory-fulfillment admin 3,081 894 10 3,975
inventory-fulfillment customer 3,076 894 10 3,970
inventory-fulfillment seller 3,087 894 10 3,981
orchestrator admin 2,971 126 1 3,097
orchestrator customer 2,967 126 1 3,093
orchestrator seller 3,063 126 1 3,189
order-management admin 3,843 1,666 18 5,509
order-management customer 3,846 1,666 18 5,512
order-management seller 3,847 1,666 18 5,513
pricing-promotions admin 3,159 891 11 4,050
pricing-promotions customer 3,153 891 11 4,044
pricing-promotions seller 3,159 891 11 4,050
product-discovery admin 3,092 1,342 13 4,434
product-discovery customer 3,086 1,342 13 4,428
product-discovery seller 3,107 1,342 13 4,449
review-sentiment admin 3,129 1,213 13 4,342
review-sentiment customer 3,125 1,213 13 4,338
review-sentiment seller 3,131 1,213 13 4,344

Regenerating

uv run --project agents/python python -m scripts.build_eval_scorecard

Run it after a baseline changes (--update-baseline, see agents/python/evals/README.md). CI runs the same command with --check and fails when this page no longer matches the baselines.


Source: docs/eval-scorecard.md — this page is generated from the repository.