Every Evaluator in the Azure AI Evaluation SDK, and When to Use Each

The package that grades your AI can’t produce an accurate list of its own graders

#

Open azure-ai-evaluation’s own README.md and you’ll find a table of built-in evaluators. It lists 15 of them. It has never been updated to include the entire agent category: no Intent Resolution, no Task Adherence, no Tool Call Accuracy, no Response Completeness, all of which have been generally available for the better part of a year. It also skips Document Retrieval, Code Vulnerability, Ungrounded Attributes, and Groundedness Pro.

Now go check the official Python API reference instead. DocumentRetrievalEvaluator, a real, public, fully stable class, isn’t in the alphabetical class index, and its dedicated reference page 404s. A reader doing everything right, checking the canonical docs, will conclude the evaluator doesn’t exist. It does. It’s imported and exported from the package right now. The docs generation pipeline just never picked it up.

Then it gets stranger. Five more evaluators (Task Completion, Task Navigation Efficiency, Tool Selection, Tool Input Accuracy, and one called ECIEvaluator) have real, working code in the SDK’s source tree. They have .prompty judge templates. They’re demonstrated in the SDK’s own sample file. And from azure.ai.evaluation import TaskCompletionEvaluator still fails, because none of them are exported from the package. You can only reach them through a private submodule path Microsoft never documents.

And five other names (Customer Satisfaction, Quality Grader, Rubric, Prohibited Actions, Sensitive Data Leakage) don’t have any Python code at all. They’re real, but they only exist as string IDs in a completely different, newer API.

None of this is a criticism of the SDK’s engineering. The underlying evaluators work, and the team ships fast (roughly one release every one to three weeks through 2025 and 2026). It’s a documentation and packaging problem, and it means you cannot learn this catalog by reading one page.

This article is the reference table I wish existed: every evaluator, organized by whether you can actually import it, with exact inputs and exact citations. Everything below is verified directly against the package source (azure-ai-evaluation 1.18.1) and cross-checked against the current docs, not paraphrased from a single page that might itself be wrong.

What’s covered

#

• The four-tier map for every evaluator name: what you can import, what’s version-fragile, what’s hidden in the source, and what lives only in the cloud catalog.
• A full reference table by category (quality, textual similarity, RAG, agent, safety, composite, graders) with exact inputs, scales, and tiers.
• The two execution generations (classic in-process SDK versus the Foundry catalog) and how the same evaluate() data contract feeds both.
• How to write custom evaluators in each generation, and the sandbox rules for the server-side ones.
• The silent-failure and non-determinism traps that make an AI-assisted evaluator lie to you in production.

Two generations, one package name

#

Before the table, one piece of context that explains half the confusion above: azure-ai-evaluation currently spans two different execution models.

Generation 1: the classic SDK. You pip install azure-ai-evaluation, import evaluator classes directly, and call evaluate() in-process. This is what the rest of this article covers.

Generation 2: the Foundry evaluator catalog. You install azure-ai-projects>=2.0.0, get an OpenAI-compatible client from your Foundry project, and call client.evals.create(testing_criteria=[...]), referencing evaluators by string ID ("builtin.groundedness", "builtin.task_adherence") instead of importing a class. This is the newer, cloud-first path, and it’s the only path for five of the names in this article.

# Generation 1 — classic SDK, import a class
from azure.ai.evaluation import GroundednessEvaluator
evaluator = GroundednessEvaluator(model_config=model_config)

# Generation 2 — Foundry catalog, reference by string ID
testing_criteria = [{
    "type": "azure_ai_evaluator",
    "name": "groundedness",
    "evaluator_name": "builtin.groundedness",
    "initialization_parameters": {"deployment_name": model_deployment},
    "data_mapping": {"context": "{{item.context}}", "response": "{{item.response}}"},
}]

Both call the same underlying grading logic for evaluators that exist in both generations. The .NET side has an analogous split: Microsoft.Extensions.AI.Evaluation is the local IEvaluator library, and the Agent Framework’s FoundryEvals is a thin bridge to the same builtin.* catalog. That comparison is a big enough topic to deserve its own article; this piece stays Python-first.

The four tiers

#

Given that split, every evaluator name in this article falls into one of four tiers. This is the organizing idea for everything that follows. Learn this legend once, then use it as a column in every table below.

• T1, Public. from azure.ai.evaluation import X works, there’s a docs page (or should be, see Document Retrieval), and it’s covered by normal semantic versioning. 27 evaluators live here, plus the 2 composite evaluators and 6 grader wrapper classes.
• T2, Exported but unstable. The import genuinely works (no trick, no private path), but there’s no official docs page, and the underscore prefix (_ToolOutputUtilizationEvaluator, _ToolCallSuccessEvaluator) is Microsoft’s own convention for “not covered by semver, can change or vanish in a patch release.” 2 evaluators, both still receiving behavioral changes as of the 1.18.1 release.
• T3, Hidden. Real code, real .prompty templates, exercised in the SDK’s own sample file, but never imported into the package’s __init__.py. You have to reach in with a private submodule path. 5 evaluators: Task Completion, Task Navigation Efficiency, Tool Selection, Tool Input Accuracy, and ECI.
• T4, Catalog-only. No Python class exists anywhere in the package source, confirmed by listing all 30 submodules under azure/ai/evaluation/_evaluators/ directly. The only way to run these is the Foundry catalog. 5 names: Customer Satisfaction, Quality Grader, Rubric, Prohibited Actions, Sensitive Data Leakage.

The reference table

#

General quality

#

Both default to a pass threshold of 3, English-only, and current docs recommend gpt-5-mini as the judge model for cost/performance balance (this superseded gpt-4o-mini in recent guidance; check current docs, this kind of recommendation moves).

Textual similarity / NLP: all six are GA

#

Similarity is the odd one out: it’s the only evaluator in this group that calls a model. The rest are deterministic string comparisons: no network call, no cost, no latency beyond your own compute. ROUGE is also the only one that returns three independently-thresholded scores instead of one; rouge_type is a required positional argument (rouge1 through rouge5, or rougeL), and there’s no sensible default, so you have to pick.

RAG evaluators

#

The mistake everyone makes here. Relevance sounds like it should need the retrieved context. How else would you know if the response was relevant to what was retrieved? It doesn’t take context at all; a 1.0.0 breaking change removed it deliberately. RetrievalEvaluator is the one that scores against context. Pass the wrong shape of data to the wrong evaluator and you get a scoring error, not a helpful message about which field you’re missing.

Groundedness measures precision: does the response stay inside the provided context? Response Completeness measures recall: does it capture everything the ground truth says it should? They’re not redundant; they catch opposite failure modes. If you want the full runtime-detection-vs-offline-evaluation distinction (a separate, frequently confused pair of Azure products that happen to share the word “Groundedness”), that’s covered end to end in Groundedness Detection vs Groundedness Evaluation. This table only covers the offline GroundednessEvaluator.

Context is a plain string for all RAG evaluators: there’s no structured list-of-chunks input. For multi-chunk retrieval, concatenate with a separator before passing it in.

Agent evaluators

#

Agent evaluation splits into three groups: system-level (did the agent accomplish the goal?), process-level (did each tool call go right?), and quality (a rubric-style catch-all). This is also where the tier system gets its heaviest workout; most of the interesting Tier 3 evaluators live here.

System evaluation:

Process evaluation (tool-call level):

Quality evaluation:

Tool Call Accuracy is the one evaluator in this whole catalog worth showing a full worked example for, because its input shape trips people up:

tool_call_accuracy_evaluator = ToolCallAccuracyEvaluator(model_config=model_config)
tool_call_accuracy_evaluator(
    query="How is the weather in New York?",
    response="The weather in New York is sunny.",
    tool_calls={
        "type": "tool_call",
        "tool_call": {
            "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
            "type": "function",
            "function": {
                "name": "fetch_weather",
                "arguments": {"location": "New York"},
            },
        },
    },
    tool_definitions={
        "id": "fetch_weather",
        "name": "fetch_weather",
        "description": "Fetches the weather information for the specified location.",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "The location to fetch weather for."}
            },
        },
    },
)

It scores 1–5 (a breaking change from an older 0–1 range), and unlike the other agent evaluators, tool_definitions is required, not optional. The rubric: 1 = irrelevant calls, 3 = relevant but with unnecessary/excessive calls, 5 = relevant and every parameter correctly passed.

For pulling this data out of an actual agent run, AIAgentConverter auto-converts Foundry Agent Service (and Semantic Kernel) thread/run data into the shape these evaluators expect: converter.convert(thread_id, run_id) for one run, or prepare_evaluation_data() to batch-export a JSONL file. It currently supports exactly five evaluators: Intent Resolution, Tool Call Accuracy, Task Adherence, Relevance, and Groundedness. The rest of the agent catalog needs hand-built input.

On the five Tier 3 evaluators: they work (real .prompty templates, exercised in Microsoft’s own samples), but treat them as unstable. The 1.18.1 CHANGELOG note reads “a follow-up will land a context-extractor helper” for the Tier 2 pair, which is direct evidence these input contracts are still actively being redesigned. If you need Task Completion or Tool Selection today, the supported path is the Foundry catalog (builtin.task_completion, builtin.tool_selection), not the private import. Treat the private path as something to know exists, not something to build production code on top of.

Safety and risk evaluators

#

Two things worth flagging explicitly. First, the scale: content-harm evaluators use a 0–7 severity scale, the opposite direction from quality’s 1–5 (lower is better here; a threshold of 3 means “pass if score ≤ 3”). If you’re building one dashboard across quality and safety metrics, don’t assume they share an axis.

Second, ECI: it looks abandoned (undocumented, not exported, no page anywhere), but it isn’t. The 1.10.0 CHANGELOG lists it alongside every fully-public RAI evaluator receiving the same evaluate_query parameter in the same release. It’s built on the identical service-backed pattern as ContentSafetyEvaluator.

It’s just deliberately kept out of the public surface, most plausibly because “Election Critical Information” is a policy-sensitive category Microsoft doesn’t want to present as a supported, GA feature. Know it exists; don’t build on it without accepting that risk.

All RAI-backed safety evaluators run against Microsoft’s hosted evaluation service rather than your own model deployment, which means they need azure_ai_project credentials instead of model_config. They’re also region-restricted more tightly than general Azure OpenAI availability (five regions for the core set, two for Groundedness Pro, one for Protected Material). Don’t assume your existing Azure OpenAI region works for all of these.

Composite evaluators

#

QAEvaluator needs all four core fields present (query, response, context, ground_truth) to populate every sub-metric. Pass a partial set and some of the six results come back empty rather than erroring.

Azure OpenAI graders

#

A separate mechanism, also split across generations. The classic SDK ships six wrapper classes (all marked experimental): AzureOpenAIGrader, AzureOpenAILabelGrader, AzureOpenAIStringCheckGrader, AzureOpenAITextSimilarityGrader, AzureOpenAIScoreModelGrader, AzureOpenAIPythonGrader. They plug into evaluate()’s evaluators= dict exactly like a built-in: pass an instance, get results merged in.

The new catalog exposes four grader types instead (label_model, score_model, string_check, text_similarity) as dict-shaped testing_criteria entries using the raw OpenAI Evals API shape, not Python classes.

The evaluate() data contract

#

evaluate(
    *,
    data: str | PathLike,                              # JSONL or CSV
    evaluators: Dict[str, Callable | AzureOpenAIGrader], # alias -> evaluator instance
    evaluation_name: str | None = None,
    target: Callable | None = None,
    evaluator_config: Dict[str, EvaluatorConfig] | None = None,
    azure_ai_project: str | AzureAIProject | None = None, # None = local only
    output_path: str | PathLike | None = None,
    fail_on_evaluator_errors: bool = False,
    tags: Dict[str, str] | None = None,
    **kwargs,
) -> EvaluationResult

Any evaluator is just a callable. This is exactly how custom evaluators plug in with zero extra ceremony, and it’s also why the SDK doesn’t need a plugin/registration system for its built-ins.

Column mapping uses two placeholder forms: ${data.<column>} pulls from your input dataset, ${outputs.<column>} pulls from a target callable’s return value if you’re evaluating live rather than a static dataset.

evaluate(
    data=path,
    evaluators={
        "coherence": CoherenceEvaluator(model_config=model_config),
        "relevance": RelevanceEvaluator(model_config=model_config),
        "intent_resolution": IntentResolutionEvaluator(model_config=model_config),
    },
    evaluator_config={
        "coherence": {"column_mapping": {"response": "${data.response}", "query": "${data.query}"}},
        "relevance": {
            "column_mapping": {
                "response": "${data.response}",
                "context": "${data.context}",
                "query": "${data.query}",
            }
        },
    },
)

The core recognized fields are query, response, context, ground_truth, and, for multi-turn or agent scenarios, conversation, a dict with a messages list. For conversation-mode input, the evaluator infers query from the first user turn and context from the following assistant turn’s context key:

conversation = {
    "messages": [
        {"content": "Which tent is the most waterproof?", "role": "user"},
        {
            "content": "The Alpine Explorer Tent is the most waterproof",
            "role": "assistant",
            "context": "From our product list the alpine explorer tent is the most waterproof.",
        },
    ]
}

One data-quality trap worth knowing: a missing or null context on a later turn is silently treated as an empty string rather than raising an error. If your groundedness scores look suspiciously good on later turns of a conversation, check whether context is actually populated there.

EvaluationResult returns three top-level keys: metrics (aggregate, e.g. groundedness.groundedness), rows (per-row results), and traces (target-invocation data, empty if you didn’t pass a target). Since a 1.13.0 standardization, every evaluator’s row-level output follows one key pattern: {name} (score), {name}_result (pass/fail), {name}_reason, {name}_threshold, plus token-usage fields. Since 1.16.7, every result item also carries a status field (completed/error/skipped). If you’re working from an article or example written before mid-2026, it won’t mention this field.

Local vs. cloud: one parameter

#

The entire difference between a local-only run and a cloud-logged one is whether azure_ai_project= is passed to evaluate(). Omit it, and the exact same evaluator code runs entirely in your process. Pass it, and evaluate() additionally populates result["studio_url"] (a link straight into the Foundry Evaluation runs UI) and logs the run for later comparison.

azure_ai_project accepts two shapes: the legacy hub-based dict (subscription_id, resource_group_name, project_name) or, the currently recommended form, a single Foundry project endpoint URL:

# Legacy, hub-based
azure_ai_project = {
    "subscription_id": "<subscription_id>",
    "resource_group_name": "<resource_group_name>",
    "project_name": "<project_name>",
}

# Recommended, Foundry-based
azure_ai_project = "https://{resource_name}.services.ai.azure.com/api/projects/{project_name}"

But “local” is a slippery word here. AI-assisted quality evaluators (Coherence, Groundedness, Relevance, and the rest) need a real Azure OpenAI or OpenAI model_config, which is a network call regardless of whether azure_ai_project is set.

And the RAI-service-backed safety evaluators need azure_ai_project credentials as a required constructor argument, not an optional flag. Their actual scoring happens on Microsoft’s hosted service no matter what, so only the orchestration loop runs locally. “Local” means “you don’t need a Foundry project to get a result,” not “nothing leaves your machine.”

For scale (CI/CD gates, predeployment testing, large datasets), Microsoft’s own guidance is to skip local orchestration and use the cloud path (azure-ai-projects) directly. It eliminates managing compute for the evaluation loop itself, supports four scenario types (dataset, CSV, model-target, agent-target), and ships a dedicated Azure DevOps extension that reports confidence intervals to distinguish a real regression from noise. That last part is genuinely useful once you’re gating deployments on eval scores rather than eyeballing a notebook.

Custom evaluators: two escape hatches, two generations

#

Classic SDK: any callable, no ceremony. A function or a class with __call__ that returns a dict is a valid evaluator. No base class, no decorator, no registration step:

class BlocklistEvaluator:
    def __init__(self, blocklist):
        self._blocklist = blocklist

    def __call__(self, *, response: str, **kwargs):
        score = any(word in self._blocklist for word in response.split())
        return {"score": score}

Pass it straight into evaluate(evaluators={"blocklist": BlocklistEvaluator(my_words)}) alongside the built-ins.

New catalog: three registered types, server-side. This is a materially different mechanism, not a variation on the same idea:

• Code-based: a function literally named grade(sample: dict, item: dict) -> float, returning 0.0–1.0. Runs sandboxed: under 256 KB, a 2-minute execution limit, no network access, with a preinstalled set of pinned packages (numpy, pandas, scikit-learn, nltk, rouge-score, and others). If it raises or times out, the service records 0.0 and moves on.
• Prompt-based: a {{variable}}-templated judge prompt returning JSON {"result": ..., "reason": ...}, scored ordinal, continuous, or binary depending on how you configure it.
• Endpoint-based: delegates scoring to your own HTTP POST endpoint (Azure Functions, App Service, Container Apps), with a documented request/response schema and a 30-second timeout. No classic-SDK equivalent exists for this one.

All three can also be built entirely in the Foundry portal with no code at all.

For a rubric/LLM-as-judge custom evaluator specifically, AzureOpenAIScoreModelGrader is the documented pattern without hand-writing raw prompty:

conversation_quality_grader = AzureOpenAIScoreModelGrader(
    model_config=model_config,
    name="Conversation Quality Assessment",
    model="gpt-4o-mini",
    input=[
        {"role": "system", "content": "You are an expert conversation quality evaluator. Assess helpfulness, completeness, accuracy, and appropriateness. Return a score between 0.0 and 1.0."},
        {"role": "user", "content": "Evaluate this conversation:\nContext: {{ item.context }}\nMessages: {{ item.conversation }}\n\nProvide a quality score from 0.0 to 1.0."},
    ],
    range=[0.0, 1.0],
    sampling_params={"temperature": 0.0},
)

Gotchas that bite in production

#

Three independent practitioners hit variants of the same underlying problem: AI-assisted evaluators fail silently.

• Rate limits become blank cells, not errors. A 480-row evaluation run hit its judge deployment’s TPM quota and simply returned nan across the affected rows, with no clear error, just missing data. Raising the deployment from 50K to 200K TPM fixed it. If your aggregate scores look implausibly good, check for silently-skipped rows before trusting the number.
• Reasoning-model judges fail without telling you why. Point model_config at a gpt-5/o1/o3 deployment and you’ll hit 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead. On the .NET side of this same problem, the evaluator was reported to swallow the error and record it as a metric row rather than failing the run, so a broken judge can pass CI for hours before anyone notices.
• Custom evaluators need numeric output, or the column vanishes. If your custom evaluator’s return dict mixes in a non-numeric value, the Foundry UI silently drops that whole column instead of erroring.

Beyond silent failures, keep two more things in mind.

The other is cost. It’s real but manageable if you plan for it. One reported compliance pipeline (roughly 60 PR evaluations a month plus weekly full audits, gpt-4o as judge) landed around £310–340/month, split roughly evenly between tokens, compute, and infrastructure. The judge deployment consumes its own Azure OpenAI quota on top of whatever your application already uses, which is worth budgeting for before wiring continuous evaluation into CI.

Takeaways

#

• The catalog you can trust is the package source, not the README or the API reference index. Both undercount what’s actually exported.
• The four tiers are the map: import Tier 1 freely, treat Tier 2 as version-fragile, reach Tier 3 only through the Foundry catalog, and accept Tier 4 lives cloud-only.
• Groundedness (precision) and Response Completeness (recall) catch opposite failures. Quality scores run 1–5, safety severity runs 0–7 in the other direction. Don’t put them on one axis untouched.
• AI-assisted evaluators fail silently: rate limits become blank cells, reasoning-model judges error on max_tokens, non-numeric custom output drops the column. Check for skipped rows before trusting an aggregate.
• LLM-judge scores aren’t deterministic even at temperature 0. Gate on an ensemble or a pass band, not a single run.

Where to start

#

You don’t need all 39 names. Two independent practitioners converged on the same starting checklist:

• Building RAG? Start with Groundedness. It catches fabrication, which is the failure mode that actually embarrasses you in production. Add Relevance once Groundedness is stable.
• Building an agent? Start with Task Adherence as your baseline, add Intent Resolution for anything conversational, and add Tool Call Accuracy the moment your agent calls more than one tool.
• Shipping anything user-facing? Run ContentSafetyEvaluator regardless of the above. It’s cheap, it’s a single composite call, and it’s the one category where “we didn’t think we needed it” is the wrong default.

Wire whichever set you pick into evaluate() with azure_ai_project set from day one. The marginal cost of cloud logging is one keyword argument, and you’ll want the Evaluation runs UI’s row-level drill-down the first time a score looks wrong and you need to know why, not just that it happened.