Groundedness Detection vs Groundedness Evaluation: Two Azure Products, One Name

Azure has two features named “Groundedness.” One is a runtime content-safety check that blocks or rewrites an ungrounded answer before the user sees it. The other is an offline evaluator that scores how grounded your answers are against a test set in CI.

They share a name and a concept (is this answer actually supported by the source material?) and almost nothing else. Different products, different APIs, different lifecycle stages, different pricing. Pick the wrong one for your RAG pipeline and you’ll spend a sprint wiring a batch-evaluation SDK into a request path it was never meant for, or trying to gate a build with an API that only runs live.

This is the third piece in the defense-in-depth series. After the input-side shields and the guardrail stack, groundedness is the output-side check that a RAG answer didn’t drift from its sources. Here’s how the two products divide the work.

What’s covered

#

• The runtime detection API in Content Safety: what it returns, and how it blocks or rewrites a live answer.
• The two offline evaluators (GroundednessEvaluator vs GroundednessProEvaluator), and which one drags in a Foundry project.
• The decision boundary for RAG, and the honest recall numbers before you make groundedness a hard gate.
• The open-source substitutes for when you’re off Azure or need higher recall than the managed service gives.

Groundedness detection: the runtime check

#

Groundedness detection lives in Azure AI Content Safety. You give it the answer, the query, and the grounding sources, and it tells you, on the request path and in real time, whether the answer is supported. It’s the RAG analogue of a Prompt Shields check: a live gate.

POST {endpoint}/contentsafety/text:detectGroundedness?api-version=2024-09-15-preview
{
  "domain": "Generic",           // or "Medical"
  "task": "QnA",                 // or "Summarization"
  "qna": { "query": "What is the current interest rate?" },
  "text": "The interest rate is 5%.",
  "groundingSources": ["As of July 2024, the interest rate is 4.5%."]
}

It returns ungroundedDetected, an ungroundedPercentage, and the specific ungrounded spans. Two modes trade speed for insight: non-reasoning (fast, binary, for the live path) and reasoning (explanations for each ungrounded span, for debugging). One feature makes it more than a detector: set "mitigating": true and it returns a correctionText, the answer rewritten against the sources, so the “5%” above comes back corrected to “4.5%.”

Three things to know before you wire it in:

• It runs without Foundry. This is a standalone Cognitive Services call: provision a plain Content Safety resource and hit the endpoint. The catch: groundedness is S0-only (the free F0 tier excludes it), it’s Preview, English-only, and available in a narrow set of regions (East US, East US 2, France Central, Sweden Central, UK South, West US).
• Correction needs a GPT-4o deployment. The mitigating and reasoning features require you to point at an Azure OpenAI GPT-4o deployment via an llmResource block. Keep it keyless: grant the Content Safety resource’s own managed identity the Cognitive Services OpenAI User role on the Azure OpenAI resource. That resource-to-resource identity relationship is easy to miss.
• As a Foundry content filter, it’s streaming-only. If you attach groundedness as an Azure OpenAI content filter (rather than calling the API yourself), it is documented as available only in streaming scenarios, default off. As the companion piece covered, it does not fire on agents at all. And if you enforce content safety at the APIM gateway, note the llm-content-safety policy has no groundedness attribute: you’d hand-write a <send-request> to the detection endpoint.

Groundedness evaluation: the offline score

#

Groundedness evaluation lives in the Azure AI Evaluation SDK. It does not run on the request path. You point it at a dataset of query/response/context rows and it produces a score you gate a build on, the RAG analogue of a unit test. There are two evaluators, and the difference matters:

from azure.ai.evaluation import GroundednessEvaluator

# Local: bring your own GPT judge, no Foundry project required
groundedness = GroundednessEvaluator(model_config={
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "api_key": os.environ["AZURE_OPENAI_KEY"],
    "azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
})
result = groundedness(response="Paris is the capital of France.", context="...")

• GroundednessEvaluator returns a 1-to-5 score and runs fully local. Its constructor takes only a model_config, so you bring your own GPT judge (Azure OpenAI or plain OpenAI), with no azure_ai_project scope. The .NET equivalent (Microsoft.Extensions.AI.Evaluation.Quality.GroundednessEvaluator, GA) takes any IChatClient. This is what you want for CI: no cloud dependency beyond a judge model you already have.
• GroundednessProEvaluator returns a boolean pass/fail, powered by Azure AI Content Safety’s own fine-tuned model rather than your GPT deployment, but it requires an azure_ai_project because it calls the Foundry-hosted evaluation service. Use it when you want a stricter, service-backed verdict and don’t want to run your own judge. It’s experimental (.NET ships it in the preview Safety package).

So the decision boundary for RAG:

The honest part: it’s a signal, not a guarantee

#

Before you make groundedness a hard gate, know how well it actually works. Independent benchmarking of Azure’s Groundedness detection against the public RAGTruth dataset put it at roughly 35% recall and 47% F1 at the response level. That means it misses well over half of hallucinations in that test, while flagging some grounded content as ungrounded. On the same benchmark, a bare GPT-4 prompt scored higher (63% F1), and a purpose-built fine-tuned detector higher still (79%).

That doesn’t make the service useless. It’s a cheap, managed, no-ops signal. But treat it the way the whole series treats every classifier: a layer that raises confidence, not a boundary you bet correctness on. For high-stakes RAG (claims, clinical, legal), pair it with a second judge and route disagreements to a human.

The open-source equivalents

#

Groundedness is a universal RAG concern, and the same runtime-vs-offline split exists off Azure. The offline tools are the equivalents of GroundednessEvaluator; the runtime detectors are the open equivalent of the detection API.

DeepEval’s split (retrieved context vs supplied ground truth) is a finer distinction than Azure’s single framing. Both runtime detectors report numbers that, benchmark-for-benchmark, look stronger than the managed service. But these are different studies on the same dataset, so treat the gap as suggestive, not a controlled head-to-head.

On an Azure-primary stack, the managed detection API and GroundednessEvaluator are the right defaults: no serving to run, keyless, and they slot straight into the guardrail stack. The open-source detectors are the option when you need higher recall than the managed service delivers, or you’re running RAG off Azure.

The bottom line

#

• Same name, opposite lifecycle stages. Detection is a live gate that can rewrite the answer; the evaluator is a CI score you gate a build on. They are not substitutes.
• Only the Pro evaluator pulls in Foundry. The 1-to-5 GroundednessEvaluator runs fully local on a judge model you already have; GroundednessProEvaluator needs an azure_ai_project.
• Treat any groundedness score as a signal, not a verdict. At ~35% recall the managed detector misses more than half of hallucinations, so layer a second judge on high-stakes routes.

Where to start

#

1. Name which one you need. Blocking or correcting a live answer → detection (Content Safety API). Gating a build → the evaluator (Evaluation SDK). They are not substitutes.
2. Run the evaluator in CI first. It’s local, needs only a judge model, and catches grounding regressions before release. Use the 1-to-5 GroundednessEvaluator unless you specifically want the strict service verdict of the Pro variant (which pulls in a Foundry project).
3. Add detection in production where a wrong answer is costly, with correctionText if you want it fixed inline rather than just flagged. Budget for S0, the narrow regions, and the GPT-4o dependency for correction.
4. Don’t make it a hard gate on its own. ~35% recall means it misses things; pair it with a second judge for high-stakes routes and keep a human in the loop.
5. If you’re off Azure or need higher recall, reach for Ragas/DeepEval in CI and a small self-hosted detector at runtime. Same split, portable.

That closes the loop on the series: shields on the way in, the stack around the agent, and groundedness on the way out. None of them a boundary alone, all of them worth layering.