Observability for AI Agents on Azure: OpenTelemetry GenAI to the Agent Dashboard

An agent I read about burned $41.20 overnight calling one tool, search_orders(customer_id="C-7841"), 187 times in a row against a result set that was empty every single time. It ran unnoticed for two and a half hours. Nothing was broken in the way monitoring understands broken: every one of those 187 calls returned a clean HTTP 200. The agent was stuck in a loop, and the only place that fact existed was in the tool-call spans nobody was watching.

That is the whole problem with agent observability. Agent failures are cognitive: a bad tool choice, a stuck reasoning loop, a weak retrieval that grounds the model in the wrong context. Cognitive failures don’t raise exceptions. Your APM dashboard shows green while the agent quietly does the wrong thing, expensively. To catch it you have to trace the one thing traditional observability never modelled: the decision chain inside a single agent run.

This is a build guide for doing that on Azure, with OpenTelemetry, Azure Monitor, and the API Management AI gateway. It includes the exact span tree, the KQL people actually paste, and the traps that waste an afternoon.

What you’ll build

#

• The OpenTelemetry GenAI span tree (invoke_agent → chat → execute_tool) wired into Azure Monitor three different ways.
• The KQL to reconstruct one agent run in order, token usage over time, and per-agent success rate.
• A second observability layer at the APIM AI gateway: token metrics and full prompt/completion logs.
• Trace continuity across the MCP boundary via W3C traceparent propagation.
• The production gotchas that cost an afternoon: PII capture flags, orphaned streaming spans, dropped error traces.

The problem is heterogeneity, not tooling

#

Walk into any enterprise running agents in 2026 and you find a product team on a no-code prompt agent, a back-end team on LangGraph, another on Google’s ADK, and someone asking for the Copilot SDK as a catch-all. Different framework, different hosting stack, different metrics, different dashboard. When an agent returns a bad answer, nobody can answer the three questions that matter: which agent did it, why, and how do we stop it happening again.

Microsoft’s own Build 2026 session on this framed the fix well, and it is not “move everyone onto one framework.” It is a two-part open standard:

Instrument once with OpenTelemetry, and every agent (any framework, any cloud) collapses into one unified trace in Azure Monitor.

There are two ingredients. The OpenTelemetry GenAI semantic conventions give you a shared vocabulary of spans (invoke_agent, chat, execute_tool) and attributes (gen_ai.*) that every framework can emit. Azure Monitor / Application Insights is the single backend that stores and correlates them. Foundry, Grafana, and your own workbooks are just views on top of the same data.

It is the same argument I made for the AI gateway and for keyless identity: adopt the portable, open layer, and keep the vendor UI on top of it rather than betting the stack on it.

One caveat to set up front, because it colours everything below.

The GenAI conventions are still Development status, not GA. Spans, agent spans, events, and metrics all carry an experimental stability badge; only error.type is Stable. The spec even moved repositories in 2026 (open-telemetry/semantic-conventions → semantic-conventions-genai), so older links redirect. Treat gen_ai.* as a fast-moving contract, not a frozen one, and expect Microsoft’s own emitters to be mid-migration (some still emit gen_ai.system where the new name is gen_ai.provider.name).

The span vocabulary

#

Everything downstream keys on three span names. Learn these and the rest of the article reads itself.

• invoke_agent {agent_name}: the top-level span for one agent turn. It is the parent of everything else in that turn, and it carries gen_ai.agent.id, gen_ai.agent.name, and gen_ai.conversation.id (the thread/session id that groups a multi-turn conversation).
• chat {model_name}: emitted when the agent calls the underlying model. It carries gen_ai.request.model, token usage, and gen_ai.response.finish_reasons. When the model decides to call a tool, that finish reason is tool_calls, and this span is where you see the decision.
• execute_tool {tool_name}: one span per tool invocation, carrying gen_ai.tool.name and gen_ai.tool.call.id.

The prompts, completions, and tool arguments/results are opt-in only. The spec marks the content attributes (gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments) as Opt-In precisely because “they are likely to contain sensitive information including user/PII data.” More on that trade-off later. For now, know that by default your spans have the shape of what happened but not the content.

Metrics come alongside the spans: gen_ai.client.token.usage (a histogram of tokens), gen_ai.client.operation.duration, and agent_framework.function.invocation.duration for tool calls.

Wire it up, three ways

#

Foundry-native, zero code. Agents built with Microsoft Agent Framework or Semantic Kernel and hosted on Foundry emit traces automatically once tracing is enabled on the project. Foundry’s observability is, in Microsoft’s words, “powered by Azure Monitor and Azure Application Insights.” The traces land in the App Insights resource connected to your project, so you reuse the same backend you already run.

Agent Framework: one builder call plus an exporter. In .NET you instrument the chat client and the agent, then export to Azure Monitor:

var agent = new AIProjectClient(new Uri(endpoint), new DefaultAzureCredential())
    .AsAIAgent(
        model: deploymentName,
        instructions: "You are a helpful assistant.",
        name: "OpenTelemetryDemoAgent",
        tools: [AIFunctionFactory.Create(GetWeatherAsync)],
        clientFactory: client => client
            .AsBuilder()
            .UseFunctionInvocation()
            .UseOpenTelemetry(sourceName: SourceName, configure: c => c.EnableSensitiveData = true)
            .Build())
    .AsBuilder()
    .UseOpenTelemetry(sourceName: SourceName, configure: c => c.EnableSensitiveData = true)
    .Build();

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName))
    .AddSource(SourceName)
    .AddAzureMonitorTraceExporter(o => o.ConnectionString = appInsightsConnectionString)
    .Build();

Enable EnableSensitiveData on one of the chat client or the agent, not both. Enabling it on both duplicates the prompt/response content across two spans. In Python the equivalent is one call, and it discovers the connection string from the Foundry project:

client = FoundryChatClient(project_endpoint=..., model=..., credential=AzureCliCredential())
await client.configure_azure_monitor(enable_sensitive_data=True, enable_live_metrics=True)

Any framework, the Microsoft OpenTelemetry Distro. For LangGraph, ADK, or the OpenAI Agents SDK, add a few lines of SDK init, point the exporter at Azure Monitor with a connection string, declare the framework so the distro applies the right auto-instrumentation, and pass the same agent ID you used to register the agent in Foundry so its traces correlate back to it. That last step is what lets Foundry stitch an agent running on GCP Cloud Run or AWS into the same trace tree as a Foundry-hosted one.

Locally, use the Aspire Dashboard. Before any of this reaches Azure, run the dashboard as a local OTLP endpoint and point your agent at it:

docker run --rm -it -d -p 18888:18888 -p 4317:18889 \
  --name aspire-dashboard mcr.microsoft.com/dotnet/aspire-dashboard:latest
# then: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

One gotcha worth stating plainly, because it is the single most common “why is nothing showing up in Application Insights”: in Agent Framework’s own Aspire ServiceDefaults sample, the UseAzureMonitor() call ships commented out. It exports to OTLP (the Aspire Dashboard) by default; you have to uncomment the Azure Monitor block and add the connection string to ship to the cloud.

The step-by-step trace of a tool call

#

This is the centrepiece. When an agent answers a question that requires a tool, it emits a tree of spans, all sharing one operation_Id. Read the tree and you can see exactly what the agent decided and did.

Read top to bottom, the temporal order is a sequence of steps:

Two things make this tree legible in Azure Monitor. First, span kind decides the table: Server-kind spans (the inbound HTTP request) land in requests/AppRequests; Client and Internal spans (invoke_agent, chat, execute_tool) land in dependencies/AppDependencies. Second, correlation is W3C trace context: every span shares one operation_Id (the trace id), and each row’s operation_ParentId equals the id of its immediate parent. That parent/child link is what turns 47 flat rows into a tree you can actually reason about.

Here is the query I reach for first: “show me one agent run, in order.” No Microsoft doc ships this verbatim for gen_ai.* spans, so treat it as assembled from confirmed building blocks, but every key in it is one I verified against a real Microsoft Learn example:

dependencies
| where operation_Id == "<operation_Id of one agent run>"
| project timestamp, id, operation_ParentId, name, duration, success,
    op       = tostring(customDimensions["gen_ai.operation.name"]),   // invoke_agent | chat | execute_tool
    agent    = tostring(customDimensions["gen_ai.agent.name"]),
    tool     = tostring(customDimensions["gen_ai.tool.name"]),
    convo    = tostring(customDimensions["gen_ai.conversation.id"]),
    inTokens = toint(customDimensions["gen_ai.usage.input_tokens"]),
    outTokens= toint(customDimensions["gen_ai.usage.output_tokens"])
| order by timestamp asc

In the Foundry and Application Insights UIs the same data renders as a trace tree. The App Insights “Agents” view now has a simple view that lays an agent run out in a “story-like” fashion: the invoked agent, the model it called, the tools it executed.

When an agent hallucinates, the first span to open is the retrieval execute_tool. Its metadata shows the exact query the agent generated and the raw content the tool handed back to the model. Nine times out of ten the grounding problem is right there.

A trap: the two Microsoft agent stacks don’t emit the same keys

#

Before you build a dashboard, know this. Microsoft Agent Framework emits token counts as gen_ai.usage.input_tokens / gen_ai.usage.output_tokens on a chat {model} span. Semantic Kernel emits a different pair: gen_ai.response.prompt_tokens / gen_ai.response.completion_tokens on a chat.completion {deployment-name} span. Same vendor, same “OpenTelemetry GenAI” banner, two different customDimensions shapes. A single dashboard query across both stacks needs both key sets, or it silently reports zero tokens for half your agents. “It follows the semantic conventions” does not mean “one query works everywhere” yet.

Observability at the gateway: APIM v2 AI gateway

#

Route your model calls through API Management’s AI gateway and you get a second, independent observability layer that doesn’t depend on the agent being instrumented at all. Two features matter here.

Token metrics. The llm-emit-token-metric policy (the provider-agnostic successor to azure-openai-emit-token-metric) emits total, prompt, and completion token counts as custom metrics, with up to five custom dimensions:

<inbound>
  <base />
  <llm-emit-token-metric namespace="llm-metrics">
    <dimension name="API ID" value="@(context.Api.Id)" />
    <dimension name="User ID" value="@(context.Request.Headers.GetValueOrDefault(&quot;x-user-id&quot;, &quot;N/A&quot;))" />
  </llm-emit-token-metric>
</inbound>

These are actual counts read from the model’s usage field, not estimates, and they are emitted asynchronously after the response returns, so the metric adds no latency to the request path. Read them back out of customMetrics:

customMetrics
| where name == "Total Tokens"
| extend d = parse_json(customDimensions)
| project timestamp, value, apiId = tostring(d.["API ID"]), userId = tostring(d.["User ID"])
| order by timestamp asc

Two caveats the docs are explicit about. Streaming responses can undercount if the stream is interrupted, and some models omit token counts when streaming unless you set include_usage: true on the request. The practitioner warning that matters more than the policy config: the metric gives you consumption data, not a billing system. Sort out a cross-charging model with finance before AI spend scales, because retroactive cross-charging is far harder to establish than a plan set up in advance.

Prompt and completion logging. Separately, a diagnostic setting streams the full LLM request/response into the ApiManagementGatewayLlmLog table (chunked at 32 KB), and the built-in “Language models” analytics workbook visualises consumption. This is your audit trail when you deliberately do not capture content in the agent’s own spans.

One honest limitation. You would think you could join the APIM token metric back to the agent span that caused it by trace id. That join is not cleanly documented. APIM’s built-in “Operation ID” dimension is the static API-operation resource identifier, not the per-request trace id, and no doc confirms whether the AppMetrics table’s native OperationId column inherits the ambient request’s trace id when the policy fires.

The documented workaround is to thread your own correlation value: send gen_ai.conversation.id as an HTTP header and emit it as a custom dimension, then join on that.

Tracing across the MCP boundary

#

The Model Context Protocol boundary is the acknowledged blind spot in agent observability. Sessions that never initialise, tool registries too large to parse, tool calls that time out: these surface at the client, even when the MCP server is hosted elsewhere, and standard tracing does not link the client-side symptom back to the model decision that triggered it.

The fix the community converged on is not to ship server-side spans across the wire (that leaks internal detail, SQL queries and all, to a potentially untrusted client). It is W3C trace context propagation via MCP’s params._meta field:

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "get-weather",
    "_meta": { "traceparent": "00-4bf92f3577b34da6...-00f067aa0ba902b7-01" }
  }
}

When both sides propagate traceparent, the server span nests as a child of the client span and the trace stays continuous across the boundary. Agent Framework injects this automatically, but only for client-opened MCP sessions (MCPStreamableHTTPTool, MCPStdioTool, MCPWebsocketTool). It does not inject for provider-hosted MCP tools, because in that case the provider’s runtime, not your agent process, issues the call.

The MCP Python SDK, notably, is OTel-instrumented with zero application code: every inbound tools/call becomes a span carrying gen_ai.operation.name = execute_tool and gen_ai.tool.name, so it merges cleanly with the agent’s own tool span rather than duplicating it.

If your MCP servers sit behind API Management, you get the richest option: APIM emits a native MCP telemetry schema for every request, no config required, writing requests rows with gen_ai.operation.name (tools/list or tools/call), gen_ai.conversation.id (the MCP session id), gen_ai.tool.name, and api.type = "Mcp". That gives you per-tool latency and error rates directly in KQL:

requests
| where customDimensions["api.type"] == "Mcp"
  and customDimensions["gen_ai.operation.name"] == "tools/call"
  and timestamp > ago(24h)
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), calls = count()
    by tool = tostring(customDimensions["gen_ai.tool.name"])
| order by p95 desc

Because APIM’s MCP rows and the agent’s invoke_agent spans both carry gen_ai.conversation.id, you can join them, if your agent sets the same session identifier on both its own span and its outbound MCP session. The frameworks don’t guarantee that alignment automatically; it is a design choice you make. It is the same identity-propagation discipline the on-behalf-of flow relies on: the trace and the token travel the same path.

The dashboards, and building your own

#

Azure gives you three ready-made surfaces. The App Insights Agents view (preview) unifies Foundry, Copilot Studio, and third-party agents, sortable by “most tokens used.” The Foundry Agent Monitoring Dashboard (preview) shows token usage, latency (it flags anything over 10 seconds), run success rate (it flags anything under 95%), plus evaluation and red-team results. And Explore in Grafana ships opinionated prebuilt dashboards for Agent Framework, workflows, and Foundry.

But here is the gap worth knowing before you plan: no repository or doc ships a single workbook that stitches agent spans, APIM token metrics, and MCP calls together. The pieces are scattered across three data sources. Assembling one is genuinely your work. The building blocks, all verified:

// Token usage over time — assembled from confirmed keys + the standard bin/timechart idiom
dependencies
| where timestamp > ago(24h)
  and customDimensions["gen_ai.operation.name"] in ("chat", "invoke_agent")
| extend inTokens  = toint(customDimensions["gen_ai.usage.input_tokens"]),
         outTokens = toint(customDimensions["gen_ai.usage.output_tokens"])
| summarize InputTokens = sum(inTokens), OutputTokens = sum(outTokens) by bin(timestamp, 1h)
| render timechart

// Per-agent success rate — flag anything under 95%, matching the Foundry dashboard's own threshold
dependencies
| where timestamp > ago(24h) and customDimensions["gen_ai.operation.name"] == "invoke_agent"
| summarize Total = count(), Success = countif(success == true) by agent = tostring(customDimensions["gen_ai.agent.name"])
| extend SuccessRatePct = round((Success * 100.0) / Total, 1)
| order by SuccessRatePct asc

Wire the same traces into continuous evaluation or trace evaluation (both preview) and you close the loop: Foundry runs evaluators against sampled production traces, and a failed eval jumps straight back into the offending trace. That is the operating loop: instrument, debug, evaluate against live traffic, optimise, all on one observability plane.

Production gotchas

#

The failure modes below are the ones that cost real time. None of them show up as an exception.

• Content capture is a PII and cost decision, not a checkbox. A single GenAI trace can run ~25 KB against ~500 bytes for a traditional trace (roughly 50×) once you log full prompts and completions. There are two capture flags across the stack, and stacking the SDKs means you must set both or content goes missing from one layer’s spans:
  • Agent Framework’s ENABLE_SENSITIVE_DATA.
  • The Azure AI Inference SDK’s AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED.
  • Redact at the application before data enters the pipeline, add a Collector-level regex/OTTL net behind it, and keep content capture on in dev only.
• Streaming can orphan your tool spans. A confirmed .NET regression set Activity.Current to null after the first streamed chunk when the parent was an invoke_agent span, so every subsequent tool and chat span appeared as a disconnected root trace instead of a child. If your tool calls show up as orphans, suspect the streaming path before you suspect your wiring.
• Application Insights truncates long messages. Practitioners have hit gen_ai.input.messages getting cut off on long system prompts, sometimes losing the user message entirely. If you need the full conversation for audit or evaluation, store it in your own structured store. Trace data is spread across three tables and was never designed to be the system of record for full payloads.
• Adaptive sampling silently drops error traces. Error-level logs are categorised as traces, not exceptions, so an “errors only” strategy can quietly lose them to sampling. Exclude them explicitly (<ExcludedTypes>Trace;Exception</ExcludedTypes>) or your rarest, most important agent failures vanish.
• Watch for the stuck loop directly. The $41 overnight loop is catchable. Put an agent.turn parent span on each run with an agent.iterations count, give each execute_tool child a tool.input_hash (a short hash of the canonicalised arguments), and alert when iterations cross a threshold or the same input hash repeats more than twice inside one turn. That catches the loop at iteration 3, not 187.
• Not every “OTel” span is a gen_ai.* span. OpenLLMetry and OpenInference are OTLP-compatible and Azure Monitor will happily ingest them, but they use their own attribute namespaces (llm.*, input.value), not gen_ai.*. A workbook keyed on gen_ai.* filters will show nothing for spans from those libraries without a mapping layer.

Worth remembering

#

• Cognitive failures show green. Every one of those 187 looping calls returned HTTP 200. If you only watch for exceptions and latency, you never see the expensive-but-“successful” agent run.
• Instrument once, view many. OpenTelemetry GenAI conventions plus one Application Insights backend give you Foundry, Grafana, and your own workbooks as interchangeable views. Don’t bet the stack on a single vendor UI.
• The conventions are a moving contract. gen_ai.* is still Development status, and the two Microsoft agent stacks emit different token keys. Confirm what your framework emits before you write a line of KQL.
• The stitched dashboard is your work. No repo ships one workbook that joins agent spans, APIM token metrics, and MCP calls. The building blocks are here; the assembly is yours.

Where to start

#

1. Enable OpenTelemetry and the Azure Monitor exporter on every agent: one connection string, one backend. Keep content capture off in production.
2. Route model calls and MCP servers through the APIM AI gateway so you get token metrics, LLM logs, and the native MCP telemetry schema without touching agent code.
3. Confirm which key set your framework emits (gen_ai.usage.* for Agent Framework, gen_ai.response.* for Semantic Kernel) before you write a line of KQL.
4. Build one workbook keyed on gen_ai.conversation.id: the run-in-order query, token usage over time, per-agent success rate. Nobody ships this for you.
5. Alert on the things monitoring can’t see: iteration counts, duplicate tool calls, and run success rate under 95%.
6. Sample tail-first: keep 100% of errors and anything over a cost threshold, sample the rest.

Capability was never the thing that made agents production-ready. Being able to answer “what did this agent actually do, and why” is what does. On Azure, that answer lives in the span tree.