The AI Gateway in Azure API Management: A Control Plane for Enterprise AI

The governance gap nobody budgets for

#

Ask three questions in any organization that has been building with AI for a year: Which teams are calling which models, and what does each one spend? Which MCP tools can our agents reach, and who approved them? If an agent misbehaves right now, how do we stop it?

In most places the honest answers are: nobody knows, nobody approved anything, and we’d have to find the team that owns it. Teams provision their own model endpoints with their own keys. Developers wire MCP servers into Copilot and Claude with credentials in config files. Agents call other agents with nothing in between. Every one of those is a reasonable local decision — and together they are an ungoverned AI estate.

The industry data says this gap is expensive. IBM’s 2025 breach-cost research found shadow AI was a factor in roughly one in five breaches studied, adding around $670K in average breach cost. Gartner projects that more than 40% of agentic AI projects will be canceled by 2027 — for cost, unclear value, and weak risk controls, not model capability.

This is the job Azure API Management’s AI gateway has grown into. It started as token limiting for Azure OpenAI. Today it is Microsoft’s answer to a much bigger question: where does ALL AI traffic — model calls, tool calls, agent-to-agent calls — get identity, policy, safety, and audit applied? Microsoft’s own Cloud Adoption Framework now recommends routing all AI agent traffic through an AI gateway as “a unified control point for policy enforcement.”

Bottom line: the value is not any single policy. It is that model calls, tool calls, and agent calls all pass one enforcement point where identity, rate limits, content safety, and audit are applied the same way — governance you already operate, extended to AI rather than rebuilt for it.

And it moves real numbers. Access Group, presented at Ignite 2025, shipped 50+ AI-powered products in 12 months and achieved ISO 42001 certification for its own AI management system after standardizing on the AI gateway. Uniper reports collapsing seven per-environment API definitions into one governed gateway pattern — an 85% reduction in API surface.

What the AI gateway actually is

#

The AI gateway is not a separate product — it is the same APIM gateway and policy engine you already run for REST APIs, extended to three kinds of AI traffic. Microsoft’s own material structures it as three pillars: Models, Tools, and Agents (Azure-Samples/AI-Gateway).

One policy engine fronting all three pillars. The same validate-azure-ad-token, rate-limit, and content-safety policies apply to a chat completion, an MCP tool call, and an A2A agent call — write once, enforce everywhere.

As one architect put it: policies say “don’t do X” — control planes make X impossible.

What each team gets

#

The reason this matters to more than the AI team is that a single governed ingress gives every department its own control surface without any of them standing up separate infrastructure.

Bottom line: the gateway is not just an AI-team tool. It is where the security, network, infrastructure, and finance controls each team already owns get applied to AI traffic — through one shared surface.

The rest of this piece walks the three pillars, then the identity, safety, and audit layers, then the network and infrastructure view the platform teams need.

Governing models: one endpoint, many providers

#

The model pillar is broader than Azure OpenAI. APIM natively manages three API schemas — OpenAI Chat Completions/Responses, Anthropic Messages, and Google Vertex — and fronts backends including Amazon Bedrock, Gemini, and Hugging Face. At Ignite, the team demoed the same token-limit and content-safety policies against a Bedrock model to make exactly that point (capabilities overview).

The unified model API (preview) takes it further: one OpenAI-format endpoint in front of all providers, an auto-generated /models discovery endpoint, cross-provider failover, and model aliases that decouple the name clients use from the deployment behind it:

from openai import OpenAI

client = OpenAI(
    base_url="https://<apim-instance>.azure-api.net/llm/v1",
    api_key="<api-management-subscription-key>",
)

response = client.chat.completions.create(
    model="gpt",  # alias — swap the backend, clients never change
    messages=[{"role": "user", "content": "What can you do?"}],
)

Aliases are an underrated governance tool. Model deprecations, A/B tests, and cost-driven swaps become gateway routing changes instead of a coordinated client-code rollout across every team.

Underneath sit the traffic controls, each deserving its own deep dive: llm-token-limit rate limits and monthly quotas per team or tenant (with deployment-level limits added in 2026), priority-based backend pools spilling PTU baseline over to PAYG, circuit breakers that honor Retry-After, and semantic caching for FAQ-shaped traffic.

Watch out: token counters are tracked per gateway, not aggregated across regions — a real factor in capacity planning. And a single oversized completion can blow an entire per-user quota in one call. Size quotas for worst-case response length, not call frequency.

Governing tools: MCP without the sprawl

#

MCP adoption inside enterprises is currently where API sprawl was fifteen years ago — every team standing up servers, credentials in config files, no inventory. The security research is not reassuring: documented one-click account-takeover vulnerabilities from MCP servers mishandling OAuth consent (Obsidian Security), and a consultancy assessment that “no single vendor closes the full loop yet” on MCP governance. The gateway approach exists precisely because you cannot trust every MCP server’s homegrown auth.

APIM gives MCP two governed on-ramps (overview): expose any REST API you already manage as an MCP server (operations become tools), or passthrough an existing MCP server behind gateway policy. Either way, MCP servers become products — bundled, subscription-gated, approval-workflowed, rate limited — exactly like your APIs. As of 2026 you can package MCP servers into APIM Products with versioning, and register them in Azure API Center’s MCP registry that VS Code and Copilot query for discovery.

Inbound access uses OAuth 2.1 with Entra ID; outbound, APIM’s credential manager injects backend OAuth tokens so tool credentials live in the gateway, not in client config files (secure MCP servers). The part teams consistently get wrong is that there are two tokens in play, and neither side ever sees the other’s — the client’s Entra token proves who is calling and dies at the gateway; the backend credential lives in the gateway and never reaches the client:

Two tokens, isolated by design: the caller’s identity token is validated and terminated at the gateway; the backend credential is injected server-side and never returns to the client. No tool secrets in client config.

Watch out (the sharp edges): REST/OpenAPI sources only for now (gRPC/GraphQL is roadmap); tools only — no MCP resources or prompts for REST-backed servers; never touch context.Response.Body in an MCP policy and zero out frontend response-body logging before enabling diagnostics, or you break MCP streaming; and MCP isn’t supported inside workspaces yet.

Governing agents: A2A and the gateway block

#

The newest pillar, and the reason “AI gateway” no longer means “LLM proxy.” Agent-to-Agent (A2A) protocol support reached GA in 2026, and it is a first-class API type — an APIM API declared with type: 'a2a':

resource api 'Microsoft.ApiManagement/service/apis@2024-10-01-preview' = {
  parent: apim
  name: agentName
  properties: {
    displayName: '${agentName} Server'
    type: 'a2a'
    isAgent: true
    a2aProperties: {
      agentCardPath: '/.well-known/agent.json'
      agentCardBackendUrl: '${APIServiceURL}/.well-known/agent.json'
    }
    jsonRpcProperties: { backendUrl: APIServiceURL, path: '/' }
    subscriptionRequired: true
    path: APIPath
    protocols: [ 'https' ]
  }
}

APIM mediates the agent’s JSON-RPC traffic and — the clever part — rewrites the agent card so the advertised hostname points at the gateway and the security requirements include your subscription key (A2A agent APIs). Traces carry OpenTelemetry GenAI attributes including gen_ai.agent.id, so per-agent observability works in multi-agent estates — with one catch: you must supply the same agent ID the agent already emits in its own traces; nothing reconciles them for you.

Consumers never learn where the agent really lives — every discovery request and every JSON-RPC call goes through the gateway, which is also what makes the block path possible:

Because all discovery and JSON-RPC traffic transits the gateway, a registered agent can be blocked at the gateway regardless of which cloud it runs on.

Because a Foundry-registered agent’s traffic all flows through the gateway, disabling it at the gateway makes the next call return 403 — even for an agent hosted on another cloud you don’t operate. That is a real incident-response pattern for a rogue or non-compliant agent. Know its limit, though: it’s a gateway-layer block. The agent’s compute keeps running (and billing) on the source cloud until you stop it there.

The Foundry integration (preview) rounds this out: models, agents, and MCP tools governed from one control-plane view, with agents running on Azure, other clouds, or on-premises registered into a central inventory (governance docs).

Identity is the spine

#

None of the above matters if callers aren’t strongly identified. The documented pattern for AI APIs is three layers, combined:

• Subscription keys — attribution and product scoping at the front door; what quota counter-key expressions hang off.
• Managed identity, gateway → backend — APIM authenticates to Azure OpenAI with its own managed identity, so no model API keys live at the application layer at all. This is the same keyless principle covered end-to-end in the Zero-Secrets Azure series — the gateway simply extends it to the ingress tier.
• OAuth on top — validate-azure-ad-token for fine-grained, who-is-this-human-or-agent authorization.

<validate-azure-ad-token tenant-id="{{TENANT_ID}}" header-name="Authorization"
    failed-validation-httpcode="401">
    <client-application-ids>
        <application-id>{{CLIENT_APP_ID}}</application-id>
    </client-application-ids>
    <required-claims>
        <claim name="roles" match="any">
            <value>OpenAI.ChatCompletion</value>
        </claim>
    </required-claims>
</validate-azure-ad-token>

Two Entra-specific lessons from the official access-controlling lab: prefer app roles over group claims — Entra drops group claims from tokens when a user belongs to too many groups (the 200-group cap), which turns “works in dev” into “random 403s in production.” And APIM subscriptions cannot be assigned to Entra security groups directly — claims-based authorization at the policy layer is the standard enterprise pattern, not a workaround.

For the CISO — two caveats worth surfacing: APIM does not validate which backend a managed-identity token is forwarded to, so a misconfigured set-backend-service could leak the gateway’s own token — treat token forwarding as your responsibility. And anyone with policy-edit rights can mint a token as the service identity, so restrict apis/policies/write and audit it (security considerations).

The frontier problem is agent identity. Security practitioners report most organizations running agents on borrowed human or service-account credentials — “shadow agents” outside any inventory (Schneider). The gateway gives you the enforcement point; you still have to do the identity work: one identity per agent, short-lived credentials, an accountable owner for each.

Responsible AI as enforcement, not a policy document

#

Most responsible-AI programs live in slide decks. The gateway makes parts of them executable: the llm-content-safety policy routes traffic through Azure AI Content Safety with per-category severity thresholds, custom block lists, and Prompt Shields — which detect both direct jailbreak attempts and instructions smuggled into grounding documents. As of 2026 the same policy applies to MCP tool calls and A2A agent traffic, not just chat completions — one enforcement point for all three pillars.

<llm-content-safety backend-id="content-safety-backend" shield-prompt="true">
    <categories output-type="EightSeverityLevels">
        <category name="Hate" threshold="4" />
        <category name="Violence" threshold="4" />
    </categories>
</llm-content-safety>

Watch out: on streaming responses, a mid-stream violation silently stops the stream rather than returning a 403 — build client handling accordingly. And semantic caching returns responses by similarity, so Microsoft recommends pairing cache lookup with content safety and rate limiting for defense in depth.

The audit backbone

#

This is the capability that turns “we should govern AI” into evidence. The dedicated ApiManagementGatewayLlmLog table captures token usage and model per request, and — opt-in, per API — full prompts and completions with byte caps (LLM logging).

This is a data-handling decision, not a logging toggle. You are persisting user prompts. Log LLM messages is off by default; prompts and completions are separate opt-ins. Decide deliberately who can query them — Azure Monitor’s Protected tables (preview) plus a Privileged Monitoring Data Reader role via PIM let you restrict raw prompt access to just-in-time, while everyone else sees token metrics only. For MCP, tool arguments and results are excluded by default for the same reason.

What the log unlocks, beyond compliance, comes as reference patterns in the AI-Gateway labs rather than one-click features: token metrics dimensioned by team and app for chargeback; budget alerts wired to a Logic App that disables an overspending subscription — a kill switch you assemble, since Azure budget alerts are informational and don’t stop spend on their own; and an evaluation loop where logged prompt/completion pairs export into Foundry to test whether a cheaper model can take over a workload before you swap it behind an alias.

On compliance specifically, be precise: the gateway sits within Azure’s standard compliance envelope, and when it fronts Microsoft Foundry, the model layer underneath is ISO/IEC 42001-certified — but APIM itself is not independently attested to that standard. What the audit trail does is let a customer (as Access Group did) certify its own AI management system.

The network and infrastructure view

#

Everything above is the policy surface. For the platform, network, and infrastructure teams who actually run the gateway, the deployment model is the other half of the decision — and it changed materially in late 2025 with the v2 tiers.

APIM now ships three tier families: Consumption, classic (Developer/Basic/Standard/Premium), and v2 (Basic v2/Standard v2/Premium v2). All three v2 tiers are GA — Premium v2 reached GA in November 2025 — on a faster, more scalable platform. There is no in-place migration from classic to v2; v2 is create-new only, and classic is not deprecated. The AI-gateway policies themselves apply to all tiers; the differences that matter are networking, resilience, and a few feature gates.

The networking model is where most designs get sloppy. There are five distinct mechanisms, and they are not interchangeable:

The last row is the practical sweet spot: combining an inbound private endpoint with outbound VNet integration on Standard v2 gives end-to-end isolation without paying for Premium v2 injection. Premium v2’s own injection is the notable simplification over classic — it no longer requires manually configuring route tables or service endpoints (inject a Premium v2 instance).

The end-to-end private path to a model backend looks like this:

Client reaches the gateway over a private endpoint; the gateway authenticates to a network-isolated Azure OpenAI with its managed identity over VNet integration. No keys, and no public data plane — two private DNS zones tie it together.

Three more things the infrastructure team needs on the table:

• Workspaces let API teams operate like tenants inside one shared instance — each with its own APIs, products, and Entra-scoped RBAC — so you get decentralized team ownership without N separate instances and N separate VNets to manage. Now available in Basic v2, Standard v2, and Premium v2 (MCP isn’t supported inside workspaces yet).
• Resilience: availability zones are supported only on Premium (classic) and Premium v2 — and on Premium v2, zone redundancy is set at creation and can’t be changed after. Premium v2 does not yet support multi-region deployment (still classic Premium only) — the single most common wrong assumption about v2. Plan around it.
• Self-hosted gateway — the containerized data plane for hybrid/on-prem — runs on Developer and Premium (classic) only, not on any v2 tier. It’s the pattern for keeping AI traffic inside the enterprise network except the single hop out to Azure OpenAI, but note semantic caching doesn’t run on it.

Bottom line for the platform team: Standard v2 (inbound private endpoint + outbound VNet integration) covers most network-isolated AI workloads; go Premium v2 for availability zones and injection; stay on classic Premium if you need multi-region or the self-hosted gateway. The GenAI landing-zone accelerator packages both the cloud and self-hosted patterns as IaC.

Tiers and previews: what to believe

#

Facts an architecture review needs on the table:

On the lock-in question competitors raise — yes, this control plane lives in Azure. If your identity is Entra, your monitoring is Azure Monitor, and your API estate is already in APIM, that is precisely the value: AI governance as an extension of governance you already operate, rather than a new platform. Microsoft’s own guidance is honest about the trade-off, too: a gateway can become a single point of failure and add latency, so weigh it against a simple, few-client environment where in-app key management might suffice. If you’re genuinely multi-cloud-first with no Azure center of gravity, evaluate the independent gateways honestly — but with the knowledge that their MCP/A2A governance is no more mature.

Where I’d start

#

Not with all of it. The sequence that works: put models behind the gateway with managed identity and token metrics first — two weeks of telemetry answers “who spends what” and funds the rest of the conversation. Add identity-scoped access with app roles next, because everything downstream assumes callers are known. Then bring MCP under governance as teams adopt tools — registry first, then OAuth enforcement. A2A governance comes last, and by the time your agents are calling each other in production, it will be the least optional item on this list.

Run that sequence and the three questions this post opened with get real answers: the token metrics and LLM logs tell you who calls which models and what it costs, the MCP registry and product subscriptions tell you which tools your agents can reach and who approved them, and when an agent misbehaves, blocking it is one action at the gateway — whichever cloud it runs on.

The Azure-Samples/AI-Gateway labs cover every capability here as deployable Bicep plus policy XML — including the MCP authorization and A2A labs this post draws on. Start with the access-controlling and token-rate-limiting labs; they’re the foundation the rest builds on. And for the keyless managed-identity backbone underneath all of it, the Zero-Secrets Azure series is the companion read.