Every team that ships generative AI hits the same questions. Which models may we call, and with which keys? Who spent the tokens? How do we stop one agent from starving everyone else, or from calling a tool it should never touch? Azure AI Gateway, a managed gateway for AI traffic, answers them in one place: apps and agents call one gateway, and the gateway decides who gets in, which model or tool answers, what each call may cost, and what gets recorded.
Transcript
Welcome to the Azure AI Gateway labs. In three minutes, you'll see what the gateway does, follow one request through it, and tour the portal.
Every team that ships AI hits the same questions. Which models may we call, with which keys? Who spent the tokens? AI Gateway answers them in one place: apps and agents call one gateway, and it routes to models and tools.
The gateway has two faces. The data plane is the runtime endpoint your code calls. The control plane is this portal and the Azure Resource Manager API, which you can drive with the CLI or Bicep.
Now follow one request. The gateway checks the caller's key or Entra token, then evaluates every policy before calling the backend. A blocked call gets a predictable status: 401, 403, 429, or 400. Either way, it's recorded.
Here's the portal, at ai.gateway.azure.com. The switcher in the top bar picks your gateway, and Home shows its runtime endpoint. The left rail holds everything you manage. Press Control K to search from anywhere.
Catalog is what developers browse: every published model and MCP server, with a playground to try it and code samples to copy.
Models opens on aliases, the names your apps send. Here, chat points at gpt-6-luna. Move it to a newer model, and no client code changes. Backend models shows what each provider serves.
MCP servers give agents one endpoint for their tools. This Microsoft Learn server federates two sources, learn and docs, and tool availability decides which tools agents can see and call.
Access decides who can call. Use Microsoft Entra ID roles, which we recommend, or API keys like this one. In the other direction, the gateway's managed identity signs in to Foundry, so apps never hold provider keys.
Policies are the guardrails. This gateway has a request limit and a token limit at gateway scope, covering every model, plus tighter limits on individual assets, like 1,000 tokens a minute on one alias.
To add one, select Add policy. Choose a type, then a scope: the whole gateway, or individual assets. Then set the limits. We'll cancel without saving.
Once Application Insights is connected, Monitoring shows model and tool calls, tokens, and estimated cost, and which policies blocked requests, like the 429s from that limited alias. Traces follow any call end to end.
That's the map. Next, provision your own gateway in Lab 01, or pick the learning path for your role. See you in the labs.
Azure AI Gateway is generally available since November 18, 2026, and runs in a growing list of regions: run the command in Find the available regions to list them. For the gateway's pricing, see the AI Gateway documentation. Connected services, such as Microsoft Foundry inference and Application Insights, are billed normally.
Before you begin
This lab is a guided reading. You don't need to deploy anything.
- If you already have a gateway, open the AI Gateway portal in another tab and follow the portal tour at the end.
- If you don't have one yet, read on. You create your first gateway in Provision your first AI gateway.
How it works
The gateway sits on the path of every model call and every tool call. It has two faces:
- Data plane: the runtime endpoints your code calls, at
<gateway-url>. - Control plane: the product portal at ai.gateway.azure.com and the Azure
Resource Manager API for the
Microsoft.ApiManagement/aigatewaysresource type, which you call with theaz aigatewayAzure CLI extension, Bicep, or any REST client.
Azure AI Gateway is a dedicated product for AI traffic, built on the Azure API Management gateway foundation. It
shares the Microsoft.ApiManagement resource provider with API Management service but has its own resource type, so
you can keep business APIs in API Management and send model and tool traffic through the gateway.
| Building block | What it is | Example |
|---|---|---|
| Gateway | An Azure AI Gateway resource (Microsoft.ApiManagement/aigateways) in your subscription | <gateway-url> |
| Backend model | A model at a provider: a chat, embedding, or decision model, such as a Foundry deployment or an Anthropic model | gpt-5-mini at Microsoft Foundry |
| Alias | The stable, gateway-wide name clients send in the model field. Its routing settings pick a backend model for each request | chat |
| MCP server | One governed MCP endpoint that federates tools from one or more sources | /default/toolservers/<server-name>/mcp |
| Policy | A rule on the gateway or on one asset: limits, budgets, safety, filters, and Agent hooks | Token rate limit, Cost limit |
| Access | How callers authenticate: API keys or Microsoft Entra ID roles | Authorization: Bearer with a key or an Entra token |
| Managed identity | How the gateway authenticates to backends | Foundry User role on the Foundry account |
| Monitoring | OpenTelemetry metrics, logs, and traces for every call | Application Insights or any OTLP endpoint |
Model calls use protocols developers already know. The OpenAI-compatible base URL is
https://<gateway-host>/default/models/openai/v1 (append /responses, /chat/completions, or /embeddings),
Anthropic Messages lives at /default/models/anthropic/v1/messages, and Jev decision models answer TypeSafe System
One requests at /default/models/typesafe/v1/systemone. The gateway passes requests through without translating
between formats, so call each model on an interface it advertises. See
Model API formats and providers.
Follow one request
Press Play to send a request through the gateway. Then switch scenarios to see where each policy stops a request and which status code the caller gets back.
Three rules explain almost every response you'll see in these labs:
- Policies run before the backend. If any applicable policy blocks a request, the gateway answers right away and never calls the backend. The one exception is an Agent hooks policy that checks a tool's result: it runs after the tool and can withhold the result, but it can't undo the tool's work.
- Each block has a predictable status.
401for missing or invalid credentials,403for IP filters and cost limits (LlmCostQuotaExceeded),429withRetry-Afterfor token and request rate limits, and400for content safety. - Everything is recorded. Allowed and blocked calls both show up in Monitoring, along with the policies that affected them.
A backend error is different from a policy block. When the serving backend fails, for example with 429 or a
5xx, the alias's failover rules can retry it or switch to another backend before the caller sees the error.
Models and aliases
A backend model is what a provider serves. An alias is what your clients call. Microsoft Foundry imports create an alias for each model by default, and you can add more. Because clients only know the alias, you can move it to a newer or cheaper model without a code change, and the alias keeps its own policies.
You can put models from Microsoft Foundry, Azure OpenAI, OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI, and custom OpenAI-compatible endpoints behind aliases. You try it in Route models from several providers behind aliases. Besides chat models, an alias can publish embedding models for search and retrieval, which you try in lab 26, or TypeSafe Jev decision models that return structured yes/no probabilities, choices, and rubric scores, which you try in lab 31. An alias can't mix chat, embedding, and decision models: all its backends must share an API endpoint.
Routing options
An alias can point to several backend models. Its routing settings decide which one serves each request:
| Routing option | What it does | Try it |
|---|---|---|
| Priorities and weights | Prefers the available backends with the lowest priority number, and spreads requests across them by weight | Lab 22 |
| Backend conditions | Limits which requests a backend can receive, with a CEL expression over the request, such as a header | Lab 27 |
| Auto model selection | Uses Model Router to choose a smaller or larger model for each prompt, for your quality and cost objective. It's still emerging: see the November 6, 2026 release | Lab 28 |
| Session affinity | Sends requests with the same session key, read by a CEL expression, to the same backend when possible, so a conversation can reuse the provider's prompt cache | Session affinity |
| Failover and retry | Retries a failed backend or switches to the next eligible one, and can exclude a failing backend for a while | Lab 29 |
Clients keep calling the same alias while you change any of these settings. See Models and routing and Routing order for how the settings combine.
MCP servers and tools
Agents need tools as much as models. An MCP server on the gateway gives agents one endpoint and one credential for tools from remote MCP servers, OpenAPI specifications, Foundry Toolbox, and MCP servers you import from Azure, such as the connector tools of a Connector Namespace. Tool names are namespaced per source, and tool availability decides which tools clients can see and call.
Policies and scopes
You add a policy at gateway scope, where it applies to every compatible asset including assets you add later, or to individual assets: a backend model, an alias, or an MCP server. Pick an asset below to see every policy that applies to it.
| Policy | Applies to | When it blocks |
|---|---|---|
| Token rate limit | Models and aliases | 429 with Retry-After |
| Request rate limit | Models, aliases, and MCP servers | 429 with Retry-After |
| Cost limit | Models and aliases | 403 LlmCostQuotaExceeded |
| Content safety | Models, aliases, and MCP servers | 400 |
| IP filter | Models, aliases, and MCP servers | 403 |
| Agent 365 tool threat protection | MCP servers | Denies the tool call |
| Agent hooks | MCP servers | Your decision service denies the tool call before it runs, or withholds its result after it runs |
At gateway scope, the policy type decides which assets are relevant. A gateway-scope cost limit, for example, tracks one budget across all models. An asset policy adds a control: it doesn't replace a policy the asset inherits from the gateway.
Agent hooks send the tool call's context to an external decision service that you run, and enforce its allow or
deny verdict, for example to refuse refunds above a limit. They're still emerging: see the
November 6, 2026 release and
Agent hooks, then try them in
lab 30.
Identities on both sides
The gateway separates who is calling from how the gateway reaches the backend. Callers use an API key or a Microsoft Entra token with one of the AI Gateway data-plane roles. The gateway uses its own managed identity, or a provider key it stores, to call backends. Callers never see backend credentials.
Managing the gateway is a third, separate permission. AI Gateway Developer can view the configuration of aliases and MCP servers, AI Gateway Contributor can manage model providers, aliases, MCP servers, routing, limits, and policies, and Contributor on the gateway resource can also change gateway settings, networking, identities, and API keys. None of these roles can call models or tools: that takes a data-plane role. See Control-plane access.
Observability
The gateway emits OpenTelemetry metrics, logs, and traces for model calls, embeddings, decisions, tool calls,
routing, and policy decisions. Traces also mark model-stream and MCP operation failures as errors, even when the
HTTP response was 200. Send them to Application Insights to light up the built-in Monitoring dashboard, or
to any OTLP endpoint.
Take a portal tour
If you have a gateway, walk through the portal now. Otherwise, use the screenshots and come back after lab 01.
Open your gateway
Go to ai.gateway.azure.com and sign in. Use the gateway switcher in the top bar to pick a gateway. The left rail lists Home and Catalog, then Models, MCP servers, Policies, and Monitoring under Manage, and Access, Managed identities, and Networking under Secure. Press Ctrl+K, or ⌘+K on macOS, to open the command palette from anywhere.
Browse models and aliases
Select Models. It opens on Aliases, which lists each alias with its backend models, providers, API formats, and model selection. The Backend models tab lists each backend model with its aliases and provider.
Look at an MCP server
Select MCP servers and open a server. The tool availability settings decide which tools clients can see.
Start a policy
Select PoliciesAdd policy. The wizard asks for a Type, then a Scope (Gateway or Individual assets), then the Details. Select Cancel to leave without saving.
Open Monitoring
Select Monitoring. When Application Insights is connected, the Overview, Models, Tools, Policies, and Traces tabs show live traffic.
Verify
Check your understanding before you move on.
An app sends "model": "chat". What must chat be?
A callable alias on the gateway. The model field must exactly match an alias, not a provider deployment name.
A caller exceeds a token rate limit. Is the backend called?
No. The gateway answers 429 Too Many Requests with Retry-After and never forwards the request.
An alias has two backends, and the first one returns 503. Does the caller see the error?
Only if no failover rule on the alias matches the failure. A matching rule can retry the same backend, switch to the
other backend, and exclude the failing one for a while. Without a matching rule, the gateway returns the 503.
You add an IP filter at gateway scope, then add a new model next week. Is the new model protected?
Yes. Gateway-scope policies apply to every compatible asset, including assets you add later.
Which credential does the gateway use to call a Microsoft Foundry model?
Its own managed identity, which needs the Foundry User role on the Foundry account. The caller's key or token is never forwarded.
You can explain the request lifecycle, the difference between a backend model and an alias, and where gateway and asset policies apply.
Clean up
Nothing to clean up. This lab doesn't create resources.
Next steps
Short on time? The 5-minute quick start creates a gateway with a Foundry model in the portal or Azure Cloud Shell and makes your first call. Then pick the path that fits your role. Each path is a sequence of labs, and your progress is saved in this browser.
| Path | Start with | You'll learn to |
|---|---|---|
| 30-minute essentials | Provision a gateway | Create a gateway, call a model, and add MCP tools |
| Platform engineer | Provision a gateway | Build a governed, multi-provider gateway with code |
| App & agent developer | First model call | Call models and tools from SDKs, agents, and hosted agents |
| Security & compliance | API keys | Control access, content, tools, and threats |
| FinOps & operations | Token limits | Cap tokens, set budgets, and charge back usage |
- Docs: AI Gateway documentation
- Microsoft Learn: What is Azure AI Gateway? and Azure AI Gateway and Azure API Management service