Azure AI Gateway Labs
Documentation Open portal
Foundations · Lab 00

How Azure AI Gateway works

A 20-minute visual tour of Azure AI Gateway. Follow one request through the gateway, meet models, aliases, MCP servers, policies, and identities, and pick the learning path that fits your role.

Beginner 20 min Portal Updated
Watch video Source Edit

What you'll do

  • Describe what Azure AI Gateway does and where it sits between your apps, agents, models, and tools
  • Follow a request through authentication, policies, alias routing, the backend, and telemetry
  • Explain the difference between a backend model, an alias, and an MCP server
  • Pick the routing option an alias needs, from priorities and weights to backend conditions, auto model selection, session affinity, and failover
  • Predict which policies apply to a request and which status code a blocked request returns
  • Choose the learning path that matches your role

Before you start

No prerequisite labs. Start here.

Tracks: Portal

Every team that ships generative AI hits the same questions. Which models may we call, and with which keys? Who spent the tokens? How do we stop one agent from starving everyone else, or from calling a tool it should never touch? Azure AI Gateway, a managed gateway for AI traffic, answers them in one place: apps and agents call one gateway, and the gateway decides who gets in, which model or tool answers, what each call may cost, and what gets recorded.

Narrated walkthrough: how Azure AI Gateway works (3 min)
Transcript

Welcome to the Azure AI Gateway labs. In three minutes, you'll see what the gateway does, follow one request through it, and tour the portal.

Every team that ships AI hits the same questions. Which models may we call, with which keys? Who spent the tokens? AI Gateway answers them in one place: apps and agents call one gateway, and it routes to models and tools.

The gateway has two faces. The data plane is the runtime endpoint your code calls. The control plane is this portal and the Azure Resource Manager API, which you can drive with the CLI or Bicep.

Now follow one request. The gateway checks the caller's key or Entra token, then evaluates every policy before calling the backend. A blocked call gets a predictable status: 401, 403, 429, or 400. Either way, it's recorded.

Here's the portal, at ai.gateway.azure.com. The switcher in the top bar picks your gateway, and Home shows its runtime endpoint. The left rail holds everything you manage. Press Control K to search from anywhere.

Catalog is what developers browse: every published model and MCP server, with a playground to try it and code samples to copy.

Models opens on aliases, the names your apps send. Here, chat points at gpt-6-luna. Move it to a newer model, and no client code changes. Backend models shows what each provider serves.

MCP servers give agents one endpoint for their tools. This Microsoft Learn server federates two sources, learn and docs, and tool availability decides which tools agents can see and call.

Access decides who can call. Use Microsoft Entra ID roles, which we recommend, or API keys like this one. In the other direction, the gateway's managed identity signs in to Foundry, so apps never hold provider keys.

Policies are the guardrails. This gateway has a request limit and a token limit at gateway scope, covering every model, plus tighter limits on individual assets, like 1,000 tokens a minute on one alias.

To add one, select Add policy. Choose a type, then a scope: the whole gateway, or individual assets. Then set the limits. We'll cancel without saving.

Once Application Insights is connected, Monitoring shows model and tool calls, tokens, and estimated cost, and which policies blocked requests, like the 429s from that limited alias. Traces follow any call end to end.

That's the map. Next, provision your own gateway in Lab 01, or pick the learning path for your role. See you in the labs.

Download the captions (WebVTT)

Note

Azure AI Gateway is generally available since November 18, 2026, and runs in a growing list of regions: run the command in Find the available regions to list them. For the gateway's pricing, see the AI Gateway documentation. Connected services, such as Microsoft Foundry inference and Application Insights, are billed normally.

Before you begin

This lab is a guided reading. You don't need to deploy anything.

How it works

The gateway sits on the path of every model call and every tool call. It has two faces:

  • Data plane: the runtime endpoints your code calls, at <gateway-url>.
  • Control plane: the product portal at ai.gateway.azure.com and the Azure Resource Manager API for the Microsoft.ApiManagement/aigateways resource type, which you call with the az aigateway Azure CLI extension, Bicep, or any REST client.

Azure AI Gateway is a dedicated product for AI traffic, built on the Azure API Management gateway foundation. It shares the Microsoft.ApiManagement resource provider with API Management service but has its own resource type, so you can keep business APIs in API Management and send model and tool traffic through the gateway.

Building blockWhat it isExample
GatewayAn Azure AI Gateway resource (Microsoft.ApiManagement/aigateways) in your subscription<gateway-url>
Backend modelA model at a provider: a chat, embedding, or decision model, such as a Foundry deployment or an Anthropic modelgpt-5-mini at Microsoft Foundry
AliasThe stable, gateway-wide name clients send in the model field. Its routing settings pick a backend model for each requestchat
MCP serverOne governed MCP endpoint that federates tools from one or more sources/default/toolservers/<server-name>/mcp
PolicyA rule on the gateway or on one asset: limits, budgets, safety, filters, and Agent hooksToken rate limit, Cost limit
AccessHow callers authenticate: API keys or Microsoft Entra ID rolesAuthorization: Bearer with a key or an Entra token
Managed identityHow the gateway authenticates to backendsFoundry User role on the Foundry account
MonitoringOpenTelemetry metrics, logs, and traces for every callApplication Insights or any OTLP endpoint

Model calls use protocols developers already know. The OpenAI-compatible base URL is https://<gateway-host>/default/models/openai/v1 (append /responses, /chat/completions, or /embeddings), Anthropic Messages lives at /default/models/anthropic/v1/messages, and Jev decision models answer TypeSafe System One requests at /default/models/typesafe/v1/systemone. The gateway passes requests through without translating between formats, so call each model on an interface it advertises. See Model API formats and providers.

Follow one request

Press Play to send a request through the gateway. Then switch scenarios to see where each policy stops a request and which status code the caller gets back.

Three rules explain almost every response you'll see in these labs:

  1. Policies run before the backend. If any applicable policy blocks a request, the gateway answers right away and never calls the backend. The one exception is an Agent hooks policy that checks a tool's result: it runs after the tool and can withhold the result, but it can't undo the tool's work.
  2. Each block has a predictable status. 401 for missing or invalid credentials, 403 for IP filters and cost limits (LlmCostQuotaExceeded), 429 with Retry-After for token and request rate limits, and 400 for content safety.
  3. Everything is recorded. Allowed and blocked calls both show up in Monitoring, along with the policies that affected them.

A backend error is different from a policy block. When the serving backend fails, for example with 429 or a 5xx, the alias's failover rules can retry it or switch to another backend before the caller sees the error.

Models and aliases

A backend model is what a provider serves. An alias is what your clients call. Microsoft Foundry imports create an alias for each model by default, and you can add more. Because clients only know the alias, you can move it to a newer or cheaper model without a code change, and the alias keeps its own policies.

You can put models from Microsoft Foundry, Azure OpenAI, OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI, and custom OpenAI-compatible endpoints behind aliases. You try it in Route models from several providers behind aliases. Besides chat models, an alias can publish embedding models for search and retrieval, which you try in lab 26, or TypeSafe Jev decision models that return structured yes/no probabilities, choices, and rubric scores, which you try in lab 31. An alias can't mix chat, embedding, and decision models: all its backends must share an API endpoint.

Routing options

An alias can point to several backend models. Its routing settings decide which one serves each request:

Routing optionWhat it doesTry it
Priorities and weightsPrefers the available backends with the lowest priority number, and spreads requests across them by weightLab 22
Backend conditionsLimits which requests a backend can receive, with a CEL expression over the request, such as a headerLab 27
Auto model selectionUses Model Router to choose a smaller or larger model for each prompt, for your quality and cost objective. It's still emerging: see the November 6, 2026 releaseLab 28
Session affinitySends requests with the same session key, read by a CEL expression, to the same backend when possible, so a conversation can reuse the provider's prompt cacheSession affinity
Failover and retryRetries a failed backend or switches to the next eligible one, and can exclude a failing backend for a whileLab 29

Clients keep calling the same alias while you change any of these settings. See Models and routing and Routing order for how the settings combine.

MCP servers and tools

Agents need tools as much as models. An MCP server on the gateway gives agents one endpoint and one credential for tools from remote MCP servers, OpenAPI specifications, Foundry Toolbox, and MCP servers you import from Azure, such as the connector tools of a Connector Namespace. Tool names are namespaced per source, and tool availability decides which tools clients can see and call.

Policies and scopes

You add a policy at gateway scope, where it applies to every compatible asset including assets you add later, or to individual assets: a backend model, an alias, or an MCP server. Pick an asset below to see every policy that applies to it.

PolicyApplies toWhen it blocks
Token rate limitModels and aliases429 with Retry-After
Request rate limitModels, aliases, and MCP servers429 with Retry-After
Cost limitModels and aliases403 LlmCostQuotaExceeded
Content safetyModels, aliases, and MCP servers400
IP filterModels, aliases, and MCP servers403
Agent 365 tool threat protectionMCP serversDenies the tool call
Agent hooksMCP serversYour decision service denies the tool call before it runs, or withholds its result after it runs

At gateway scope, the policy type decides which assets are relevant. A gateway-scope cost limit, for example, tracks one budget across all models. An asset policy adds a control: it doesn't replace a policy the asset inherits from the gateway.

Agent hooks send the tool call's context to an external decision service that you run, and enforce its allow or deny verdict, for example to refuse refunds above a limit. They're still emerging: see the November 6, 2026 release and Agent hooks, then try them in lab 30.

Identities on both sides

The gateway separates who is calling from how the gateway reaches the backend. Callers use an API key or a Microsoft Entra token with one of the AI Gateway data-plane roles. The gateway uses its own managed identity, or a provider key it stores, to call backends. Callers never see backend credentials.

Managing the gateway is a third, separate permission. AI Gateway Developer can view the configuration of aliases and MCP servers, AI Gateway Contributor can manage model providers, aliases, MCP servers, routing, limits, and policies, and Contributor on the gateway resource can also change gateway settings, networking, identities, and API keys. None of these roles can call models or tools: that takes a data-plane role. See Control-plane access.

Observability

The gateway emits OpenTelemetry metrics, logs, and traces for model calls, embeddings, decisions, tool calls, routing, and policy decisions. Traces also mark model-stream and MCP operation failures as errors, even when the HTTP response was 200. Send them to Application Insights to light up the built-in Monitoring dashboard, or to any OTLP endpoint.

Take a portal tour

If you have a gateway, walk through the portal now. Otherwise, use the screenshots and come back after lab 01.

1

Open your gateway

Go to ai.gateway.azure.com and sign in. Use the gateway switcher in the top bar to pick a gateway. The left rail lists Home and Catalog, then Models, MCP servers, Policies, and Monitoring under Manage, and Access, Managed identities, and Networking under Secure. Press Ctrl+K, or ⌘+K on macOS, to open the command palette from anywhere.

2

Browse models and aliases

Select Models. It opens on Aliases, which lists each alias with its backend models, providers, API formats, and model selection. The Backend models tab lists each backend model with its aliases and provider.

Models opens on Aliases. The Backend models and Model providers tabs list what the aliases point to.
3

Look at an MCP server

Select MCP servers and open a server. The tool availability settings decide which tools clients can see.

MCP servers: Tool availability allows tools by exact name or name pattern, and Blocked tools removes tools from the list.
4

Start a policy

Select PoliciesAdd policy. The wizard asks for a Type, then a Scope (Gateway or Individual assets), then the Details. Select Cancel to leave without saving.

Policies: pick a Type, then a Scope, then the Details.
5

Open Monitoring

Select Monitoring. When Application Insights is connected, the Overview, Models, Tools, Policies, and Traces tabs show live traffic.

Monitoring: requests, outcomes, tokens, and estimated cost.

Verify

Check your understanding before you move on.

An app sends "model": "chat". What must chat be?

A callable alias on the gateway. The model field must exactly match an alias, not a provider deployment name.

A caller exceeds a token rate limit. Is the backend called?

No. The gateway answers 429 Too Many Requests with Retry-After and never forwards the request.

An alias has two backends, and the first one returns 503. Does the caller see the error?

Only if no failover rule on the alias matches the failure. A matching rule can retry the same backend, switch to the other backend, and exclude the failing one for a while. Without a matching rule, the gateway returns the 503.

You add an IP filter at gateway scope, then add a new model next week. Is the new model protected?

Yes. Gateway-scope policies apply to every compatible asset, including assets you add later.

Which credential does the gateway use to call a Microsoft Foundry model?

Its own managed identity, which needs the Foundry User role on the Foundry account. The caller's key or token is never forwarded.

Checkpoint

You can explain the request lifecycle, the difference between a backend model and an alias, and where gateway and asset policies apply.

Clean up

Nothing to clean up. This lab doesn't create resources.

Next steps

Short on time? The 5-minute quick start creates a gateway with a Foundry model in the portal or Azure Cloud Shell and makes your first call. Then pick the path that fits your role. Each path is a sequence of labs, and your progress is saved in this browser.

PathStart withYou'll learn to
30-minute essentialsProvision a gatewayCreate a gateway, call a model, and add MCP tools
Platform engineerProvision a gatewayBuild a governed, multi-provider gateway with code
App & agent developerFirst model callCall models and tools from SDKs, agents, and hosted agents
Security & complianceAPI keysControl access, content, tools, and threats
FinOps & operationsToken limitsCap tokens, set budgets, and charge back usage

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.