Azure AI Gateway Labs
Documentation Open portal
Agents · Lab 32Emerging

Use gateway models in Microsoft Foundry projects

Connect a Microsoft Foundry project to Azure AI Gateway, so that the project's playground, agents, and code call the platform team's models with the project's managed identity, with no model deployments or keys in the project. Then cap the project's tokens with a gateway policy and manage the connection.

Intermediate 35 min PortalCLIBicepTerraformCodeNotebook Updated
Notebook Source Edit

What you'll do

  • Explain the hub-and-spoke model, where Foundry projects consume models from a shared gateway
  • Give a Foundry project's managed identity the AI Gateway Models User role on the gateway
  • Add a gateway connection to a Foundry project in the Foundry portal, with the Azure CLI, with Bicep and azd, or with Terraform
  • Call the gateway's models from the Foundry playground, a Foundry agent, curl, PowerShell, and the Python SDK
  • Cap a project's tokens with a gateway policy and read the 429 that Foundry returns

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform, Code, Notebook

Your platform team runs one Azure AI Gateway for the company. It publishes the approved models behind aliases and governs every call with token limits, budgets, and logs. App teams build agents in their own Microsoft Foundry projects, and they shouldn't need model deployments, quota, or keys to get started. In this lab you connect an app team's Foundry project to the gateway, so that the project's playground, its agents, and its code call the gateway's aliases with the project's managed identity. Then you cap the project's tokens with a gateway policy and see how Foundry reports the limit.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway and model, the app team's Foundry project, the connection, and the roles, so you can skip this section. Run that track in Connect a Foundry project to the gateway first: it sets the variables that the next sections and Verify use. You need only the one-time setup above, Python 3.10 or later, and jq.

Note

Gateway connections in Microsoft Foundry are an emerging capability of Azure AI Gateway. Labels and behavior can change; see Connect Microsoft Foundry to AI Gateway and Connect Microsoft Foundry agents to AI Gateway.

  • A gateway with a model alias from Publish a Foundry model and make your first call. This lab calls it from a new Foundry project. The project doesn't deploy models of its own.
  • Permission to create a resource group and a Foundry resource in your subscription, and to assign roles on them and on the gateway, for example Owner, or Contributor with User Access Administrator.
  • Python 3.10 or later for the code track, PowerShell 7.4 or later for the PowerShell steps, and jq and curl 7.84 or later for the Bash steps.
  • Run the scripts and the sample from the lab's folder, ~/aigateway/labs/32-foundry-gateway-connection.

Set these variables once. Every track and the Verify section reuse them.

# Name your gateway from lab 01 and the alias from lab 02 that the app team's project calls
export RG="<resource-group>"
export GATEWAY="<gateway>"
export AI_GATEWAY_MODEL="<model-name>"
echo "📝 The app team's project will call alias $AI_GATEWAY_MODEL on gateway $GATEWAY in $RG"

# Read the gateway's resource ID, runtime URL, and region, and the backend model behind the alias
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
export LOCATION="$(az aigateway show -g "$RG" -n "$GATEWAY" --query location -o tsv)"
export MODEL_ID="$(az aigateway model-alias show -g "$RG" --gateway-name "$GATEWAY" -n "$AI_GATEWAY_MODEL" \
  --query "properties.backendModels[0].resourceId" -o tsv)"
echo "🆔 Gateway: $GATEWAY_ID"
echo "🌐 Gateway URL: $AI_GATEWAY_URL"
echo "🤖 $AI_GATEWAY_MODEL routes to ${MODEL_ID##*/modelProviders/}"

# Name the app team's resource group, Foundry resource, and project. The Foundry resource's name is also its
# subdomain, so it must be globally unique
export APP_RG="rg-aigw-app-team"
export APP_FOUNDRY="$GATEWAY-apps"
export PROJECT="app-team"
echo "📝 Project $PROJECT on Foundry resource $APP_FOUNDRY in $APP_RG ($LOCATION)"

# Foundry names the connection <gateway>-model-gateway and shows the model as <connection>/<alias>
export CONNECTION="$GATEWAY-model-gateway"
export FOUNDRY_PROJECT_ENDPOINT="https://$APP_FOUNDRY.services.ai.azure.com/api/projects/$PROJECT"
export FOUNDRY_MODEL="$CONNECTION/$AI_GATEWAY_MODEL"
echo "🌐 Project endpoint: $FOUNDRY_PROJECT_ENDPOINT"
echo "🤖 Model in Foundry: $FOUNDRY_MODEL"

How it works

The gateway is the hub and each Foundry project is a spoke. A gateway connection in the project points at the gateway's OpenAI-compatible endpoint and lists the aliases that the project may use. Foundry shows each alias as a connected model deployment named <connection>/<alias>, next to the project's own deployments. Nothing is deployed in the project: when the playground, an agent, or your code asks for that model, Foundry calls the gateway with a token of the project's managed identity. The gateway checks the identity's role, applies the alias's policies, such as a token limit, and routes the call to the backend model that the platform team publishes.

Connection settingValue in this lab
Name<gateway>-model-gateway. The Foundry portal sets it, and you can't change it
Connection typeAdd consuming project on the gateway's page in ManageGateways. Foundry stores the category ApiManagement and lists the connection as API Management
Base URL<gateway URL>/default/models/openai/v1, the gateway's OpenAI v1 endpoint
Upstream APIThe Responses API: Foundry adds /responses, with no api-version and no deployment in the path
AuthenticationManaged identity, stored as ProjectManagedIdentity, with the audience https://azure-api.net/aigateway
ModelsThe aliases that you pick, each with its model name and version. Foundry calls them <connection>/<alias>

Two identities need a role, and neither gets a key:

IdentityRoleScopeWhy
The project's managed identityAI Gateway Models UserThe gateway's default workspace, All assetsFoundry calls the gateway as the project. The Foundry portal doesn't assign this role: the platform team does, in the AI Gateway portal
You, or your app's identityFoundry UserThe app team's Foundry resourceTo use the project's playground and agents, and to call the project from code
CallerHow it names the model
Playground and agents in the Foundry portalThey pick <connection>/<alias> from the model list
CodePOST <project endpoint>/openai/v1/responses with a Microsoft Entra token for https://ai.azure.com, and "model": "<connection>/<alias>"
Agents in codePromptAgentDefinition(model="<connection>/<alias>") in the azure-ai-projects SDK

Each project calls with its own identity, so a token limit counted per Identity gives every project its own allowance on the same alias.

Connect a Foundry project to the gateway

Create the app team's Foundry project, give its managed identity access to the gateway's models, and add the gateway connection.

Choose your track
1

Create the app team's resource group

The Foundry portal's Create a project dialog lists only resource groups that exist, so create the app team's resource group first, in the gateway's region:

# Create the app team's resource group, in the gateway's region
az group create -n "$APP_RG" -l "$LOCATION" -o none &&
  echo "📦 Resource group $APP_RG is ready"
2

Create a Foundry project

  1. Open the Foundry portal, and sign in. If it shows No access, select your avatar › Switch, and pick the directory of your subscription.

  2. Select the project name ⌄ in the top bar › Create new project.

  3. Enter app-team in Project name, and open Advanced options:

    FieldValue
    Foundry resourceThe APP_FOUNDRY value, such as aigw-contoso-apps. The portal suggests <project>-resource
    RegionThe gateway's region, the LOCATION value
    SubscriptionYour subscription
    Resource grouprg-aigw-app-team
  4. Select Create. After about a minute, the portal shows All set. Let's build your agents. Select Let's go. The portal also adds a Log Analytics workspace and Application Insights to the resource group, and connects them to the project.

3

Read the project's identity

Read the project's managed identity, which the gateway authorizes, and give yourself Foundry User on the new Foundry resource, to use its playground and agents and to call it from code:

# Read the Foundry resource's ID and the project's identity
export APP_FOUNDRY_ID="$(az cognitiveservices account show -g "$APP_RG" -n "$APP_FOUNDRY" --query id -o tsv)"
export PROJECT_PRINCIPAL_ID="$(az cognitiveservices account project show -g "$APP_RG" -n "$APP_FOUNDRY" \
  --project-name "$PROJECT" --query identity.principalId -o tsv)"
echo "🆔 Project identity: $PROJECT_PRINCIPAL_ID"

# Give yourself Foundry User on the Foundry resource, to use the project's playground and agents and call it from code
az role assignment create --assignee-object-id "$(az ad signed-in-user show --query id -o tsv)" \
  --assignee-principal-type User --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" --scope "$APP_FOUNDRY_ID" -o none &&
  echo "🔐 You have the Foundry User role on $APP_FOUNDRY"
4

Give the project access to the gateway's models

The platform team grants this role in the AI Gateway portal. Select your gateway, then AccessEntra IDAdd role assignment:

  1. On Select identities, open the Managed identities tab. Filter Resource type to Microsoft Foundry project, search for app-team, check <Foundry resource>/app-team, and select Next.
  2. On Choose access, select Invoke gateway assets only, then Next.
  3. On Choose scope, select All models, then Next.
  4. On Review, select Assign.
The project's managed identity gets AI Gateway Models User on all assets.
Checkpoint

AccessEntra ID lists the project with the AI Gateway Models User role on All assets. The role reaches the runtime within about a minute.

5

Open the gateway in Foundry

Back in the Foundry portal, in the app-team project, select ManageGateways. The AI Gateway tab lists the AI Gateways of the Subscription at the top, with the number of Foundry resources behind each and its Gateway region. To find yours in a long list, type its name in Search, or pick the lab's Foundry resource in the Foundry resources filter. Select your gateway.

Foundry lists the AI Gateways in your subscription.

The gateway's page shows what the gateway publishes, on the Provider, Models, and MCP servers tabs. Foundry reads them from the gateway: you add providers and models, and set policies and access, in the AI Gateway portal, which Open AI Gateway portal opens. Here, you choose the projects that consume the gateway's models, through connected models.

Add a consuming project to the gateway.
6

Select the project

Select Add consuming project. Connect project to gateway opens on Project:

  1. Select Select. In Select Foundry project, pick your Foundry resource, the APP_FOUNDRY value, in Resource, and app-team in Project. Select Select.

    Pick the project that consumes the gateway's models.
  2. The wizard fills Base URL with the gateway's OpenAI v1 endpoint, …/default/models/openai/v1. Check Use the Responses API for upstream requests: Request URL then ends with /responses. Select Next.

    Foundry sends the project's requests to the gateway's Responses API.
7

Choose managed identity

On Authentication, select Managed identity: the wizard starts on API key. Foundry reminds you that the project's identity needs the AI Gateway Models User role, which you assigned. Audience shows https://azure-api.net/aigateway, the audience of the gateway's tokens. Select Next.

The project's managed identity authenticates to the gateway. No key is stored.
8

Pick the models

On Model configuration, select Select models. Configure models lists the gateway's models with their Type and Endpoints. Check your alias, and select Select.

Foundry discovers the models that the gateway publishes.

Models to be connected lists the alias with its version and Foundry resource. Select Connect: the portal returns to the gateway's page.

Select Connect to add the connection to the project.
Checkpoint

On the gateway's Models tab, your alias counts app-team under Consuming projects. Pick app-team in All consuming projects to list the models that the project consumes. Foundry names the connection <gateway>-model-gateway, and the model's Foundry name is <gateway>-model-gateway/<alias>.

The gateway's models, and the projects that consume them.

Call gateway models from Foundry

Call the alias through the project: from the playground, from a Foundry agent, and from code. Foundry calls the gateway with the project's managed identity, and the gateway routes the alias to its backend model. New roles reach the gateway within about a minute: if the first call returns 403 with Access to model is denied, wait a minute and try again.

Choose your track
1

Chat in the playground

In the Foundry portal, select BuildModelsDeploymentsConnections. The list shows your alias with its model, version, Connection name, and the gateway it's Connected to.

The project lists the gateway's model as a connected model.

Select your alias's row, then Open in playground. The playground's title is the model's Foundry name, such as aigw-contoso-model-gateway/gpt-5-mini, with a Connections tag. Ask In one sentence, what does an AI gateway do for an app team?

The playground calls the gateway's alias through the project's connection.

Save as agent is turned off for connected models. Create the agent from Agents instead.

2

Build an agent on the gateway model

  1. Select AgentsNew agentBuild an agent.
  2. In Create an agent, enter gateway-helper in Agent name, keep Interaction mode on Text, and select Create agent and open playground. It takes about 20 seconds.
  3. The agent starts on model-router. Open Model, and pick your alias under Connections: the picker shows it as AI Gateway · <gateway>-model-gateway. Select Save, which saves version 2 of the agent.
  4. The agent starts with a toolbox that has the Web search tool, so ask it Without searching the web, why route an agent's model calls through a gateway?
A Foundry prompt agent runs on the gateway's model.
Checkpoint

The playground and the agent answer, and the model picker shows <gateway>-model-gateway/<alias>. The project has no model deployments of its own.

Govern Foundry traffic with a token limit

The platform team governs the project's calls at the gateway, without touching the project. Add a token limit of 300 tokens per minute for each caller identity to the backend model behind the alias. Each project calls with its own managed identity, so each project gets its own 300 tokens. The limit is deliberately low, so that two or three requests reach it.

Choose your track
1

Add the token limit

In the AI Gateway portal, select your gateway, then PoliciesAdd policy:

  1. On Type, under Cost and rate limits, select Token rate limit, then Next.
  2. On Scope, select Individual assets, select the row of type Model for the backend model behind your alias, then Next.
  3. On Details, set Time period to Per minute, Allowed tokens to 300, and Counter dimensions to Identity. Select Create. The portal confirms Policy created.

The limit reaches the runtime in up to a minute.

2

Reach the limit in the playground

Wait a minute, then open the playground of your connected model, as in Call gateway models from Foundry, and ask Write a short paragraph about token limits. until it shows an error, usually on the third or fourth request. The answer that crosses the limit still arrives, and the next request gets the error:

Foundry passes the gateway's 429 to the playground.
Checkpoint

The playground shows Model could not be accessed - Received 429 status code from model, with the gateway's message Token limit is exceeded. Try again in N seconds.

The gateway counts the tokens of each response, so the request that crosses the limit still passes, and the next ones get 429 until the minute ends. Foundry passes the gateway's 429 and its Retry-After header on, and wraps the gateway's message in its own error, with the code rate_limit_exceeded:

Text
Rate limit exceeded. Model could not be accessed - Received 429 status code from model. ModelGateway response:
{'statusCode': 429, 'message': 'Token limit is exceeded. Try again in 35 seconds.'} Request ID: <gateway request ID>

The request ID is the gateway's, so the platform team can find the call in the gateway's logs. The error ends with a link to Azure OpenAI quotas, but the limit is the gateway's policy, not a Foundry quota.

Verify

1

Check the project's role on the gateway

# List the roles of the project's identity on the gateway's default workspace
az role assignment list --assignee "$PROJECT_PRINCIPAL_ID" --scope "$GATEWAY_ID/workspaces/default" \
  --query "[].{role: roleDefinitionName, scope: scope}" -o json

Expected output (your names differ):

Text
[
  {
    "role": "AI Gateway Models User",
    "scope": "/subscriptions/<subscription-id>/resourceGroups/rg-aigw-lab32t/providers/Microsoft.ApiManagement/aigateways/aigw-guximp/workspaces/default"
  }
]
2

Call Chat Completions through the project

Foundry serves connected models through the Responses API. The project's Chat Completions endpoint doesn't find them:

# Get a Microsoft Entra token for Foundry
TOKEN="$(az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv)"

# Send a Chat Completions request for the connected model through the project
curl -sS "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"model\": \"$FOUNDRY_MODEL\", \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}]}" |
  jq -r '"🚦 \(.error.code): \(.error.message)"'

Expected output:

Text
🚦 DeploymentNotFound: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.

The message is Foundry's generic one: the model is a connection, not a deployment, so waiting doesn't help.

3

Read a 429 through the project

With the token limit from Govern Foundry traffic with a token limit in place, send requests until the gateway answers 429, and read Foundry's error:

# Get a Microsoft Entra token for Foundry, and build one request that uses about 150 to 300 tokens
TOKEN="$(az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv)"
BODY="$(jq -nc --arg model "$FOUNDRY_MODEL" '{model: $model, input: "Write a short paragraph about token limits."}')"

# Send up to six requests, and stop at the first 429
for i in 1 2 3 4 5 6; do
  REPLY="$(curl -sS -w '\n%{http_code} %header{retry-after}' "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY")"
  read -r CODE WAIT <<< "$(tail -n 1 <<< "$REPLY")"
  [ "$CODE" = 429 ] && break
done

# Print the status, the Retry-After header, and Foundry's error code
echo "🚦 #$i: $CODE, retry after $WAIT s"
sed '$d' <<< "$REPLY" | jq -r '"🚨 \(.error.code): \(.error.message | split(" Request ID")[0])"'

On the code track, the sample sends a burst with the SDK's retries off. By default, the OpenAI client retries a 429 twice before it raises the error:

Python
# Send four requests in a row with the SDK, and print each status
python infra/call_model.py --burst 4

Expected output of the Bash loop (your values differ):

Text
🚦 #3: 429, retry after 35 s
🚨 rate_limit_exceeded: Rate limit exceeded. Model could not be accessed - Received 429 status code from model. ModelGateway response: {'statusCode': 429, 'message': 'Token limit is exceeded. Try again in 35 seconds.'}

Expected output of the Python burst (your values differ):

Text
1: 200 290 tokens
2: 429 Error code: 429 - Rate limit exceeded. Model could not be accessed - Received 429 status code from model. ModelGateway response: {'statusCode': 429, 'message': 'Token limit is exceeded. Try again in 23 seconds.'}
3: 429 Error code: 429 - Rate limit exceeded. Model could not be accessed - Received 429 status code from model. ModelGateway response: {'statusCode': 429, 'message': 'Token limit is exceeded. Try again in 19 seconds.'}
4: 429 Error code: 429 - Rate limit exceeded. Model could not be accessed - Received 429 status code from model. ModelGateway response: {'statusCode': 429, 'message': 'Token limit is exceeded. Try again in 16 seconds.'}
Checkpoint

The project's identity holds AI Gateway Models User on the gateway's default workspace, Chat Completions gets DeploymentNotFound, and the token limit returns rate_limit_exceeded with the gateway's Token limit is exceeded message.

Go further

  • Connect more projects. Each app team's project needs its own connection and its own role on the gateway. Run Add consuming project again on the gateway's page for each project: the Models tab then counts every project that consumes each model. With the Identity counter, a second project still had its full allowance while the first one got 429.

  • Connect from the project's model list instead. BuildModelsConnectionsConnect to a modelGatewayAI Gateway tier creates the same kind of connection, but names it after the gateway, so the model's Foundry name is <gateway>/<alias>. Pick one path per project, and use its name in your code.

  • Manage the connection. ManageProject detailsConnections lists the gateway connection with the category ApiManagement, the authentication Project Managed Identity, and No key available. Use Edit connection to add aliases, or Delete connection to remove the gateway's models from the project.

    Project details lists the connection, which holds no key.
  • Use an API key where managed identity isn't an option. The wizard's Authentication step also offers API key: paste a gateway key from AccessAPI keys, as in Publish a Foundry model and make your first call. Foundry stores the key in the connection, and every caller of the project shares it.

  • Make the gateway the only path. Assign Azure Policy on the spoke projects to block direct model deployments, and let only the gateway's managed identity call the backend models, as in How Foundry and gateway controls work together.

  • Know the limits. Connected models work with prompt agents in Foundry Agent Service, but not with the Microsoft Copilot harness, Foundry IQ, agent optimization, agent memory, guardrails, real-time audio, or vision. For private networking, see Configure private AI gateway networking.

  • Run your own agent code instead. A hosted agent can call the gateway directly with its own identity, as in Deploy a Foundry hosted agent behind AI Gateway.

Clean up

Remove the token limit and the project's role on the gateway, then delete the app team's project and Foundry resource. Remove the role first, while the project's identity still exists.

If the model had a token limit before this lab, such as the 10,000 tokens a minute per identity that lab 02's Bicep track adds, set it back instead of removing it. In the portal, delete the limit, then add it again as in Govern Foundry traffic with a token limit, with Allowed tokens set to 10000. In the Azure CLI, run set-token-limit again with --token-limit 10000 instead of policy remove.

Choose your track
1

Remove the token limit and the role

In the AI Gateway portal, select your gateway:

  1. Select Policies. In the Token rate limit row, select More actionsDelete, and confirm.
  2. Select AccessEntra ID. In the project's row, select More actionsRemove role assignment, and confirm with Remove.
2

Delete the app team's project

Delete the project with its connection, then the Foundry resource, purge it to free its name, and delete the resource group:

# Delete the project with its connection, then the app team's Foundry resource
az cognitiveservices account project delete -g "$APP_RG" -n "$APP_FOUNDRY" --project-name "$PROJECT" &&
  echo "🚮 Deleted project $PROJECT and its connection"
az cognitiveservices account delete -g "$APP_RG" -n "$APP_FOUNDRY" &&
  echo "🚮 Deleted Foundry resource $APP_FOUNDRY"

# Purge the deleted Foundry resource to free its name, then delete the app team's resource group
az cognitiveservices account purge -g "$APP_RG" -n "$APP_FOUNDRY" -l "$LOCATION" &&
  echo "🚮 Purged $APP_FOUNDRY"
az group delete -n "$APP_RG" --yes &&
  echo "🚮 Deleted resource group $APP_RG"

If you built the gateway-helper agent in the portal, it goes with the project. Keep the gateway from lab 01: the next labs reuse it.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.