Your team wants to run an agent in Microsoft Foundry, but the platform team needs every model call and every tool call to be governed and visible in one place, with no keys to rotate or leak. In this lab you deploy a small Microsoft Agent Framework agent to Foundry Hosted Agents. Its Foundry project has no model deployments, and the agent holds no secrets: it reaches its model and its tools only through AI Gateway, which authorizes its Microsoft Entra identities with an Azure role.
Before you begin
One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.
- Choose where to run commandsEvery track except the portal
- Sign in to AzureOn your own machine. Cloud Shell is already signed in.
- Install the
aigatewayextensionThe Azure CLI tracks and the Verify steps - Clone the labs repoThe Bicep + azd and Terraform tracks, the notebooks, and the code samples
You've done the one-time setup that this lab uses.
- Complete Publish a Foundry model and make your first call and
Federate MCP tools behind one endpoint. The agent calls the alias from lab 02 and the
learnMCP server from lab 04. - An Azure subscription where you can create a resource group and a Foundry resource, for example as Owner. azd gives you the Foundry User role on the project it creates, so that you can run the agent locally.
- Permission to create role assignments on the gateway: Owner, User Access Administrator, or Role Based Access Control Administrator. Contributor can manage the gateway but can't grant roles.
- The Azure Developer CLI (azd) 1.37.0 or later, and Python 3.13 or later. With uv installed, azd uses it to create the agent's virtual environment.
azd creates a Foundry resource, and the hosted agent and its model calls are billable. Run Clean up when you're done.
How it works
The agent lives in infra/, a complete azd project:
| File | What it does |
|---|---|
infra/azure.yaml | Declares a Foundry project with no model deployments; gateway-mcp, a connection to the gateway's MCP server; gateway-tools, a Foundry toolbox that exposes that connection; and the hosted agent gateway-agent: Python 3.13, code deploy, 0.5 CPU, 1 GiB, Responses protocol |
infra/src/gateway-agent/main.py | The agent: a model client that points at the gateway, and the toolbox |
infra/src/gateway-agent/requirements.txt | Pinned dependencies, compiled from requirements.in. Foundry builds the agent from this file. |
The agent has two outbound paths. Both end at AI Gateway, and neither uses a key:
- Model. An
OpenAIChatClientpoints at the gateway's model endpoint,/default/models/openai/v1/, and sends your lab 02 alias asmodel. Instead of a key, it sends a Microsoft Entra token for the gateway's scope,https://ai.gateway.azure.com/.default, asAuthorization: Bearer. In Foundry, the token belongs to the agent's own identity. Locally, it belongs to you. - Tools.
FoundryToolboxcalls thegateway-toolstoolbox in Foundry. The toolbox reaches the gateway's/default/toolservers/learn/mcpendpoint through thegateway-mcpconnection, which signs in with the Foundry project's managed identity for the audiencehttps://ai.gateway.azure.com. The agent never holds a credential for the MCP server.
Three identities call the gateway, and each one needs the AI Gateway User role, which lets it invoke models and call MCP tools:
| Identity | Shown in AccessEntra ID as | Calls | Exists |
|---|---|---|---|
| The agent's identity, a Microsoft Entra Agent ID | <account>-<project>-gateway-agent-AgentIdentity (Service principal) | The model endpoint, from Foundry | After the first deploy |
| The Foundry project's managed identity | <account>/projects/<project> (Service principal) | The MCP endpoint, through the connection | After provisioning |
| You | Your account (User) | The model endpoint, when the agent runs locally | Already |
So you grant the role in two rounds: the project and yourself before you deploy, then the agent once Foundry has created its identity.
| Piece | Name in this lab | Owner |
|---|---|---|
| Model alias | Your alias from lab 02, in AI_GATEWAY_MODEL | AI Gateway |
| MCP server | learn, at /default/toolservers/learn/mcp | AI Gateway |
| Role | AI Gateway User, caad7a21-134e-43f4-a550-c5ffc17e051c, on the gateway | AI Gateway |
| Foundry resource and project | In resource group rg-<env>; the project is named after the azd environment | Foundry, created by azd |
| Connection and toolbox | gateway-mcp and gateway-tools | Foundry, created by azd |
| Hosted agent | gateway-agent | Foundry, created by azd |
Foundry runs the agent. AI Gateway decides what each identity may call, counts its tokens, and records every call.
Set your variables
Pick one track. You use the same track to grant the roles later.
Load the gateway values in your shell
Copy the gateway's runtime URL from its Overview page in the AI Gateway portal, and set the alias from lab 02:
# Set the gateway's runtime URL from its Overview page, and the alias that the agent sends as its model
export AI_GATEWAY_URL="<gateway-url>"
export AI_GATEWAY_MODEL="<model-name>" # the alias from lab 02
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"# Set the gateway's runtime URL from its Overview page, and the alias that the agent sends as its model
$env:AI_GATEWAY_URL = "<gateway-url>"
$env:AI_GATEWAY_MODEL = "<model-name>" # the alias from lab 02
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"Read the gateway values
# Set your resource group, your gateway, and the alias that the agent sends as its model
export RG="<resource-group>"
export GATEWAY="<gateway>"
export AI_GATEWAY_MODEL="<model-name>" # the alias from lab 02
echo "📝 Gateway $GATEWAY in $RG, alias $AI_GATEWAY_MODEL"
# Read the gateway's runtime URL, which the agent calls for its model and its tools
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
# Read the gateway's resource ID, the scope of the role assignments
GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
echo "🆔 Gateway ID: $GATEWAY_ID"# Set your resource group, your gateway, and the alias that the agent sends as its model
$RG = "<resource-group>"
$GATEWAY = "<gateway>"
$env:AI_GATEWAY_MODEL = "<model-name>" # the alias from lab 02
Write-Host "📝 Gateway $GATEWAY in $RG, alias $env:AI_GATEWAY_MODEL"
# Read the gateway's runtime URL, which the agent calls for its model and its tools
$env:AI_GATEWAY_URL = "https://$(az aigateway show -g $RG -n $GATEWAY --query properties.frontend.defaultHostname -o tsv)"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
# Read the gateway's resource ID, the scope of the role assignments
$GATEWAY_ID = az aigateway show -g $RG -n $GATEWAY --query id -o tsv
Write-Host "🆔 Gateway ID: $GATEWAY_ID"Your shell has AI_GATEWAY_URL and AI_GATEWAY_MODEL, and, on the Azure CLI track, GATEWAY_ID.
Configure the azd project
Get the lab files
Move into the lab's infra folder and install the Foundry extensions for azd:
# Move into the lab's azd project
cd ~/aigateway/labs/16-foundry-hosted-agent/infra &&
echo "📂 Lab folder: ~/aigateway/labs/16-foundry-hosted-agent/infra"
# Install the Foundry extensions for azd, which add the azd ai agent, connection, and toolbox support
azd ext install microsoft.foundry &&
echo "📦 The Foundry extensions for azd are installed"microsoft.foundry installs the Foundry extensions for azd, including azure.ai.agents, azure.ai.connections,
and azure.ai.toolboxes, the three that azure.yaml requires. If you installed them before,
run azd ext upgrade --all.
Create an azd environment
The environment name also names the resource group, rg-<env>, and the Foundry project. Choose a region where
Foundry Hosted Agents are available, for example eastus2. The environment uses the subscription that the Azure CLI
signed in to, the same one as your gateway.
# Create the azd environment, which also names the resource group, rg-aigw-lab16, and the Foundry project
azd env new aigw-lab16 --location eastus2 --subscription "$(az account show --query id -o tsv)" &&
echo "📝 azd environment aigw-lab16 deploys to eastus2"Pass the gateway settings to Foundry
azure.yaml uses three azd environment values. The connection gets the MCP endpoint and the
gateway's audience, and the agent gets the model endpoint and the toolbox's endpoint, which azd fills in on deploy:
gateway-mcp:
host: azure.ai.connection
category: RemoteTool
authType: ProjectManagedIdentity
audience: https://ai.gateway.azure.com
target: ${AI_GATEWAY_URL}/default/toolservers/${AI_GATEWAY_MCP_SERVER}/mcp
gateway-agent:
env:
AI_GATEWAY_URL: ${AI_GATEWAY_URL}
AI_GATEWAY_MODEL: ${AI_GATEWAY_MODEL}
TOOLBOX_ENDPOINT: ${TOOLBOX_GATEWAY_TOOLS_MCP_ENDPOINT}Set them from your shell:
# Store the gateway's runtime URL and the alias in the azd environment
azd env set AI_GATEWAY_URL "$AI_GATEWAY_URL" &&
echo "📝 AI_GATEWAY_URL: $AI_GATEWAY_URL"
azd env set AI_GATEWAY_MODEL "$AI_GATEWAY_MODEL" &&
echo "📝 AI_GATEWAY_MODEL: $AI_GATEWAY_MODEL"
# Point the connection at the learn MCP server from lab 04
azd env set AI_GATEWAY_MCP_SERVER learn &&
echo "📝 AI_GATEWAY_MCP_SERVER: learn"# Store the gateway's runtime URL and the alias in the azd environment
azd env set AI_GATEWAY_URL $env:AI_GATEWAY_URL
if ($?) { Write-Host "📝 AI_GATEWAY_URL: $env:AI_GATEWAY_URL" }
azd env set AI_GATEWAY_MODEL $env:AI_GATEWAY_MODEL
if ($?) { Write-Host "📝 AI_GATEWAY_MODEL: $env:AI_GATEWAY_MODEL" }
# Point the connection at the learn MCP server from lab 04
azd env set AI_GATEWAY_MCP_SERVER learn
if ($?) { Write-Host "📝 AI_GATEWAY_MCP_SERVER: learn" }azd keeps the values in .azure/aigw-lab16/.env, which Git ignores. None of them is a secret.
Provision the Foundry project
# Create the resource group, the Foundry resource, and a project with hosted agents enabled
azd provision &&
echo "🏭 The Foundry project is ready"It takes about two minutes, and ends with:
SUCCESS: Your application was provisioned in Azure in 1 minute 22 seconds.
🏭 The Foundry project is readyazd provision succeeded, and azd env get-value AZURE_AI_PROJECT_ID prints the project's resource ID.
Give the project and yourself access
The project's managed identity calls the MCP endpoint for the toolbox, and you call the model endpoint when you run the agent locally. Grant both AI Gateway User on the gateway, with the track you picked.
Find the identities
Print what to look for in the portal: your object ID, and the project's name as the Managed identities tab shows it:
# Print your object ID, and the name of the Foundry project's managed identity
echo "👤 Your object ID: $(az ad signed-in-user show --query id -o tsv)"
echo "🏭 Foundry project: $(azd env get-value AZURE_AI_ACCOUNT_NAME)/$(azd env get-value AZURE_AI_PROJECT_NAME)"Select both identities
- In the AI Gateway portal, open your gateway and select AccessEntra IDAdd role assignment.
- On Select identities, in Search by name, email, or object ID, enter your object ID and check your account.
- Open the Managed identities tab. In Search managed identities, enter the project's name, and check
<account>/<project>, a System-assigned managed identity. The wizard keeps both selections: it shows 2 selected. Select Next.
Choose access and scope
- On Choose access, select Invoke gateway assets only, then Next.
- On Choose scope, select All models and tools, then Next.
- On Review, Role assignments to create lists both identities with the role AI Gateway User and the scope All assets. Select Assign.
Assign the role
# Read the object IDs of the Foundry project's managed identity and of your account
PROJECT_PRINCIPAL_ID="$(az resource show --ids "$(azd env get-value AZURE_AI_PROJECT_ID)" \
--query identity.principalId -o tsv)"
MY_OBJECT_ID="$(az ad signed-in-user show --query id -o tsv)"
echo "🏭 Project identity: $PROJECT_PRINCIPAL_ID"
echo "👤 Your object ID: $MY_OBJECT_ID"
# Grant both AI Gateway User on the gateway: the project calls the MCP endpoint, and you call the model locally
az role assignment create --assignee-object-id "$PROJECT_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope "$GATEWAY_ID" -o none &&
echo "🔐 The Foundry project has AI Gateway User on $GATEWAY"
az role assignment create --assignee-object-id "$MY_OBJECT_ID" --assignee-principal-type User \
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope "$GATEWAY_ID" -o none &&
echo "🔐 You have AI Gateway User on $GATEWAY"# Read the object IDs of the Foundry project's managed identity and of your account
$PROJECT_PRINCIPAL_ID = az resource show --ids (azd env get-value AZURE_AI_PROJECT_ID) `
--query identity.principalId -o tsv
$MY_OBJECT_ID = az ad signed-in-user show --query id -o tsv
Write-Host "🏭 Project identity: $PROJECT_PRINCIPAL_ID"
Write-Host "👤 Your object ID: $MY_OBJECT_ID"
# Grant both AI Gateway User on the gateway: the project calls the MCP endpoint, and you call the model locally
az role assignment create --assignee-object-id $PROJECT_PRINCIPAL_ID --assignee-principal-type ServicePrincipal `
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope $GATEWAY_ID -o none
if ($?) { Write-Host "🔐 The Foundry project has AI Gateway User on $GATEWAY" }
az role assignment create --assignee-object-id $MY_OBJECT_ID --assignee-principal-type User `
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope $GATEWAY_ID -o none
if ($?) { Write-Host "🔐 You have AI Gateway User on $GATEWAY" }If you already have AI Gateway User on the gateway, for example from the quick start, the second command succeeds without a change.
AccessEntra ID lists your account and <account>/projects/<project> with the role AI Gateway User,
and Models and MCP servers under Access.
Read the agent
main.py is the whole agent. The model client and the toolbox are the only
places it talks to the network, and neither holds a key:
credential = DefaultAzureCredential(
exclude_managed_identity_credential=not os.environ.get("FOUNDRY_HOSTING_ENVIRONMENT"),
)
get_gateway_token = get_bearer_token_provider(credential, AI_GATEWAY_SCOPE)
async def gateway_token() -> str:
return get_gateway_token()
client = OpenAIChatClient(
model=model,
base_url=f"{gateway_url}/default/models/openai/v1/",
api_key=gateway_token,
)
tools = FoundryToolbox(credential, timeout=60)DefaultAzureCredentialsigns in as the agent's identity in Foundry, which setsFOUNDRY_HOSTING_ENVIRONMENTin the agent's container. Locally, it skips the managed identity probe and uses your Azure CLI or azd sign-in.- The OpenAI SDK calls
gateway_tokenbefore each request and sends the result asAuthorization: Bearer, so the token refreshes on its own. FoundryToolboxreads the toolbox's endpoint fromTOOLBOX_ENDPOINT, and signs in to Foundry with the same credential. The toolbox, not the agent, signs in to the gateway's MCP endpoint.HandshakeClientopens the toolbox connection with the MCPinitializehandshake. Otherwise the MCP SDK picks the stateless version2026-07-28, which the toolbox offers, and the toolbox speaks it to the gateway'slearnserver. That server keeps the versions of its Microsoft Learn backend, so it answers with403.- The instructions ask the model to answer from the search results, and to fetch a whole article only when they
don't answer the question. An article is long: with the 10,000 tokens per minute that the quick start and
lab 02 give the model deployment, the model call that reads it can fail with
429.
ResponsesHostServer serves the agent on the Responses protocol at port 8088, the same way locally and in Foundry,
and creates the agent for each request. The agent sets store to false on its model calls, because the hosting
server already keeps the conversation history.
"""A Foundry hosted agent whose model calls and MCP tool calls both go through AI Gateway, without keys.
The agent has no Foundry model deployment and no direct route to a tool:
- Model: it calls the gateway's OpenAI-compatible model endpoint with an alias and a Microsoft Entra token for the
gateway's audience.
- Tools: it calls a Foundry toolbox, which reaches the gateway's MCP server through a RemoteTool connection that signs
in with the Foundry project's managed identity.
In Foundry, the tokens come from the agent's managed identity. Locally, `azd ai agent run` uses your Azure CLI sign-in.
To run it without azd, copy .env.example to .env, install requirements.txt in a virtual environment, and run
`python main.py`.
"""
import os
import mcp
from agent_framework import Agent
from agent_framework.openai import OpenAIChatClient
from agent_framework_foundry_hosting import FoundryToolbox, ResponsesHostServer
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from dotenv import load_dotenv
load_dotenv()
class HandshakeClient(mcp.Client):
"""An MCP client that opens its connection with the initialize handshake.
By default, the MCP SDK negotiates the stateless MCP version 2026-07-28 when the server offers it, and Foundry
toolboxes do. The toolbox then speaks 2026-07-28 to the gateway's MCP server, and a server that supports only
earlier versions, like one whose backend is the Microsoft Learn MCP server, answers with HTTP 403. The handshake
keeps the whole chain on a version that the gateway's MCP server supports.
"""
def __init__(self, *args, mode: str = "legacy", **kwargs) -> None:
super().__init__(*args, mode=mode, **kwargs)
# Agent Framework creates its MCP clients from mcp.Client, so this applies to the toolbox connection.
mcp.Client = HandshakeClient
# The token scope for AI Gateway's runtime endpoints
AI_GATEWAY_SCOPE = "https://ai.gateway.azure.com/.default"
INSTRUCTIONS = """
You answer questions about Microsoft and Azure technologies.
- Search the Microsoft Learn tools before you answer, and cite the URLs you used.
- Answer from the search results. Fetch a whole article only when they don't answer the question: articles are long,
and their tokens count toward the model's limits.
- If the tools don't cover the question, say so instead of guessing.
- Keep answers under 150 words.
"""
def required(name: str) -> str:
value = os.environ.get(name, "").strip()
if not value:
raise RuntimeError(f"Set {name}. With azd, run: azd env set {name} <value>")
return value
# Foundry sets FOUNDRY_HOSTING_ENVIRONMENT in the agent's container. Locally, skip the managed identity probe, so that
# the credential goes straight to your Azure CLI or azd sign-in.
credential = DefaultAzureCredential(
exclude_managed_identity_credential=not os.environ.get("FOUNDRY_HOSTING_ENVIRONMENT"),
)
def create_agent() -> Agent:
gateway_url = required("AI_GATEWAY_URL").rstrip("/")
model = required("AI_GATEWAY_MODEL")
get_gateway_token = get_bearer_token_provider(credential, AI_GATEWAY_SCOPE)
async def gateway_token() -> str:
return get_gateway_token()
# The SDK sends the token as Authorization: Bearer, and asks for a fresh one before each call.
client = OpenAIChatClient(
model=model,
base_url=f"{gateway_url}/default/models/openai/v1/",
api_key=gateway_token,
)
# FoundryToolbox reads the toolbox endpoint from TOOLBOX_ENDPOINT, and signs in to Foundry with the same credential.
tools = FoundryToolbox(credential, timeout=60)
return Agent(
client=client,
name="gateway-docs-agent",
instructions=INSTRUCTIONS,
tools=tools,
# Foundry keeps the conversation history, so the model call doesn't need to store it.
default_options={"store": False},
)
if __name__ == "__main__":
# The server creates the agent for each request, and closes its toolbox connection afterward.
ResponsesHostServer(agent=create_agent).run()On GitHub, open main.py and azure.yaml.
Deploy to Foundry
Deploy the connection, the toolbox, and the agent
# Create the MCP connection and the toolbox, upload the agent's code, and create the hosted agent
azd deploy --all &&
echo "🚀 The hosted agent gateway-agent is deployed to Foundry"azd creates the gateway-mcp connection, gives you Foundry User on the project for the toolbox, creates the
gateway-tools toolbox, uploads src/gateway-agent as a code package, and creates the hosted agent. Foundry builds
it from requirements.txt and gives it its own identity. It takes about three minutes:
gateway-mcp: Deploying (Upserting connection "gateway-mcp") [1s]
gateway-tools: Deploying (Assigning Foundry User on the toolbox's Foundry project.) [9s]
gateway-tools: Deploying (Upserting toolbox "gateway-tools") [31s]
gateway-agent: Deploying (Creating new agent from code package) [34s]
gateway-agent: Done [2m38s]
...
SUCCESS: Your application was deployed to Azure in 2 minutes 38 seconds.
🚀 The hosted agent gateway-agent is deployed to FoundryThe first deploy can also print WARNING: gateway-agent (predeploy): AZURE_CONTAINER_REGISTRY_ENDPOINT not set; skipping ACR role check. The agent is deployed as code, so the deploy still succeeds.
Give the agent access
Foundry created the agent's identity on deploy. It stays the same when you deploy new versions, so you grant it once. Use the track you picked.
Find the agent's identity
# Print the object ID of the agent's identity, which azd saved on deploy
echo "🤖 Agent identity: $(azd env get-value AGENT_GATEWAY_AGENT_INSTANCE_IDENTITY_PRINCIPAL_ID)"Assign the role
- Select AccessEntra IDAdd role assignment.
- On Select identities, in Search by name, email, or object ID, enter the agent's object ID. Check
<account>-<project>-gateway-agent-AgentIdentity, a Service principal, and select Next. - On Choose access, select Invoke gateway assets only, then Next.
- On Choose scope, select All models and tools, then Next.
- On Review, select Assign.
Assign the role
# Read the object ID of the agent's identity, which azd saved on deploy
AGENT_PRINCIPAL_ID="$(azd env get-value AGENT_GATEWAY_AGENT_INSTANCE_IDENTITY_PRINCIPAL_ID)"
echo "🤖 Agent identity: $AGENT_PRINCIPAL_ID"
# Grant the agent AI Gateway User on the gateway, so that it can call the model
az role assignment create --assignee-object-id "$AGENT_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope "$GATEWAY_ID" -o none &&
echo "🔐 The agent has AI Gateway User on $GATEWAY"# Read the object ID of the agent's identity, which azd saved on deploy
$AGENT_PRINCIPAL_ID = azd env get-value AGENT_GATEWAY_AGENT_INSTANCE_IDENTITY_PRINCIPAL_ID
Write-Host "🤖 Agent identity: $AGENT_PRINCIPAL_ID"
# Grant the agent AI Gateway User on the gateway, so that it can call the model
az role assignment create --assignee-object-id $AGENT_PRINCIPAL_ID --assignee-principal-type ServicePrincipal `
--role caad7a21-134e-43f4-a550-c5ffc17e051c --scope $GATEWAY_ID -o none
if ($?) { Write-Host "🔐 The agent has AI Gateway User on $GATEWAY" }The role takes a couple of minutes to apply. Run the agent locally in the meantime.
AccessEntra ID lists three identities with AI Gateway User: you, the project, and
<account>-<project>-gateway-agent-AgentIdentity.
Run the agent locally
Start the agent
From the infra folder:
# Start the agent locally: azd installs requirements.txt in a virtual environment and opens Agent Inspector
echo "🚀 Starting the agent on http://localhost:8088 (Ctrl+C to stop)"
azd ai agent runazd creates a virtual environment, installs requirements.txt, starts the agent on http://localhost:8088 with the
values from the azd environment, and opens Agent Inspector, where you can chat with the agent and see its tool
calls. Add --no-client to skip Agent Inspector.
Ask a question
In a second terminal, from the same folder:
# Ask the local agent a question: it calls the model as you, and the Learn tools through the toolbox
echo "🤖 The local agent's answer:"
azd ai agent invoke --local "Find the Microsoft Learn article that explains Azure API Management key concepts. Reply with its title and URL in one sentence."Expected output: an answer that cites a learn.microsoft.com URL, for example:
🤖 The local agent's answer:
Target: localhost:8088 (local)
...
[local] The Microsoft Learn page "What is Azure API Management?" (Key concepts) explains Azure API Management key concepts: https://learn.microsoft.com/azure/api-management/api-management-key-conceptsBehind that answer, the agent made a model call that asked for a tool, one or more Learn tool calls through the toolbox, and a model call that wrote the answer, all through the gateway. The model calls carried your token, and the tool calls carried the project's. Stop the agent with Ctrl+C.
The local agent answers with a Microsoft Learn URL.
Invoke the hosted agent
Ask the hosted agent
# Ask the hosted agent the same question in Foundry, where it calls the model with its own identity
echo "🤖 The hosted agent's answer:"
azd ai agent invoke gateway-agent "Find the Microsoft Learn article that explains Azure API Management key concepts. Reply with its title and URL in one sentence."Expected output: the same kind of answer, now from Foundry. The first call can take 30 seconds or more while the agent starts.
🤖 The hosted agent's answer:
Agent: gateway-agent (remote)
...
[gateway-agent] The Microsoft Learn article is "What is Azure API Management?" — https://learn.microsoft.com/azure/api-management/api-management-key-conceptsazd keeps the conversation, so a second invoke, such as "Shorten that to ten words.", continues it. Add
--new-conversation to start over.
Read the agent's logs
When a hosted call fails, azd prints only ERROR: agent failed (server_error): An internal server error occurred.
The agent's logs show the gateway's answer:
# Fetch the last 300 log lines of the agent's latest session, and keep the errors from the gateway
echo "📊 Errors in the agent's logs:"
azd ai agent monitor gateway-agent --tail 300 | grep "Error code"# Fetch the last 300 log lines of the agent's latest session, and keep the errors from the gateway
Write-Host "📊 Errors in the agent's logs:"
azd ai agent monitor gateway-agent --tail 300 | Select-String "Error code"If the agent's role hasn't applied yet, the logs show a 403:
... openai.PermissionDeniedError: Error code: 403 - Access denied. You do not have permission to access the requested resource. Ask an administrator to grant the required Azure role.Wait a minute and invoke again. A successful call prints nothing here.
Find the agent in Foundry
# Show the hosted agent's status, its identity, its Responses endpoint, and its Playground URL
echo "📊 Hosted agent gateway-agent:"
azd ai agent show gateway-agentThe output shows Status active, the Instance Identity Principal ID that you granted the role, the
agent's Responses endpoint, and a Playground URL. Open that URL to chat with the agent in the Foundry portal.
azd ai agent show gateway-agent reports active, and azd ai agent invoke returns an answer with a Learn URL.
See the agent in Monitoring
Each identity is a caller identity of its own. With monitoring set up as in Monitor usage, cost, and traces, invoke the agent a few times, wait a few minutes, then open Monitoring in the portal:
- Models: Model usage by caller identity counts the agent's identity apart from you and your other apps, with its calls, tokens, and estimated cost for your alias.
- Tools: the
learntool server under Usage by tool server, with the tools the toolbox called under Top tools within tool servers. The toolbox makes those calls, so they belong to the project's identity. - Traces: one row per model or tool call, with the policies the gateway applied.
Cap the agent's tokens
Give the agent's alias a strict token limit counted per caller identity, then watch the gateway enforce it without any change to the agent. Use the track you picked.
Add the token limit
- Select PoliciesAdd policy.
- On Type, select Token rate limit.
- On Scope, select Individual assets and pick your alias.
- On Details, keep Time period at Per minute, set Allowed tokens to
1000, keep Counter dimensions at Identity, and select Create.
Add the token limit
Find the alias by the name that apps send, which works even when a portal-created alias contains /.
az aigateway model-alias update --policies replaces the alias's whole policy array, so these commands keep its
other policies:
# Find the alias's resource name from the name that apps send
ALIAS_NAME="$(az aigateway model-alias list -g "$RG" --gateway-name "$GATEWAY" \
--query "[?properties.aliasName=='$AI_GATEWAY_MODEL'].name | [0]" -o tsv)"
echo "🔖 Alias resource name: $ALIAS_NAME"
# Write the alias's other policies, plus a limit of 1,000 tokens per minute counted per caller identity, to a file
az aigateway model-alias show -g "$RG" --gateway-name "$GATEWAY" -n "$ALIAS_NAME" -o json \
| jq -c '[(.properties.policies // [])[] | select(.type != "tokenLimit") | del(.scope)] + [{type:"tokenLimit",period:"minute",count:1000,counterKey:["identity"]}]' > policies.json
# Replace the alias's whole policy array with the file's, and print the new array
az aigateway model-alias update -g "$RG" --gateway-name "$GATEWAY" -n "$ALIAS_NAME" --policies @policies.json \
--query properties.policies &&
echo "🔐 Token limit of 1,000 tokens per minute per caller identity is on $AI_GATEWAY_MODEL"# Find the alias's resource name from the name that apps send
$ALIAS_NAME = az aigateway model-alias list -g $RG --gateway-name $GATEWAY `
--query "[?properties.aliasName=='$($env:AI_GATEWAY_MODEL)'].name | [0]" -o tsv
Write-Host "🔖 Alias resource name: $ALIAS_NAME"
# Write the alias's other policies, plus a limit of 1,000 tokens per minute counted per caller identity, to a file
$alias = az aigateway model-alias show -g $RG --gateway-name $GATEWAY -n $ALIAS_NAME -o json | ConvertFrom-Json
$p = @($alias.properties.policies | Where-Object { $_ -and $_.type -ne 'tokenLimit' } |
Select-Object * -ExcludeProperty scope) +
@(@{ type = 'tokenLimit'; period = 'minute'; count = 1000; counterKey = @('identity') })
ConvertTo-Json -InputObject $p -Depth 10 -Compress | Set-Content policies.json
# Replace the alias's whole policy array with the file's, and print the new array
az aigateway model-alias update -g $RG --gateway-name $GATEWAY -n $ALIAS_NAME --policies "@policies.json" `
--query properties.policies
if ($?) { Write-Host "🔐 Token limit of 1,000 tokens per minute per caller identity is on $env:AI_GATEWAY_MODEL" }Trigger the limit
Wait about a minute for the policy to apply, then invoke the hosted agent twice:
# Invoke the hosted agent twice, each time in a new conversation: the first invoke uses well over 1,000 tokens
echo "🚦 Expect the second invoke to fail with a 429 from the gateway's token limit"
azd ai agent invoke gateway-agent --new-conversation "Summarize Azure API Management in three sentences with Learn links."
azd ai agent invoke gateway-agent --new-conversation "Summarize Azure API Management in three sentences with Learn links."The gateway counts tokens when a call completes, so the first invoke usually succeeds and uses well over 1,000
tokens, because Learn tool results are long. The second invoke fails with ERROR: agent failed (server_error): An internal server error occurred., and the agent's logs show the gateway's 429:
# Fetch the agent's latest logs, and keep the errors from the gateway
echo "📊 Errors in the agent's logs:"
azd ai agent monitor gateway-agent --tail 300 | grep "Error code"# Fetch the agent's latest logs, and keep the errors from the gateway
Write-Host "📊 Errors in the agent's logs:"
azd ai agent monitor gateway-agent --tail 300 | Select-String "Error code"📊 Errors in the agent's logs:
... openai.RateLimitError: Error code: 429 - Token limit is exceeded. Try again in 431 seconds.
...The counter belongs to the agent's identity. Run the agent locally and it still answers: locally, it signs in as you, a different caller identity with its own counter. The wait in the message can be several minutes, because the first invoke spent several times the 1,000-token budget. For more on limits, see Throttle tokens per caller with token limits.
Remove the limit
In the portal, select Policies. On the row of the Token rate limit for your alias, select More actionsDelete, and confirm with Delete. With the CLI, remove the tokenLimit policy the same way you added it:
# Write the alias's policies, without the token limit, to a file
az aigateway model-alias show -g "$RG" --gateway-name "$GATEWAY" -n "$ALIAS_NAME" -o json \
| jq -c '[(.properties.policies // [])[] | select(.type != "tokenLimit") | del(.scope)]' > policies.json
# Replace the alias's whole policy array with the file's, then delete the file
az aigateway model-alias update -g "$RG" --gateway-name "$GATEWAY" -n "$ALIAS_NAME" --policies @policies.json \
--query properties.policies &&
echo "🚮 The token limit is removed from $AI_GATEWAY_MODEL"
rm policies.json# Write the alias's policies, without the token limit, to a file
$alias = az aigateway model-alias show -g $RG --gateway-name $GATEWAY -n $ALIAS_NAME -o json | ConvertFrom-Json
$p = @($alias.properties.policies | Where-Object { $_ -and $_.type -ne 'tokenLimit' } |
Select-Object * -ExcludeProperty scope)
ConvertTo-Json -InputObject $p -Depth 10 -Compress | Set-Content policies.json
# Replace the alias's whole policy array with the file's, then delete the file
az aigateway model-alias update -g $RG --gateway-name $GATEWAY -n $ALIAS_NAME --policies "@policies.json" `
--query properties.policies
if ($?) { Write-Host "🚮 The token limit is removed from $env:AI_GATEWAY_MODEL" }
Remove-Item policies.jsonIf your alias had its own token limit before this lab, the CLI commands remove it too: add it back with the count it had. When the alias has no policies left, the update prints no policies, only the 🚮 line.
Verify
azd ai agent invoke --localandazd ai agent invokeboth return an answer with a Microsoft Learn URL.- The Foundry project has no model deployments and the azd environment holds no gateway key, yet the agent answers: its model and tool calls go through the gateway with Microsoft Entra tokens.
- AccessEntra ID lists the agent's identity, the project, and you with AI Gateway User.
- With the token limit in place, the agent got
429from the gateway without any code change.
Go further
- Narrow the roles. AI Gateway User covers models and tools. The agent's identity only calls the model, and the project's managed identity only calls the MCP server, so assign them AI Gateway Models User and AI Gateway Tools User instead: in the wizard, pick All models or All tools as the scope. Running locally, you need both. Go keyless with Microsoft Entra ID and RBAC scopes roles further, to one alias or MCP server.
- Use an API key where Entra isn't an option. Create a key in AccessAPI keys, and send it as the
SDK's API key, which the SDK sends as
Authorization: Bearer <key>. MCP clients send the same header. Don't add anapi-keyheader as well: the gateway rejects a request that has both with400and "Specify either an API key or an Authorization header, not both". - Lock down egress. Foundry can restrict where a hosted agent may connect. Allow only your gateway's host name, so the gateway is the agent's only way out. See the egress control sample.
- Use another framework. Swap the agent's internals for the OpenAI Agents SDK or LangGraph, as in Connect agent frameworks to models and tools. The gateway settings stay the same.
- Skip the agent code. A gateway connection gives a Foundry project's playground and prompt agents the gateway's models, with no code to deploy, as in Use gateway models in Microsoft Foundry projects.
- Charge the agent back. Attribute its tokens and estimated cost with Attribute tokens and cost with OpenTelemetry.
Clean up
Remove the roles of the agent and the project first, while their identities still exist, then delete the Foundry resources. Use the track you picked.
Remove the role assignments
In AccessEntra ID, on the rows of <account>-<project>-gateway-agent-AgentIdentity and
<account>/projects/<project>, select More actions, then Remove role assignment, and confirm with
Remove.
Remove the role assignments
The query finds the roles at the gateway and below it, so it also removes the ones you assigned in the portal:
# Read the object IDs of the agent's identity and of the project's managed identity
AGENT_PRINCIPAL_ID="$(azd env get-value AGENT_GATEWAY_AGENT_INSTANCE_IDENTITY_PRINCIPAL_ID)"
PROJECT_PRINCIPAL_ID="$(az resource show --ids "$(azd env get-value AZURE_AI_PROJECT_ID)" \
--query identity.principalId -o tsv)"
# Delete each identity's role assignments on the gateway and its assets
for id in "$AGENT_PRINCIPAL_ID" "$PROJECT_PRINCIPAL_ID"; do
ids="$(az role assignment list --all -o tsv --query \
"[?principalId=='$id' && (ends_with(scope, '/aigateways/$GATEWAY') || contains(scope, '/aigateways/$GATEWAY/'))].id")"
# xargs passes one ID per line, the same way in Bash and zsh
if [ -n "$ids" ]; then
echo "$ids" | xargs az role assignment delete --ids && echo "🚮 Removed the gateway roles of $id"
fi
done# Read the object IDs of the agent's identity and of the project's managed identity
$AGENT_PRINCIPAL_ID = azd env get-value AGENT_GATEWAY_AGENT_INSTANCE_IDENTITY_PRINCIPAL_ID
$PROJECT_PRINCIPAL_ID = az resource show --ids (azd env get-value AZURE_AI_PROJECT_ID) `
--query identity.principalId -o tsv
# Delete each identity's role assignments on the gateway and its assets
foreach ($id in @($AGENT_PRINCIPAL_ID, $PROJECT_PRINCIPAL_ID)) {
$ids = @(az role assignment list --all -o tsv --query `
"[?principalId=='$id' && (ends_with(scope, '/aigateways/$GATEWAY') || contains(scope, '/aigateways/$GATEWAY/'))].id")
if ($ids) { az role assignment delete --ids @ids; if ($?) { Write-Host "🚮 Removed the gateway roles of $id" } }
}Keep your own AI Gateway User role if you continue with other labs. Go keyless with Microsoft Entra ID and RBAC shows how to remove it.
Delete the Foundry resources
Delete the Foundry resource group, with the connection, the toolbox, and the hosted agent. --purge also purges the
soft-deleted Foundry resource so you can reuse its name:
# Delete the resource group with the Foundry resource and the hosted agent, and purge the soft-deleted resource
azd down --purge --force &&
echo "🚮 The Foundry resource group and the hosted agent are deleted"Delete the .azure folder in infra if you no longer need the environment. Your gateway, its alias, and the
learn MCP server aren't affected. To do the lab again soon after, create the azd environment with a new name, such
as aigw-lab16b: in a Foundry project that you recreate with the same name, azd deploy --all can fail with
get_toolbox: the Foundry project has not synchronized.
Next steps
- Connect agent frameworks to models and tools: wire other frameworks to the gateway the same way.
- Go keyless with Microsoft Entra ID and RBAC: scope roles to one alias or MCP server, and sign users in to call the gateway on their behalf.
- Monitor usage, cost, and traces: tour the dashboards in depth.
- Docs: Connect Microsoft Foundry to AI Gateway and Deploy a hosted agent