Your teams build agents with different frameworks, and each one wires its own model keys and tool servers. Nobody can say which agent called what, and a policy change means touching every codebase. In this lab you point three popular frameworks at one gateway: two URLs and one header, and every model and tool call the agent makes is governed and observable in the same place.
Before you begin
One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.
- Choose where to run commandsEvery track except the portal
- Clone the labs repoThe Bicep + azd and Terraform tracks, the notebooks, and the code samples
You've done the one-time setup that this lab uses.
- Complete Publish a Foundry model and make your first call. You need the
gateway URL, a callable alias, and a runtime key. Create a dedicated key for this agent under AccessAPI keys, for example
docs-agent, so its traffic is easy to find. - Complete Federate MCP tools behind one endpoint. The samples use the
learnMCP server from that lab. Any MCP server on your gateway works. - Python 3.10 or later. The samples were tested together on Python 3.13 with the versions in infra/requirements.txt.
- Optional: monitoring from Monitor usage, cost, and traces, to see the agent's calls.
- Run the scripts and samples from the lab's folder,
~/aigateway/labs/15-agent-frameworks.
How it works
Agent frameworks talk to models through an OpenAI-compatible client and to tools through an MCP client. Point both at
the gateway. The model client takes the runtime key as its API key, which it sends as Authorization: Bearer <key>.
The MCP client sends the same key in the same header.
The framework still owns orchestration, memory, and state. The gateway authenticates each call, applies the policies on the alias, the MCP server, and the gateway, routes the alias to its provider, and records telemetry for every call that leaves the agent. Changing a model, a policy, or a tool list then needs no agent code change.
| Framework | Model client | MCP tools | Model API |
|---|---|---|---|
| Microsoft Agent Framework | OpenAIChatClient | MCPStreamableHTTPTool | Responses |
| OpenAI Agents SDK | OpenAIResponsesModel with AsyncOpenAI | MCPServerStreamableHttp | Responses |
| LangChain and LangGraph | ChatOpenAI with use_responses_api=True | MultiServerMCPClient (langchain-mcp-adapters) | Responses |
All three use the same two endpoints. The model endpoint also serves Chat Completions, but some reasoning models only accept function tools on Responses, so the samples use Responses:
- Model:
<gateway-url>/default/models/openai/v1 - MCP:
<gateway-url>/default/toolservers/<server-name>/mcp
Set up the samples
Install the packages
From the lab folder, create a virtual environment in infra and install the pinned packages.
# Move into the samples folder
cd infra &&
echo "📂 Samples folder: $(pwd)"
# Create a virtual environment in .venv and activate it, so that the packages stay out of your global Python
python3 -m venv .venv
source .venv/bin/activate &&
echo "📦 Virtual environment .venv is active"
# Install the pinned packages for Agent Framework, the OpenAI Agents SDK, LangChain, and LangGraph
pip install -r requirements.txt &&
echo "📦 The packages in requirements.txt are installed"# Move into the samples folder
cd infra
if ($?) { Write-Host "📂 Samples folder: $(Get-Location)" }
# Create a virtual environment in .venv and activate it, so that the packages stay out of your global Python
python -m venv .venv
.venv\Scripts\Activate.ps1
if ($?) { Write-Host "📦 Virtual environment .venv is active" }
# Install the pinned packages for Agent Framework, the OpenAI Agents SDK, LangChain, and LangGraph
pip install -r requirements.txt
if ($?) { Write-Host "📦 The packages in requirements.txt are installed" }Configure the environment
Copy .env.example to .env and fill in your values. The samples build both endpoints from AI_GATEWAY_URL. The
agent access docs name the full endpoints
AI_GATEWAY_MODEL_ENDPOINT and AI_GATEWAY_MCP_ENDPOINT. They point to the same URLs.
# Copy this file to .env in the same folder and fill in the values. Never commit .env.
# Values already set in your shell take precedence over the file.
# Gateway URL from the gateway's overview, without a trailing path.
AI_GATEWAY_URL=<gateway-url>
# A callable model alias, for example the one you used in lab 02.
AI_GATEWAY_MODEL=<model-name>
# A runtime key from Access > API keys. Prefer one key per agent so Monitoring can tell agents apart.
AI_GATEWAY_API_KEY=<runtime-access-key>
# The MCP server you published in lab 04.
AI_GATEWAY_MCP_SERVER=learnOn GitHub, open infra/.env.example.
# Copy the template to .env, then open .env and fill in your four values
cp .env.example .env &&
echo "📄 .env is ready to fill in"# Copy the template to .env, then open .env and fill in your four values
Copy-Item .env.example .env
if ($?) { Write-Host "📄 .env is ready to fill in" }.env holds a secret. Keep it out of source control, and in production read the key from your platform's secret
store or switch to Microsoft Entra ID.
pip list shows agent-framework-core, openai-agents, and langchain-mcp-adapters, and .env has your four
values.
Point the model client at the gateway
Each framework takes the gateway's model endpoint as its base URL, the alias as the model name, and the runtime key as its API key.
Configure the model client
# The SDK sends the key as a bearer token. Adding an api-key header as well gets a 400.
client = OpenAIChatClient(
model=model,
base_url=f"{gateway_url}/default/models/openai/v1/",
api_key=api_key,
)client = AsyncOpenAI(
base_url=f"{gateway_url}/default/models/openai/v1",
api_key=api_key,
)
# ...
model=OpenAIResponsesModel(model=model, openai_client=client),model = ChatOpenAI(
model=model_name,
base_url=f"{gateway_url}/default/models/openai/v1",
api_key=api_key,
use_responses_api=True,
)Don't add an api-key default header to the model client as well: the gateway rejects a request that has both an
api-key and an Authorization header with 400.
The OpenAI Agents SDK sample turns off the SDK's built-in tracing, which otherwise exports traces to OpenAI. The gateway already records each model and tool call.
Add the gateway's MCP tools
The MCP client connects to one MCP server on the gateway and sends the same key. The agent only sees the tools that the server publishes, after the gateway's tool controls.
Connect to the MCP endpoint
mcp_tool = MCPStreamableHTTPTool(
name="gateway-tools",
url=f"{gateway_url}/default/toolservers/{server}/mcp",
static_headers={"Authorization": f"Bearer {api_key}"},
approval_mode="never_require",
request_timeout=60,
)async with MCPServerStreamableHttp(
name="gateway-tools",
params={
"url": f"{gateway_url}/default/toolservers/{server}/mcp",
"headers": {"Authorization": f"Bearer {api_key}"},
"timeout": 60,
},
cache_tools_list=True,
client_session_timeout_seconds=60,
) as mcp_server:mcp_client = MultiServerMCPClient(
{
"gateway-tools": {
"transport": "streamable_http",
"url": f"{gateway_url}/default/toolservers/{server}/mcp",
"headers": {"Authorization": f"Bearer {api_key}"},
}
}
)
tools = await mcp_client.get_tools()The learn tools only read documentation, so the Agent Framework sample runs them without asking for approval. For
tools that change data, set approval_mode="always_require" on the MCP tool and handle the approval requests in your
app. Also narrow the tool list on the gateway, as in
Control which tools agents can call, so every agent gets the same limits.
Build the agent
async with mcp_tool:
agent = Agent(client=client, name="docs-agent", instructions=INSTRUCTIONS, tools=mcp_tool)
result = await agent.run(question)
print(result.text)agent = Agent(
name="docs-agent",
instructions=INSTRUCTIONS,
model=OpenAIResponsesModel(model=model, openai_client=client),
mcp_servers=[mcp_server],
)
result = await Runner.run(agent, question)
print(result.final_output)agent = create_agent(model, tools, system_prompt=INSTRUCTIONS)
result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]})
print(result["messages"][-1].text)create_agent from LangChain builds a LangGraph graph that loops between the model and the tools. LangGraph keeps
orchestration and state, while AI Gateway applies policies and records telemetry for every call leaving the agent.
Run the agents
Run each sample
Run the sample for each framework from the infra folder, with the virtual environment active. Pass a question as
arguments to override the default one.
# Run the Agent Framework agent: its model and MCP tool calls both go through the gateway
python agent_framework_agent.py
echo "🤖 The Agent Framework agent's answer is above"# Run the OpenAI Agents SDK agent: its model and MCP tool calls both go through the gateway
python openai_agents_agent.py
echo "🤖 The OpenAI Agents SDK agent's answer is above"# Run the LangGraph agent: its model and MCP tool calls both go through the gateway
python langgraph_agent.py
echo "🤖 The LangGraph agent's answer is above"Expected output: a line such as Model <model-name> and MCP server learn through <gateway-url>
(the LangGraph sample also prints how many tools it loaded), then a short answer about AI Gateway that cites
Microsoft Learn URLs. The answer differs between runs and frameworks.
Review the full samples
"""A Microsoft Agent Framework agent whose model and MCP tool calls both go through AI Gateway.
An OpenAIChatClient (Responses API) points at the gateway's OpenAI-compatible endpoint and sends the runtime key as
its API key. Tools come from the gateway's MCP endpoint for one MCP server, with the same key as a bearer token.
Usage:
cp .env.example .env # then fill in the values
python agent_framework_agent.py ["your question"]
"""
import asyncio
import os
import sys
from agent_framework import Agent, MCPStreamableHTTPTool
from agent_framework.openai import OpenAIChatClient
from dotenv import load_dotenv
INSTRUCTIONS = (
"You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
"documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."
def settings() -> tuple[str, str, str, str]:
load_dotenv()
missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
if missing:
sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server
async def main() -> None:
gateway_url, model, api_key, server = settings()
question = " ".join(sys.argv[1:]) or QUESTION
# The SDK sends the key as a bearer token. Adding an api-key header as well gets a 400.
client = OpenAIChatClient(
model=model,
base_url=f"{gateway_url}/default/models/openai/v1/",
api_key=api_key,
)
mcp_tool = MCPStreamableHTTPTool(
name="gateway-tools",
url=f"{gateway_url}/default/toolservers/{server}/mcp",
static_headers={"Authorization": f"Bearer {api_key}"},
approval_mode="never_require",
request_timeout=60,
)
async with mcp_tool:
agent = Agent(client=client, name="docs-agent", instructions=INSTRUCTIONS, tools=mcp_tool)
print(f"Model {model} and MCP server {server} through {gateway_url}\n")
result = await agent.run(question)
print(result.text)
if __name__ == "__main__":
asyncio.run(main())"""An OpenAI Agents SDK agent whose model and MCP tool calls both go through AI Gateway.
The agent uses the Responses model class with an AsyncOpenAI client pointed at the gateway's
OpenAI-compatible endpoint, and an MCP server connection to the gateway's MCP endpoint. The client sends the runtime
key as its API key, and the MCP connection sends it as a bearer token.
Usage:
cp .env.example .env # then fill in the values
python openai_agents_agent.py ["your question"]
"""
import asyncio
import os
import sys
from agents import Agent, OpenAIResponsesModel, Runner, set_tracing_disabled
from agents.mcp import MCPServerStreamableHttp
from dotenv import load_dotenv
from openai import AsyncOpenAI
INSTRUCTIONS = (
"You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
"documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."
def settings() -> tuple[str, str, str, str]:
load_dotenv()
missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
if missing:
sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server
async def main() -> None:
gateway_url, model, api_key, server = settings()
question = " ".join(sys.argv[1:]) or QUESTION
# The SDK exports its own traces to OpenAI by default. AI Gateway already records every model and tool call.
set_tracing_disabled(True)
client = AsyncOpenAI(
base_url=f"{gateway_url}/default/models/openai/v1",
api_key=api_key,
)
async with MCPServerStreamableHttp(
name="gateway-tools",
params={
"url": f"{gateway_url}/default/toolservers/{server}/mcp",
"headers": {"Authorization": f"Bearer {api_key}"},
"timeout": 60,
},
cache_tools_list=True,
client_session_timeout_seconds=60,
) as mcp_server:
agent = Agent(
name="docs-agent",
instructions=INSTRUCTIONS,
model=OpenAIResponsesModel(model=model, openai_client=client),
mcp_servers=[mcp_server],
)
print(f"Model {model} and MCP server {server} through {gateway_url}\n")
result = await Runner.run(agent, question)
print(result.final_output)
if __name__ == "__main__":
asyncio.run(main())"""A LangChain agent (built on LangGraph) whose model and MCP tool calls both go through AI Gateway.
ChatOpenAI points at the gateway's OpenAI-compatible endpoint, and langchain-mcp-adapters loads tools from the
gateway's MCP endpoint over streamable HTTP. The model takes the runtime key as its API key, and the MCP client
sends it as a bearer token. LangGraph keeps orchestration and state; the gateway applies policies and records
telemetry for every call that leaves the agent.
Usage:
cp .env.example .env # then fill in the values
python langgraph_agent.py ["your question"]
"""
import asyncio
import os
import sys
from dotenv import load_dotenv
from langchain.agents import create_agent
from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain_openai import ChatOpenAI
INSTRUCTIONS = (
"You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
"documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."
def settings() -> tuple[str, str, str, str]:
load_dotenv()
missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
if missing:
sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server
async def main() -> None:
gateway_url, model_name, api_key, server = settings()
question = " ".join(sys.argv[1:]) or QUESTION
model = ChatOpenAI(
model=model_name,
base_url=f"{gateway_url}/default/models/openai/v1",
api_key=api_key,
use_responses_api=True,
)
mcp_client = MultiServerMCPClient(
{
"gateway-tools": {
"transport": "streamable_http",
"url": f"{gateway_url}/default/toolservers/{server}/mcp",
"headers": {"Authorization": f"Bearer {api_key}"},
}
}
)
tools = await mcp_client.get_tools()
agent = create_agent(model, tools, system_prompt=INSTRUCTIONS)
print(f"Model {model_name} and MCP server {server} ({len(tools)} tools) through {gateway_url}\n")
result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]})
print(result["messages"][-1].text)
if __name__ == "__main__":
asyncio.run(main())On GitHub, open agent_framework_agent.py, openai_agents_agent.py, and langgraph_agent.py.
All three samples print an answer that cites at least one learn.microsoft.com URL, which means each agent called
a model and an MCP tool through the gateway.
Verify
Prove the gateway is in the path
Run one sample with a wrong key. The gateway rejects the agent's first call with 401, before any model or tool
runs.
# Run the LangGraph sample with a wrong key, for this command only, because your shell's value wins over .env
echo "🚦 Expect 401 Unauthorized: the gateway rejects the agent's first call"
AI_GATEWAY_API_KEY="not-a-key" python langgraph_agent.py# Run the LangGraph sample with a wrong key, then remove it from your shell so that the samples read .env again
Write-Host "🚦 Expect 401 Unauthorized: the gateway rejects the agent's first call"
$env:AI_GATEWAY_API_KEY = "not-a-key"; python langgraph_agent.py; Remove-Item Env:AI_GATEWAY_API_KEYExpected output: a traceback that ends with a 401 Unauthorized error for the MCP endpoint, because each sample
connects to the MCP server before it calls the model. A key set in your shell takes precedence over .env.
See the agent in Monitoring
If you set up monitoring in Monitor usage, cost, and traces, wait a few minutes and
open Monitoring. On Overview, Model calls and Tool calls both grew. On Tools, the learn
server shows up in Usage by tool server, and Tool usage by caller identity shows the agent's key. A run
usually makes at least two model calls: one that asks for a tool and one that writes the answer. Open
Traces to see each call with its gateway policies.
Go further
- Give each agent an identity. Create one runtime key per agent, as in
Issue and rotate runtime API keys, so Monitoring and cost limits can tell agents apart.
For production, drop keys: send a Microsoft Entra token for
https://ai.gateway.azure.comfrom the agent's managed identity or agent identity, with the AI Gateway User role, as in Go keyless with Microsoft Entra ID and RBAC. - Govern the agent, not the code. Add a token limit to the alias with
Throttle tokens per caller with token limits, or a request rate limit to the MCP
server. The agent gets
429responses without any code change, so handleRetry-Afterin long-running agents. - Swap models behind the alias. Retarget the alias to another model or provider, as in
Publish models from any provider behind stable aliases. The agents keep
working with the same
AI_GATEWAY_MODEL. - Use .NET. The .NET hosted agent sample connects Microsoft Agent Framework for .NET to a gateway. With Azure AI Gateway, pass the runtime key as the client's API key, which the client sends as a bearer token.
- Use the GitHub Copilot SDK. Give
create_sessionaproviderof typeopenaiwith the model endpoint asbase_url,wire_apiset toresponses, and the key in anAuthorization: Bearerheader, and list the MCP endpoint undermcp_servers, as in the Copilot SDK tab of the agent access docs. Its requests carry the Copilot system prompt and tools, so one request can exceed a small token limit, such as 10,000 tokens per minute, and get429. - Other SDKs. For other frameworks, use their model and MCP configuration points with the same endpoints, and confirm the endpoint formats and authentication they support.
Clean up
- Delete
infra/.env, because it holds a key, andinfra/.venvif you don't need it. - If you created a key just for this lab, revoke it under AccessAPI keys.
Keep the gateway, the alias, and the learn MCP server. Other labs reuse them.
Next steps
- Deploy a Foundry hosted agent behind AI Gateway: run an Agent Framework agent in Microsoft Foundry with the same gateway wiring.
- Go keyless with Microsoft Entra ID and RBAC: replace the agent's key with an identity. Lab 16's hosted agent already runs without keys.
- Attribute tokens and cost with OpenTelemetry: report each agent's tokens and estimated cost.
- Docs: Connect agents to AI Gateway