Azure AI Gateway Labs
Documentation Open portal
Agents · Lab 15

Connect agent frameworks to models and tools

Point Microsoft Agent Framework, the OpenAI Agents SDK, and LangChain with LangGraph at the gateway's OpenAI-compatible model endpoint and its MCP endpoint, so every model and tool call an agent makes passes through gateway policies and shows up in Monitoring.

Intermediate 30 min CodeNotebook Updated
Notebook Source Edit

What you'll do

  • Configure a framework's model client with the gateway's OpenAI-compatible endpoint, an alias, and a runtime key
  • Load governed tools from the gateway's MCP endpoint in each framework
  • Run the same agent in Microsoft Agent Framework, the OpenAI Agents SDK, and LangGraph
  • Prove that the gateway authenticates and records every model and tool call the agent makes

Your teams build agents with different frameworks, and each one wires its own model keys and tool servers. Nobody can say which agent called what, and a policy change means touching every codebase. In this lab you point three popular frameworks at one gateway: two URLs and one header, and every model and tool call the agent makes is governed and observable in the same place.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

How it works

Agent frameworks talk to models through an OpenAI-compatible client and to tools through an MCP client. Point both at the gateway. The model client takes the runtime key as its API key, which it sends as Authorization: Bearer <key>. The MCP client sends the same key in the same header.

The framework still owns orchestration, memory, and state. The gateway authenticates each call, applies the policies on the alias, the MCP server, and the gateway, routes the alias to its provider, and records telemetry for every call that leaves the agent. Changing a model, a policy, or a tool list then needs no agent code change.

FrameworkModel clientMCP toolsModel API
Microsoft Agent FrameworkOpenAIChatClientMCPStreamableHTTPToolResponses
OpenAI Agents SDKOpenAIResponsesModel with AsyncOpenAIMCPServerStreamableHttpResponses
LangChain and LangGraphChatOpenAI with use_responses_api=TrueMultiServerMCPClient (langchain-mcp-adapters)Responses

All three use the same two endpoints. The model endpoint also serves Chat Completions, but some reasoning models only accept function tools on Responses, so the samples use Responses:

  • Model: <gateway-url>/default/models/openai/v1
  • MCP: <gateway-url>/default/toolservers/<server-name>/mcp

Set up the samples

1

Install the packages

From the lab folder, create a virtual environment in infra and install the pinned packages.

# Move into the samples folder
cd infra &&
  echo "📂 Samples folder: $(pwd)"

# Create a virtual environment in .venv and activate it, so that the packages stay out of your global Python
python3 -m venv .venv
source .venv/bin/activate &&
  echo "📦 Virtual environment .venv is active"

# Install the pinned packages for Agent Framework, the OpenAI Agents SDK, LangChain, and LangGraph
pip install -r requirements.txt &&
  echo "📦 The packages in requirements.txt are installed"
2

Configure the environment

Copy .env.example to .env and fill in your values. The samples build both endpoints from AI_GATEWAY_URL. The agent access docs name the full endpoints AI_GATEWAY_MODEL_ENDPOINT and AI_GATEWAY_MCP_ENDPOINT. They point to the same URLs.

infra/.env.exampleGitHub
# Copy this file to .env in the same folder and fill in the values. Never commit .env.
# Values already set in your shell take precedence over the file.

# Gateway URL from the gateway's overview, without a trailing path.
AI_GATEWAY_URL=<gateway-url>
# A callable model alias, for example the one you used in lab 02.
AI_GATEWAY_MODEL=<model-name>
# A runtime key from Access > API keys. Prefer one key per agent so Monitoring can tell agents apart.
AI_GATEWAY_API_KEY=<runtime-access-key>
# The MCP server you published in lab 04.
AI_GATEWAY_MCP_SERVER=learn

On GitHub, open infra/.env.example.

# Copy the template to .env, then open .env and fill in your four values
cp .env.example .env &&
  echo "📄 .env is ready to fill in"
Important

.env holds a secret. Keep it out of source control, and in production read the key from your platform's secret store or switch to Microsoft Entra ID.

Checkpoint

pip list shows agent-framework-core, openai-agents, and langchain-mcp-adapters, and .env has your four values.

Point the model client at the gateway

Each framework takes the gateway's model endpoint as its base URL, the alias as the model name, and the runtime key as its API key.

1

Configure the model client

# The SDK sends the key as a bearer token. Adding an api-key header as well gets a 400.
client = OpenAIChatClient(
    model=model,
    base_url=f"{gateway_url}/default/models/openai/v1/",
    api_key=api_key,
)

Don't add an api-key default header to the model client as well: the gateway rejects a request that has both an api-key and an Authorization header with 400.

Note

The OpenAI Agents SDK sample turns off the SDK's built-in tracing, which otherwise exports traces to OpenAI. The gateway already records each model and tool call.

Add the gateway's MCP tools

The MCP client connects to one MCP server on the gateway and sends the same key. The agent only sees the tools that the server publishes, after the gateway's tool controls.

1

Connect to the MCP endpoint

mcp_tool = MCPStreamableHTTPTool(
    name="gateway-tools",
    url=f"{gateway_url}/default/toolservers/{server}/mcp",
    static_headers={"Authorization": f"Bearer {api_key}"},
    approval_mode="never_require",
    request_timeout=60,
)

The learn tools only read documentation, so the Agent Framework sample runs them without asking for approval. For tools that change data, set approval_mode="always_require" on the MCP tool and handle the approval requests in your app. Also narrow the tool list on the gateway, as in Control which tools agents can call, so every agent gets the same limits.

2

Build the agent

async with mcp_tool:
    agent = Agent(client=client, name="docs-agent", instructions=INSTRUCTIONS, tools=mcp_tool)
    result = await agent.run(question)
    print(result.text)

create_agent from LangChain builds a LangGraph graph that loops between the model and the tools. LangGraph keeps orchestration and state, while AI Gateway applies policies and records telemetry for every call leaving the agent.

Run the agents

1

Run each sample

Run the sample for each framework from the infra folder, with the virtual environment active. Pass a question as arguments to override the default one.

# Run the Agent Framework agent: its model and MCP tool calls both go through the gateway
python agent_framework_agent.py
echo "🤖 The Agent Framework agent's answer is above"

Expected output: a line such as Model <model-name> and MCP server learn through <gateway-url> (the LangGraph sample also prints how many tools it loaded), then a short answer about AI Gateway that cites Microsoft Learn URLs. The answer differs between runs and frameworks.

2

Review the full samples

infra/agent_framework_agent.pyGitHub
"""A Microsoft Agent Framework agent whose model and MCP tool calls both go through AI Gateway.

An OpenAIChatClient (Responses API) points at the gateway's OpenAI-compatible endpoint and sends the runtime key as
its API key. Tools come from the gateway's MCP endpoint for one MCP server, with the same key as a bearer token.

Usage:
    cp .env.example .env      # then fill in the values
    python agent_framework_agent.py ["your question"]
"""

import asyncio
import os
import sys

from agent_framework import Agent, MCPStreamableHTTPTool
from agent_framework.openai import OpenAIChatClient
from dotenv import load_dotenv

INSTRUCTIONS = (
    "You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
    "documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."


def settings() -> tuple[str, str, str, str]:
    load_dotenv()
    missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
    if missing:
        sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
    gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
    server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
    return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server


async def main() -> None:
    gateway_url, model, api_key, server = settings()
    question = " ".join(sys.argv[1:]) or QUESTION

    # The SDK sends the key as a bearer token. Adding an api-key header as well gets a 400.
    client = OpenAIChatClient(
        model=model,
        base_url=f"{gateway_url}/default/models/openai/v1/",
        api_key=api_key,
    )

    mcp_tool = MCPStreamableHTTPTool(
        name="gateway-tools",
        url=f"{gateway_url}/default/toolservers/{server}/mcp",
        static_headers={"Authorization": f"Bearer {api_key}"},
        approval_mode="never_require",
        request_timeout=60,
    )

    async with mcp_tool:
        agent = Agent(client=client, name="docs-agent", instructions=INSTRUCTIONS, tools=mcp_tool)
        print(f"Model {model} and MCP server {server} through {gateway_url}\n")
        result = await agent.run(question)
        print(result.text)


if __name__ == "__main__":
    asyncio.run(main())
infra/openai_agents_agent.pyGitHub
"""An OpenAI Agents SDK agent whose model and MCP tool calls both go through AI Gateway.

The agent uses the Responses model class with an AsyncOpenAI client pointed at the gateway's
OpenAI-compatible endpoint, and an MCP server connection to the gateway's MCP endpoint. The client sends the runtime
key as its API key, and the MCP connection sends it as a bearer token.

Usage:
    cp .env.example .env      # then fill in the values
    python openai_agents_agent.py ["your question"]
"""

import asyncio
import os
import sys

from agents import Agent, OpenAIResponsesModel, Runner, set_tracing_disabled
from agents.mcp import MCPServerStreamableHttp
from dotenv import load_dotenv
from openai import AsyncOpenAI

INSTRUCTIONS = (
    "You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
    "documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."


def settings() -> tuple[str, str, str, str]:
    load_dotenv()
    missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
    if missing:
        sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
    gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
    server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
    return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server


async def main() -> None:
    gateway_url, model, api_key, server = settings()
    question = " ".join(sys.argv[1:]) or QUESTION

    # The SDK exports its own traces to OpenAI by default. AI Gateway already records every model and tool call.
    set_tracing_disabled(True)

    client = AsyncOpenAI(
        base_url=f"{gateway_url}/default/models/openai/v1",
        api_key=api_key,
    )

    async with MCPServerStreamableHttp(
        name="gateway-tools",
        params={
            "url": f"{gateway_url}/default/toolservers/{server}/mcp",
            "headers": {"Authorization": f"Bearer {api_key}"},
            "timeout": 60,
        },
        cache_tools_list=True,
        client_session_timeout_seconds=60,
    ) as mcp_server:
        agent = Agent(
            name="docs-agent",
            instructions=INSTRUCTIONS,
            model=OpenAIResponsesModel(model=model, openai_client=client),
            mcp_servers=[mcp_server],
        )
        print(f"Model {model} and MCP server {server} through {gateway_url}\n")
        result = await Runner.run(agent, question)
        print(result.final_output)


if __name__ == "__main__":
    asyncio.run(main())
infra/langgraph_agent.pyGitHub
"""A LangChain agent (built on LangGraph) whose model and MCP tool calls both go through AI Gateway.

ChatOpenAI points at the gateway's OpenAI-compatible endpoint, and langchain-mcp-adapters loads tools from the
gateway's MCP endpoint over streamable HTTP. The model takes the runtime key as its API key, and the MCP client
sends it as a bearer token. LangGraph keeps orchestration and state; the gateway applies policies and records
telemetry for every call that leaves the agent.

Usage:
    cp .env.example .env      # then fill in the values
    python langgraph_agent.py ["your question"]
"""

import asyncio
import os
import sys

from dotenv import load_dotenv
from langchain.agents import create_agent
from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain_openai import ChatOpenAI

INSTRUCTIONS = (
    "You answer questions about Microsoft technologies. Use the Microsoft Learn tools to find current "
    "documentation and cite the URLs you used. Keep answers under 120 words."
)
QUESTION = "What is Azure AI Gateway? Answer in three bullet points."


def settings() -> tuple[str, str, str, str]:
    load_dotenv()
    missing = [name for name in ("AI_GATEWAY_URL", "AI_GATEWAY_MODEL", "AI_GATEWAY_API_KEY") if not os.environ.get(name)]
    if missing:
        sys.exit(f"Set {', '.join(missing)} in .env or your shell. See .env.example.")
    gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
    server = os.environ.get("AI_GATEWAY_MCP_SERVER", "learn")
    return gateway_url, os.environ["AI_GATEWAY_MODEL"], os.environ["AI_GATEWAY_API_KEY"], server


async def main() -> None:
    gateway_url, model_name, api_key, server = settings()
    question = " ".join(sys.argv[1:]) or QUESTION

    model = ChatOpenAI(
        model=model_name,
        base_url=f"{gateway_url}/default/models/openai/v1",
        api_key=api_key,
        use_responses_api=True,
    )

    mcp_client = MultiServerMCPClient(
        {
            "gateway-tools": {
                "transport": "streamable_http",
                "url": f"{gateway_url}/default/toolservers/{server}/mcp",
                "headers": {"Authorization": f"Bearer {api_key}"},
            }
        }
    )
    tools = await mcp_client.get_tools()

    agent = create_agent(model, tools, system_prompt=INSTRUCTIONS)
    print(f"Model {model_name} and MCP server {server} ({len(tools)} tools) through {gateway_url}\n")
    result = await agent.ainvoke({"messages": [{"role": "user", "content": question}]})
    print(result["messages"][-1].text)


if __name__ == "__main__":
    asyncio.run(main())

On GitHub, open agent_framework_agent.py, openai_agents_agent.py, and langgraph_agent.py.

Checkpoint

All three samples print an answer that cites at least one learn.microsoft.com URL, which means each agent called a model and an MCP tool through the gateway.

Verify

1

Prove the gateway is in the path

Run one sample with a wrong key. The gateway rejects the agent's first call with 401, before any model or tool runs.

# Run the LangGraph sample with a wrong key, for this command only, because your shell's value wins over .env
echo "🚦 Expect 401 Unauthorized: the gateway rejects the agent's first call"
AI_GATEWAY_API_KEY="not-a-key" python langgraph_agent.py

Expected output: a traceback that ends with a 401 Unauthorized error for the MCP endpoint, because each sample connects to the MCP server before it calls the model. A key set in your shell takes precedence over .env.

2

See the agent in Monitoring

If you set up monitoring in Monitor usage, cost, and traces, wait a few minutes and open Monitoring. On Overview, Model calls and Tool calls both grew. On Tools, the learn server shows up in Usage by tool server, and Tool usage by caller identity shows the agent's key. A run usually makes at least two model calls: one that asks for a tool and one that writes the answer. Open Traces to see each call with its gateway policies.

Go further

  • Give each agent an identity. Create one runtime key per agent, as in Issue and rotate runtime API keys, so Monitoring and cost limits can tell agents apart. For production, drop keys: send a Microsoft Entra token for https://ai.gateway.azure.com from the agent's managed identity or agent identity, with the AI Gateway User role, as in Go keyless with Microsoft Entra ID and RBAC.
  • Govern the agent, not the code. Add a token limit to the alias with Throttle tokens per caller with token limits, or a request rate limit to the MCP server. The agent gets 429 responses without any code change, so handle Retry-After in long-running agents.
  • Swap models behind the alias. Retarget the alias to another model or provider, as in Publish models from any provider behind stable aliases. The agents keep working with the same AI_GATEWAY_MODEL.
  • Use .NET. The .NET hosted agent sample connects Microsoft Agent Framework for .NET to a gateway. With Azure AI Gateway, pass the runtime key as the client's API key, which the client sends as a bearer token.
  • Use the GitHub Copilot SDK. Give create_session a provider of type openai with the model endpoint as base_url, wire_api set to responses, and the key in an Authorization: Bearer header, and list the MCP endpoint under mcp_servers, as in the Copilot SDK tab of the agent access docs. Its requests carry the Copilot system prompt and tools, so one request can exceed a small token limit, such as 10,000 tokens per minute, and get 429.
  • Other SDKs. For other frameworks, use their model and MCP configuration points with the same endpoints, and confirm the endpoint formats and authentication they support.

Clean up

  • Delete infra/.env, because it holds a key, and infra/.venv if you don't need it.
  • If you created a key just for this lab, revoke it under AccessAPI keys.

Keep the gateway, the alias, and the learn MCP server. Other labs reuse them.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.