Azure AI Gateway Labs
Documentation Open portal
Observability & FinOps ยท Lab 13

Monitor usage, cost, and traces

Send the gateway's OpenTelemetry metrics, logs, and traces to Application Insights, then use the built-in Monitoring dashboard to see usage, estimated cost per model and API key, policy impact, and gateway-aware traces.

Beginner 25 min PortalCLIBicepTerraform Updated
Source Edit

What you'll do

  • Explain how the gateway exports metrics, logs, and traces independently over OTLP
  • Connect the gateway to Application Insights in the portal, with the Azure CLI, with Bicep and azd, or with Terraform
  • Read health, outcomes, policy impact, tokens, and estimated cost on the Monitoring dashboard
  • Follow one request through the trace waterfall, including its policy child spans
  • Query gateway telemetry in Log Analytics with KQL

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform

Your gateway now serves several apps. Finance asks what the models cost per team, an app owner reports intermittent failures, and nobody can tell whether a policy or the backend rejected those calls. In this lab you send the gateway's OpenTelemetry signals to Application Insights and answer all three questions from the built-in Monitoring dashboard, down to a single request's trace.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, model, key, and Application Insights, so you can skip this section. You need only the one-time setup above, and permission to create role assignments, for example Owner on the subscription.

  • Complete Publish a Foundry model and make your first call. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
  • Permission to change the gateway, for example Contributor on its resource group, and permission to create role assignments, for example Owner or Role Based Access Control Administrator. Setup grants the gateway's managed identity a role on the telemetry destination. To reuse an existing Application Insights resource, you also need Application Insights Contributor on it.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/13-monitoring.

Set these variables once. The tracks and the traffic script reuse them.

# Name your gateway from lab 02, and the alias that the traffic script calls
export RG="<resource-group>"
export GATEWAY="<gateway>"
export AI_GATEWAY_MODEL="<model-name>"     # a callable alias, for example the one you used in lab 02

# Read the gateway's runtime URL
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "๐ŸŒ Runtime URL: $AI_GATEWAY_URL"

# Paste the runtime key you created in Access โ€บ API keys, without showing it: don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "๐Ÿ”‘ Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
Important

Did you configure monitoring before the 2026-08-28 release? If the Monitoring page flags the exporter as legacy, select SettingsGeneralDisable monitoring, then select Configure to run the setup again. You can select the same Application Insights resource.

How it works

The gateway emits three OpenTelemetry signals and exports each one independently over OTLP, to Azure Application Insights or to any OTLP endpoint (for example Datadog, Splunk, Grafana Cloud, or your own collector). Configure only the signals you need. Telemetry follows the OpenTelemetry GenAI and MCP semantic conventions, such as gen_ai.request.model, gen_ai.operation.name, and mcp.method.name, and adds gateway attributes under azure.ai_gateway.*: the API key, the routing decision, the estimated cost, and the policy type.

A failed operation marks its span Error even when the HTTP response is 200: for example an MCP tool call that returns a tool error. The span's error.type names the failure, so you can find these calls even though no HTTP error reached the client.

When the destination is Application Insights and all three signals are on, the Monitoring page turns that data into a dashboard. Estimated cost is observed tokens ร— public pricing: use it to spot trends and compare callers, then reconcile with provider billing or Azure Cost Management for financial reporting.

SignalWhat the gateway sendsWhere it lands with Application InsightsDashboard uses it for
MetricsToken usage and estimated token cost, by model, token type, API key, and callerAzure Monitor workspace (managed Prometheus) linked to Application InsightsTokens and Estimated model cost
TracesOne root span per request, with child spans for each stage and each policy that ranOTelSpans table in the Log Analytics workspaceCalls, outcomes, blocked traces, the trace waterfall
LogsRequest records with status, caller, model, routing, estimated cost, and the token usage the backend reportedOTelLogs table in the Log Analytics workspaceCorrelated logs and caller attribution

Connect Application Insights

Pick one track. Each ends with the gateway exporting all three signals to an Application Insights resource named after the gateway, <gateway>-insights.

Choose your track
1

Start the setup

Open the AI Gateway portal and select your gateway from the gateway switcher. Select Monitoring. If monitoring isn't configured yet, a banner reads Start by configuring telemetry. Select Configure. The Configure telemetry wizard has three steps: Destination, Authentication, and Review and apply.

2

Choose a destination

On Destination, keep New Azure Application Insights resource. The wizard suggests the resource name <gateway>-insights, adds a short random suffix if that name is taken, and creates it in the gateway's region and resource group. Select Next.

Tip

The other choices are Existing Azure Application Insights resource, for a component you already have, and Custom OpenTelemetry destination, which takes separate Metrics endpoint, Logs endpoint, and Traces endpoint URLs. Leave an endpoint empty to turn that signal off. The built-in dashboard only reads Application Insights, so with a custom destination you use your own platform's views. See Send telemetry to a custom destination.

3

Review authentication

On Authentication, Managed identity shows System-assigned managed identity: the gateway exports with its managed identity. If the gateway also has a user-assigned identity, you can pick it here instead. A note confirms that the Monitoring Metrics Publisher role is assigned on the destination's Data Collection Rule when you apply.

Leave Include recommended token claims in telemetry selected. It adds four Microsoft Entra caller claims to the telemetry: tid and oid in logs, traces, and metrics, and appid and azp in logs and traces. Attribute tokens and cost with OpenTelemetry builds on them. Select Next.

4

Apply the configuration

On Review and apply, check the Destination, the Signals (all three Discovered automatically), the Identity, and the Token claim attributes, then select Apply. The button shows Applyingโ€ฆ while the wizard registers the resource providers, creates Application Insights on a managed Log Analytics workspace, discovers the OTLP endpoints, assigns the role, and creates the exporter. It can take a few minutes. If it fails, the wizard shows the error: fix the cause and select Apply again. If an Azure Policy assignment blocks a step, check the Activity Log for the denied operation.

Checkpoint

The Monitoring page shows the Overview, Models, Tools, Policies, and Traces tabs, with Open in Azure portal and Settings in the header.

Generate traffic

The dashboard needs data. Send a few requests through the gateway so every view has something to show.

1

Run the traffic script

From the lab folder, run the script for your shell. It sends ten chat completions, then a few calls that fail on purpose, so you have failures to investigate: one to an alias that doesn't exist, which the gateway rejects, and one with an invalid message role, which the model provider rejects. If you federated the learn tool server in Federate MCP tools behind one endpoint, pass its name too: the script then calls a tool without its required argument. That call returns HTTP 200, but the tool reports an error.

# Send ten chat completions, then the failing calls: drop "learn" if you don't have lab 04's tool server
echo "๐Ÿ“Š Sending 10 requests to $AI_GATEWAY_MODEL, then the calls that fail on purpose:"
./infra/generate-traffic.sh "$AI_GATEWAY_MODEL" 10 learn

On GitHub, open generate-traffic.sh or generate-traffic.ps1.

Expected output, with the requests in the middle left out:

Text
Request  Target                             Status       Expected
1        gpt-5-mini                         200          200
...
10       gpt-5-mini                         200          200
fail-1   does-not-exist                     404          404, unknown alias
fail-2   gpt-5-mini                         400          400, from the provider
fail-3   learn: learn_microsoft_docs_fetch  200 isError  200 isError, tool error

Telemetry can take a few minutes to reach the dashboard.

Tip

To find your own requests later, send a W3C traceparent header from your app, or let an OpenTelemetry SDK send it. The gateway continues your trace, so the trace ID your app logs is the Trace ID on the Traces tab and the TraceId in KQL.

Tour the dashboard

Each tab answers a different question. Use the time range buttons (1H, 24H, 7D, 30D) to match your traffic.

1

Overview: is the gateway healthy, and what does it cost?

Select MonitoringOverview. The cards summarize the range: Estimated model cost, Model calls, Tool calls, Trace failure rate, and Blocked traces. Below them, Model and tool calls over time stacks both kinds of traffic, Top models ranks models by estimated cost with their 4xx and 5xx error rates, and Top tool servers ranks MCP servers by calls. Recent traces requiring attention and Top blocking policies take you straight to problems.

Overview: estimated model cost, model and tool calls, failure rate, blocked traces, and the top models and tool servers.
2

Models: who uses which model, and how many tokens?

Select Models. The cards show Estimated model cost, Total tokens, Active aliases, Active caller identities, and Peak throughput. Estimated cost over time splits cost by token type, and Token breakdown compares input, output, cached, and reasoning tokens. The Usage table lists each Alias with its Responding models, Caller identities, Estimated cost, Estimated cost share, Total tokens, and Calls: switch between By alias and By model. Model usage by caller identity breaks cost, tokens, and calls down by API key or Microsoft Entra caller. That's your first view of cost by API key and model.

Note

If the page asks for a Prometheus access token, follow its Azure CLI instructions while you're signed in to the destination's tenant. The token is used only to query metrics.

3

Tools: which MCP tools are called?

Select Tools. The cards show Total tool calls, Active tool servers, Active tools, Active caller identities, and Peak throughput. Tool calls over time by response status code charts the calls, then Usage by tool server, Top tools within tool servers, and Tool usage by caller identity rank them. It stays empty until agents call tools through the gateway, as in Federate MCP tools behind one endpoint.

4

Policies: what did governance block?

Select Policies. The table lists each Policy type with Allowed requests, Allowed caller identities, Blocked requests, Blocked caller identities, and Last executed. After Throttle tokens per caller, the token limit's 429 responses show up as blocked requests.

5

Traces: find the failed request

Select Traces. Each row shows the Trace ID, Time, Assets, Caller identities, Outcome, Applied policies, and Duration. Outcome is Succeeded, Completed with issues, Blocked by policy, Failed, or Disconnected. Narrow the list with the caller identity, outcome, and asset type filters (they're under Filters in a narrow window). Find the fail-2 request, which the provider rejected: set the outcome filter to Failed, or paste a trace ID in Search.

Checkpoint

Overview shows the model calls you sent and a non-zero Estimated model cost, Models shows your API key as a caller identity, and Traces lists the fail-2 request as Failed.

Follow a request through the trace waterfall

A gateway-aware trace shows where time went and which policies ran, so you can tell a policy rejection from a backend failure at a glance.

1

Open a successful trace

On Traces, select a successful model call. Trace details shows the Trace ID, the number of Spans and Logs, and the Duration. For a model call, Model identity compares the Alias that the caller sent with the Responding model that answered, for example gpt-5-mini and gpt-5-mini-2025-08-07.

2

Read the waterfall

Trace waterfall draws parent-first spans on one timeline. The root span is the model request, for example Model ยท <model-name> on POST /default/models/*. Under it:

Child spanWhat it measures
inboundWork before the backend call. Each policy that ran is a child span, such as policy tokenLimit with its configuration.
backendThe call to the provider: a chat <alias> span with the routing decision, and its HTTP POST on /openai/v1/chat/completions.
outboundWork on the response before it returns to the client.
response transferSending the response back to the client.
The waterfall splits gateway time from backend time, and each policy that ran appears as its own span.
3

Read the correlated logs

Under Correlated logs, expand the request log and switch between Code and Tree. Look for:

  • gen_ai.operation.name (for example chat), gen_ai.request.model (the alias you sent), and gen_ai.response.model (the versioned model that answered).
  • azure.ai_gateway.client.usage.prompt_tokens, ...completion_tokens, and ...total_tokens: the usage the backend reported for this one call.
  • azure.ai_gateway.client.token.cost and azure.ai_gateway.currency: the call's estimated cost.
  • azure.ai_gateway.api_key_id: the API key that made the call.
  • azure.ai_gateway.routing.backend, ...routing.attempt, and ...routing.circuit.state: which backend served the call, on which attempt, and the state of its circuit breaker.
Checkpoint

You can point to the policy spans under inbound, compare backend time with total time, and read the token usage of one request in its correlated log.

Query telemetry with KQL

The dashboard reads the OTelSpans and OTelLogs tables in the Log Analytics workspace behind Application Insights. You can query them directly to build your own reports and alerts.

1

Open Logs

The tables are in a Log Analytics workspace: the Logs page of the Application Insights resource doesn't resolve OTelSpans. In the Azure portal, open the workspace, select Logs, and switch the editor to KQL mode:

  • Portal track: the wizard's Application Insights resource stores its data in a managed workspace, managed-<insights>-ws, in a managed resource group whose name starts with ai_<insights>_. The resource's Overview page links to it.
  • Azure CLI, Bicep + azd, and Terraform tracks: <gateway>-logs.
2

Run the starter queries

Copy a query from infra/queries.kql and run it. Start with query 2 (calls by alias, responding models, and API key) and query 3 (token usage and estimated cost by alias and API key). Query 8 lists the failed operations that the next section explores.

infra/queries.kqlGitHub
// Starting-point KQL queries for AI Gateway telemetry. Adapt them to your traffic before you rely on them.
//
// Where to run them: the Log Analytics workspace behind your Application Insights resource, Logs, KQL mode.
// Select one query and run it.
//
// AI Gateway exports OpenTelemetry (OTLP) signals. Traces land in the OTelSpans table and logs in the
// OTelLogs table; span and log attributes are in the dynamic Attributes column. Attribute names follow the
// OpenTelemetry GenAI and MCP semantic conventions (gen_ai.request.model, gen_ai.response.model,
// gen_ai.operation.name, gen_ai.tool.name, mcp.method.name) plus gateway attributes (azure.ai_gateway.*).
// See https://ai.gateway.azure.com/docs/observability/telemetry-reference, and run `OTelSpans | take 10`
// to check the names your data carries. Token and cost metrics also go to the Azure Monitor workspace
// (Prometheus), which these queries don't use.
//
// A request's span is the Server span that carries http.route, for example POST /default/models/*, and its
// request log in OTelLogs has the trace ID and the span ID of that span. A client that sends a W3C traceparent
// header puts all its requests in one trace, so the queries match spans to their request span, not to the trace.
// A request that the gateway retried or failed over logs one record per attempt, with the same span ID.

// 1. Model and tool server requests over time, by HTTP outcome
OTelSpans
| where TimeGenerated > ago(24h)
| where Kind == "Server" and isnotempty(Attributes["http.route"])
| extend Route = tostring(Attributes["http.route"]), Status = toint(Attributes["http.response.status_code"])
| extend Asset = case(Route contains "/models", "model", Route contains "/toolservers/", "tool server", "other"),
         Outcome = case(Status between (200 .. 399), "2xx-3xx", Status between (400 .. 499), "4xx", Status >= 500, "5xx", "unknown")
| summarize Requests = count() by bin(TimeGenerated, 15m), Series = strcat(Asset, " ", Outcome)
| render timechart

// 2. Model calls by alias and API key: responding models, errors, and latency
//    The log of a request's last attempt gives its outcome.
let requests = OTelSpans
| where TimeGenerated > ago(24h)
| where Kind == "Server" and isnotempty(Attributes["http.route"])
| project TraceId, SpanId, DurationMs;
OTelLogs
| where TimeGenerated > ago(24h)
| where isnotempty(Attributes["gen_ai.request.model"])
| extend Attempt = coalesce(toint(Attributes["azure.ai_gateway.routing.attempt"]), 1)
| summarize arg_max(Attempt, Attributes) by TraceId, SpanId
| extend Alias = tostring(Attributes["gen_ai.request.model"]),
         RespondingModel = tostring(Attributes["gen_ai.response.model"]),
         ApiKeyId = tostring(Attributes["azure.ai_gateway.api_key_id"]),
         Status = toint(Attributes["http.response.status_code"])
| join kind=leftouter requests on TraceId, SpanId
| summarize Calls = count(), Errors4xx = countif(Status between (400 .. 499)), Errors5xx = countif(Status >= 500),
            RespondingModels = make_set_if(RespondingModel, isnotempty(RespondingModel), 5),
            P95DurationMs = round(percentile(DurationMs, 95))
    by Alias, ApiKeyId
| order by Calls desc

// 3. Token usage and estimated cost reported for each alias and API key (from request logs)
//    Chat Completions reports prompt/completion tokens; the Responses API reports input/output tokens.
//    Cached input tokens are part of the input tokens. Missing usage means "unknown", not zero.
//    The estimated cost uses the gateway's model prices, in the currency the log reports.
OTelLogs
| where TimeGenerated > ago(24h)
| extend Alias = tostring(Attributes["gen_ai.request.model"]),
         ApiKeyId = tostring(Attributes["azure.ai_gateway.api_key_id"]),
         InputTokens = coalesce(tolong(Attributes["azure.ai_gateway.client.usage.prompt_tokens"]), tolong(Attributes["azure.ai_gateway.client.usage.input_tokens"])),
         OutputTokens = coalesce(tolong(Attributes["azure.ai_gateway.client.usage.completion_tokens"]), tolong(Attributes["azure.ai_gateway.client.usage.output_tokens"])),
         CachedInputTokens = coalesce(tolong(Attributes["azure.ai_gateway.client.usage.prompt_tokens_details.cached_tokens"]), tolong(Attributes["azure.ai_gateway.client.usage.input_tokens_details.cached_tokens"])),
         Cost = todouble(Attributes["azure.ai_gateway.client.token.cost"]),
         Currency = tostring(Attributes["azure.ai_gateway.currency"])
| where isnotnull(InputTokens) or isnotnull(OutputTokens)
| summarize Requests = count(), InputTokens = sum(InputTokens), CachedInputTokens = sum(CachedInputTokens),
            OutputTokens = sum(OutputTokens), EstimatedCost = round(sum(Cost), 6)
    by Alias, ApiKeyId, Currency
| extend TotalTokens = InputTokens + OutputTokens
| order by TotalTokens desc

// 4. Policy impact: which policies ran, and how often the request ended in 4xx
//    A 4xx on a request where a policy ran usually means that policy rejected it. Open the trace to confirm.
//    A policy span sits under a phase span, such as inbound, under the request span.
let spans = OTelSpans
| where TimeGenerated > ago(24h);
let requests = spans
| where Kind == "Server" and isnotempty(Attributes["http.route"])
| project TraceId, RequestSpanId = SpanId, Status = toint(Attributes["http.response.status_code"]);
let parents = spans | project TraceId, SpanId, ParentSpanId;
spans
| extend PolicyType = tostring(Attributes["azure.ai_gateway.policy.type"])
| where isnotempty(PolicyType)
| project TraceId, PolicyType, TimeGenerated, S1 = ParentSpanId
| lookup kind=leftouter (parents | project TraceId, S1 = SpanId, S2 = ParentSpanId) on TraceId, S1
| lookup kind=leftouter (parents | project TraceId, S2 = SpanId, S3 = ParentSpanId) on TraceId, S2
| mv-expand RequestSpanId = pack_array(S1, S2, S3) to typeof(string)
| join kind=inner requests on TraceId, RequestSpanId
| summarize Requests = dcount(RequestSpanId), RequestsEndingIn4xx = dcountif(RequestSpanId, Status between (400 .. 499)),
            LastExecuted = max(TimeGenerated)
    by PolicyType
| order by RequestsEndingIn4xx desc

// 5. Slowest model calls: total time versus time spent waiting for the backend
//    The backend span is a child of the request span. A request that a policy rejected has none.
let backend = OTelSpans
| where TimeGenerated > ago(24h)
| where Name == "backend"
| project TraceId, SpanId = ParentSpanId, BackendMs = DurationMs;
OTelSpans
| where TimeGenerated > ago(24h)
| where Kind == "Server" and tostring(Attributes["http.route"]) contains "/models/"
| project TimeGenerated, TraceId, SpanId, Model = tostring(Attributes["gen_ai.request.model"]), TotalMs = DurationMs
| join kind=leftouter backend on TraceId, SpanId
| extend GatewayMs = TotalMs - BackendMs
| project TimeGenerated, TraceId, Model, TotalMs, BackendMs, GatewayMs
| top 20 by TotalMs desc

// 6. MCP tool calls by tool server and tool, with failed calls
//    A tool that returns isError: true still answers HTTP 200; its tools/call span has the status Error.
OTelSpans
| where TimeGenerated > ago(24h)
| where Kind == "Server" and tostring(Attributes["mcp.method.name"]) == "tools/call"
| extend ToolServer = tostring(Attributes["azure.ai_gateway.mcp.tool_server.name"]),
         ToolName = tostring(Attributes["gen_ai.tool.name"]),
         ErrorType = tostring(Attributes["error.type"])
| summarize Calls = count(), Failed = countif(StatusCode == "Error"),
            ErrorTypes = make_set_if(ErrorType, isnotempty(ErrorType), 5)
    by ToolServer, ToolName
| order by Calls desc

// 7. Every span of one trace, in order (paste a trace ID from Monitoring โ€บ Traces)
let traceId = "<trace-id>";
OTelSpans
| where TimeGenerated > ago(30d)
| where TraceId == traceId
| project TimeGenerated, SpanId, ParentSpanId, Name, Kind, DurationMs, StatusCode,
          ErrorType = tostring(Attributes["error.type"]),
          PolicyType = tostring(Attributes["azure.ai_gateway.policy.type"]), Attributes
| order by TimeGenerated asc

// 8. Failed operations: requests with at least one span whose status is Error, whatever the HTTP status
//    Model-stream errors and MCP tool errors can answer HTTP 200 and still mark a span as Error.
//    error.type names the cause, for example tool_error or LlmAliasNotFound. A request to a path that the
//    gateway doesn't serve has no http.route; its request span is named after the method, for example
//    HTTP GET, and its error.type is OperationNotFound.
//    A request that the gateway retried or failed over is listed too, with the spans of its failed attempt, even
//    when a later attempt answered.
//    Each Error span belongs to the request span among itself and its three nearest ancestors, for example
//    HTTP POST โ€บ chat <model> โ€บ backend โ€บ request span.
let spans = OTelSpans
| where TimeGenerated > ago(24h);
let requests = spans
| where Kind == "Server" and (isnotempty(Attributes["http.route"]) or Name startswith "HTTP ")
| project TraceId, RequestSpanId = SpanId, Start = TimeGenerated, Request = Name,
          HttpStatus = toint(Attributes["http.response.status_code"]);
let parents = spans | project TraceId, SpanId, ParentSpanId;
spans
| where StatusCode == "Error"
| project TraceId, FailedSpan = Name, ErrorType = tostring(Attributes["error.type"]), S0 = SpanId, S1 = ParentSpanId
| lookup kind=leftouter (parents | project TraceId, S1 = SpanId, S2 = ParentSpanId) on TraceId, S1
| lookup kind=leftouter (parents | project TraceId, S2 = SpanId, S3 = ParentSpanId) on TraceId, S2
| mv-expand RequestSpanId = pack_array(S0, S1, S2, S3) to typeof(string)
| join kind=inner requests on TraceId, RequestSpanId
| summarize FailedSpans = make_set(FailedSpan, 5), ErrorTypes = make_set_if(ErrorType, isnotempty(ErrorType), 5)
    by Start, TraceId, Request, HttpStatus
| order by Start desc

On GitHub, open infra/queries.kql.

To run a query from the command line instead, pass the workspace's customer ID to az monitor log-analytics query. The command comes from the log-analytics extension, which the Azure CLI offers to install the first time. On the Azure CLI, Bicep + azd, and Terraform tracks, LAW_CUSTOMER_ID already holds the ID. Otherwise, set it to the Workspace ID on the workspace's Overview page. This short query counts the model calls of the last hour by alias and status code:

# Count the model calls of the last hour by alias and status code, from the gateway's spans in Log Analytics
echo "๐Ÿ“Š Model calls in the last hour, by alias and status code:"
az monitor log-analytics query -w "$LAW_CUSTOMER_ID" -o table --analytics-query \
  'OTelSpans | where TimeGenerated > ago(1h) | where Kind =~ "Server" and tostring(Attributes["http.route"]) contains "/models/" | summarize Calls = count() by Model = tostring(Attributes["gen_ai.request.model"]), Status = toint(Attributes["http.response.status_code"])'

These queries are starting points. Attribute names follow the OpenTelemetry GenAI conventions plus gateway attributes. Run OTelSpans | take 10 to see what your data carries, and adapt names before you build alerts on them.

Checkpoint

Query 2 returns one row per alias and API key with your call count, and query 3 returns input and output token totals that match the order of magnitude on the Models tab.

Find failed operations

HTTP status alone doesn't identify every failure. A model stream can fail after it has started, and an MCP tool can report an error in a successful response. In both cases, the gateway marks the span of the failed operation with the status Error and keeps http.response.status_code unchanged, so a 200 response can still carry a failure. The error.type attribute names the cause, for example tool_error for a tool that failed. Docs: Operation failures.

1

Open the failed tool call

On Traces, set the outcome filter to Completed with issues and open the fail-3 request, a Tools: 1 row. The request log under Correlated logs says 200 OK, and the root span Tool ยท learn is only Completed. The spans below it show the failure: backend, Tool ยท microsoft_docs_fetch, and tools/call microsoft_docs_fetch are Failed, and the last two carry tool_error.

A tool error answers 200 OK: the root span is Completed, and the spans that failed carry tool_error.
2

Compare the three failures

Each failing call that the traffic script sent leaves a different trace:

RequestHTTP statusOutcome on TracesSpans with the status Errorerror.type
fail-1, an unknown alias404Not listedinboundLlmAliasNotFound
fail-2, rejected by the provider400Failedchat <alias> and HTTP POST400
fail-3, a tool error200Completed with issuestools/call microsoft_docs_fetchtool_error

The fail-1 request doesn't name a model on the gateway, so it never appears on Traces: find it with KQL. The request log of fail-2 shows why it cost nothing: azure.ai_gateway.routing.failure.type is http_response, and azure.ai_gateway.client.token.cost is 0.

3

Query failed operations

In Logs, run query 8 from infra/queries.kql. It lists every request with at least one span whose status is Error, whatever its HTTP status, with the failed spans and their error.type values:

Text
Request                              HttpStatus  FailedSpans                          ErrorTypes
POST /default/toolservers/learn/mcp  200         ["tools/call microsoft_docs_fetch"]  ["tool_error"]
POST /default/models/*               400         ["HTTP POST","chat gpt-5-mini"]      ["400"]
POST /default/models                 404         ["inbound"]                          ["LlmAliasNotFound"]

The list can also show HTTP GET or HTTP POST requests with the status 404 and the error.type OperationNotFound: calls to a path that the gateway doesn't serve, such as a probe of the runtime URL. Their request span has no http.route. A model call that the gateway retried or failed over shows up too, with the spans of its failed attempt, even when a later attempt answered 200. Query 8 gives each request its own row, even when your app sends one traceparent for several requests.

Query 6 counts the failed calls of each tool, and query 7 shows the error.type of each span of one trace. The telemetry reference also lists rpc.response.status_code, the JSON-RPC error code of a failed MCP call, when the tool server reports one.

4

Read the routing attributes

The spans and logs also record why the gateway sent a call where it did. The azure.ai_gateway.routing.* attributes on the chat <alias> span and in the request log name the backend, the attempt, the circuit breaker state, the priority and weight, and the classified failure of a failed attempt. Requests to a Jev decision model set gen_ai.operation.name to decisions. Docs: Routing attributes and Jev decisions.

Checkpoint

You found the fail-3 tool call that answered 200 with Failed spans, and query 8 lists all three failing calls with their error.type.

Verify

1

Match the dashboard to your traffic

Run the traffic script once more with five requests, wait a few minutes, and refresh MonitoringOverview with 1H selected. Model calls grows by at least five.

2

Confirm all signals are flowing

Open SettingsEndpoints. Metrics endpoint, Logs endpoint, and Traces endpoint are all on: if any endpoint is disabled, the built-in dashboard is unavailable. SettingsGeneral shows your Application Insights resource as the Telemetry destination.

To check the exporter from the CLI, list the gateway's telemetry exporters:

# List the gateway's telemetry exporters, with the kind of each one
echo "๐Ÿ“Š Telemetry exporters on $GATEWAY:"
az aigateway telemetry-exporter list -g "$RG" --gateway-name "$GATEWAY" --query "[].{name:name, kind:properties.telemetryExporterType}" -o table

Every track creates an exporter of kind OpenTelemetry, which the 2026-11-01 API returns as properties.telemetryExporterType. A legacy exporter, from Troubleshooting, shows the kind ApplicationInsights in this list.

Go further

  • Query the token metrics. Token and cost metrics land in the Azure Monitor workspace linked to Application Insights as azure.ai_gateway.client.token.usage and azure.ai_gateway.client.token.cost, with labels such as gen_ai.request.model and gen_ai.token.type. Docs show PromQL in its underscore form, for example sum by (gen_ai_request_model) (gen_ai_client_token_usage). Check which names your backend received before you build on them. Attribute tokens and cost with OpenTelemetry queries them.
  • Alert on what you can act on. Start with token usage spikes, backend authentication failures, and telemetry export failures. Route each alert to an owner who can investigate the gateway, backend, policy, or destination.
  • Send telemetry somewhere else. In the Configure telemetry wizard, choose Custom OpenTelemetry destination to export to an OpenTelemetry Collector or any receiver that accepts OTLP/HTTP with binary Protobuf. Store any credential headers as secrets. The dashboard doesn't show data from a custom destination. Docs: Send telemetry to a custom destination.
  • Capture payloads carefully. SettingsLogging has Log request and response payloads. It's off by default because prompts and responses can be sensitive and add volume.

Clean up

Application Insights and Log Analytics bill for ingestion and retention. If you continue with Attribute tokens and cost with OpenTelemetry, keep everything.

Choose your track
1

Disable monitoring

On the Monitoring page, select SettingsGeneralDisable monitoring. This stops the export but doesn't delete the destination.

2

Delete the destination

In the Azure portal, delete the Application Insights resource that the wizard created, <gateway>-insights or the name you kept, if nothing else uses it. Azure deletes the managed Log Analytics workspace and the data collection endpoint and rule along with it, then its managed resource group once the Azure Monitor workspace in it is gone. Azure can take a long time to delete that workspace, and a deny assignment keeps you from deleting it yourself.

Keep the gateway, model, and key from lab 02: the next labs reuse them. azd down and terraform destroy don't touch them, because the Bicep + azd and Terraform tracks created their own gateways.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.