Chat apps and agents stream their replies: the model sends text a few tokens at a time, so people start reading after a second or two instead of waiting for the whole answer. Streaming used to be a blind spot for governance, because a streamed Chat Completions response only reports token usage when the client asks for it. In this lab you stream through the gateway with curl, PowerShell, and the OpenAI SDKs, read every event the client receives, and then prove that a token limit still counts every streamed token.
Before you begin
One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.
- Choose where to run commandsEvery track except the portal
- Sign in to AzureOn your own machine. Cloud Shell is already signed in.
- Install the
aigatewayextensionThe Azure CLI tracks and the Verify steps - Clone the labs repoThe Bicep + azd and Terraform tracks, the notebooks, and the code samples
You've done the one-time setup that this lab uses.
On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, model, key,
and the token-limited streaming alias, so you can skip this section. Run that track in Put a token limit on a
streaming alias first: it sets the variables that Stream a
response and Verify use. You need only the one-time setup above.
- Complete Publish a Foundry model and make your first call. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
- For the PowerShell steps: PowerShell 7.4 or later.
- For the Code track: Python 3.10 or later, or Node.js 22.18 or later.
- Permission to change the gateway, for example Contributor on the resource group, for the token limit steps.
- Run the scripts and samples from the lab's folder,
~/aigateway/labs/23-streaming-responses.
Set these variables once, from the lab folder (labs/23-streaming-responses). Every track and the Verify
section reuse them.
# Name your gateway, its model provider, and the backend model from lab 02, and the alias that this lab creates
export RG="<resource-group>"
export GATEWAY="<gateway>" # 24 characters or fewer
export PROVIDER="<provider-name>" # for example foundry-models
export MODEL="<model-name>" # the backend model name under that provider
export STREAM_ALIAS="$MODEL-stream" # the token-limited alias that this lab creates
echo "📝 Gateway $GATEWAY in $RG, model $MODEL on $PROVIDER, streaming alias $STREAM_ALIAS"
# Read the gateway's resource ID and its runtime URL
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🆔 Gateway ID: $GATEWAY_ID"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
# Read a starting guess for the alias: confirm it on the model's Aliases tab (lab 02), and replace it if it differs
export AI_GATEWAY_MODEL="$(az aigateway model show -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
--query properties.deployment.modelName -o tsv)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
# Paste the runtime key that you created in Access › API keys, without showing it: don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"# Name your gateway, its model provider, and the backend model from lab 02, and the alias that this lab creates
$RG = "<resource-group>"
$GATEWAY = "<gateway>" # 24 characters or fewer
$PROVIDER = "<provider-name>" # for example foundry-models
$MODEL = "<model-name>" # the backend model name under that provider
$STREAM_ALIAS = "$MODEL-stream" # the token-limited alias that this lab creates
Write-Host "📝 Gateway $GATEWAY in $RG, model $MODEL on $PROVIDER, streaming alias $STREAM_ALIAS"
# Read the gateway's resource ID and its runtime URL
$GATEWAY_ID = az aigateway show -g $RG -n $GATEWAY --query id -o tsv
$env:AI_GATEWAY_URL = "https://$(az aigateway show -g $RG -n $GATEWAY --query properties.frontend.defaultHostname -o tsv)"
Write-Host "🆔 Gateway ID: $GATEWAY_ID"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
# Read a starting guess for the alias: confirm it on the model's Aliases tab (lab 02), and replace it if it differs
$env:AI_GATEWAY_MODEL = az aigateway model show -g $RG --gateway-name $GATEWAY --provider-name $PROVIDER -n $MODEL `
--query properties.deployment.modelName -o tsv
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
# Paste the runtime key that you created in Access › API keys, without showing it: don't commit it or write it to a file
$env:AI_GATEWAY_API_KEY = Read-Host "AI Gateway API key" -MaskInput
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"The Code track only needs AI_GATEWAY_URL, AI_GATEWAY_MODEL, and AI_GATEWAY_API_KEY. If you skip the token
limit steps, you can set those three by hand and skip the Azure CLI.
How it works
Add "stream": true to a Chat Completions request and the gateway answers with Content-Type: text/event-stream. The body is a series of server-sent events (SSE): each one is a data: line with a JSON chunk,
followed by a blank line. The gateway passes each chunk on as the model produces it, and the response carries
x-accel-buffering: no, which tells proxies such as NGINX not to hold it back.
This is what a streamed reply from a Foundry model looks like through the gateway:
| Chunk | choices | What it carries |
|---|---|---|
| Prompt filter results | Empty [] | Content safety results for the prompt. id and model are empty strings. |
| Role | One choice | delta.role is assistant and delta.content is an empty string. |
| Content deltas | One choice | delta.content holds the next few characters of the reply. usage is null. |
| Finish | One choice | finish_reason is stop, or length when the reply hits max_completion_tokens. |
| Usage | Empty [] | usage with prompt_tokens, completion_tokens, and total_tokens. Sent when the client sets stream_options.include_usage. |
data: [DONE] | Not JSON | The end of the stream. Stop reading. |
Two chunks have an empty choices array, so code that reads choices[0] on every chunk crashes on the first one.
Where the gateway counts tokens. A token limit or cost limit is checked before the gateway forwards the
request. When a counter is exhausted, the client gets an ordinary JSON error, such as 429 with Retry-After, and no
stream starts. Once a stream starts, it runs to the end, even if it takes the counter past the limit; the next request
is throttled. While the stream flows, the gateway records its token usage for token limits, cost limits, and
Monitoring.
Usage without include_usage. Since the September 24, 2026 release,
the gateway makes sure it has a usage record for every streamed Chat Completions response, even when the client
doesn't set stream_options.include_usage, so the counters stay accurate. The release notes ask clients that parse
the events themselves to accept a usage-only chunk before [DONE], even when they didn't ask for one. In our test
run in November 2026, a client that didn't set include_usage received no usage chunk, yet the token limit counted
its tokens exactly as it did for a client that asked. So write your client to work either way: skip chunks with an
empty choices array, and take usage from whichever chunk carries it.
The Responses API streams typed events instead: each data: line follows an event: line, and the last event,
response.completed, always carries the usage. A streamed response is stored like any other, so you can continue
it with previous_response_id. The gateway sends that follow-up to the backend that stores the response, even
when the alias has several backend models. See
Session affinity.
Stream a response
Pick a track. Both stream the same request through the gateway and show the text as it arrives, the events, and the usage.
Watch the raw events
curl buffers its output by default; -N turns that off, so each event prints the moment it arrives. PowerShell's
Invoke-RestMethod and Invoke-WebRequest wait for the whole body, so the PowerShell script reads the stream
with HttpClient.
# Ask for a streamed reply: -N turns off curl's buffering, so that each event prints the moment it arrives
echo "🤖 Raw server-sent events from the $AI_GATEWAY_MODEL alias:"
curl -sSN "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
-d "{\"model\":\"$AI_GATEWAY_MODEL\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in five words.\"}]}"# Ask for a streamed reply: with -Raw, the script prints every server-sent event line the moment it arrives
Write-Host "🤖 Raw server-sent events from the $env:AI_GATEWAY_MODEL alias:"
./infra/stream-chat.ps1 -Raw -Prompt "Say hello in five words."Expected output (shortened; your text and IDs differ):
🤖 Raw server-sent events from the gpt-5-mini alias:
data: {"choices":[],"created":0,"id":"","model":"","object":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{…}}]}
data: {"choices":[{"delta":{"content":"","refusal":null,"role":"assistant"},"finish_reason":null,"index":0,…}],"model":"gpt-5-mini-2025-08-07","object":"chat.completion.chunk",…,"usage":null}
data: {"choices":[{"delta":{"content":"Hi"},"finish_reason":null,"index":0,…}],…,"usage":null}
data: {"choices":[{"delta":{"content":" there"},"finish_reason":null,"index":0,…}],…,"usage":null}
…
data: {"choices":[{"delta":{},"finish_reason":"stop","index":0,…}],…,"usage":null}
data: [DONE]Stream with the script
The script prints the reply as it streams, then counts the events, measures the time to the first token, and shows the usage chunk if the stream had one.
#!/usr/bin/env bash
# Streams a Chat Completions response through AI Gateway, prints the reply as it arrives, then prints how many
# server-sent events came back, how many had an empty choices array, the time to the first token, and the usage
# chunk if the stream ended with one.
#
# Usage:
# export AI_GATEWAY_URL="<gateway-url>"
# export AI_GATEWAY_MODEL="<model-name>"
# export AI_GATEWAY_API_KEY="..." # never commit this value
# ./stream-chat.sh [--usage] [--raw] ["Your prompt"]
#
# --usage sets stream_options.include_usage to true, so the stream ends with a usage chunk
# --raw prints every server-sent event line instead of the reply text
# Requires curl and jq 1.6 or later.
set -euo pipefail
USAGE=false
RAW=false
while [[ "${1:-}" == --* ]]; do
case "$1" in
--usage) USAGE=true ;;
--raw) RAW=true ;;
*) echo "Unknown option: $1" >&2; exit 2 ;;
esac
shift
done
PROMPT="${1:-Explain in three short sentences why streaming makes chat apps feel faster.}"
: "${AI_GATEWAY_URL:?Set AI_GATEWAY_URL to the gateway runtime URL, for example https://aigw-labs-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com}"
: "${AI_GATEWAY_MODEL:?Set AI_GATEWAY_MODEL to the callable model alias}"
: "${AI_GATEWAY_API_KEY:?Set AI_GATEWAY_API_KEY to a runtime key from Access > API keys}"
# reasoning_effort none spends the budget on the reply instead of hidden reasoning, so the first token arrives sooner.
BODY=$(jq -n --arg m "$AI_GATEWAY_MODEL" --arg p "$PROMPT" --argjson usage "$USAGE" '
{model: $m, stream: true, max_completion_tokens: 300, reasoning_effort: "none",
messages: [{role: "user", content: $p}]}
+ (if $usage then {stream_options: {include_usage: true}} else {} end)')
headers=$(mktemp)
trap 'rm -f "$headers"' EXIT
# -N turns off curl's output buffering. jq reads the raw lines as they arrive and keeps running totals.
curl -sSN -D "$headers" "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" -d "$BODY" \
| jq -nRrj --unbuffered --argjson raw "$RAW" '
def secs: (. * 100 | round) / 100 | tostring + " s";
def summary:
"\n\n"
+ (if .events == 0 then "Response body \(.other)\n"
else "Events \(.events) (\(.empty) with an empty choices array)\n"
+ "First token \(if .ttft then (.ttft | secs) else "-" end)\n"
+ "Usage chunk "
+ (if .usage then "prompt \(.usage.prompt_tokens), completion \(.usage.completion_tokens), total \(.usage.total_tokens)"
else "none: run again with --usage to ask for one" end) + "\n"
end);
foreach (inputs, null) as $line (
{t0: now, events: 0, empty: 0, ttft: null, usage: null, other: "", out: ""};
.out = ""
| if $line == null then .end = true
elif ($line | startswith("data: ")) then
($line[6:]) as $data
| if $data == "[DONE]" then .out = (if $raw then $line + "\n" else "" end)
else ($data | fromjson) as $chunk
| ([$chunk.choices[]?.delta.content // empty] | join("")) as $text
| .events += 1
| .empty += (if ($chunk.choices // []) == [] then 1 else 0 end)
| .usage = ($chunk.usage // .usage)
| .ttft = (if .ttft == null and $text != "" then now - .t0 else .ttft end)
| .out = (if $raw then $line + "\n" else $text end)
end
else .other += $line
end;
if .end then summary else .out end)'
status=$(awk 'toupper($1) ~ /^HTTP/ {code=$2} END {print code}' "$headers")
type=$(awk -F': ' 'tolower($1)=="content-type" {gsub("\r","",$2); print $2}' "$headers")
served=$(awk -F': ' 'tolower($1)=="x-ms-served-model" {gsub("\r","",$2); print $2}' "$headers")
retry=$(awk -F': ' 'tolower($1)=="retry-after" {gsub("\r","",$2); print $2}' "$headers")
printf 'HTTP status %s\n' "$status"
printf 'Content-Type %s\n' "${type:--}"
printf 'Served by %s\n' "${served:--}"
[[ -n "$retry" ]] && printf 'Retry-After %s\n' "$retry"
[[ "$status" == 200 ]]# Stream a reply, then print the event count, the time to the first token, the usage chunk, and the headers
echo "🤖 Reply from the $AI_GATEWAY_MODEL alias, as it streams:"
./infra/stream-chat.sh# Stream a reply, then print the event count, the time to the first token, the usage chunk, and the headers
Write-Host "🤖 Reply from the $env:AI_GATEWAY_MODEL alias, as it streams:"
./infra/stream-chat.ps1On GitHub, open stream-chat.sh or stream-chat.ps1.
Expected output (your text and numbers differ):
🤖 Reply from the gpt-5-mini alias, as it streams:
Streaming shows replies as they're generated, so users see progress right away. They can start reading before the full answer is ready. That makes the app feel more responsive than waiting for one complete message.
Events 43 (1 with an empty choices array)
First token 1.77 s
Usage chunk none: run again with --usage to ask for one
HTTP status 200
Content-Type text/event-stream; charset=utf-8
Served by gpt-5-mini-2025-08-07Ask for usage at the end of the stream
Run the script again with the option that sets "stream_options": {"include_usage": true}:
# Stream again with stream_options.include_usage set, so that the stream ends with a usage chunk
echo "🤖 Reply from the $AI_GATEWAY_MODEL alias, with a usage chunk at the end:"
./infra/stream-chat.sh --usage# Stream again with stream_options.include_usage set, so that the stream ends with a usage chunk
Write-Host "🤖 Reply from the $env:AI_GATEWAY_MODEL alias, with a usage chunk at the end:"
./infra/stream-chat.ps1 -UsageExpected output:
🤖 Reply from the gpt-5-mini alias, with a usage chunk at the end:
Streaming shows the reply as it's generated instead of waiting for the whole answer. You can start reading right away. That makes the app feel more responsive.
Events 35 (2 with an empty choices array)
First token 1.74 s
Usage chunk prompt 19, completion 34, total 53
HTTP status 200
Content-Type text/event-stream; charset=utf-8
Served by gpt-5-mini-2025-08-07The second empty-choices chunk is the usage chunk, the last one before [DONE]. stream_options is only valid
together with "stream": true; without it, the gateway returns 400.
Stream the Responses API
The Responses API names every event. List the event types in order, then read the usage from response.completed:
# Stream a Responses API reply into a file, so that you can count its events
curl -sSN "${AI_GATEWAY_URL%/}/default/models/openai/v1/responses" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
-d "{\"model\":\"$AI_GATEWAY_MODEL\",\"stream\":true,\"input\":\"Say hello in five words.\"}" > events.txt &&
echo "🤖 Stream received from the $AI_GATEWAY_MODEL alias"
# Count each run of the same event type, in order
echo "📊 Event types, in order:"
grep '^event:' events.txt | uniq -c
# Read the usage from the response.completed event, then delete the file
echo "📊 Usage from response.completed:"
sed -n 's/^data: {/{/p' events.txt | jq -c 'select(.type == "response.completed") | .response.usage'
rm events.txt# Build a streamed Responses API request
$body = @{ model = $env:AI_GATEWAY_MODEL; stream = $true; input = "Say hello in five words." } | ConvertTo-Json
# Send it: Invoke-WebRequest waits for the whole stream, which is fine here because you only count its events
$r = Invoke-WebRequest -Method Post -Uri "$($env:AI_GATEWAY_URL.TrimEnd('/'))/default/models/openai/v1/responses" `
-Headers @{ Authorization = "Bearer $env:AI_GATEWAY_API_KEY" } -ContentType "application/json" -Body $body
if ($?) { Write-Host "🤖 Stream received from the $env:AI_GATEWAY_MODEL alias" }
# Count each run of the same event type, in order, the way uniq -c does
Write-Host "📊 Event types, in order:"
$lines = $r.Content -split "`n"
$prev = $null; $n = 0
foreach ($e in $lines -match '^event:') { if ($e -eq $prev) { $n++ } else { if ($prev) { '{0,4} {1}' -f $n, $prev }; $prev = $e; $n = 1 } }
'{0,4} {1}' -f $n, $prev
# Read the usage from the response.completed event
Write-Host "📊 Usage from response.completed:"
($lines -match '^data: \{' | ForEach-Object { $_.Substring(6) | ConvertFrom-Json } |
Where-Object type -eq 'response.completed').response.usage | ConvertTo-Json -CompressExpected output (your counts differ):
🤖 Stream received from the gpt-5-mini alias
📊 Event types, in order:
1 event: response.created
1 event: response.in_progress
1 event: response.output_item.added
1 event: response.output_item.done
1 event: response.output_item.added
1 event: response.content_part.added
7 event: response.output_text.delta
1 event: response.output_text.done
1 event: response.content_part.done
1 event: response.output_item.done
1 event: response.completed
📊 Usage from response.completed:
{"input_tokens":12,"input_tokens_details":{"cache_write_tokens":0,"cached_tokens":0},"output_tokens":66,"output_tokens_details":{"reasoning_tokens":53},"total_tokens":78}The first output_item pair is the model's reasoning item; the second is the message that carries the text. The
reasoning_tokens count shows why the first token can take a moment with a reasoning model.
Install the SDK
Each sample reads AI_GATEWAY_URL, AI_GATEWAY_MODEL, and AI_GATEWAY_API_KEY from the environment. Run the
commands from the lab folder.
# Move into the samples' folder, and install their requirements
cd infra
pip install -r requirements.txt # the openai package
echo "📦 The OpenAI SDK is installed"# Move into the samples' folder, and install their dependencies
cd infra
npm install # the openai npm package
echo "📦 The OpenAI SDK is installed"Stream from code
Pass stream=True and iterate over the chunks. Skip chunks without choices, and keep the usage from the chunk that
has it.
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Explain in two sentences why streaming makes chat apps feel faster."}],
stream=True,
stream_options={"include_usage": True},
)
usage = None
for chunk in stream:
if chunk.usage:
usage = chunk.usage
if not chunk.choices: # the prompt filter chunk and the usage chunk
continue
print(chunk.choices[0].delta.content or "", end="", flush=True)
print(f"\nTokens used: {usage.total_tokens if usage else 'no usage chunk'}")const stream = await client.chat.completions.create({
model,
messages: [{ role: "user", content: "Explain in two sentences why streaming makes chat apps feel faster." }],
stream: true,
stream_options: { include_usage: true },
});
let usage: OpenAI.CompletionUsage | undefined;
for await (const chunk of stream) {
if (chunk.usage) usage = chunk.usage;
// choices[0] is undefined on the prompt filter chunk and the usage chunk.
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
console.log(`\nTokens used: ${usage?.total_tokens ?? "no usage chunk"}`);The Responses API streams typed events. Print response.output_text.delta events and read the usage from
response.completed:
for event in client.responses.create(model=model, input="Write a one-sentence status update.", stream=True):
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.completed":
print(f"\nTokens used: {event.response.usage.total_tokens}")const events = await client.responses.create({ model, input: "Write a one-sentence status update.", stream: true });
for await (const event of events) {
if (event.type === "response.output_text.delta") process.stdout.write(event.delta);
else if (event.type === "response.completed") console.log(`\nTokens used: ${event.response.usage?.total_tokens}`);
}The full samples create the client, time the first token, and print both streams: open stream.py and stream.ts.
Run the sample
# Run the sample: it streams a Chat Completions reply, then a Responses API reply
echo "🤖 Streamed replies through the gateway:"
python stream.py# Run the sample: it streams a Chat Completions reply, then a Responses API reply
echo "🤖 Streamed replies through the gateway:"
node stream.ts # Node.js 22.18 or later; on older versions run: npx tsx stream.tsExpected output (your text differs):
🤖 Streamed replies through the gateway:
Chat Completions:
Streaming shows each response as it's generated instead of making you wait for the whole message, so you see progress right away. It also lets you start reading or reacting before the response is complete.
First token: 1.77 s
Tokens used: 74 (prompt 18, completion 56)
Responses API:
Streaming is running smoothly, with no interruptions reported.
Tokens used: 66 (input 16, output 50)The reply appears a few words at a time, and you can name the chunk that carries the usage in each API.
Put a token limit on a streaming alias
To find out whether the gateway counts streamed tokens, give a new alias, <model-name>-stream, a deliberately low
limit: 1,000 tokens per minute for each caller identity. It targets the same backend model as your alias from
lab 02, so clients that call that alias aren't affected.
Create the alias
- Open the AI Gateway portal and select your gateway in the top bar.
- Select ModelsAliasesAddAlias for existing models.
- On Basics, set Alias name to
<model-name>-stream, for examplegpt-5-mini-stream. Select Next. - Keep the defaults on Auto model selection (preview), and select Next.
- On Select models, select your model in the list. Select Next.
- Keep the defaults on Session affinity, Priorities and weights, and Failure handling, then select Create on Review.
Add the token limit
- Select PoliciesAdd policy. On the Type step, under Cost and rate limits, select Token rate limit, then Next.
- On the Scope step, select Individual assets, select the
<model-name>-streamalias, then Next. - On the Details step, keep Time period at Per minute and Counter dimensions at Identity, set
Allowed tokens to
1000, and select Create.
ModelsAliases lists <model-name>-stream, and Policies lists 1,000 tokens / minute per caller
identity, a Token rate limit on that alias.
Create the alias with its token limit
An alias carries its own policies array, so one command creates the alias and its limit together.
# Create or update the streaming alias for the same backend model, with its own token limit of 1,000 tokens per minute
# for each caller identity, and show the alias and its policies
echo "🔖 Alias $STREAM_ALIAS and its policies:"
az aigateway model-alias set -g "$RG" --gateway-name "$GATEWAY" -n "$STREAM_ALIAS" \
--description "Streaming lab: 1,000 tokens per minute for each caller" \
--backend-model resource-id="$GATEWAY_ID/workspaces/default/modelProviders/$PROVIDER/models/$MODEL" \
--policies '[{"type": "tokenLimit", "count": 1000, "period": "minute", "counterKey": ["identity"]}]' \
--only-show-errors --query "{alias:name, policies:properties.policies}" -o json# Write the token limit of 1,000 tokens per minute for each caller identity to a file, so that PowerShell
# passes the JSON to az unchanged
ConvertTo-Json -Depth 10 -Compress -InputObject @(
@{ type = "tokenLimit"; count = 1000; period = "minute"; counterKey = @("identity") }
) | Set-Content policies.json
# Create or update the streaming alias for the same backend model, with that limit, and show it and its policies
Write-Host "🔖 Alias $STREAM_ALIAS and its policies:"
az aigateway model-alias set -g $RG --gateway-name $GATEWAY -n $STREAM_ALIAS `
--description "Streaming lab: 1,000 tokens per minute for each caller" `
--backend-model resource-id="$GATEWAY_ID/workspaces/default/modelProviders/$PROVIDER/models/$MODEL" `
--policies "@policies.json" `
--only-show-errors --query "{alias:name, policies:properties.policies}" -o json
Remove-Item policies.jsonaz aigateway model-alias set creates the alias, or updates it if it exists, so you can run the step again.
The output shows your alias and one policy with "type": "tokenLimit", "count": 1000, "period": "minute",
"counterKey": ["identity"], and "scope": "resource".
What you need: only the one-time setup from Before you begin. This track doesn't use your gateway from lab 02:
azd up creates a new one, so it works on its own.
Review the template
infra/main.bicep creates the resource group rg-<environment name> and calls the quick start's
resources.bicep for the gateway, the Foundry model, its default alias, and the
app-first-call key. Then it reuses the template from
Throttle tokens per caller with token limits,
limited-alias.bicep, which publishes a token-limited alias for an
existing model, with the streaming alias name and the limit of 1,000 tokens per minute for each caller identity:
// Lab 23 with azd: "azd up" creates the resource group rg-<environment name>, the quick start's gateway, Foundry
// account, gpt-5-mini alias, and app-first-call key, then reuses lab 06's limited-alias.bicep to publish the
// gpt-5-mini-stream alias with a limit of 1,000 tokens per minute for each caller. The key value is never an output.
targetScope = 'subscription'
@description('azd environment name. The resource group is rg-<environmentName>.')
@minLength(1)
@maxLength(60)
param environmentName string
@description('Region for every resource. Pick one where AI Gateway and the lab\'s Foundry models are available. List them with az provider show --namespace Microsoft.ApiManagement (Setup guide, Find the available regions).')
@metadata({ azd: { type: 'location' } })
param location string
var suffix = take(uniqueString(subscription().id, environmentName, location), 6)
var tags = { 'azd-env-name': environmentName }
resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = {
name: 'rg-${environmentName}'
location: location
tags: tags
}
// The quick start's template: a gateway, a Foundry account with gpt-5-mini, the gpt-5-mini alias, and a key.
module base '../../quickstart/infra/resources.bicep' = {
name: 'aigw-base'
scope: rg
params: {
location: location
tags: tags
gatewayName: 'aigw-${suffix}'
foundryName: 'aigw${suffix}'
}
}
// Lab 06's template: a second alias for the same backend model, with a deliberately low token limit.
module streamAlias '../../06-token-limits/infra/limited-alias.bicep' = {
name: 'aigw-stream-alias'
scope: rg
params: {
gatewayName: base.outputs.gatewayName
providerName: base.outputs.providerName
modelName: base.outputs.modelName
aliasName: '${base.outputs.modelName}-stream'
allowedTokens: 1000
period: 'minute'
counterKey: [
'identity'
]
}
}
output AZURE_RESOURCE_GROUP string = rg.name
output AI_GATEWAY_NAME string = base.outputs.gatewayName
output AI_GATEWAY_ID string = base.outputs.gatewayId
output AI_GATEWAY_URL string = base.outputs.gatewayUrl
output AI_GATEWAY_PROVIDER string = base.outputs.providerName
output AI_GATEWAY_BACKEND_MODEL string = base.outputs.modelName
output AI_GATEWAY_MODEL string = base.outputs.aliasName
output AI_GATEWAY_STREAM_ALIAS string = streamAlias.outputs.aliasName
output FOUNDRY_ACCOUNT_NAME string = base.outputs.foundryNameDeploy with azd up
Move into the lab's folder, create an azd environment named aigw-lab23 in your current subscription, then provision
everything. It takes about three minutes.
# Move into the lab's folder, which holds the azd project
cd ~/aigateway/labs/23-streaming-responses &&
echo "📂 Lab folder: $(pwd)"
# Create an azd environment in your current subscription, so that azd doesn't prompt you
azd env new aigw-lab23 --location eastus2 --subscription "$(az account show --query id --output tsv)" &&
echo "📝 azd environment aigw-lab23 deploys to eastus2"
# Provision the gateway, the Foundry model, its default alias, the key, and the token-limited streaming alias
azd up &&
echo "🚀 Gateway deployed: $(azd env get-value AI_GATEWAY_URL)"# Move into the lab's folder, which holds the azd project
Set-Location ~/aigateway/labs/23-streaming-responses
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }
# Create an azd environment in your current subscription, so that azd doesn't prompt you
azd env new aigw-lab23 --location eastus2 --subscription (az account show --query id --output tsv)
if ($?) { Write-Host "📝 azd environment aigw-lab23 deploys to eastus2" }
# Provision the gateway, the Foundry model, its default alias, the key, and the token-limited streaming alias
azd up
if ($?) { Write-Host "🚀 Gateway deployed: $(azd env get-value AI_GATEWAY_URL)" }For another region, pick one where AI Gateway and the gpt-5-mini model are available, as in Find the available
regions.
Read the outputs and the key
Set the variables that the rest of the lab uses from azd's environment values, and read the key from the gateway, so that it never shows on screen.
# Read the template's outputs into the variables that the rest of the lab uses
export RG="$(azd env get-value AZURE_RESOURCE_GROUP)"
export GATEWAY="$(azd env get-value AI_GATEWAY_NAME)"
export PROVIDER="$(azd env get-value AI_GATEWAY_PROVIDER)"
export MODEL="$(azd env get-value AI_GATEWAY_BACKEND_MODEL)"
export STREAM_ALIAS="$(azd env get-value AI_GATEWAY_STREAM_ALIAS)"
export GATEWAY_ID="$(azd env get-value AI_GATEWAY_ID)"
export AI_GATEWAY_URL="$(azd env get-value AI_GATEWAY_URL)"
export AI_GATEWAY_MODEL="$(azd env get-value AI_GATEWAY_MODEL)"
# Read the key from the gateway, so that it never shows on screen or lands in a file
export AI_GATEWAY_API_KEY="$(az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" \
--query primaryKey --output tsv)"
echo "📝 Gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
echo "🔖 Streaming alias: $STREAM_ALIAS"# Read the template's outputs into the variables that the rest of the lab uses
$RG = azd env get-value AZURE_RESOURCE_GROUP
$GATEWAY = azd env get-value AI_GATEWAY_NAME
$PROVIDER = azd env get-value AI_GATEWAY_PROVIDER
$MODEL = azd env get-value AI_GATEWAY_BACKEND_MODEL
$STREAM_ALIAS = azd env get-value AI_GATEWAY_STREAM_ALIAS
$GATEWAY_ID = azd env get-value AI_GATEWAY_ID
$env:AI_GATEWAY_URL = azd env get-value AI_GATEWAY_URL
$env:AI_GATEWAY_MODEL = azd env get-value AI_GATEWAY_MODEL
# Read the key from the gateway, so that it never shows on screen or lands in a file
$env:AI_GATEWAY_API_KEY = az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" `
--query primaryKey --output tsv
Write-Host "📝 Gateway $GATEWAY in $RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"
Write-Host "🔖 Streaming alias: $STREAM_ALIAS"AI_GATEWAY_MODEL is the default alias, gpt-5-mini, which has no token limit, so the steps in
Stream a response aren't throttled. Verify calls the streaming alias.
azd up ended with SUCCESS, and the last command printed 🔖 Streaming alias: gpt-5-mini-stream. In
the portal, ModelsAliases lists it next to gpt-5-mini, and Policies lists a Token rate limit
scoped to it. If you came here from Before you begin, go back to Stream a response now.
What you need: only the one-time setup from Before you begin, plus Terraform 1.9 or later, as in
Deploy a lab with Terraform. This track doesn't use your gateway from
lab 02: terraform apply creates a new one, so it works on its own.
Review the configuration
terraform/main.tf creates the resource group rg-<environment_name> and calls the quick start's
resources module for the gateway, the Foundry model, its default alias, and the
app-first-call key:
# Lab 23 with Terraform: creates the resource group rg-<environment_name>, the quick start's gateway, Foundry
# account, gpt-5-mini alias, and app-first-call key, then the gpt-5-mini-stream alias with a limit of 1,000 tokens
# per minute for each caller (stream-alias.tf). The key value is never an output.
data "azapi_client_config" "current" {}
locals {
suffix = substr(sha1("${data.azapi_client_config.current.subscription_id}/${var.environment_name}/${var.location}"), 0, 6)
tags = { "environment-name" = var.environment_name }
}
resource "azurerm_resource_group" "rg" {
name = "rg-${var.environment_name}"
location = var.location
tags = local.tags
}
# The quick start's module: a gateway, a Foundry account with gpt-5-mini, the gpt-5-mini alias, and a key.
module "base" {
source = "../../quickstart/terraform/resources"
subscription_id = data.azapi_client_config.current.subscription_id
resource_group_name = azurerm_resource_group.rg.name
resource_group_id = azurerm_resource_group.rg.id
location = var.location
tags = local.tags
gateway_name = "aigw-${local.suffix}"
foundry_name = "aigw${local.suffix}"
}terraform/stream-alias.tf publishes the streaming alias with the azapi provider, which sends the AI Gateway API
version, 2026-11-01, as is. Like the Bicep template, it targets the same backend model as the default alias and
carries a limit of 1,000 tokens per minute for each caller identity:
# The streaming alias, like lab 06's limited alias: a second alias for the same backend model, with a deliberately low
# limit of 1,000 tokens per minute for each caller identity. Clients that call the gpt-5-mini alias aren't affected.
resource "azapi_resource" "stream_alias" {
type = "Microsoft.ApiManagement/aigateways/workspaces/aliases@2026-11-01"
name = "${module.base.model_name}-stream"
parent_id = module.base.workspace_id
body = {
properties = {
backendModels = [
{ resourceId = module.base.model_id }
]
policies = [
{
type = "tokenLimit"
count = 1000
period = "minute"
counterKey = ["identity"]
}
]
}
}
schema_validation_enabled = false
}Deploy with terraform apply
Move into the lab's Terraform folder, select your current subscription, then plan and apply the configuration. It takes about three minutes.
# Move into the lab's Terraform folder
cd ~/aigateway/labs/23-streaming-responses/terraform &&
echo "📂 Terraform folder: $(pwd)"
# Deploy to the Azure CLI's current subscription
export ARM_SUBSCRIPTION_ID="$(az account show --query id --output tsv)" &&
echo "📍 Subscription: $ARM_SUBSCRIPTION_ID"
# Download the providers, then plan and apply the gateway, the Foundry model, its default alias, the key, and the
# token-limited streaming alias
terraform init &&
terraform plan -out tfplan &&
terraform apply tfplan &&
echo "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)"# Move into the lab's Terraform folder
Set-Location ~/aigateway/labs/23-streaming-responses/terraform
if ($?) { Write-Host "📂 Terraform folder: $(Get-Location)" }
# Deploy to the Azure CLI's current subscription
$env:ARM_SUBSCRIPTION_ID = az account show --query id --output tsv
if ($?) { Write-Host "📍 Subscription: $env:ARM_SUBSCRIPTION_ID" }
# Download the providers, then plan and apply the gateway, the Foundry model, its default alias, the key, and the
# token-limited streaming alias
terraform init
if ($?) { terraform plan -out tfplan }
if ($?) { terraform apply tfplan }
if ($?) { Write-Host "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)" }The configuration deploys to eastus2. For another region where AI Gateway and the gpt-5-mini model are
available, as in Find the available regions, run
export TF_VAR_location=<region> (PowerShell: $env:TF_VAR_location = "<region>") before terraform plan.
Read the outputs and the key
Set the variables that the rest of the lab uses from the Terraform outputs, read the key from the gateway, so that it never shows on screen or lands in the Terraform state, then move back to the lab folder, where the scripts are.
# Read the configuration's outputs into the variables that the rest of the lab uses
export RG="$(terraform output -raw azure_resource_group)"
export GATEWAY="$(terraform output -raw ai_gateway_name)"
export PROVIDER="$(terraform output -raw ai_gateway_provider)"
export MODEL="$(terraform output -raw ai_gateway_backend_model)"
export STREAM_ALIAS="$(terraform output -raw ai_gateway_stream_alias)"
export GATEWAY_ID="$(terraform output -raw ai_gateway_id)"
export AI_GATEWAY_URL="$(terraform output -raw ai_gateway_url)"
export AI_GATEWAY_MODEL="$(terraform output -raw ai_gateway_model)"
# Read the key from the gateway, so that it never lands in a file or the Terraform state
export AI_GATEWAY_API_KEY="$(az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" \
--query primaryKey --output tsv)"
echo "📝 Gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
echo "🔖 Streaming alias: $STREAM_ALIAS"
# Move back to the lab folder, where the scripts are
cd .. && echo "📂 Lab folder: $(pwd)"# Read the configuration's outputs into the variables that the rest of the lab uses
$RG = terraform output -raw azure_resource_group
$GATEWAY = terraform output -raw ai_gateway_name
$PROVIDER = terraform output -raw ai_gateway_provider
$MODEL = terraform output -raw ai_gateway_backend_model
$STREAM_ALIAS = terraform output -raw ai_gateway_stream_alias
$GATEWAY_ID = terraform output -raw ai_gateway_id
$env:AI_GATEWAY_URL = terraform output -raw ai_gateway_url
$env:AI_GATEWAY_MODEL = terraform output -raw ai_gateway_model
# Read the key from the gateway, so that it never lands in a file or the Terraform state
$env:AI_GATEWAY_API_KEY = az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" `
--query primaryKey --output tsv
Write-Host "📝 Gateway $GATEWAY in $RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"
Write-Host "🔖 Streaming alias: $STREAM_ALIAS"
# Move back to the lab folder, where the scripts are
Set-Location ..
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }AI_GATEWAY_MODEL is the default alias, gpt-5-mini, which has no token limit, so the steps in
Stream a response aren't throttled. Verify calls the streaming alias.
terraform apply ended with Apply complete! Resources: 10 added, 0 changed, 0 destroyed., and the
outputs command printed 🔖 Streaming alias: gpt-5-mini-stream. If you came here from Before you begin, go
back to Stream a response now.
Verify
Send bursts of streamed requests that each use about 220 tokens: first without asking for usage, then with it. If the gateway only counted the tokens that clients report, the first burst would never be throttled.
Stream a burst without asking for usage
For each request, the script prints the status, Retry-After, the number of events, and the usage chunk, if any.
# Send 8 streamed requests to the streaming alias, without asking for usage
echo "🚦 Expect 429 Too Many Requests after about 1,000 tokens, with no usage chunks:"
./infra/stream-burst.sh "$STREAM_ALIAS" 8# Send 8 streamed requests to the streaming alias, without asking for usage
Write-Host "🚦 Expect 429 Too Many Requests after about 1,000 tokens, with no usage chunks:"
./infra/stream-burst.ps1 -Model $STREAM_ALIAS -Requests 8On GitHub, open stream-burst.sh or stream-burst.ps1.
Expected output (your counts differ):
🚦 Expect 429 Too Many Requests after about 1,000 tokens, with no usage chunks:
Request Status Retry-After Events Usage chunk
1 200 - 209 none
2 200 - 187 none
3 200 - 207 none
4 200 - 200 none
5 200 - 196 none
6 200 - 191 none
7 429 2 - Token limit is exceeded. Try again in 2 seconds.
8 429 1 - Token limit is exceeded. Try again in 1 seconds.No stream carried a usage chunk, yet the gateway throttled request 7, after about 1,300 tokens. If it had counted only the prompts, about 25 tokens each, the alias would have allowed roughly 40 requests.
A new alias or policy can take up to a minute to apply, and the counters catch up within a few seconds, so a fast
burst can go a few hundred tokens past the limit. If every request returns 200, wait a minute and run it again.
Stream a burst with usage
Wait a minute for the window to reset, then run the same burst with include_usage:
# Wait a minute, so that the token limit's window resets
echo "⏳ Waiting 60 seconds for the token limit's window to reset"
sleep 60
# Send the same burst with stream_options.include_usage, so that each stream ends with a usage chunk
echo "🚦 Expect 429 Too Many Requests at the same point, now with usage chunks:"
./infra/stream-burst.sh --usage "$STREAM_ALIAS" 8# Wait a minute, so that the token limit's window resets
Write-Host "⏳ Waiting 60 seconds for the token limit's window to reset"
Start-Sleep 60
# Send the same burst with stream_options.include_usage, so that each stream ends with a usage chunk
Write-Host "🚦 Expect 429 Too Many Requests at the same point, now with usage chunks:"
./infra/stream-burst.ps1 -Model $STREAM_ALIAS -Requests 8 -UsageExpected output:
⏳ Waiting 60 seconds for the token limit's window to reset
🚦 Expect 429 Too Many Requests at the same point, now with usage chunks:
Request Status Retry-After Events Usage chunk
1 200 - 183 211 tokens
2 200 - 186 214 tokens
3 200 - 202 230 tokens
4 200 - 181 209 tokens
5 200 - 180 208 tokens
6 200 - 188 216 tokens
7 429 5 - Token limit is exceeded. Try again in 5 seconds.
8 429 5 - Token limit is exceeded. Try again in 5 seconds.The throttle point is the same: the gateway counted the streams the same way whether or not the client asked for
usage. If a request after the 429 returns 200, it was sent after the Retry-After wait.
Inspect a throttled streaming request
Run the burst once more and, while the alias is throttled, look at a full response:
# Send one streamed request while the alias is throttled: -i prints the status line and the headers too
echo "🚦 Expect 429 Too Many Requests while the limit is active, as plain JSON:"
curl -sS -i "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
-d "{\"model\":\"$STREAM_ALIAS\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}"Expected output, when the limit is active:
🚦 Expect 429 Too Many Requests while the limit is active, as plain JSON:
HTTP/2 429
content-type: application/json
retry-after: 3
content-length: 84
{ "statusCode": 429, "message": "Token limit is exceeded. Try again in 3 seconds." }The throttled answer is plain JSON, not an event stream, because the gateway refused the request before it reached
the model. Check the status code before you start reading events, and honor Retry-After.
Find the streamed calls in Monitoring
If you set up telemetry in Monitor usage, cost, and traces, open Monitoring. On Overview, Top models lists the streaming alias with its calls and estimated cost next to the non-streamed calls, and Top blocking policies counts the throttled requests under Token limit. On Traces, the throttled requests show the outcome Blocked by policy, and the streams before them show Token limit under Applied policies.
Both bursts were throttled at the same request, the burst without include_usage included, and the throttled
request returned a JSON 429 with Retry-After instead of a stream.
Go further
- Cap spend for streams. A cost limit is estimated from the same usage record, so streamed calls count against budgets too. Try it in Enforce cost budgets.
- Shorten the time to the first token. Reasoning models think before they write. For chat that needs a quick
first word, set
reasoning_efforttolowornonewhere the model supports it, as the scripts in this lab do. - Stream from an agent. Agent frameworks stream through the same endpoint. See Connect agent frameworks to models and tools.
- Check your proxies. Anything between the client and the gateway, such as a reverse proxy or an API layer of your
own, must pass
text/event-streamresponses through without buffering. - Find streams that fail after they start. A stream that fails partway, for example with a Responses
response.failedevent, has already returned HTTP200. With tracing on, the gateway marks the span asErrorand keeps the status code. See Operation failures.
Clean up
Delete the streaming alias. Its token limit goes with it, and the backend model and your alias from lab 02 stay.
Delete the streaming alias
Open <model-name>-stream in ModelsAliases and select Delete.
Delete the streaming alias
# Delete the streaming alias, with its token limit: the backend model and your alias from lab 02 stay
az aigateway model-alias delete -g "$RG" --gateway-name "$GATEWAY" -n "$STREAM_ALIAS" --yes &&
echo "🚮 Alias $STREAM_ALIAS is deleted"# Delete the streaming alias, with its token limit: the backend model and your alias from lab 02 stay
az aigateway model-alias delete -g $RG --gateway-name $GATEWAY -n $STREAM_ALIAS --yes
if ($?) { Write-Host "🚮 Alias $STREAM_ALIAS is deleted" }Delete everything azd created
In the aigateway/labs/23-streaming-responses folder, delete the resource group with the gateway, the Foundry
account, and both aliases. --purge also purges the soft-deleted Foundry
account. It takes about three minutes.
# Delete everything that azd up created, and purge the Foundry account
azd down --purge --force &&
echo "🚮 Everything that azd up created is deleted"Delete everything Terraform created
In the aigateway/labs/23-streaming-responses folder, delete the resource group with the gateway, the
Foundry account, and both aliases. The azurerm provider also purges
the soft-deleted Foundry account. It takes one to two minutes.
# Delete everything that terraform apply created, and purge the soft-deleted Foundry account
terraform -chdir=terraform destroy -auto-approve &&
echo "🚮 Everything that terraform apply created is deleted"Keep the gateway, model, and key from lab 02: the next labs reuse them. azd down and terraform destroy don't
touch them, because the Bicep + azd and Terraform tracks created their own gateways.
Next steps
- Throttle tokens per caller with token limits: give every caller its own allowance.
- Enforce cost budgets with per-caller overrides: cap spend in dollars.
- Monitor usage, cost, and traces: see streamed and throttled requests side by side.
- September 24 release: usage for streamed Chat Completions responses