Azure AI Gateway Labs
Documentation Open portal
Models & routing · Lab 23New

Stream responses and count every token

Stream Chat Completions and Responses API replies through the gateway, read the server-sent events that your client receives, and prove that a token limit counts streamed tokens even when the client never asks for usage.

Intermediate 20 min CLICodePortalBicepTerraformNotebook Updated
Notebook Source Edit

What you'll do

  • Read the server-sent events that a streamed Chat Completions response returns through the gateway
  • Get token usage at the end of a stream with include_usage, and skip chunks that have no choices
  • Stream the Responses API and read usage from the response.completed event
  • Prove that a token limit counts streamed tokens whether or not the client asks for usage

Before you start

Tracks: Azure CLI, Code, Portal, Bicep + azd, Terraform, Notebook

Chat apps and agents stream their replies: the model sends text a few tokens at a time, so people start reading after a second or two instead of waiting for the whole answer. Streaming used to be a blind spot for governance, because a streamed Chat Completions response only reports token usage when the client asks for it. In this lab you stream through the gateway with curl, PowerShell, and the OpenAI SDKs, read every event the client receives, and then prove that a token limit still counts every streamed token.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, model, key, and the token-limited streaming alias, so you can skip this section. Run that track in Put a token limit on a streaming alias first: it sets the variables that Stream a response and Verify use. You need only the one-time setup above.

  • Complete Publish a Foundry model and make your first call. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
  • For the PowerShell steps: PowerShell 7.4 or later.
  • For the Code track: Python 3.10 or later, or Node.js 22.18 or later.
  • Permission to change the gateway, for example Contributor on the resource group, for the token limit steps.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/23-streaming-responses.

Set these variables once, from the lab folder (labs/23-streaming-responses). Every track and the Verify section reuse them.

# Name your gateway, its model provider, and the backend model from lab 02, and the alias that this lab creates
export RG="<resource-group>"
export GATEWAY="<gateway>"                 # 24 characters or fewer
export PROVIDER="<provider-name>"          # for example foundry-models
export MODEL="<model-name>"                # the backend model name under that provider
export STREAM_ALIAS="$MODEL-stream"        # the token-limited alias that this lab creates
echo "📝 Gateway $GATEWAY in $RG, model $MODEL on $PROVIDER, streaming alias $STREAM_ALIAS"

# Read the gateway's resource ID and its runtime URL
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🆔 Gateway ID: $GATEWAY_ID"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"

# Read a starting guess for the alias: confirm it on the model's Aliases tab (lab 02), and replace it if it differs
export AI_GATEWAY_MODEL="$(az aigateway model show -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
  --query properties.deployment.modelName -o tsv)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"

# Paste the runtime key that you created in Access › API keys, without showing it: don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
Tip

The Code track only needs AI_GATEWAY_URL, AI_GATEWAY_MODEL, and AI_GATEWAY_API_KEY. If you skip the token limit steps, you can set those three by hand and skip the Azure CLI.

How it works

Add "stream": true to a Chat Completions request and the gateway answers with Content-Type: text/event-stream. The body is a series of server-sent events (SSE): each one is a data: line with a JSON chunk, followed by a blank line. The gateway passes each chunk on as the model produces it, and the response carries x-accel-buffering: no, which tells proxies such as NGINX not to hold it back.

This is what a streamed reply from a Foundry model looks like through the gateway:

ChunkchoicesWhat it carries
Prompt filter resultsEmpty []Content safety results for the prompt. id and model are empty strings.
RoleOne choicedelta.role is assistant and delta.content is an empty string.
Content deltasOne choicedelta.content holds the next few characters of the reply. usage is null.
FinishOne choicefinish_reason is stop, or length when the reply hits max_completion_tokens.
UsageEmpty []usage with prompt_tokens, completion_tokens, and total_tokens. Sent when the client sets stream_options.include_usage.
data: [DONE]Not JSONThe end of the stream. Stop reading.

Two chunks have an empty choices array, so code that reads choices[0] on every chunk crashes on the first one.

Where the gateway counts tokens. A token limit or cost limit is checked before the gateway forwards the request. When a counter is exhausted, the client gets an ordinary JSON error, such as 429 with Retry-After, and no stream starts. Once a stream starts, it runs to the end, even if it takes the counter past the limit; the next request is throttled. While the stream flows, the gateway records its token usage for token limits, cost limits, and Monitoring.

Usage without include_usage. Since the September 24, 2026 release, the gateway makes sure it has a usage record for every streamed Chat Completions response, even when the client doesn't set stream_options.include_usage, so the counters stay accurate. The release notes ask clients that parse the events themselves to accept a usage-only chunk before [DONE], even when they didn't ask for one. In our test run in November 2026, a client that didn't set include_usage received no usage chunk, yet the token limit counted its tokens exactly as it did for a client that asked. So write your client to work either way: skip chunks with an empty choices array, and take usage from whichever chunk carries it.

The Responses API streams typed events instead: each data: line follows an event: line, and the last event, response.completed, always carries the usage. A streamed response is stored like any other, so you can continue it with previous_response_id. The gateway sends that follow-up to the backend that stores the response, even when the alias has several backend models. See Session affinity.

Stream a response

Pick a track. Both stream the same request through the gateway and show the text as it arrives, the events, and the usage.

Choose your track
1

Watch the raw events

curl buffers its output by default; -N turns that off, so each event prints the moment it arrives. PowerShell's Invoke-RestMethod and Invoke-WebRequest wait for the whole body, so the PowerShell script reads the stream with HttpClient.

# Ask for a streamed reply: -N turns off curl's buffering, so that each event prints the moment it arrives
echo "🤖 Raw server-sent events from the $AI_GATEWAY_MODEL alias:"
curl -sSN "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in five words.\"}]}"

Expected output (shortened; your text and IDs differ):

Text
🤖 Raw server-sent events from the gpt-5-mini alias:
data: {"choices":[],"created":0,"id":"","model":"","object":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{…}}]}

data: {"choices":[{"delta":{"content":"","refusal":null,"role":"assistant"},"finish_reason":null,"index":0,…}],"model":"gpt-5-mini-2025-08-07","object":"chat.completion.chunk",…,"usage":null}

data: {"choices":[{"delta":{"content":"Hi"},"finish_reason":null,"index":0,…}],…,"usage":null}

data: {"choices":[{"delta":{"content":" there"},"finish_reason":null,"index":0,…}],…,"usage":null}

…

data: {"choices":[{"delta":{},"finish_reason":"stop","index":0,…}],…,"usage":null}

data: [DONE]
2

Stream with the script

The script prints the reply as it streams, then counts the events, measures the time to the first token, and shows the usage chunk if the stream had one.

infra/stream-chat.shGitHub
#!/usr/bin/env bash
# Streams a Chat Completions response through AI Gateway, prints the reply as it arrives, then prints how many
# server-sent events came back, how many had an empty choices array, the time to the first token, and the usage
# chunk if the stream ended with one.
#
# Usage:
#   export AI_GATEWAY_URL="<gateway-url>"
#   export AI_GATEWAY_MODEL="<model-name>"
#   export AI_GATEWAY_API_KEY="..."      # never commit this value
#   ./stream-chat.sh [--usage] [--raw] ["Your prompt"]
#
#   --usage  sets stream_options.include_usage to true, so the stream ends with a usage chunk
#   --raw    prints every server-sent event line instead of the reply text
# Requires curl and jq 1.6 or later.
set -euo pipefail

USAGE=false
RAW=false
while [[ "${1:-}" == --* ]]; do
  case "$1" in
    --usage) USAGE=true ;;
    --raw) RAW=true ;;
    *) echo "Unknown option: $1" >&2; exit 2 ;;
  esac
  shift
done
PROMPT="${1:-Explain in three short sentences why streaming makes chat apps feel faster.}"
: "${AI_GATEWAY_URL:?Set AI_GATEWAY_URL to the gateway runtime URL, for example https://aigw-labs-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com}"
: "${AI_GATEWAY_MODEL:?Set AI_GATEWAY_MODEL to the callable model alias}"
: "${AI_GATEWAY_API_KEY:?Set AI_GATEWAY_API_KEY to a runtime key from Access > API keys}"

# reasoning_effort none spends the budget on the reply instead of hidden reasoning, so the first token arrives sooner.
BODY=$(jq -n --arg m "$AI_GATEWAY_MODEL" --arg p "$PROMPT" --argjson usage "$USAGE" '
  {model: $m, stream: true, max_completion_tokens: 300, reasoning_effort: "none",
   messages: [{role: "user", content: $p}]}
  + (if $usage then {stream_options: {include_usage: true}} else {} end)')

headers=$(mktemp)
trap 'rm -f "$headers"' EXIT

# -N turns off curl's output buffering. jq reads the raw lines as they arrive and keeps running totals.
curl -sSN -D "$headers" "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" -d "$BODY" \
| jq -nRrj --unbuffered --argjson raw "$RAW" '
    def secs: (. * 100 | round) / 100 | tostring + " s";
    def summary:
      "\n\n"
      + (if .events == 0 then "Response body       \(.other)\n"
         else "Events              \(.events) (\(.empty) with an empty choices array)\n"
            + "First token         \(if .ttft then (.ttft | secs) else "-" end)\n"
            + "Usage chunk         "
            + (if .usage then "prompt \(.usage.prompt_tokens), completion \(.usage.completion_tokens), total \(.usage.total_tokens)"
               else "none: run again with --usage to ask for one" end) + "\n"
         end);
    foreach (inputs, null) as $line (
      {t0: now, events: 0, empty: 0, ttft: null, usage: null, other: "", out: ""};
      .out = ""
      | if $line == null then .end = true
        elif ($line | startswith("data: ")) then
          ($line[6:]) as $data
          | if $data == "[DONE]" then .out = (if $raw then $line + "\n" else "" end)
            else ($data | fromjson) as $chunk
              | ([$chunk.choices[]?.delta.content // empty] | join("")) as $text
              | .events += 1
              | .empty += (if ($chunk.choices // []) == [] then 1 else 0 end)
              | .usage = ($chunk.usage // .usage)
              | .ttft = (if .ttft == null and $text != "" then now - .t0 else .ttft end)
              | .out = (if $raw then $line + "\n" else $text end)
            end
        else .other += $line
        end;
      if .end then summary else .out end)'

status=$(awk 'toupper($1) ~ /^HTTP/ {code=$2} END {print code}' "$headers")
type=$(awk -F': ' 'tolower($1)=="content-type" {gsub("\r","",$2); print $2}' "$headers")
served=$(awk -F': ' 'tolower($1)=="x-ms-served-model" {gsub("\r","",$2); print $2}' "$headers")
retry=$(awk -F': ' 'tolower($1)=="retry-after" {gsub("\r","",$2); print $2}' "$headers")
printf 'HTTP status         %s\n' "$status"
printf 'Content-Type        %s\n' "${type:--}"
printf 'Served by           %s\n' "${served:--}"
[[ -n "$retry" ]] && printf 'Retry-After         %s\n' "$retry"
[[ "$status" == 200 ]]
# Stream a reply, then print the event count, the time to the first token, the usage chunk, and the headers
echo "🤖 Reply from the $AI_GATEWAY_MODEL alias, as it streams:"
./infra/stream-chat.sh

On GitHub, open stream-chat.sh or stream-chat.ps1.

Expected output (your text and numbers differ):

Text
🤖 Reply from the gpt-5-mini alias, as it streams:
Streaming shows replies as they're generated, so users see progress right away. They can start reading before the full answer is ready. That makes the app feel more responsive than waiting for one complete message.

Events              43 (1 with an empty choices array)
First token         1.77 s
Usage chunk         none: run again with --usage to ask for one
HTTP status         200
Content-Type        text/event-stream; charset=utf-8
Served by           gpt-5-mini-2025-08-07
3

Ask for usage at the end of the stream

Run the script again with the option that sets "stream_options": {"include_usage": true}:

# Stream again with stream_options.include_usage set, so that the stream ends with a usage chunk
echo "🤖 Reply from the $AI_GATEWAY_MODEL alias, with a usage chunk at the end:"
./infra/stream-chat.sh --usage

Expected output:

Text
🤖 Reply from the gpt-5-mini alias, with a usage chunk at the end:
Streaming shows the reply as it's generated instead of waiting for the whole answer. You can start reading right away. That makes the app feel more responsive.

Events              35 (2 with an empty choices array)
First token         1.74 s
Usage chunk         prompt 19, completion 34, total 53
HTTP status         200
Content-Type        text/event-stream; charset=utf-8
Served by           gpt-5-mini-2025-08-07

The second empty-choices chunk is the usage chunk, the last one before [DONE]. stream_options is only valid together with "stream": true; without it, the gateway returns 400.

4

Stream the Responses API

The Responses API names every event. List the event types in order, then read the usage from response.completed:

# Stream a Responses API reply into a file, so that you can count its events
curl -sSN "${AI_GATEWAY_URL%/}/default/models/openai/v1/responses" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"stream\":true,\"input\":\"Say hello in five words.\"}" > events.txt &&
  echo "🤖 Stream received from the $AI_GATEWAY_MODEL alias"

# Count each run of the same event type, in order
echo "📊 Event types, in order:"
grep '^event:' events.txt | uniq -c

# Read the usage from the response.completed event, then delete the file
echo "📊 Usage from response.completed:"
sed -n 's/^data: {/{/p' events.txt | jq -c 'select(.type == "response.completed") | .response.usage'
rm events.txt

Expected output (your counts differ):

Text
🤖 Stream received from the gpt-5-mini alias
📊 Event types, in order:
   1 event: response.created
   1 event: response.in_progress
   1 event: response.output_item.added
   1 event: response.output_item.done
   1 event: response.output_item.added
   1 event: response.content_part.added
   7 event: response.output_text.delta
   1 event: response.output_text.done
   1 event: response.content_part.done
   1 event: response.output_item.done
   1 event: response.completed
📊 Usage from response.completed:
{"input_tokens":12,"input_tokens_details":{"cache_write_tokens":0,"cached_tokens":0},"output_tokens":66,"output_tokens_details":{"reasoning_tokens":53},"total_tokens":78}

The first output_item pair is the model's reasoning item; the second is the message that carries the text. The reasoning_tokens count shows why the first token can take a moment with a reasoning model.

Checkpoint

The reply appears a few words at a time, and you can name the chunk that carries the usage in each API.

Put a token limit on a streaming alias

To find out whether the gateway counts streamed tokens, give a new alias, <model-name>-stream, a deliberately low limit: 1,000 tokens per minute for each caller identity. It targets the same backend model as your alias from lab 02, so clients that call that alias aren't affected.

Choose your track
1

Create the alias

  1. Open the AI Gateway portal and select your gateway in the top bar.
  2. Select ModelsAliasesAddAlias for existing models.
  3. On Basics, set Alias name to <model-name>-stream, for example gpt-5-mini-stream. Select Next.
  4. Keep the defaults on Auto model selection (preview), and select Next.
  5. On Select models, select your model in the list. Select Next.
  6. Keep the defaults on Session affinity, Priorities and weights, and Failure handling, then select Create on Review.
2

Add the token limit

  1. Select PoliciesAdd policy. On the Type step, under Cost and rate limits, select Token rate limit, then Next.
  2. On the Scope step, select Individual assets, select the <model-name>-stream alias, then Next.
  3. On the Details step, keep Time period at Per minute and Counter dimensions at Identity, set Allowed tokens to 1000, and select Create.
Policies › Add policy opens a three-step wizard: Type, Scope, Details.
Checkpoint

ModelsAliases lists <model-name>-stream, and Policies lists 1,000 tokens / minute per caller identity, a Token rate limit on that alias.

Verify

Send bursts of streamed requests that each use about 220 tokens: first without asking for usage, then with it. If the gateway only counted the tokens that clients report, the first burst would never be throttled.

1

Stream a burst without asking for usage

For each request, the script prints the status, Retry-After, the number of events, and the usage chunk, if any.

# Send 8 streamed requests to the streaming alias, without asking for usage
echo "🚦 Expect 429 Too Many Requests after about 1,000 tokens, with no usage chunks:"
./infra/stream-burst.sh "$STREAM_ALIAS" 8

On GitHub, open stream-burst.sh or stream-burst.ps1.

Expected output (your counts differ):

Text
🚦 Expect 429 Too Many Requests after about 1,000 tokens, with no usage chunks:
Request  Status  Retry-After  Events  Usage chunk
1        200     -            209     none
2        200     -            187     none
3        200     -            207     none
4        200     -            200     none
5        200     -            196     none
6        200     -            191     none
7        429     2            -       Token limit is exceeded. Try again in 2 seconds.
8        429     1            -       Token limit is exceeded. Try again in 1 seconds.

No stream carried a usage chunk, yet the gateway throttled request 7, after about 1,300 tokens. If it had counted only the prompts, about 25 tokens each, the alias would have allowed roughly 40 requests.

Note

A new alias or policy can take up to a minute to apply, and the counters catch up within a few seconds, so a fast burst can go a few hundred tokens past the limit. If every request returns 200, wait a minute and run it again.

2

Stream a burst with usage

Wait a minute for the window to reset, then run the same burst with include_usage:

# Wait a minute, so that the token limit's window resets
echo "⏳ Waiting 60 seconds for the token limit's window to reset"
sleep 60

# Send the same burst with stream_options.include_usage, so that each stream ends with a usage chunk
echo "🚦 Expect 429 Too Many Requests at the same point, now with usage chunks:"
./infra/stream-burst.sh --usage "$STREAM_ALIAS" 8

Expected output:

Text
⏳ Waiting 60 seconds for the token limit's window to reset
🚦 Expect 429 Too Many Requests at the same point, now with usage chunks:
Request  Status  Retry-After  Events  Usage chunk
1        200     -            183     211 tokens
2        200     -            186     214 tokens
3        200     -            202     230 tokens
4        200     -            181     209 tokens
5        200     -            180     208 tokens
6        200     -            188     216 tokens
7        429     5            -       Token limit is exceeded. Try again in 5 seconds.
8        429     5            -       Token limit is exceeded. Try again in 5 seconds.

The throttle point is the same: the gateway counted the streams the same way whether or not the client asked for usage. If a request after the 429 returns 200, it was sent after the Retry-After wait.

3

Inspect a throttled streaming request

Run the burst once more and, while the alias is throttled, look at a full response:

Bash
# Send one streamed request while the alias is throttled: -i prints the status line and the headers too
echo "🚦 Expect 429 Too Many Requests while the limit is active, as plain JSON:"
curl -sS -i "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$STREAM_ALIAS\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}"

Expected output, when the limit is active:

Text
🚦 Expect 429 Too Many Requests while the limit is active, as plain JSON:
HTTP/2 429
content-type: application/json
retry-after: 3
content-length: 84

{ "statusCode": 429, "message": "Token limit is exceeded. Try again in 3 seconds." }

The throttled answer is plain JSON, not an event stream, because the gateway refused the request before it reached the model. Check the status code before you start reading events, and honor Retry-After.

4

Find the streamed calls in Monitoring

If you set up telemetry in Monitor usage, cost, and traces, open Monitoring. On Overview, Top models lists the streaming alias with its calls and estimated cost next to the non-streamed calls, and Top blocking policies counts the throttled requests under Token limit. On Traces, the throttled requests show the outcome Blocked by policy, and the streams before them show Token limit under Applied policies.

Checkpoint

Both bursts were throttled at the same request, the burst without include_usage included, and the throttled request returned a JSON 429 with Retry-After instead of a stream.

Go further

  • Cap spend for streams. A cost limit is estimated from the same usage record, so streamed calls count against budgets too. Try it in Enforce cost budgets.
  • Shorten the time to the first token. Reasoning models think before they write. For chat that needs a quick first word, set reasoning_effort to low or none where the model supports it, as the scripts in this lab do.
  • Stream from an agent. Agent frameworks stream through the same endpoint. See Connect agent frameworks to models and tools.
  • Check your proxies. Anything between the client and the gateway, such as a reverse proxy or an API layer of your own, must pass text/event-stream responses through without buffering.
  • Find streams that fail after they start. A stream that fails partway, for example with a Responses response.failed event, has already returned HTTP 200. With tracing on, the gateway marks the span as Error and keeps the status code. See Operation failures.

Clean up

Delete the streaming alias. Its token limit goes with it, and the backend model and your alias from lab 02 stay.

Choose your track
1

Delete the streaming alias

Open <model-name>-stream in ModelsAliases and select Delete.

Keep the gateway, model, and key from lab 02: the next labs reuse them. azd down and terraform destroy don't touch them, because the Bicep + azd and Terraform tracks created their own gateways.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.