Azure AI Gateway Labs
Documentation Open portal
Governance & cost · Lab 06

Throttle tokens per caller with token limits

Cap how many tokens each app, agent, or user can consume per minute, hour, or day, and watch the gateway answer with 429 and Retry-After before your backend capacity runs out.

Beginner 25 min PortalCLIRESTBicepTerraformNotebook Updated
Notebook Source Edit

What you'll do

  • Explain how a token limit counts tokens per caller and per time window
  • Add a token rate limit to a model in the portal, with the Azure CLI, or with the REST API
  • Publish a separately limited alias with Bicep and azd, without touching existing clients
  • Trigger a 429 response, read Retry-After, and find the throttled calls in Monitoring

Before you start

Tracks: Portal, Azure CLI, REST, Bicep + azd, Terraform, Notebook

Your gateway now serves several apps from one model deployment. One batch job sends long prompts in a tight loop and eats the deployment's tokens-per-minute (TPM) capacity, so every other app starts to fail. In this lab you give each caller its own token allowance at the gateway, so a noisy neighbor gets a 429 with Retry-After and everyone else keeps working.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, model, and key, so you can skip this section. You need only the one-time setup above.

  • Complete Publish a Foundry model and make your first call. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
  • Permission to change the gateway, for example Contributor on the resource group.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/06-token-limits.

Set these variables once. Every track and the Verify section reuse them.

# Name the resource group, the gateway, the model provider, and the model that you published in lab 02
export RG="<resource-group>"
export GATEWAY="<gateway>"                 # 24 characters or fewer
export PROVIDER="<provider-name>"          # for example foundry-models
export MODEL="<model-name>"                # the backend model name under that provider
echo "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"

# Read the gateway's runtime URL
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"

# Guess the alias from the model's deployment name
# Confirm it on the model's Aliases tab (lab 02), and replace it if it differs
export AI_GATEWAY_MODEL="$(az aigateway model show -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" \
  -n "$MODEL" --query properties.deployment.modelName -o tsv)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"

# Paste the runtime key that you created in Access › API keys, without showing it
# Don't commit the key or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
Tip

Not sure which alias your clients call? Open the model in Models and check its Aliases tab. The value in the request model field must match an alias exactly.

How it works

A token limit is a policy that the gateway evaluates before it forwards a request. It keeps a counter of the tokens that each caller consumed in the current window. While the counter is under the limit, the request goes through and the tokens in the response are added to the counter. When the counter reaches the limit, the gateway answers 429 Too Many Requests with a Retry-After header and never calls the backend.

SettingPortal labelJSON fieldValues
AllowanceAllowed tokenscountPositive integer
WindowTime periodperiodminute (TPM), hour, day (quota)
Who gets a counterCounter dimensionscounterKeyA required array: ["identity"] (each API key, or each Entra tenant + object ID), ["ipAddress"], both, or [] for one allowance that all callers share

You can attach a token limit to a model (applies to every alias that targets it), to an alias (only callers that use that name), or to the gateway (every compatible model and alias, including ones you add later). A gateway limit keeps one counter per caller across all of those assets, not one per model. Every applicable policy is checked, and any one of them can block the request.

Note

The backend deployment has its own capacity, shown as Token rate limit (TPM) on the model's details page. The gateway limit is how you share that capacity fairly. Set per-caller limits so that the sum of your busiest callers stays below the deployment's TPM.

Add a token limit to the model

Pick one track. Each track produces the same policy on your model: 1,000 tokens per minute for each caller identity. The limit is deliberately low so you can trigger it in a few requests. The Bicep + azd and Terraform tracks take a different route, explained in their first step.

Choose your track
1

Start the policy wizard

Open the AI Gateway portal and select your gateway from the gateway switcher. Select PoliciesAdd policy. On the Type step, under Cost and rate limits, select Token rate limit, then Next.

Policies › Add policy opens a three-step wizard: Type, Scope, Details.
2

Choose where the limit applies

On the Scope step, select Individual assets ("Apply only to selected models, aliases, or MCP servers"). Select the row of type Model for your model, then Next.

Tip

Gateway scope applies the limit to every compatible asset, including models you add later. It's a good default for production. You'll try it in Go further.

3

Configure the allowance

On the Details step, under Policy details, set:

FieldValue
Time periodPer minute
Allowed tokens1000 (the default is 10000)
Counter dimensionsIdentity (the default)

Select Create. The portal confirms Policy created.

Checkpoint

Policies lists a Token rate limit for your model, with 1,000 tokens / minute per caller identity under Configuration.

Verify

Send a burst of requests that each use about 225 tokens. With a 1,000-token budget, the first few succeed, and then the gateway starts answering 429.

1

Run the burst

From the lab folder, run the script for your shell. For each request it prints the status, Retry-After, the tokens used, and the backend deployment's own x-ratelimit-remaining-tokens header.

# Send eight chat completions to the alias, and print the status, Retry-After, and tokens of each one
echo "🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:"
./infra/trigger-token-limit.sh "$AI_GATEWAY_MODEL" 8

On GitHub, open trigger-token-limit.sh or trigger-token-limit.ps1.

Expected output (your token counts will differ):

Text
🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:
Request  Status  Retry-After  Tokens used  Backend tokens left
1        200     -            211          9977
2        200     -            208          9977
3        200     -            205          9977
4        200     -            215          9977
5        200     -            225          9977
6        200     -            203          9977
7        429     3            -            -
8        429     3            -            -

The gateway stops this caller once it passes 1,000 tokens, while the backend deployment still has most of its capacity left: that's the point of a per-caller limit. Retry-After says how many seconds to wait, and a request sent after that wait can succeed again. The gateway doesn't add its own remaining-tokens header, so watch the status and Retry-After.

Note

The gateway counts a request's tokens after the response, and its counters catch up within a few seconds, so a fast burst can go a few hundred tokens past the budget before the first 429. A new or changed policy can also take up to a minute to apply. If every request returns 200, wait a minute and run the burst again.

2

Inspect one throttled response

Look at the full headers and body of a single throttled call:

Bash
# Send one raw chat completion while you're throttled, and show its status line, headers, and body
echo "🚦 Expect HTTP 429 with a Retry-After header:"
curl -i "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}"

The status line is HTTP/2 429, and the body is { "statusCode": 429, "message": "Token limit is exceeded. Try again in 5 seconds." }, with your own wait. Retry-After tells the client how many seconds to wait. Well-behaved SDKs, including the OpenAI SDKs, honor Retry-After automatically when they retry.

3

Prove the counter is per caller

Create a second runtime key in AccessAPI keys, set it as AI_GATEWAY_API_KEY in a new terminal, and run the burst again while the first key is still throttled. The second key starts with a full allowance, because the Identity counter dimension gives every key its own counter.

4

Find the throttled calls in Monitoring

If you configured Application Insights in Monitor usage, cost, and traces, open Monitoring. The dashboard shows 429 outcomes and policy impact next to token consumption, and the Traces tab shows the policy span that blocked each request.

Monitoring shows throttled requests and token use per model and API key.
Checkpoint

You saw 200 responses until the allowance ran out, then 429 with Retry-After. A second key wasn't affected, and the backend received no calls while the limit was active.

Go further

  • Set a gateway-wide default. Add a second Token rate limit with Gateway scope and a generous allowance, such as 100000 per minute. New models get a guardrail the moment you import them, and you keep the stricter model or alias limits on top. Each caller gets one allowance across all models and aliases.
  • Add a daily quota. Combine a per-minute limit (smooths bursts) with a Per day limit (caps total use). Both are checked on every request.
  • Limit by IP address. For a public demo with one shared key, use the IP address counter dimension so each visitor gets their own counter.
  • Budget in dollars instead of tokens. Token counts don't reflect model prices. To cap spend, continue with Enforce cost budgets.

Clean up

The policy doesn't cost anything, but a 1,000-token limit will throttle real traffic. Remove it, or raise it to a production value, before you continue.

Choose your track
1

Remove the policy

Select Policies. On the row of the Token rate limit you created, select More actions (⋮) › Delete, then Delete again in Delete this policy?. If you created a second runtime key for testing, revoke it in AccessAPI keys.

Keep the gateway, model, and key from lab 02: the next labs reuse them. azd down and terraform destroy don't touch them, because the Bicep + azd and Terraform tracks created their own gateways.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.