Your gateway now serves several apps from one model deployment. One batch job sends long prompts in a tight
loop and eats the deployment's tokens-per-minute (TPM) capacity, so every other app starts to fail. In this lab
you give each caller its own token allowance at the gateway, so a noisy neighbor gets a 429 with
Retry-After and everyone else keeps working.
Before you begin
One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.
- Choose where to run commandsEvery track except the portal
- Sign in to AzureOn your own machine. Cloud Shell is already signed in.
- Install the
aigatewayextensionThe Azure CLI tracks and the Verify steps - Clone the labs repoThe Bicep + azd and Terraform tracks, the notebooks, and the code samples
You've done the one-time setup that this lab uses.
On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, model, and
key, so you can skip this section. You need only the one-time setup above.
- Complete Publish a Foundry model and make your first call. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
- Permission to change the gateway, for example Contributor on the resource group.
- Run the scripts and samples from the lab's folder,
~/aigateway/labs/06-token-limits.
Set these variables once. Every track and the Verify section reuse them.
# Name the resource group, the gateway, the model provider, and the model that you published in lab 02
export RG="<resource-group>"
export GATEWAY="<gateway>" # 24 characters or fewer
export PROVIDER="<provider-name>" # for example foundry-models
export MODEL="<model-name>" # the backend model name under that provider
echo "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
# Read the gateway's runtime URL
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
# Guess the alias from the model's deployment name
# Confirm it on the model's Aliases tab (lab 02), and replace it if it differs
export AI_GATEWAY_MODEL="$(az aigateway model show -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" \
-n "$MODEL" --query properties.deployment.modelName -o tsv)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
# Paste the runtime key that you created in Access › API keys, without showing it
# Don't commit the key or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"# Name the resource group, the gateway, the model provider, and the model that you published in lab 02
$RG = "<resource-group>"
$GATEWAY = "<gateway>" # 24 characters or fewer
$PROVIDER = "<provider-name>" # for example foundry-models
$MODEL = "<model-name>" # the backend model name under that provider
Write-Host "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
# Read the gateway's runtime URL
$env:AI_GATEWAY_URL = "https://$(az aigateway show -g $RG -n $GATEWAY --query properties.frontend.defaultHostname -o tsv)"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
# Guess the alias from the model's deployment name
# Confirm it on the model's Aliases tab (lab 02), and replace it if it differs
$env:AI_GATEWAY_MODEL = az aigateway model show -g $RG --gateway-name $GATEWAY --provider-name $PROVIDER `
-n $MODEL --query properties.deployment.modelName -o tsv
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
# Paste the runtime key that you created in Access › API keys, without showing it
# Don't commit the key or write it to a file
$env:AI_GATEWAY_API_KEY = Read-Host "AI Gateway API key" -MaskInput
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"Not sure which alias your clients call? Open the model in Models and check its Aliases tab. The value
in the request model field must match an alias exactly.
How it works
A token limit is a policy that the gateway evaluates before it forwards a request. It keeps a counter of
the tokens that each caller consumed in the current window. While the counter is under the limit, the request
goes through and the tokens in the response are added to the counter. When the counter reaches the limit, the
gateway answers 429 Too Many Requests with a Retry-After header and never calls the backend.
| Setting | Portal label | JSON field | Values |
|---|---|---|---|
| Allowance | Allowed tokens | count | Positive integer |
| Window | Time period | period | minute (TPM), hour, day (quota) |
| Who gets a counter | Counter dimensions | counterKey | A required array: ["identity"] (each API key, or each Entra tenant + object ID), ["ipAddress"], both, or [] for one allowance that all callers share |
You can attach a token limit to a model (applies to every alias that targets it), to an alias (only callers that use that name), or to the gateway (every compatible model and alias, including ones you add later). A gateway limit keeps one counter per caller across all of those assets, not one per model. Every applicable policy is checked, and any one of them can block the request.
The backend deployment has its own capacity, shown as Token rate limit (TPM) on the model's details page. The gateway limit is how you share that capacity fairly. Set per-caller limits so that the sum of your busiest callers stays below the deployment's TPM.
Add a token limit to the model
Pick one track. Each track produces the same policy on your model: 1,000 tokens per minute for each caller identity. The limit is deliberately low so you can trigger it in a few requests. The Bicep + azd and Terraform tracks take a different route, explained in their first step.
Start the policy wizard
Open the AI Gateway portal and select your gateway from the gateway switcher. Select PoliciesAdd policy. On the Type step, under Cost and rate limits, select Token rate limit, then Next.
Choose where the limit applies
On the Scope step, select Individual assets ("Apply only to selected models, aliases, or MCP servers"). Select the row of type Model for your model, then Next.
Gateway scope applies the limit to every compatible asset, including models you add later. It's a good default for production. You'll try it in Go further.
Configure the allowance
On the Details step, under Policy details, set:
| Field | Value |
|---|---|
| Time period | Per minute |
| Allowed tokens | 1000 (the default is 10000) |
| Counter dimensions | Identity (the default) |
Select Create. The portal confirms Policy created.
Policies lists a Token rate limit for your model, with 1,000 tokens / minute per caller identity under Configuration.
Set the token limit
Policies live in the model's properties.policies array. az aigateway model policy set-token-limit adds the
token limit, or replaces the one that's there, and keeps every other policy on the model.
# Add a token limit to the model: 1,000 tokens per minute, with its own counter for each caller identity
az aigateway model policy set-token-limit -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
--token-limit 1000 --token-period minute --token-counter-key Identity &&
echo "🔐 Token limit of 1,000 tokens per minute is on $MODEL"# Add a token limit to the model: 1,000 tokens per minute, with its own counter for each caller identity
az aigateway model policy set-token-limit -g $RG --gateway-name $GATEWAY --provider-name $PROVIDER -n $MODEL `
--token-limit 1000 --token-period minute --token-counter-key Identity
if ($?) { Write-Host "🔐 Token limit of 1,000 tokens per minute is on $MODEL" }The command prints the model's policies. --token-period accepts minute (the default), hour, or day.
--token-counter-key accepts Identity (the default), IPAddress, or both, as the portal's Counter dimensions
allows: --token-counter-key Identity IPAddress gives each caller identity and IP address its own counter.
Confirm the policy
# List the model's policies, to check the token limit
echo "🔐 Policies on ${MODEL}:"
az aigateway model policy list -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" -o json# List the model's policies, to check the token limit
Write-Host "🔐 Policies on ${MODEL}:"
az aigateway model policy list -g $RG --gateway-name $GATEWAY --provider-name $PROVIDER -n $MODEL -o jsonThe output contains "type": "tokenLimit" with "count": 1000, "period": "minute",
"counterKey": ["identity"], and "scope": "resource", which the service adds to every model policy.
Read the current policies
Policies live inline in the model's properties.policies array. A PATCH replaces the whole array, so
always read it first and keep the policies you didn't mean to change.
# Read your subscription ID, and build the model's management URL, with the API version
SUB="$(az account show --query id -o tsv)"
URL="https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.ApiManagement/aigateways/$GATEWAY/workspaces/default/modelProviders/$PROVIDER/models/$MODEL?api-version=2026-11-01"
echo "🆔 Subscription: $SUB"
# Read the model's policies, so that you can keep the ones you don't change
echo "🔐 Policies on ${MODEL}:"
az rest --method get --url "$URL" --query properties.policiesReplace the token limit and keep everything else
# Read the model, keep every policy except the token limit, and add the new one:
# 1,000 tokens per minute, with its own counter for each caller identity
POLICIES="$(az rest --method get --url "$URL" | jq -c '
[(.properties.policies // [])[] | select(.type != "tokenLimit")]
+ [{"type":"tokenLimit","count":1000,"period":"minute","counterKey":["identity"]}]')"
echo "📝 Policies to write: $(jq length <<< "$POLICIES"), including the token limit"
# Replace the model's whole policies array with the merged one
az rest --method patch --url "$URL" --body "{\"properties\":{\"policies\":$POLICIES}}" &&
echo "🔐 Token limit of 1,000 tokens per minute is on $MODEL"# Read your subscription ID, and build the model's management URL, with the API version
$SUB = az account show --query id -o tsv
$URL = "https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.ApiManagement/aigateways/$GATEWAY/workspaces/default/modelProviders/$PROVIDER/models/$MODEL`?api-version=2026-11-01"
Write-Host "🆔 Subscription: $SUB"
# Read the model, keep every policy except the token limit, and add the new one:
# 1,000 tokens per minute, with its own counter for each caller identity
$current = az rest --method get --url $URL | ConvertFrom-Json
$policies = @($current.properties.policies | Where-Object { $_ -and $_.type -ne 'tokenLimit' })
$policies += [pscustomobject]@{ type = 'tokenLimit'; count = 1000; period = 'minute'; counterKey = @('identity') }
Write-Host "📝 Policies to write: $($policies.Count), including the token limit"
# Write the request body to a file, which --body reads with the @ prefix, so that its quotes reach az intact
# Then replace the model's whole policies array, and delete the file
$body = @{ properties = @{ policies = $policies } } | ConvertTo-Json -Depth 10 -Compress
$body | Set-Content -Path patch.json
az rest --method patch --url $URL --body '@patch.json'
if ($?) { Write-Host "🔐 Token limit of 1,000 tokens per minute is on $MODEL" }
Remove-Item patch.jsonRunning the GET again returns your other policies unchanged plus one tokenLimit entry with
"counterKey": ["identity"] and "scope": "resource".
What you need: only the one-time setup from Before you begin. This track doesn't use your gateway from lab 02:
azd up creates a new one, so it works on its own.
Choose what to declare
A Bicep deployment sends a PUT, which replaces the whole resource. Redeclaring a model you imported in the
portal risks dropping settings you didn't copy. Instead, this template adds a new alias,
<model-name>-limited, that targets the same backend model and carries its own token limit. Existing clients
keep calling the original alias. Callers that you move to the new alias get the allowance.
This is also a common production pattern: one backend model, several aliases, and a different limit for each
audience (for example gpt-chat for interactive apps and gpt-batch for offline jobs).
Review the templates
infra/main.bicep creates the resource group rg-<environment name> and calls the quick start's
resources.bicep for the gateway, the Foundry model, its default alias, and the
app-first-call key. Then it calls infra/limited-alias.bicep with the token limit:
// Lab 06 with azd: "azd up" creates the resource group rg-<environment name>, the quick start's gateway, Foundry
// model, default alias, and runtime API key, then a second alias for the same model with its own token limit.
// The runtime API key is never an output.
targetScope = 'subscription'
@description('azd environment name. The resource group is rg-<environmentName>.')
@minLength(1)
@maxLength(60)
param environmentName string
@description('Region for every resource. Pick one where AI Gateway and the lab\'s Foundry models are available. List them with az provider show --namespace Microsoft.ApiManagement (Setup guide, Find the available regions).')
@metadata({ azd: { type: 'location' } })
param location string
@description('Tokens allowed per period for each caller. azd reads it from TOKEN_LIMIT.')
@minValue(1)
param allowedTokens int = 1000
@description('Token limit window. azd reads it from TOKEN_LIMIT_PERIOD.')
@allowed([ 'minute', 'hour', 'day' ])
param period string = 'minute'
@description('Who gets a token counter. azd reads it from TOKEN_LIMIT_COUNTER_KEY.')
@allowed([ 'identity', 'ipAddress' ])
param counterKey array = [ 'identity' ]
var suffix = take(uniqueString(subscription().id, environmentName, location), 6)
var tags = { 'azd-env-name': environmentName }
resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = {
name: 'rg-${environmentName}'
location: location
tags: tags
}
// The quick start's template: a gateway, a Foundry model behind the default alias, and the app-first-call key.
module base '../../quickstart/infra/resources.bicep' = {
name: 'aigw-base'
scope: rg
params: {
location: location
tags: tags
gatewayName: 'aigw-${suffix}'
foundryName: 'aigw${suffix}'
}
}
module limitedAlias 'limited-alias.bicep' = {
name: 'aigw-limited-alias'
scope: rg
params: {
gatewayName: base.outputs.gatewayName
providerName: base.outputs.providerName
modelName: base.outputs.modelName
allowedTokens: allowedTokens
period: period
counterKey: counterKey
}
}
output AZURE_RESOURCE_GROUP string = rg.name
output AI_GATEWAY_NAME string = base.outputs.gatewayName
output AI_GATEWAY_ID string = base.outputs.gatewayId
output AI_GATEWAY_URL string = base.outputs.gatewayUrl
output AI_GATEWAY_PROVIDER string = base.outputs.providerName
output AI_GATEWAY_BACKEND_MODEL string = base.outputs.modelName
output AI_GATEWAY_MODEL string = limitedAlias.outputs.aliasName
output FOUNDRY_ACCOUNT_NAME string = base.outputs.foundryNameinfra/limited-alias.bicep adds the alias. It only references the gateway, the provider, and the model, so you can
also deploy it on its own to a gateway you already have, with az deployment group create:
// Publishes a token-limited alias for an existing AI Gateway model. main.bicep calls it after the quick start's
// template, and you can also deploy it on its own to a gateway you already have. The alias points at the same backend
// model as your default alias, but carries its own token limit, so a "batch" or "trial" audience gets a smaller
// allowance without touching the model.
@description('Name of the existing AI Gateway (Microsoft.ApiManagement/aigateways).')
param gatewayName string
@description('Name of the existing model provider in the gateway, for example foundry-models.')
param providerName string
@description('Name of the existing backend model registration under the provider.')
param modelName string
@description('Callable alias to create. Clients send this value in the request "model" field.')
param aliasName string = '${modelName}-limited'
@description('Tokens allowed per period for each caller.')
@minValue(1)
param allowedTokens int = 1000
@description('Token limit window.')
@allowed([ 'minute', 'hour', 'day' ])
param period string = 'minute'
@description('Who gets a token counter: identity (each API key or Entra principal), ipAddress (each client IP address), or both.')
@allowed([ 'identity', 'ipAddress' ])
param counterKey array = [ 'identity' ]
#disable-next-line BCP081
resource gateway 'Microsoft.ApiManagement/aigateways@2026-11-01' existing = {
name: gatewayName
}
#disable-next-line BCP081
resource workspace 'Microsoft.ApiManagement/aigateways/workspaces@2026-11-01' existing = {
parent: gateway
name: 'default'
}
#disable-next-line BCP081
resource provider 'Microsoft.ApiManagement/aigateways/workspaces/modelProviders@2026-11-01' existing = {
parent: workspace
name: providerName
}
#disable-next-line BCP081
resource model 'Microsoft.ApiManagement/aigateways/workspaces/modelProviders/models@2026-11-01' existing = {
parent: provider
name: modelName
}
#disable-next-line BCP081
resource limitedAlias 'Microsoft.ApiManagement/aigateways/workspaces/aliases@2026-11-01' = {
parent: workspace
name: aliasName
properties: {
backendModels: [
{ resourceId: model.id }
]
policies: [
{
type: 'tokenLimit'
count: allowedTokens
period: period
counterKey: counterKey
}
]
}
}
output aliasName string = limitedAlias.name
output chatCompletionsUrl string = 'https://${gateway.properties.frontend.defaultHostname}/default/models/openai/v1/chat/completions'Deploy with azd up
Move into the lab's folder, create an azd environment named aigw-lab06 in your current subscription, then provision
everything. It takes about three minutes.
# Move into the lab's folder, which holds the azd project
cd ~/aigateway/labs/06-token-limits &&
echo "📂 Lab folder: $(pwd)"
# Create an azd environment in your current subscription, so that azd doesn't prompt you
azd env new aigw-lab06 --location eastus2 --subscription "$(az account show --query id --output tsv)" &&
echo "📝 azd environment aigw-lab06 deploys to eastus2"
# Provision the gateway, the Foundry model, its default alias, the limited alias with its token limit, and the key
azd up &&
echo "🚀 Gateway deployed: $(azd env get-value AI_GATEWAY_URL)"# Move into the lab's folder, which holds the azd project
Set-Location ~/aigateway/labs/06-token-limits
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }
# Create an azd environment in your current subscription, so that azd doesn't prompt you
azd env new aigw-lab06 --location eastus2 --subscription (az account show --query id --output tsv)
if ($?) { Write-Host "📝 azd environment aigw-lab06 deploys to eastus2" }
# Provision the gateway, the Foundry model, its default alias, the limited alias with its token limit, and the key
azd up
if ($?) { Write-Host "🚀 Gateway deployed: $(azd env get-value AI_GATEWAY_URL)" }For another region, pick one where AI Gateway and the gpt-5-mini model are available, as in Find the available
regions.
Read the outputs and the key
Set the variables that Verify uses from azd's environment values, and read the key from the gateway, so that it never shows on screen.
# Read the template's outputs: the resource group, the gateway, the provider, the backend model, the runtime URL,
# and the limited alias that apps send in the model field
export RG="$(azd env get-value AZURE_RESOURCE_GROUP)"
export GATEWAY="$(azd env get-value AI_GATEWAY_NAME)"
export PROVIDER="$(azd env get-value AI_GATEWAY_PROVIDER)"
export MODEL="$(azd env get-value AI_GATEWAY_BACKEND_MODEL)"
export AI_GATEWAY_URL="$(azd env get-value AI_GATEWAY_URL)"
export AI_GATEWAY_MODEL="$(azd env get-value AI_GATEWAY_MODEL)"
echo "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
# Read the key from the gateway, so that it never lands in a file or a deployment log
export AI_GATEWAY_API_KEY="$(az aigateway api-key list-secrets --ids "$(azd env get-value AI_GATEWAY_ID)/apiKeys/app-first-call" \
--query primaryKey --output tsv)"
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"# Read the template's outputs: the resource group, the gateway, the provider, the backend model, the runtime URL,
# the limited alias that apps send in the model field, and the gateway's resource ID
$RG = azd env get-value AZURE_RESOURCE_GROUP
$GATEWAY = azd env get-value AI_GATEWAY_NAME
$PROVIDER = azd env get-value AI_GATEWAY_PROVIDER
$MODEL = azd env get-value AI_GATEWAY_BACKEND_MODEL
$env:AI_GATEWAY_URL = azd env get-value AI_GATEWAY_URL
$env:AI_GATEWAY_MODEL = azd env get-value AI_GATEWAY_MODEL
$GATEWAY_ID = azd env get-value AI_GATEWAY_ID
Write-Host "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
# Read the key from the gateway, so that it never lands in a file or a deployment log
$env:AI_GATEWAY_API_KEY = az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" `
--query primaryKey --output tsv
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"azd up ended with SUCCESS, and the last command printed the limited alias,
🔖 Model alias: gpt-5-mini-limited. In the portal, Models shows it next to gpt-5-mini, and Policies
lists a Token rate limit scoped to it.
Change the limit
The parameters come from azd environment values, with the defaults in infra/main.parameters.json. To count tokens
per client IP address instead of per key, set TOKEN_LIMIT_COUNTER_KEY to a JSON array and provision again. Only
the alias changes. TOKEN_LIMIT and TOKEN_LIMIT_PERIOD work the same way.
# Count tokens per client IP address instead of per key, in the azd environment
azd env set TOKEN_LIMIT_COUNTER_KEY '["ipAddress"]' &&
echo "📝 TOKEN_LIMIT_COUNTER_KEY is set to ipAddress"
# Provision again, which updates only the limited alias
azd provision &&
echo "🔐 The token limit on the limited alias counts tokens per client IP address"# Count tokens per client IP address instead of per key, in the azd environment
azd env set TOKEN_LIMIT_COUNTER_KEY '["ipAddress"]'
if ($?) { Write-Host "📝 TOKEN_LIMIT_COUNTER_KEY is set to ipAddress" }
# Provision again, which updates only the limited alias
azd provision
if ($?) { Write-Host "🔐 The token limit on the limited alias counts tokens per client IP address" }Use '["identity","ipAddress"]' for both. Set it back to '["identity"]' before Verify, which proves the
counter is per key.
What you need: only the one-time setup from Before you begin, plus Terraform 1.9 or later, as in
Deploy a lab with Terraform. This track doesn't use your gateway from
lab 02: terraform apply creates a new one, so it works on its own.
Review the configuration
Like the Bicep template, the configuration adds a new alias, <model-name>-limited, that targets the same
backend model and carries its own token limit, rather than redeclaring a model that you imported in the portal.
Existing clients keep calling the original alias, and callers that you move to the new alias get the allowance.
terraform/main.tf creates the resource group rg-<environment_name> and calls the quick start's
resources module for the gateway, the Foundry model, its default alias, and the
app-first-call key:
# Lab 06 with Terraform: creates the resource group rg-<environment_name>, the quick start's gateway, Foundry model,
# default alias, and runtime API key, then a second alias for the same model with its own token limit
# (limited-alias.tf). The runtime API key is never an output.
data "azapi_client_config" "current" {}
locals {
suffix = substr(sha1("${data.azapi_client_config.current.subscription_id}/${var.environment_name}/${var.location}"), 0, 6)
tags = { "environment-name" = var.environment_name }
}
resource "azurerm_resource_group" "rg" {
name = "rg-${var.environment_name}"
location = var.location
tags = local.tags
}
# The quick start's module: a gateway, a Foundry model behind the default alias, and the app-first-call key.
module "base" {
source = "../../quickstart/terraform/resources"
subscription_id = data.azapi_client_config.current.subscription_id
resource_group_name = azurerm_resource_group.rg.name
resource_group_id = azurerm_resource_group.rg.id
location = var.location
tags = local.tags
gateway_name = "aigw-${local.suffix}"
foundry_name = "aigw${local.suffix}"
}terraform/limited-alias.tf adds the alias with the azapi provider, which sends the AI Gateway API version,
2026-11-01, as is. terraform/variables.tf declares the limit's settings with the same defaults as the Bicep
template:
# A token-limited alias for the quick start's model. It points at the same backend model as the default alias, but
# carries its own token limit, so a "batch" or "trial" audience gets a smaller allowance without touching the model.
resource "azapi_resource" "limited_alias" {
type = "Microsoft.ApiManagement/aigateways/workspaces/aliases@2026-11-01"
name = "${module.base.model_name}-limited"
parent_id = module.base.workspace_id
body = {
properties = {
backendModels = [
{ resourceId = module.base.model_id }
]
policies = [
{
type = "tokenLimit"
count = var.token_limit
period = var.token_limit_period
counterKey = var.token_limit_counter_key
}
]
}
}
schema_validation_enabled = false
}Deploy with terraform apply
Move into the lab's Terraform folder, select your current subscription, then plan and apply the configuration. It takes about five minutes, most of it while the new gateway activates.
# Move into the lab's Terraform folder
cd ~/aigateway/labs/06-token-limits/terraform &&
echo "📂 Terraform folder: $(pwd)"
# Deploy to the Azure CLI's current subscription
export ARM_SUBSCRIPTION_ID="$(az account show --query id --output tsv)" &&
echo "📍 Subscription: $ARM_SUBSCRIPTION_ID"
# Download the providers, then plan and apply the gateway, the Foundry model, its default alias, the limited alias
# with its token limit, and the key
terraform init &&
terraform plan -out tfplan &&
terraform apply tfplan &&
echo "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)"# Move into the lab's Terraform folder
Set-Location ~/aigateway/labs/06-token-limits/terraform
if ($?) { Write-Host "📂 Terraform folder: $(Get-Location)" }
# Deploy to the Azure CLI's current subscription
$env:ARM_SUBSCRIPTION_ID = az account show --query id --output tsv
if ($?) { Write-Host "📍 Subscription: $env:ARM_SUBSCRIPTION_ID" }
# Download the providers, then plan and apply the gateway, the Foundry model, its default alias, the limited alias
# with its token limit, and the key
terraform init
if ($?) { terraform plan -out tfplan }
if ($?) { terraform apply tfplan }
if ($?) { Write-Host "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)" }The configuration deploys to eastus2. For another region where AI Gateway and the gpt-5-mini model are
available, as in Find the available regions, run
export TF_VAR_location=<region> (PowerShell: $env:TF_VAR_location = "<region>") before terraform plan.
Read the outputs and the key
Set the variables that Verify uses from the Terraform outputs, and read the key from the gateway, so that it never shows on screen or lands in the Terraform state.
# Read the configuration's outputs: the resource group, the gateway, the provider, the backend model, the runtime
# URL, and the limited alias that apps send in the model field
export RG="$(terraform output -raw azure_resource_group)"
export GATEWAY="$(terraform output -raw ai_gateway_name)"
export PROVIDER="$(terraform output -raw ai_gateway_provider)"
export MODEL="$(terraform output -raw ai_gateway_backend_model)"
export AI_GATEWAY_URL="$(terraform output -raw ai_gateway_url)"
export AI_GATEWAY_MODEL="$(terraform output -raw ai_gateway_model)"
echo "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
# Read the key from the gateway, so that it never lands in a file or the Terraform state
export AI_GATEWAY_API_KEY="$(az aigateway api-key list-secrets --ids "$(terraform output -raw ai_gateway_id)/apiKeys/app-first-call" \
--query primaryKey --output tsv)"
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
# Move back to the lab's folder, which holds the scripts
cd .. && echo "📂 Lab folder: $(pwd)"# Read the configuration's outputs: the resource group, the gateway, the provider, the backend model, the runtime
# URL, the limited alias that apps send in the model field, and the gateway's resource ID
$RG = terraform output -raw azure_resource_group
$GATEWAY = terraform output -raw ai_gateway_name
$PROVIDER = terraform output -raw ai_gateway_provider
$MODEL = terraform output -raw ai_gateway_backend_model
$env:AI_GATEWAY_URL = terraform output -raw ai_gateway_url
$env:AI_GATEWAY_MODEL = terraform output -raw ai_gateway_model
$GATEWAY_ID = terraform output -raw ai_gateway_id
Write-Host "📝 Model $MODEL from $PROVIDER, on gateway $GATEWAY in $RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
# Read the key from the gateway, so that it never lands in a file or the Terraform state
$env:AI_GATEWAY_API_KEY = az aigateway api-key list-secrets --ids "$GATEWAY_ID/apiKeys/app-first-call" `
--query primaryKey --output tsv
Write-Host "🔑 Key: in AI_GATEWAY_API_KEY ($($env:AI_GATEWAY_API_KEY.Length) characters, not shown)"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
# Move back to the lab's folder, which holds the scripts
Set-Location ..
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }terraform apply ended with Apply complete! Resources: 10 added, 0 changed, 0 destroyed., and the output
shows the limited alias, 🔖 Model alias: gpt-5-mini-limited. In the portal, Models shows it next to
gpt-5-mini, and Policies lists a Token rate limit scoped to it.
The next sections run from the lab's folder, with -chdir=terraform for the terraform commands.
Change the limit
The settings are Terraform variables, with the defaults in terraform/variables.tf. To count tokens per client IP
address instead of per key, pass token_limit_counter_key to terraform plan and apply the plan. Only the alias
changes. token_limit and token_limit_period work the same way.
# Plan the change: count tokens per client IP address instead of per key
terraform -chdir=terraform plan -var 'token_limit_counter_key=["ipAddress"]' -out tfplan &&
echo "📝 The plan updates only the limited alias"
# Apply the plan
terraform -chdir=terraform apply tfplan &&
echo "🔐 The token limit on the limited alias counts tokens per client IP address"# Plan the change: count tokens per client IP address instead of per key
terraform -chdir=terraform plan -var 'token_limit_counter_key=["ipAddress"]' -out tfplan
if ($?) { Write-Host "📝 The plan updates only the limited alias" }
# Apply the plan
terraform -chdir=terraform apply tfplan
if ($?) { Write-Host "🔐 The token limit on the limited alias counts tokens per client IP address" }The plan ended with Plan: 0 to add, 1 to change, 0 to destroy., for azapi_resource.limited_alias.
Use 'token_limit_counter_key=["identity","ipAddress"]' for both. Before Verify, which proves the counter is
per key, plan and apply again without -var, to go back to the default, ["identity"].
Verify
Send a burst of requests that each use about 225 tokens. With a 1,000-token budget, the first few succeed, and
then the gateway starts answering 429.
Run the burst
From the lab folder, run the script for your shell. For each request it prints the status, Retry-After, the
tokens used, and the backend deployment's own x-ratelimit-remaining-tokens header.
# Send eight chat completions to the alias, and print the status, Retry-After, and tokens of each one
echo "🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:"
./infra/trigger-token-limit.sh "$AI_GATEWAY_MODEL" 8# Send eight chat completions to the alias, and print the status, Retry-After, and tokens of each one
Write-Host "🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:"
./infra/trigger-token-limit.ps1 -Model $env:AI_GATEWAY_MODEL -Requests 8On GitHub, open trigger-token-limit.sh or trigger-token-limit.ps1.
Expected output (your token counts will differ):
🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:
Request Status Retry-After Tokens used Backend tokens left
1 200 - 211 9977
2 200 - 208 9977
3 200 - 205 9977
4 200 - 215 9977
5 200 - 225 9977
6 200 - 203 9977
7 429 3 - -
8 429 3 - -The gateway stops this caller once it passes 1,000 tokens, while the backend deployment still has most of its capacity
left: that's the point of a per-caller limit. Retry-After says how many seconds to wait, and a request sent after
that wait can succeed again. The gateway doesn't add its own remaining-tokens header, so watch the status and
Retry-After.
The gateway counts a request's tokens after the response, and its counters catch up within a few seconds, so a
fast burst can go a few hundred tokens past the budget before the first 429. A new or changed policy can also
take up to a minute to apply. If every request returns 200, wait a minute and run the burst again.
Inspect one throttled response
Look at the full headers and body of a single throttled call:
# Send one raw chat completion while you're throttled, and show its status line, headers, and body
echo "🚦 Expect HTTP 429 with a Retry-After header:"
curl -i "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
-d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}"The status line is HTTP/2 429, and the body is
{ "statusCode": 429, "message": "Token limit is exceeded. Try again in 5 seconds." }, with your own wait.
Retry-After tells the client how many seconds to wait. Well-behaved SDKs, including the OpenAI SDKs, honor
Retry-After automatically when they retry.
Prove the counter is per caller
Create a second runtime key in AccessAPI keys, set it as AI_GATEWAY_API_KEY in a new terminal, and
run the burst again while the first key is still throttled. The second key starts with a full allowance,
because the Identity counter dimension gives every key its own counter.
Find the throttled calls in Monitoring
If you configured Application Insights in Monitor usage, cost, and traces, open
Monitoring. The dashboard shows 429 outcomes and policy impact next to token consumption, and the
Traces tab shows the policy span that blocked each request.
You saw 200 responses until the allowance ran out, then 429 with Retry-After. A second key wasn't
affected, and the backend received no calls while the limit was active.
Go further
- Set a gateway-wide default. Add a second Token rate limit with Gateway scope and a generous
allowance, such as
100000per minute. New models get a guardrail the moment you import them, and you keep the stricter model or alias limits on top. Each caller gets one allowance across all models and aliases. - Add a daily quota. Combine a per-minute limit (smooths bursts) with a Per day limit (caps total use). Both are checked on every request.
- Limit by IP address. For a public demo with one shared key, use the IP address counter dimension so each visitor gets their own counter.
- Budget in dollars instead of tokens. Token counts don't reflect model prices. To cap spend, continue with Enforce cost budgets.
Clean up
The policy doesn't cost anything, but a 1,000-token limit will throttle real traffic. Remove it, or raise it to a production value, before you continue.
Remove the policy
Select Policies. On the row of the Token rate limit you created, select More actions (⋮) › Delete, then Delete again in Delete this policy?. If you created a second runtime key for testing, revoke it in AccessAPI keys.
Remove the policy
# Remove only the token limit from the model, and keep every other policy
az aigateway model policy remove -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
--remove-token-limit true &&
echo "🚮 Token limit is removed from $MODEL"# Remove only the token limit from the model, and keep every other policy
az aigateway model policy remove -g $RG --gateway-name $GATEWAY --provider-name $PROVIDER -n $MODEL `
--remove-token-limit true
if ($?) { Write-Host "🚮 Token limit is removed from $MODEL" }This removes only the token limit and keeps every other policy on the model. To keep a limit for real traffic
instead, run the Set the token limit step again with a production count, such as 100000.
Remove the policy
# Read the model, and keep every policy except the token limit
POLICIES="$(az rest --method get --url "$URL" | jq -c '[(.properties.policies // [])[] | select(.type != "tokenLimit")]')"
echo "📝 Policies to keep: $(jq length <<< "$POLICIES")"
# Replace the model's whole policies array with the remaining policies
az rest --method patch --url "$URL" --body "{\"properties\":{\"policies\":$POLICIES}}" &&
echo "🚮 Token limit is removed from $MODEL"Delete everything azd created
In the aigateway/labs/06-token-limits folder, delete the resource group with the gateway, the Foundry account, and
the limited alias. --purge also purges the soft-deleted Foundry account. It
takes about ten minutes.
# Delete everything that azd up created, and purge the soft-deleted Foundry account
azd down --purge --force &&
echo "🚮 Everything that azd up created is deleted"To keep this gateway for other labs instead, skip azd down and raise the limit to a production value:
azd env set TOKEN_LIMIT 100000, then azd provision.
Delete everything Terraform created
In the aigateway/labs/06-token-limits folder, delete the resource group with the gateway, the Foundry
account, and the limited alias. The azurerm provider also purges the
soft-deleted Foundry account. It takes a few minutes.
# Delete everything that terraform apply created, and purge the soft-deleted Foundry account
terraform -chdir=terraform destroy -auto-approve &&
echo "🚮 Everything that terraform apply created is deleted"To keep this gateway for other labs instead, skip terraform destroy and raise the limit to a production value:
terraform -chdir=terraform apply -var token_limit=100000.
Keep the gateway, model, and key from lab 02: the next labs reuse them. azd down and terraform destroy don't
touch them, because the Bicep + azd and Terraform tracks created their own gateways.
Next steps
- Enforce cost budgets with per-key overrides: cap spend in dollars per caller.
- Monitor usage, cost, and traces: see which callers hit their limits.
- Publish model aliases across providers: give each audience its own alias and limit.
- Docs: Token rate limit, Limit requests by rate, tokens, or cost, and Policies in AI Gateway.