Azure AI Gateway Labs
Documentation Open portal
Governance & cost · Lab 09

Set gateway-wide policy baselines

Apply a baseline of token limit, request rate limit, content safety, and IP filter policies at Gateway scope so every model and MCP server, including the ones you add later, starts with the same guardrails, then layer stricter asset policies on top.

Intermediate 25 min PortalCLIBicepTerraform Updated
Source Edit

What you'll do

  • Explain how Gateway scope policies and asset policies layer, and why any applicable policy can block a call
  • Apply a token limit, request rate limit, content safety, and IP filter baseline at Gateway scope
  • Keep the baseline in a reviewed policy file and apply it with a merge-safe script or with Terraform
  • Verify the baseline from your own IP address and see a stricter model policy take effect on top of it

Before you start

Tracks: Portal, Azure CLI, Policy as code, Terraform

Your gateway now serves several models and MCP servers, and teams add new ones every week. Some assets have carefully tuned policies, and some have none at all. The platform and security teams agree on a baseline that every asset gets by default: a generous token limit, a request rate limit, Balanced content safety, and runtime access only from approved networks. In this lab you apply that baseline once, at Gateway scope, and keep tighter asset policies on top of it.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

  • Complete Throttle tokens per caller with token limits and Screen prompts and responses with content safety, including their Clean up sections. You need a gateway, an imported model, a callable alias, and a runtime API key from AccessAPI keys.
  • Permission to change the gateway, for example Contributor on the resource group.
  • The public IP addresses or CIDR ranges your callers use. Once the IP filter is on, calls from anywhere else get 403.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/09-gateway-baselines.

Set these variables once. Every track and the Verify section reuse them.

# Name your gateway, its model provider, and the backend model that the Verify section calls
export RG="<resource-group>"
export GATEWAY="<gateway>"                 # 24 characters or fewer
export PROVIDER="<provider-name>"          # for example foundry-models
export MODEL="<model-name>"                # the backend model name under that provider

# Read the gateway's runtime URL, which apps call
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"

# Read the alias that apps send in the model field: for Foundry imports, it matches the deployment's model name
export AI_GATEWAY_MODEL="$(az aigateway model show -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" \
  -n "$MODEL" --query properties.deployment.modelName -o tsv)"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"

# Paste the runtime key you created in Access › API keys, without showing it: don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"

# Read the public IP address that this machine uses on the internet
export MY_IP="$(curl -s https://api.ipify.org)"
echo "🌐 Public IP address: $MY_IP"

# Set the ranges that the IP filter allows
export ALLOWED_CIDRS="$MY_IP/32"           # comma-separated, for example "$MY_IP/32,198.51.100.0/24"
echo "📝 Ranges the IP filter allows: $ALLOWED_CIDRS"

How it works

A policy with Gateway scope is stored on the gateway's default workspace instead of on one asset. It applies to every compatible model, alias, and MCP server, including assets you add later, without another policy update. Asset policies keep working alongside it. The gateway evaluates every applicable policy before it forwards a request, and if any one of them blocks, the client gets an error and the backend is never called. So an asset policy can make the rules tighter for one asset, but it can't loosen the baseline.

This lab's baseline:

PolicyApplies toBaseline valueBlocked calls get
Token rate limitModels100,000 tokens per minute, per caller identity429 with Retry-After
Request rate limitModels and MCP servers600 calls per 60 seconds, per caller identity429 with Retry-After
Content safetyModels and MCP serversBalanced (Medium) for all four categories400
IP filterModels and MCP serversAllow listed IPs only: your CIDR ranges403

A Gateway scope Cost limit is one budget across all models; you can add it in Enforce cost budgets with per-caller overrides. Agent 365 tool threat protection can't be set at Gateway scope; it applies to individual MCP servers.

Note

At the time of writing, AI Gateway resources save the content safety policy but don't block calls with it yet, as Screen prompts and responses with content safety explains. Keep it in the baseline so it applies once the gateway enforces it, and rely on your Foundry deployments' content filters until then.

Warning

The IP filter applies to runtime calls. You keep managing the gateway in the portal and with the Azure CLI, so you can always fix the list. But every caller outside it gets 403: teammates, CI pipelines, hosted agents, and your own machine when you switch networks or VPN. Remove it in Clean up.

Apply the baseline

Pick one track. Each track creates the same four policies at Gateway scope. The Azure CLI track sets three of them with one az aigateway policy set command and adds content safety with az rest. The Policy as code track keeps them in a reviewed file and applies the whole file with a script, and the Terraform track applies the same file with the azapi provider.

Choose your track
1

Add the token rate limit

Open the AI Gateway portal and select your gateway from the gateway switcher. Select PoliciesAdd policy. On the Type step, select Token rate limit, then Next.

On the Scope step, select Gateway ("Apply to the entire gateway workspace, including applicable assets added later."), then Next.

On the Scope step, Gateway applies the policy to every compatible asset, including assets added later.

On the Details step, under Parameters, set:

FieldValue
Time periodPer minute
Allowed tokens100000
Counter dimensionsIdentity

Select Create.

2

Add the request rate limit

Select Add policy again. Select Request rate limit, then Gateway scope, and set:

FieldValue
Calls600
Renewal period60 seconds
Counter dimensionsIdentity

Select Create.

3

Add content safety

The Add policy wizard doesn't list Content safety for AI Gateway resources, so create this policy with the Azure CLI, for example in Azure Cloud Shell, with the variables from Before you begin. Gateway scope policies live in the properties.policies array of the gateway's default workspace. The commands read that array, add Balanced (Medium) content safety on all four categories, and write the array back, so every other Gateway scope policy stays:

# Build the management URL of the gateway's default workspace, which holds the Gateway scope policies
GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
URL="https://management.azure.com$GATEWAY_ID/workspaces/default?api-version=2026-11-01"

# Read the policies, drop any content safety policy, and add one at Medium (Balanced) in all four categories
POLICIES="$(az rest --method get --url "$URL" | jq -c '
  [(.properties.policies // [])[] | select(.type != "contentSafety")]
  + [{"type":"contentSafety","scope":"resource","hateSeverity":"Medium","violenceSeverity":"Medium","sexualSeverity":"Medium","selfHarmSeverity":"Medium"}]')"

# Send the merged array back: a PATCH replaces the whole policies array
az rest --method patch --url "$URL" --body "{\"properties\":{\"policies\":$POLICIES}}" --output none &&
  echo "🔐 Content safety on $GATEWAY is Balanced (Medium) for all four categories at Gateway scope"

Reload Policies. The policy shows as Content safety with Gateway scope. To change a category later, open its More actions menu and select Edit.

4

Add the IP filter

Select Add policy. Under Security, select IP filter, then Gateway scope, and set:

FieldValue
ActionAllow listed IPs only
IP addresses / CIDR rangesYour ALLOWED_CIDRS value, for example 198.51.100.7/32

Use CIDR notation for every entry: /32 for a single IPv4 address. Check the value against echo "$MY_IP" before you select Create.

Checkpoint

Policies lists four policies whose scope is Gateway: Token rate limit, Request rate limit, Content safety, and IP filter.

Verify

Check that the baseline covers your model and MCP servers, that your own calls still pass, and that a tighter asset policy still takes effect on top of it.

1

Review the baseline in the portal

Select Policies. The four baseline policies show Gateway in the Scope column. Policies you add to a model, alias, or MCP server show that asset instead.

2

Call the model from your IP address

From the lab folder, send one chat completion:

# Send one chat completion from this machine through the baseline
echo "🤖 Expect Status: 200 and a reply, because your IP address is on the allow list:"
./infra/check-call.sh "$AI_GATEWAY_MODEL"

On GitHub, open check-call.sh or check-call.ps1.

Expected output:

Text
🤖 Expect Status: 200 and a reply, because your IP address is on the allow list:
Status: 200
Reply:  A gateway-wide policy baseline is a default set of rules that applies to every asset ...

Your call passed all four baseline policies: your IP address is on the allow list, you're far below both rate limits, and the prompt is benign.

Warning

If check-call keeps returning 403, the gateway sees a different public IP address than api.ipify.org reported. A VPN, a corporate proxy, or a client such as Global Secure Access can send your Azure traffic through another egress address, and some networks rotate through a pool of egress addresses, so a /32 can pass one call and block the next. Add your organization's egress ranges to ALLOWED_CIDRS and apply the baseline again, or run the lab from Azure Cloud Shell, where both addresses match.

If you completed Federate MCP tools behind one endpoint, run ../04-federate-mcp-tools/infra/list-tools.sh learn (PowerShell: ../04-federate-mcp-tools/infra/list-tools.ps1 -Server learn). The MCP server is covered by the same IP filter, request rate limit, and content safety policies, and it still lists its tools.

3

Layer a stricter model policy

Add a deliberately low token limit to your model only, the same one you used in lab 06. az aigateway model policy set-token-limit keeps every other policy on the model:

# Add a low token limit to the model only: 1,000 tokens per minute for each caller
az aigateway model policy set-token-limit -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
  --token-limit 1000 --output none &&
  echo "🔐 Token limit of 1,000 tokens per minute for each caller is on $MODEL"

Then send a burst with the script from lab 06:

# Send a burst of eight requests with the script from lab 06
echo "🚦 Expect 429 Too Many Requests once the 1,000-token allowance runs out:"
../06-token-limits/infra/trigger-token-limit.sh "$AI_GATEWAY_MODEL" 8

Once the 1,000-token allowance runs out, usually after four to six requests, the gateway returns 429. A fast burst can go a few hundred tokens past the budget before the first 429. The baseline allows 100,000 tokens per minute, but the model's tighter limit is evaluated too, and it blocks. If every request returns 200, wait a minute for the new limit to reach the gateway and run the burst again. Remove the model limit and run the burst again:

# Remove only the token limit from the model, and keep its other policies
az aigateway model policy remove -g "$RG" --gateway-name "$GATEWAY" --provider-name "$PROVIDER" -n "$MODEL" \
  --remove-token-limit true --output none &&
  echo "🚮 Token limit is removed from $MODEL"

All eight requests now return 200, because only the generous baseline applies.

4

See the IP filter block a call (optional)

Point the allow list at a documentation address that can't be yours, call the model, then restore your range.

  • Portal: in Policies, select Edit on the IP filter policy, set the ranges to 203.0.113.10/32, and save.
  • Azure CLI: az aigateway policy set-ip-filter -g "$RG" --gateway-name "$GATEWAY" --action Allow --allowed-ips "[203.0.113.10/32]" --output none
  • Script (Bash): ALLOWED_CIDRS="203.0.113.10/32" ./infra/apply-baseline.sh
  • Script (PowerShell): ./infra/apply-baseline.ps1 -ResourceGroup $RG -Gateway $GATEWAY -AllowedCidrs "203.0.113.10/32"
  • Terraform: in the lab folder, terraform -chdir=terraform plan -var allowed_cidrs=203.0.113.10/32 -out tfplan, then terraform -chdir=terraform apply tfplan

Wait about a minute, then run check-call again. It now returns Status: 403, and the request never reaches the model. Restore your range in the portal, with the same set-ip-filter command and --allowed-ips "[${ALLOWED_CIDRS// /}]", or by running the script again without the override (./infra/apply-baseline.sh, or the PowerShell script without -AllowedCidrs), or by planning and applying again without -var in Terraform. Then check that check-call returns 200. Policy changes take up to a minute to reach the gateway, so if you still see the old result, wait and retry.

5

Watch outcomes in Monitoring

If you configured Application Insights in Monitor usage, cost, and traces, open Monitoring and compare 400, 403, and 429 outcomes before and after the baseline. A sudden rise in 403 usually means a caller's network isn't on the allow list.

Checkpoint

Policies shows four Gateway scope policies. Your own call returned 200, a stricter model limit still returned 429 on top of the baseline, and you know how a blocked IP address looks (403).

Go further

Clean up

Remove the baseline before you continue, especially the IP filter: later labs call the gateway from hosted agents and other networks, which would get 403. In your own environment, you can keep the other three policies and replace the IP filter with your organization's egress ranges.

Choose your track
1

Remove the baseline

Select Policies. For each of the four policies with Gateway scope, select Delete. Remove the IP filter first.

Keep the gateway, model, and key. The next labs reuse them.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.