Azure AI Gateway Labs
Documentation Open portal
Models & routing ยท Lab 29New

Fail over to a backup model when a backend fails

Put a deliberately tiny deployment and a healthy backup behind one alias, watch throttling errors reach clients, then add a failure rule that retries on the backup and keeps the throttled backend out for 30 seconds, so every request succeeds.

Intermediate 30 min PortalCLIBicepTerraform Updated
Source Edit

What you'll do

  • Explain how failure rules, same-backend retries, temporary exclusion, and the retry limit decide where a failed request goes next
  • See throttling errors reach clients when an alias has priorities but no failure rules
  • Fail over to a backup model on HTTP 429 with a CEL failure rule, and prove it with the x-ms-served-model header
  • Keep a throttled backend out of the alias for 30 seconds, and watch it come back

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform

Your chat app calls the chat-failover alias, which prefers a gpt-5-mini deployment. When that deployment runs out of capacity, it returns HTTP 429, and so do your users' requests. In this lab you make the problem easy to see with a deliberately tiny primary deployment, then add one failure rule to the alias: a 429 from a backend moves the request to a healthy backup, gpt-6.1-sol, and keeps the throttled backend out of the alias for 30 seconds. The app keeps sending "model": "chat-failover" and never retries on its own.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, both deployments, the alias, and a key, so you can skip this section. You need only the one-time setup above.

  • Complete Publish a Foundry model and make your first call. You need the gateway, the app-first-call runtime key, and the Foundry account. Lab 02 also gave the gateway the Foundry User role on your Foundry account, which covers the two new deployments in this lab.
  • Recommended: Route an alias across models by priority and weight explains priority and weight. This lab builds on them.
  • Permission to change the gateway, for example Contributor on its resource group, and to create deployments in the Foundry account.
  • Run the scripts from the lab's folder, ~/aigateway/labs/29-model-failover. The Bash scripts and snippets need curl and jq, which Cloud Shell has.

Set these variables once. Every track and the Verify section reuse them.

# Name your gateway, the Foundry account from lab 02, the two backend models, and the alias that clients call
export RG="<resource-group>"
export GATEWAY="<gateway>"
export FOUNDRY_RG="<foundry-resource-group>"     # resource group of the Foundry account from lab 02
export FOUNDRY_ACCOUNT="<foundry-account-name>"
export PROVIDER="foundry-models"                 # the gateway provider of your Foundry account
export PRIMARY="gpt-5-mini-tiny"                 # the preferred deployment, deliberately tiny
export PRIMARY_VERSION="2025-08-07"
export BACKUP="gpt-6.1-sol"                      # the healthy backup deployment
export BACKUP_VERSION="2026-09-29"
export ALIAS="chat-failover"                     # the name that clients send in "model"
echo "๐Ÿ“ Alias $ALIAS will prefer $PRIMARY and fall back to $BACKUP on gateway $GATEWAY in $RG"

# Read the resource IDs of the gateway and the Foundry account, and the gateway's runtime URL
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export FOUNDRY_ID="$(az cognitiveservices account show -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "๐Ÿ†” Gateway: $GATEWAY_ID"
echo "๐ŸŒ Runtime URL: $AI_GATEWAY_URL"

# Build the management URL of the gateway's default workspace, and the API version, for the az rest calls
export WS="https://management.azure.com$GATEWAY_ID/workspaces/default"
export API="api-version=2026-11-01"

# Read the app-first-call key from the gateway, so that it never shows on screen or lands in a file
export AI_GATEWAY_API_KEY="$(az aigateway api-key list-secrets -g "$RG" --gateway-name "$GATEWAY" -n app-first-call \
  --query primaryKey -o tsv)"
echo "๐Ÿ”‘ Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"

# Keep a jq filter that prints the alias's backend models, its retry limit, and its failure rules
export ROUTES='.properties
  | (.backendModels[] | "\(.resourceId | split("/") | last)  priority \(.priority)  weight \(.weight)"),
    "Maximum retries per request: \(.maxRetryAttempts // 0)",
    (if ((.failures // []) | length) == 0 then "Failure rules: none"
     else (.failures | to_entries[] | "Rule \(.key + 1): when \(.value.when)  retries \(.value.retryAttempts // 0)"
       + "  exclude for \(.value.excludeFor // "this request only")") end)'

The CLI and Bicep tracks of lab 02 name the provider foundry-models. A portal import names it after your Foundry account. To check, list the backend models. The Provider column shows the provider of each one.

# List the backend models on the gateway, with the provider that holds each one
echo "โœจ Backend models on $GATEWAY:"
az aigateway model list -g "$RG" --gateway-name "$GATEWAY" -o table

Deploy the two models in your Foundry account. The primary gets capacity 1, the smallest there is, so that it returns HTTP 429 after a couple of requests; a production deployment would be much larger. If you kept gpt-6.1-sol from lab 22 or 27, skip the second command. Model inference is billed per use.

# Deploy the primary: gpt-5-mini with 1,000 tokens and 1 request per minute of Global Standard capacity
az cognitiveservices account deployment create -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" \
  --deployment-name "$PRIMARY" --model-name gpt-5-mini --model-version "$PRIMARY_VERSION" \
  --model-format OpenAI --sku-name GlobalStandard --sku-capacity 1 \
  --query "{name:name, capacity:sku.capacity, state:properties.provisioningState}" &&
  echo "โœจ Model deployment $PRIMARY is ready in $FOUNDRY_ACCOUNT"

# Deploy the backup: gpt-6.1-sol with 10,000 tokens and 10 requests per minute
az cognitiveservices account deployment create -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" \
  --deployment-name "$BACKUP" --model-name "$BACKUP" --model-version "$BACKUP_VERSION" \
  --model-format OpenAI --sku-name GlobalStandard --sku-capacity 10 \
  --query "{name:name, capacity:sku.capacity, state:properties.provisioningState}" &&
  echo "โœจ Model deployment $BACKUP is ready in $FOUNDRY_ACCOUNT"

How it works

An alias without failure rules sends each request to one backend model, picked by priority and weight, and returns whatever that backend answers, errors included. Failure rules tell the gateway what to do when a backend fails:

SettingAlias fieldPortalWhat it does
Rule conditionfailures[].whenFailure handlingAdd ruleRule conditionA CEL expression over the backend's response. Rules run in order, and only the first matching rule applies. A failure that matches no rule goes to the client.
Same backend retriesfailures[].retryAttemptsSame backend retriesExtra attempts on the failed backend before switching. The default, 0, switches at once.
Temporary exclusionfailures[].excludeForTemporarily exclude the backend after retriesExclusion duration (seconds)Keeps the failed backend out of later requests to the alias for a while, up to one day. Without it, the backend is skipped only for the current request.
Maximum retries per requestmaxRetryAttemptsRequest retry limitMaximum retries per requestCaps the extra sends of one request across all backend models, from 0 to 25. 3 allows four sends in total. The portal suggests 3. The API default, 0, leaves a matching rule nowhere to send the request: in our tests, the client then got an empty HTTP 503 instead of the backend's 429.

When a rule matches and its same-backend retries are used up, the gateway skips the failed backend for the rest of the request and runs the alias's routing again over the remaining backend models: another model in the same priority group first, then the next priority group. So priorities pick the order of the backups, and the rules decide when to move to them.

Conditions are CEL expressions. response is null when the backend sent no HTTP response at all, such as a timeout or a refused connection, so check response != null before you read response.status_code. Over the API, in Bicep, and in Terraform, wrap each expression in @cel(โ€ฆ); the portal adds the wrapper.

Not every failure can be recovered:

  • Output already sent. The gateway retries or switches only before any output reaches the client, and only when it can replay the request safely. A stream that fails after its first tokens isn't retried.
  • Stored conversations. A Responses API request with previous_response_id stays on the backend that stored the earlier response, and doesn't switch.
  • Policies. Alias policies run once per request. A same-backend retry doesn't run that model's policies again; a switch runs the new model's policies before calling it.

Create the alias

Register both deployments in the gateway, then create the chat-failover alias with gpt-5-mini-tiny at priority 0 and gpt-6.1-sol at priority 1, without failure rules for now.

Choose your track
1

Import the two models

  1. Open the AI Gateway portal and select your gateway in the top bar.

  2. Select ModelsAddModels from Microsoft Foundry.

  3. In Resource groups, select the resource group of your Foundry account. Keep Managed identity as the Authentication method, and clear Create an alias for each model.

    Clear Create an alias for each model: you create one alias for both models in the next step.
  4. In Models, make sure only your Foundry account is selected. Its row reads This model provider already exists in your gateway. 2 new models are available. Select Add models. The banner reads Models imported. 2 model(s) were created. No aliases were created. Select Close.

Checkpoint

ModelsBackend models lists gpt-5-mini-tiny and gpt-6.1-sol, next to gpt-5-mini from lab 02. The portal names the new backend model gpt-6-1-sol and shows its display name, gpt-6.1-sol.

2

Create the alias

  1. Select ModelsAddAlias for existing models.

  2. On Basics, set Alias name to chat-failover and Description to Prefers the tiny deployment. Select Next.

  3. On Auto model selection (preview), leave the option cleared, and select Next.

  4. On Select models, select gpt-5-mini-tiny and gpt-6.1-sol. Select Next.

    Both backend models come from the same Foundry provider and support the same API formats.
  5. On Session affinity, clear Enable session affinity. Select Next.

    Without session affinity, priority alone decides which backend model comes first.
  6. On Priorities and weights, set Priority to 0 for gpt-5-mini-tiny and 1 for gpt-6.1-sol. Keep both weights at 1. Select Next.

    Each model is alone in its priority group, so each has a 100% share within its priority.
  7. On Failure handling, keep the defaults, without rules, and select Next. You add a rule after you've seen the errors.

  8. On Review, select Create.

Checkpoint

The chat-failover alias opens. Its overview shows Session affinity Disabled and Routing order Priorities only, and its Backend models tab shows gpt-5-mini-tiny at priority 0 and gpt-6.1-sol at priority 1.

See the 429s without failover

Send 20 requests to chat-failover, one about every second, like a busy chat app. For each request, the script prints the HTTP status, the x-ms-served-model response header, which names the model that answered, and the Retry-After header of errors.

# Send 20 requests to the alias, with a 1-second pause between them, and count the outcomes
echo "๐Ÿšฆ Sending a burst to $ALIAS without failure rules"
./infra/burst.sh "$ALIAS" 20 1
Text
Sending 20 requests with "model": "chat-failover"
#    Time   Status  Served by                  Retry-After
1    3s     200     gpt-5-mini-2025-08-07      -
2    6s     200     gpt-5-mini-2025-08-07      -
3    8s     429     -                          30
4    10s    429     -                          30
5    12s    429     -                          30
6    16s    200     gpt-6.1-sol-2026-09-29     -
...
12   42s    200     gpt-6.1-sol-2026-09-29     -
13   44s    429     -                          30
14   46s    429     -                          30
15   48s    429     -                          30
16   52s    200     gpt-6.1-sol-2026-09-29     -
...
20   67s    200     gpt-6.1-sol-2026-09-29     -

Outcomes:
  12 200 gpt-6.1-sol-2026-09-29
   6 429
   2 200 gpt-5-mini-2025-08-07

The first two requests fit in the primary's capacity. Then the primary answers 429, and the gateway passes each 429 to the client, with the primary's Retry-After: 30: priority prefers a backend, it doesn't fail over. The 429 body reads Your requests to gpt-5-mini for gpt-5-mini-tiny in <region> have exceeded request rate limit.

In our tests, after three 429s in a row, the gateway also stopped choosing the primary for about 30 seconds on its own, so gpt-6.1-sol served the next requests, and then the cycle repeated. The product docs don't describe this protection, and it doesn't save the requests that reach the throttled primary: six of 20 requests failed. A failure rule saves them.

Checkpoint

Some requests show 429 with Retry-After 30, and none of them show a served model. Your counts can differ slightly.

Turn on failover

Add one failure rule to the alias: when a backend answers 429, switch the request to the next backend model at once, and keep the throttled backend out of the alias for 30 seconds. Cap each request at three extra sends.

Choose your track
1

Add a failure rule

  1. In ModelsAliases, open chat-failover, and select Edit. The alias opens on one page, with a section for each step.

  2. In Failure handling, under Rules, select Add rule.

  3. In Add failure rule, replace the Rule condition with this expression, without the @cel() wrapper:

    Text
    response != null && response.status_code == 429

    Keep Same backend retries at 0, keep Temporarily exclude the backend after retries selected, and keep Exclusion duration (seconds) at 30. Select Apply.

    With 0 same-backend retries, the first 429 switches the request and excludes the backend for 30 seconds.
  4. Under Request retry limit, check that Maximum retries per request is 3. The field appears once the alias has a rule. Select Save.

    Rules run in order, and only the first matching rule applies.
Checkpoint

The alias overview shows Failure handling with Maximum retries per request 3, and Rule 1 with a Custom condition, 0 (default) same-backend retries, and a 30 s exclusion.

The overview summarizes the retry limit and each rule.

Watch the failover

Wait a minute, so that the primary has capacity again, then send the same burst.

# Send the same burst of 20 requests, now with the failure rule
echo "๐Ÿšฆ Sending a burst to $ALIAS with failover"
./infra/burst.sh "$ALIAS" 20 1
Text
Sending 20 requests with "model": "chat-failover"
#    Time   Status  Served by                  Retry-After
1    2s     200     gpt-5-mini-2025-08-07      -
2    4s     200     gpt-5-mini-2025-08-07      -
3    8s     200     gpt-6.1-sol-2026-09-29     -
4    12s    200     gpt-6.1-sol-2026-09-29     -
...
18   68s    200     gpt-6.1-sol-2026-09-29     -
19   71s    200     gpt-5-mini-2025-08-07      -
20   75s    200     gpt-6.1-sol-2026-09-29     -

Outcomes:
  17 200 gpt-6.1-sol-2026-09-29
   3 200 gpt-5-mini-2025-08-07

Every request succeeds. Request 3 reached the primary, got 429, matched the rule, and the gateway sent it again to gpt-6.1-sol before it answered the client, so the client saw only the 200. The rule also excluded the primary for 30 seconds, so the next requests went straight to the backup. Each time the exclusion ends, the gateway tries the primary again: if it's still throttled, one more 429 excludes it for another 30 seconds; once it has capacity, it serves requests again, as request 19 shows.

Checkpoint

All 20 requests show 200, with no Retry-After. Most show gpt-6.1-sol-2026-09-29, and the first ones show gpt-5-mini-2025-08-07.

See what the primary saw

The client can't tell that a request was retried. The primary's own metrics can: Foundry counts its requests by status code. Metrics take two or three minutes to appear, so wait, then print the primary's requests per minute, in UTC, for the last 15 minutes.

# Print the requests to the primary deployment per minute (UTC) in the last 15 minutes, by HTTP status code
echo "๐Ÿ“Š Requests to $PRIMARY per minute, by status code:"
az monitor metrics list --resource "$FOUNDRY_ID" --metric AzureOpenAIRequests \
  --filter "ModelDeploymentName eq '$PRIMARY' and StatusCode eq '*'" \
  --interval PT1M --aggregation Total --offset 15m -o json |
  jq -r '.value[0].timeseries[]
    | (.metadatavalues[] | select(.name.value | ascii_downcase == "statuscode") | .value) as $code
    | .data[] | select((.total // 0) > 0) | "\(.timeStamp[11:16])  status \($code)  \(.total | floor)"' | sort
Text
๐Ÿ“Š Requests to gpt-5-mini-tiny per minute, by status code:
03:15  status 200  2
03:16  status 429  6
03:18  status 200  2
03:18  status 429  2

The first burst shows the six 429s that reached your clients. The second burst shows only two or three: each 429 kept the primary out for 30 seconds, and the backup served the requests in between. The requests that got 429 from the primary were retried on the backup, so no client saw them.

Checkpoint

The primary shows fewer 429s for the second burst than for the first, even though every client request got 200. If the output is empty or misses the last minutes, wait a minute and run it again.

Verify

1

Read the alias

# Read the alias, and print its backend models, retry limit, and failure rules
echo "๐Ÿ”– Alias $ALIAS routes to:"
az aigateway model-alias show -g "$RG" --gateway-name "$GATEWAY" -n "$ALIAS" | jq -r "$ROUTES"
Checkpoint
Text
gpt-5-mini-tiny  priority 0  weight 1
gpt-6.1-sol  priority 1  weight 1
Maximum retries per request: 3
Rule 1: when @cel(response != null && response.status_code == 429)  retries 0  exclude for PT30S

On the portal track, the backup shows as gpt-6-1-sol, the name that the import gave it.

2

Check that the primary comes back

Wait a minute after the last burst, so that the exclusion has ended and the primary has capacity again, then send six requests.

# Send 6 requests to the alias, and count the outcomes
echo "๐Ÿšฆ Sending a short burst to $ALIAS"
./infra/burst.sh "$ALIAS" 6 1
Checkpoint

All six requests show 200. The first ones show gpt-5-mini-2025-08-07: the primary is back in the alias. The rest show gpt-6.1-sol-2026-09-29, after the primary runs out of capacity again.

Go further

  • Add the recommended rules. The product docs suggest three rules, in this order: response == null || response.status_code == 408 with 1 same-backend retry and a 60-second exclusion, for timeouts and lost connections; response != null && response.status_code == 429 with a 30-second exclusion, for throttling; and response != null && response.status_code in [500, 502, 503, 504] with a 30-second exclusion, for server errors. Only the first matching rule applies, so a catch-all rule such as true goes last.
  • Compare without the exclusion. In the portal, edit the rule and clear Temporarily exclude the backend after retries, or send the rule without excludeFor with az rest. Clients still get only 200s, but every request now tries the throttled primary first. In our tests, the primary got 17 429s during a burst of 20 requests, instead of 3.
  • Prefer purchased capacity. Put a provisioned deployment at priority 0 and a pay-as-you-go deployment of the same model at priority 1, with the 429 rule, so that the provisioned capacity serves first and the overflow goes to pay as you go.
  • Back up across regions or providers. Register the same model from a Foundry account in another region, or from another provider, at priority 1, and add the 5xx rule, so that an outage in one region doesn't reach your users.
  • Trace the attempts. Send the gateway's telemetry to Application Insights, as in Monitor usage, cost, and traces. The product's telemetry reference lists the routing attributes that can describe each backend attempt, such as azure.ai_gateway.routing.attempt, azure.ai_gateway.routing.backend, azure.ai_gateway.routing.failure.type, and azure.ai_gateway.routing.exclusion.reason.

Clean up

Keep the gateway, the runtime key, and gpt-5-mini from lab 02. Later labs use them. Remove what you added in this lab: the alias first, then the two backend models, then the two Foundry deployments.

Choose your track
1

Remove the alias and the two models

Open chat-failover in ModelsAliases and select Delete. Requests that use the alias fail immediately; the backend models stay in the gateway. Then delete gpt-5-mini-tiny and gpt-6.1-sol from ModelsBackend models. Finally, delete both deployments from your Foundry account, for example with the last two commands of the CLI track.

azd down and terraform destroy don't touch the gateway, key, and gpt-5-mini from lab 02, because the Bicep + azd and Terraform tracks created their own gateways.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.