You've built every piece by hand in earlier labs: a keyless Foundry backend, aliases, MCP servers, per-caller limits, a gateway-wide baseline, and monitoring. Now the platform team wants the whole setup as code. Dev and prod should differ only in their parameters, every change should go through a pull request, and a pipeline should deploy it without a stored secret. In this lab you deploy a modular reference architecture, apply the baseline, prove it works with a smoke test, and wire it all into GitHub Actions.
Before you begin
One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.
- Choose where to run commandsEvery track except the portal
- Sign in to AzureOn your own machine. Cloud Shell is already signed in.
- Install the
aigatewayextensionThe Azure CLI tracks and the Verify steps - Register the resource providersOnce per subscription, before you create a gateway with the Azure CLI
- Clone the labs repoThe Bicep + azd and Terraform tracks, the notebooks, and the code samples
You've done the one-time setup that this lab uses.
- Complete Set gateway-wide policy baselines, Monitor usage, cost, and traces, and Call backends with managed identities. This lab builds on what you learned there, not on the resources you created: it deploys a new, separate environment.
- For the Terraform track: Terraform 1.9 or later, as in Deploy a lab with Terraform.
- The Azure CLI installs Bicep the first time you deploy. The telemetry script and the smoke test read the gateway
with the
aigatewayextension. Itsaz aigateway policycommands take each setting as a flag, not a policy array, so to apply the whole baseline from a file, the baseline script calls the management API withaz rest. - Owner, or Contributor plus Role Based Access Control Administrator, on the subscription. The template creates two resource groups and several role assignments.
- Quota for
gpt-5-miniwith theGlobalStandardSKU in the region you deploy to. The dev environment uses 50 capacity units (50,000 tokens per minute). - For Automate deployments with GitHub Actions: a GitHub
repository where you can create environments and variables, the GitHub CLI signed
in with
gh auth login, and permission to create an app registration in Microsoft Entra ID.
Run every command from this lab's folder, ~/aigateway/labs/21-enterprise-reference, so the infra/ paths
resolve. The Terraform track moves into terraform/ to deploy, then back. Set these variables once:
# Set the region and the deployment name that the template and the rest of the lab use
export LOCATION="eastus2" # deployment metadata, the Foundry account, and the monitoring resources
export ENV_NAME="aigw-dev" # deployment name, the same as environmentName in main.bicepparam
echo "📝 Deployment $ENV_NAME in $LOCATION"
# Read your own object ID: main.bicepparam, or TF_VAR_principal_id on the Terraform track, gives you data-plane
# access to the Foundry account, the alias, and the MCP server
export AZURE_PRINCIPAL_ID="$(az ad signed-in-user show --query id -o tsv)"
echo "👤 Your principal ID: $AZURE_PRINCIPAL_ID"
# Show the subscription that you deploy to, so that you can check it before you go on
echo "🆔 Subscription that you deploy to:"
az account show --query "{subscription:name, id:id}" -o table# Set the region and the deployment name that the template and the rest of the lab use
$env:LOCATION = "eastus2" # deployment metadata, the Foundry account, and the monitoring resources
$env:ENV_NAME = "aigw-dev" # deployment name, the same as environmentName in main.bicepparam
Write-Host "📝 Deployment $env:ENV_NAME in $env:LOCATION"
# Read your own object ID: main.bicepparam, or TF_VAR_principal_id on the Terraform track, gives you data-plane
# access to the Foundry account, the alias, and the MCP server
$env:AZURE_PRINCIPAL_ID = az ad signed-in-user show --query id -o tsv
Write-Host "👤 Your principal ID: $env:AZURE_PRINCIPAL_ID"
# Show the subscription that you deploy to, so that you can check it before you go on
Write-Host "🆔 Subscription that you deploy to:"
az account show --query "{subscription:name, id:id}" -o tableHow it works
One subscription-scope template, infra/main.bicep, creates two resource groups and composes small modules.
Each module owns one concern, so the people who own that concern can review and change it on their own, and no
module outputs a secret. Parameter files hold everything that differs between environments. The Terraform track
has the same shape: a root module, terraform/main.tf, calls one module in terraform/modules/ per Bicep module,
and dev.tfvars and prod.tfvars replace the parameter files.
Gateway-wide policies aren't in Bicep or Terraform. They live on the gateway's default workspace, and declaring
that workspace in a template would replace it. Instead, a script merges them in after each deployment, as in
Set gateway-wide policy baselines.
| File | Resource group | Deploys |
|---|---|---|
modules/monitoring.bicep | …-gateway | Log Analytics workspace and a workspace-based Application Insights component, <gateway>-insights, with OTLP ingestion on |
modules/foundry.bicep | …-foundrymodels | Foundry account with key authentication turned off, and the gpt-5-mini deployment |
modules/gateway.bicep | …-gateway | The AI Gateway (Microsoft.ApiManagement/aigateways, system-assigned identity) and Foundry User for the identity on the Foundry account |
modules/models.bicep | …-gateway | A Foundry model provider that authenticates with the managed identity, the backend model with a token limit and content safety, the alias chat, and AI Gateway Models User on the alias |
modules/tools.bicep | …-gateway | The Microsoft Learn MCP server learn, and AI Gateway Tools User on it |
modules/keys.bicep | …-gateway | One runtime API key per app: web-chat and batch-jobs |
modules/telemetry.bicep | …-gateway | The gateway's OpenTelemetry exporter to the component, with the recommended token claims. scripts/connect-telemetry.sh deploys it |
scripts/connect-telemetry.sh | Not applicable | Waits for the component's OTLP endpoints, grants the gateway's identity Monitoring Metrics Publisher on its Data Collection Rule, then deploys modules/telemetry.bicep |
scripts/apply-baselines.sh | Not applicable | Gateway scope Request rate limit and Cost limit from baselines.json |
On the Terraform track, each module is a folder with the same name, such as terraform/modules/gateway. The
telemetry module also does what connect-telemetry.sh does, so terraform apply connects telemetry in the same
run. Both tracks apply the baseline with scripts/apply-baselines.sh.
Make the architecture decisions
Each decision below is already built into the templates. In your own environment, change the parameters first, and change the shape only when your organization needs something different.
Identity: no secrets between services or in the pipeline
Each trust boundary uses its own mechanism, and none of them needs a stored secret:
| Boundary | Mechanism | Where |
|---|---|---|
| Gateway to Foundry | The gateway's system-assigned identity has Foundry User on the account. The account sets disableLocalAuth: true, so Foundry keys don't work at all. | gateway.bicep, foundry.bicep |
| Apps to gateway | One runtime API key per app, so every app gets its own policy counters, its own row in Monitoring, and its own rotation. Or keyless Microsoft Entra calls through AI Gateway Models User on the alias and AI Gateway Tools User on the MCP server, granted to groups. | keys.bicep, models.bicep, tools.bicep |
| Pipeline to Azure | An OpenID Connect federated credential. GitHub exchanges a short-lived token for an Azure token, so there's no client secret to rotate or leak. | deploy-workflow.yml |
Roles are granted at the narrowest scope that works: the alias instead of the whole gateway, and one MCP server
instead of every tool. Templates output key names only, never key values. The smoke test reads a key with
az aigateway api-key list-secrets when it needs one.
Policies: a gateway floor plus model limits
| Layer | Declared in | Policies | Counted per |
|---|---|---|---|
| Gateway | baselines.json, applied by scripts/apply-baselines.sh | Request rate limit: 120 calls per 60 seconds, on models and MCP servers. Cost limit: $20 per day, one budget across all models. | Caller identity |
| Model | models.bicep | Token rate limit: 20,000 tokens per minute in dev. Content safety: Balanced (Medium) for all four categories. | Caller identity |
The gateway evaluates every applicable policy before it forwards a request, and any one of them can block it.
The platform team owns the gateway floor, and app teams tune the model layer through parameters. Blocked calls
get 429 from a rate limit, 403 with LlmCostQuotaExceeded from the cost limit, and 400 from content
safety.
The baseline script reads the workspace's policies array, updates policies of the same type in place, appends
missing ones, and writes the array back with PATCH. Updating in place keeps a cost limit's id, its
portal-only overrides, and the spend it has already counted.
The portal writes "scope": "workspace" on Gateway scope rate and cost limits, so baselines.json does too.
If the API rejects a field, create the policy once in the portal at Gateway scope, read the Gateway scope
policies with az aigateway policy list -g "$RG" --gateway-name "$GATEWAY", and copy that shape into
baselines.json.
Observability: the destination in Bicep, the exporter after it
monitoring.bicep creates the Log Analytics workspace and the Application Insights component, with retention
set per environment, and turns on OTLP ingestion. A few minutes after the deployment, Azure creates the component's
OTLP endpoints and a Data Collection Rule in a managed resource group. The gateway's OpenTelemetry exporter, in
telemetry.bicep, needs those endpoints, and the gateway's managed identity needs Monitoring Metrics Publisher
on the rule before the exporter is created. Bicep can't wait for the endpoints, and it can't assign the role,
because the managed resource group has a deny assignment that blocks deployments. So scripts/connect-telemetry.sh
runs after each deployment: it waits for the endpoints, assigns the role with the Azure CLI, then deploys
telemetry.bicep. The result matches the portal's Configure telemetry wizard: metrics, logs, and traces, with
the recommended token claims.
Terraform can wait, so on the Terraform track the telemetry module does the same work inside terraform apply.
It runs the same wait with the Azure CLI, reads the endpoints once and keeps them in the state, assigns the role,
then creates the exporter.
The exporter sends to the component's OTLP endpoints and doesn't link to the Application Insights resource, yet
the Monitoring dashboard still reads its data. The other exporter type, ApplicationInsights, which the
reference sample uses, is legacy: the portal flags it because it no longer activates the OTLP signals the dashboard
needs. An exporter's type can't change, so the script replaces an ApplicationInsights exporter that an earlier
deployment created.
Naming, tags, and resource groups
- Deterministic names. A resource token, the first eight characters of
uniqueString(subscription, environmentName, location), goes into every name. Deploying again updates the same resources instead of creating new ones. - Two resource groups.
rg-<env>-<token>-gatewayholds the gateway and its monitoring, andrg-<env>-<token>-foundrymodelsholds model capacity. You can secure, scale, and delete them separately, and only the gateway's identity needs access to Foundry. - A unique gateway name. The gateway is
aigw-<token>. Its runtime URL starts with the name's first 10 characters and a hash, likehttps://aigw-abcde-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com. SetaiGatewayNameto choose your own: up to 24 letters, digits, and hyphens, starting with a letter. - Tags. Resource groups and top-level resources get
environmentandworkload: ai-gateway, plus thetagsfrom the parameter file, such ascostCenter. - Regions.
aiGatewayLocationdefaults toeastus2. Set it to any region where AI Gateway is available, as in Find the available regions. The other resources follow the deployment's--location, so pick one where the model is available. - On the Terraform track, the token is six characters of a hash of the subscription, the environment name,
and the location. The resource groups are
rg-<env>-gatewayandrg-<env>-foundrymodels, the environment tag isenvironment-name, and the gateway's region,ai_gateway_location, defaults tolocation, so one region drives every resource.
Environments: one template, two parameter files
| Setting | main.bicepparam (dev) | prod.bicepparam (prod) |
|---|---|---|
environmentName | aigw-dev | aigw-prod |
Developer access (principalId) | You, from AZURE_PRINCIPAL_ID | None |
Model capacity (modelCapacity) | 50 | 200 |
Tokens per minute per caller (tokensPerMinutePerCaller) | 20,000 | 50,000 |
Microsoft Entra access (modelUsers, toolUsers) | Your user only | Groups you add |
Log retention (logRetentionInDays) | 30 days | 90 days |
costCenter tag | dev | platform |
The Terraform track's dev.tfvars and prod.tfvars set the same values, with the variable names in snake case,
such as model_capacity and tokens_per_minute_per_caller. Their environment names are aigw-dev-tf and
aigw-prod-tf, so a Terraform deployment never touches a Bicep one.
Put prod in its own subscription so quotas, role assignments, and Azure Policy stay isolated from dev. The template is subscription-scoped, so it works unchanged. Use a separate deployment identity for each subscription.
Delivery: GitHub Actions with OpenID Connect
The workflow installs the aigateway extension, previews every change with what-if, deploys, connects
telemetry, applies the baseline, and runs the smoke test. A prod GitHub environment with required reviewers gates production, and a
concurrency group per environment stops two runs from deploying to the same environment at once. The deployment
identity needs Contributor and Role Based Access Control Administrator on the subscription, because the
template creates role assignments. To narrow that, you can add a condition that only lets it assign the roles the
template needs, as described in
Delegate Azure access management to others.
With Terraform, the same workflow runs terraform plan and terraform apply instead of what-if and the
deployment, and keeps the Terraform state in an Azure Storage container that every run shares.
Review the templates
main.bicep wires the modules together and passes outputs from one to the next, so Bicep deploys them in the
right order: monitoring and Foundry first, then the gateway, then models, tools, and keys.
// Enterprise reference deployment for Azure AI Gateway. One subscription-scope template creates two resource groups
// and composes small modules:
//
// rg-<env>-<token>-foundrymodels foundry.bicep Foundry account (keyless) + model deployment
// rg-<env>-<token>-gateway monitoring.bicep Log Analytics + Application Insights with OTLP ingestion
// gateway.bicep aigateways resource + Foundry User role for the gateway identity
// models.bicep Foundry provider + backend model (tokenLimit, contentSafety) + alias
// tools.bicep Microsoft Learn MCP server
// keys.bicep one runtime API key per app
//
// After deployment, scripts/connect-telemetry.sh (or .ps1) connects the gateway to Application Insights with
// telemetry.bicep, an OpenTelemetry exporter, once Azure has created the OTLP endpoints. Then
// scripts/apply-baselines.sh (or .ps1) applies the gateway-wide baseline policies.
targetScope = 'subscription'
import { principal } from './modules/types.bicep'
@minLength(1)
@maxLength(64)
@description('Environment name, for example aigw-dev or aigw-prod. Used in resource group names, tags, and the deterministic resource token.')
param environmentName string
@description('Region for the resource groups, the Foundry account, and the monitoring resources. The model and SKU must be available here.')
param location string = deployment().location
@description('Region for the gateway. Pick one where AI Gateway is available. List them with az provider show --namespace Microsoft.ApiManagement (Setup guide, Find the available regions).')
param aiGatewayLocation string = 'eastus2'
@description('Optional Microsoft Entra object ID of a developer. When set, the developer gets data-plane roles on the Foundry account, the alias, and the MCP server. Leave empty in production.')
param principalId string = ''
@description('Extra tags merged into the default tags on every resource group and resource.')
param tags object = {}
@description('Gateway name, max 24 characters. Defaults to aigw-<token>. Its first 10 characters start the runtime host name.')
param aiGatewayName string = ''
@description('Foundry model name to deploy.')
param modelName string = 'gpt-5-mini'
@description('Foundry model version to deploy.')
param modelVersion string = '2025-08-07'
@description('ARM-safe name of the Foundry deployment. It is also the backend model name in the gateway.')
param modelDeploymentName string = 'gpt-mini-latest'
@description('Model format for the Foundry deployment.')
param modelFormat string = 'OpenAI'
@description('Model deployment SKU name.')
param modelSkuName string = 'GlobalStandard'
@description('Deployment capacity, in thousands of tokens per minute for Global Standard.')
@minValue(1)
param modelCapacity int = 50
@description('Tokens per minute that each caller identity (API key or Microsoft Entra principal) can use on the backend model.')
@minValue(1)
param tokensPerMinutePerCaller int = 20000
@allowed([
'None'
'Low'
'Medium'
'High'
])
@description('Content safety threshold for every category on the backend model. The threshold is the lowest severity that is blocked: Low is strictest, None turns the category off.')
param contentSafetySeverity string = 'Medium'
@description('Client-facing alias that apps send in the request "model" field.')
param aliasName string = 'chat'
@description('Apps that get their own runtime API key. Each key is named after the app.')
param appNames string[] = [
'web-chat'
'batch-jobs'
]
@description('Microsoft Entra principals (usually groups) that get the AI Gateway Models User role on the alias.')
param modelUsers principal[] = []
@description('Microsoft Entra principals (usually groups) that get the AI Gateway Tools User role on the Microsoft Learn MCP server.')
param toolUsers principal[] = []
@description('Days to keep gateway telemetry in Log Analytics.')
@minValue(30)
@maxValue(730)
param logRetentionInDays int = 30
var resourceToken = take(toLower(uniqueString(subscription().id, environmentName, location)), 8)
var environmentLabel = toLower(environmentName)
var allTags = union({
environment: environmentName
workload: 'ai-gateway'
}, tags)
var gatewayResourceGroupName = 'rg-${environmentLabel}-${resourceToken}-gateway'
var foundryModelsResourceGroupName = 'rg-${environmentLabel}-${resourceToken}-foundrymodels'
var effectiveAiGatewayName = !empty(aiGatewayName) ? aiGatewayName : 'aigw-${resourceToken}'
var modelServicesName = 'aimodels-${resourceToken}'
var appInsightsName = take('${effectiveAiGatewayName}-insights', 64)
var logAnalyticsWorkspaceName = take('${effectiveAiGatewayName}-logs', 63)
var developer = empty(principalId) ? [] : [
{
principalId: principalId
principalType: 'User'
}
]
var gatewayModelDeployments = [
{
deploymentName: modelDeploymentName
modelName: modelName
modelFormat: modelFormat
modelVersion: modelVersion
skuName: modelSkuName
capacity: modelCapacity
description: 'Default chat model for every app behind the gateway.'
}
]
resource gatewayRg 'Microsoft.Resources/resourceGroups@2021-04-01' = {
name: gatewayResourceGroupName
location: location
tags: allTags
}
resource foundryModelsRg 'Microsoft.Resources/resourceGroups@2021-04-01' = {
name: foundryModelsResourceGroupName
location: location
tags: allTags
}
module monitoring './modules/monitoring.bicep' = {
name: 'monitoring-${resourceToken}'
scope: gatewayRg
params: {
location: location
tags: allTags
logAnalyticsWorkspaceName: logAnalyticsWorkspaceName
appInsightsName: appInsightsName
retentionInDays: logRetentionInDays
}
}
module foundry './modules/foundry.bicep' = {
name: 'foundry-models-${resourceToken}'
scope: foundryModelsRg
params: {
location: location
tags: allTags
aiServicesName: modelServicesName
principalId: principalId
modelDeployments: gatewayModelDeployments
}
}
module gateway './modules/gateway.bicep' = {
name: 'ai-gateway-${resourceToken}'
scope: gatewayRg
params: {
location: aiGatewayLocation
gatewayName: effectiveAiGatewayName
tags: allTags
foundryResourceIds: [
foundry.outputs.aiServicesId
]
}
}
// Every module below takes gateway.outputs.gatewayName, so it starts only after the gateway and the Foundry User
// role assignment exist.
module models './modules/models.bicep' = {
name: 'ai-gateway-models-${resourceToken}'
scope: gatewayRg
params: {
gatewayName: gateway.outputs.gatewayName
providerName: 'foundry'
providerDisplayName: 'Foundry'
providerDescription: 'Managed-identity Foundry provider for the ${environmentName} environment.'
foundryEndpoint: foundry.outputs.aiServicesEndpoint
foundryResourceIds: [
foundry.outputs.aiServicesId
]
foundryModels: [for (deployment, i) in gatewayModelDeployments: {
armName: deployment.deploymentName
description: deployment.description
resourceId: foundry.outputs.modelDeploymentIds[i]
modelName: deployment.modelName
modelVersion: deployment.modelVersion
tokenLimit: tokensPerMinutePerCaller
supportedEndpoints: [
'/openai/v1/chat/completions'
'/openai/v1/responses'
]
}]
contentSafetySeverity: contentSafetySeverity
aliasName: aliasName
aliasUsers: concat(developer, modelUsers)
}
}
module tools './modules/tools.bicep' = {
name: 'ai-gateway-tools-${resourceToken}'
scope: gatewayRg
params: {
gatewayName: gateway.outputs.gatewayName
toolUsers: concat(developer, toolUsers)
}
}
module keys './modules/keys.bicep' = {
name: 'ai-gateway-keys-${resourceToken}'
scope: gatewayRg
params: {
gatewayName: gateway.outputs.gatewayName
appNames: appNames
}
}
output AZURE_LOCATION string = location
output AZURE_TENANT_ID string = tenant().tenantId
output AI_GATEWAY_NAME string = gateway.outputs.gatewayName
output AI_GATEWAY_LOCATION string = aiGatewayLocation
output AI_GATEWAY_RESOURCE_GROUP string = gatewayRg.name
output AI_GATEWAY_RESOURCE_ID string = gateway.outputs.gatewayId
output AI_GATEWAY_URL string = gateway.outputs.gatewayUrl
output AI_GATEWAY_PRINCIPAL_ID string = gateway.outputs.gatewayPrincipalId
output AI_GATEWAY_MODEL string = models.outputs.aliasName
output MCP_ENDPOINT string = tools.outputs.mcpEndpoint
output API_KEY_NAMES array = keys.outputs.apiKeyNames
output FOUNDRY_MODELS_RESOURCE_GROUP string = foundryModelsRg.name
output FOUNDRY_MODELS_NAME string = foundry.outputs.aiServicesName
output FOUNDRY_MODELS_RESOURCE_ID string = foundry.outputs.aiServicesId
output APPLICATION_INSIGHTS_NAME string = monitoring.outputs.appInsightsName
output LOG_ANALYTICS_WORKSPACE_ID string = monitoring.outputs.logAnalyticsWorkspaceIdOn GitHub, open infra/main.bicep.
gateway.bicep is the only module that declares the gateway itself. The others reference it as existing and
only add child resources to it.
// The AI Gateway itself: a Microsoft.ApiManagement/aigateways resource with a system-assigned managed identity, in
// the default SKU (sku is omitted, as in the reference sample), and the Foundry User role for the gateway identity
// on every Foundry account it calls.
targetScope = 'resourceGroup'
@description('Location of the gateway. Pick one where AI Gateway is available. List them with az provider show --namespace Microsoft.ApiManagement (Setup guide, Find the available regions).')
param location string
@description('Gateway name, max 24 characters. Its first 10 characters start the runtime host name.')
@maxLength(24)
param gatewayName string
@description('Tags applied to the gateway.')
param tags object = {}
@description('Optional subnet resource ID for outbound virtual network integration. The subnet must be delegated to Microsoft.App/environments. You can add, change, or remove it after the gateway is created.')
param backendSubnetId string = ''
@description('Resource IDs of the Foundry accounts the gateway calls with its managed identity.')
param foundryResourceIds array = []
#disable-next-line BCP081
resource aiGateway 'Microsoft.ApiManagement/aigateways@2026-11-01' = {
name: gatewayName
location: location
tags: tags
identity: {
type: 'SystemAssigned'
}
properties: empty(backendSubnetId) ? {} : { backend: { subnet: { id: backendSubnetId } } }
}
module foundryUserRoleAssignments './foundry-user-role.bicep' = [for (foundryResourceId, i) in foundryResourceIds: {
name: 'foundry-rbac-${i}-${uniqueString(foundryResourceId, gatewayName)}'
scope: resourceGroup(split(foundryResourceId, '/')[2], split(foundryResourceId, '/')[4])
params: {
foundryAccountName: last(split(foundryResourceId, '/'))
gatewayPrincipalId: aiGateway.identity.principalId
gatewayResourceId: aiGateway.id
}
}]
output gatewayName string = aiGateway.name
output gatewayId string = aiGateway.id
output gatewayUrl string = 'https://${aiGateway.properties.frontend.defaultHostname}'
output runtimeHostname string = aiGateway.properties.frontend.defaultHostname
output gatewayPrincipalId string = aiGateway.identity.principalId
output foundryUserRoleAssignmentNames array = [for (foundryResourceId, i) in foundryResourceIds: foundryUserRoleAssignments[i].outputs.roleAssignmentName]On GitHub, open infra/modules/gateway.bicep.
Each deployment sends a PUT for the gateway with exactly the properties in gateway.bicep. If you change
gateway settings outside Bicep, add them to this module. Otherwise, the next deployment can reset them.
Outbound virtual network integration and Public network access are the exceptions: a deployment that
doesn't set them keeps the gateway's current values. gateway.bicep accepts the integration subnet as
backendSubnetId, but main.bicep doesn't pass it. To
use it, add a parameter to main.bicep, or set it in the portal under Networking or with
az aigateway network update --subnet. See
Lock down backend traffic with private networking.
models.bicep declares the model-level policies right on the backend model, so they're versioned with it:
// Publishes Foundry model deployments through the gateway: one Foundry model provider that authenticates with the
// gateway's managed identity, one backend model per deployment with model-level policies (a per-caller token limit
// and content safety), a stable client-facing alias, and optional AI Gateway Models User role assignments on that
// alias for Microsoft Entra callers.
targetScope = 'resourceGroup'
import { principal } from './types.bicep'
@description('Name of the existing gateway in this resource group.')
param gatewayName string
@description('Name of the model provider to create.')
param providerName string = 'foundry'
@description('Display name of the model provider.')
param providerDisplayName string = 'Foundry'
@description('Description of the model provider.')
param providerDescription string = ''
@description('Endpoint of the Foundry account, for example https://<foundry-account-name>.cognitiveservices.azure.com/.')
param foundryEndpoint string
@description('Resource IDs of the Foundry accounts behind the provider.')
param foundryResourceIds array
@description('Audience the gateway requests a managed identity token for when it calls Foundry.')
param foundryManagedIdentityResource string = 'https://cognitiveservices.azure.com/'
@description('Backend models: objects with armName, description, resourceId, modelName, modelVersion, tokenLimit, and supportedEndpoints.')
param foundryModels array
@description('Content safety threshold for every category on every backend model. The threshold is the lowest severity that is blocked.')
@allowed([
'None'
'Low'
'Medium'
'High'
])
param contentSafetySeverity string = 'Medium'
@description('Client-facing alias that apps send in the request "model" field.')
param aliasName string = 'chat'
@description('Index in foundryModels of the backend model the alias points to.')
@minValue(0)
param aliasModelIndex int = 0
@description('Microsoft Entra principals that get the AI Gateway Models User role on the alias.')
param aliasUsers principal[] = []
var normalizedFoundryEndpoint = endsWith(foundryEndpoint, '/') ? foundryEndpoint : '${foundryEndpoint}/'
var aiGatewayModelsUserRoleDefinitionId = 'a6b68e3d-a1f0-4c60-8678-237c050cf4ea'
#disable-next-line BCP081
resource aiGateway 'Microsoft.ApiManagement/aigateways@2026-11-01' existing = {
name: gatewayName
}
#disable-next-line BCP081
resource defaultWorkspace 'Microsoft.ApiManagement/aigateways/workspaces@2026-11-01' existing = {
parent: aiGateway
name: 'default'
}
#disable-next-line BCP081
resource foundryProvider 'Microsoft.ApiManagement/aigateways/workspaces/modelProviders@2026-11-01' = {
parent: defaultWorkspace
name: providerName
properties: {
modelProviderType: 'Foundry'
displayName: providerDisplayName
description: empty(providerDescription) ? null : providerDescription
foundry: {
endpoint: {
url: normalizedFoundryEndpoint
}
resourceIds: foundryResourceIds
}
authentication: {
managedIdentity: {
resource: foundryManagedIdentityResource
}
}
}
}
#disable-next-line BCP081
resource foundryCatalogModels 'Microsoft.ApiManagement/aigateways/workspaces/modelProviders/models@2026-11-01' = [for model in foundryModels: {
parent: foundryProvider
name: model.armName
properties: {
description: model.?description ?? null
displayName: model.modelName
supportedEndpoints: model.supportedEndpoints
deployment: {
resourceId: model.resourceId
modelName: last(split(model.resourceId, '/'))
modelVersion: model.?modelVersion ?? null
}
policies: [
{
type: 'tokenLimit'
period: 'minute'
count: model.?tokenLimit ?? 10000
counterKey: [ 'identity' ]
}
{
type: 'contentSafety'
hateSeverity: contentSafetySeverity
violenceSeverity: contentSafetySeverity
sexualSeverity: contentSafetySeverity
selfHarmSeverity: contentSafetySeverity
}
]
}
}]
#disable-next-line BCP081
resource alias 'Microsoft.ApiManagement/aigateways/workspaces/aliases@2026-11-01' = {
parent: defaultWorkspace
name: aliasName
properties: {
backendModels: [
{ resourceId: foundryCatalogModels[aliasModelIndex].id }
]
}
}
resource aliasModelsUsers 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for user in aliasUsers: {
scope: alias
name: guid(alias.id, user.principalId, aiGatewayModelsUserRoleDefinitionId)
properties: {
principalId: user.principalId
principalType: user.principalType
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', aiGatewayModelsUserRoleDefinitionId)
}
}]
output providerName string = foundryProvider.name
output modelNames array = [for (model, i) in foundryModels: foundryCatalogModels[i].name]
output aliasName string = alias.name
output aliasId string = alias.idOn GitHub, open infra/modules/models.bicep.
The gateway floor is a plain policy file that goes through code review like any template:
[
{
"type": "requestRateLimit",
"count": 120,
"period": "minute",
"counterKey": ["identity"],
"scope": "workspace"
},
{
"type": "costLimit",
"amount": 20,
"period": "day",
"counterKey": ["identity"],
"scope": "workspace"
}
]On GitHub, open infra/baselines.json.
The other files are short. Open them on GitHub: monitoring.bicep, foundry.bicep, foundry-user-role.bicep, tools.bicep, keys.bicep, telemetry.bicep, types.bicep, main.bicepparam, and prod.bicepparam.
terraform/main.tf is the root module. It creates the two resource groups, then calls the modules in
terraform/modules/ and passes outputs from one to the next, so Terraform creates them in the same order as Bicep:
monitoring and Foundry first, then the gateway, then models, tools, keys, and telemetry. Like the Bicep modules,
the AI Gateway resources go through the azapi provider at API version 2026-11-01, and no output holds a key.
# Lab 21 with Terraform: the enterprise reference environment that infra/main.bicep deploys, as a root module that
# wires the modules in modules/ together. Pick the environment with a variable file:
# terraform plan -var-file dev.tfvars -out tfplan
# It creates two resource groups, rg-<environment_name>-gateway and rg-<environment_name>-foundrymodels, then the
# monitoring, the Foundry models, the gateway, the backend model and its alias, the MCP server, one runtime API key
# per app, and the OpenTelemetry exporter. The runtime API keys are never outputs.
data "azapi_client_config" "current" {}
locals {
suffix = substr(sha1("${data.azapi_client_config.current.subscription_id}/${var.environment_name}/${var.location}"), 0, 6)
ai_gateway_location = var.ai_gateway_location != "" ? var.ai_gateway_location : var.location
ai_gateway_name = var.ai_gateway_name != "" ? var.ai_gateway_name : "aigw-${local.suffix}"
tags = merge({
"environment-name" = var.environment_name
workload = "ai-gateway"
}, var.tags)
# The developer from principal_id, when set, gets data-plane access next to the Entra groups of the environment.
developer = var.principal_id == "" ? [] : [{ principal_id = var.principal_id, principal_type = "User" }]
}
resource "azurerm_resource_group" "gateway" {
name = "rg-${var.environment_name}-gateway"
location = var.location
tags = local.tags
}
resource "azurerm_resource_group" "foundry_models" {
name = "rg-${var.environment_name}-foundrymodels"
location = var.location
tags = local.tags
}
# Log Analytics and Application Insights, with OTLP ingestion on, in the gateway's resource group.
module "monitoring" {
source = "./modules/monitoring"
resource_group_name = azurerm_resource_group.gateway.name
resource_group_id = azurerm_resource_group.gateway.id
location = var.location
tags = local.tags
workspace_name = substr("${local.ai_gateway_name}-logs", 0, 63)
app_insights_name = substr("${local.ai_gateway_name}-insights", 0, 64)
retention_in_days = var.log_retention_in_days
}
# The Foundry account and its model deployment, in their own resource group.
module "foundry" {
source = "./modules/foundry"
resource_group_name = azurerm_resource_group.foundry_models.name
location = var.location
tags = local.tags
name = "aimodels-${local.suffix}"
principal_id = var.principal_id
model_deployments = {
(var.model_deployment_name) = {
model_name = var.model_name
model_format = var.model_format
model_version = var.model_version
sku_name = var.model_sku_name
capacity = var.model_capacity
}
}
}
# The gateway, with the Foundry User role on the account for its managed identity.
module "gateway" {
source = "./modules/gateway"
resource_group_id = azurerm_resource_group.gateway.id
location = local.ai_gateway_location
tags = local.tags
gateway_name = local.ai_gateway_name
foundry_resource_ids = [module.foundry.id]
}
# The Foundry provider, the backend model with its token limit and content safety, the alias, and who can call it.
module "models" {
source = "./modules/models"
gateway_id = module.gateway.id
provider_description = "Managed-identity Foundry provider for the ${var.environment_name} environment."
foundry_endpoint = module.foundry.endpoint
foundry_resource_ids = [module.foundry.id]
models = {
(var.model_deployment_name) = {
description = "Default chat model for every app behind the gateway."
resource_id = module.foundry.deployment_ids[var.model_deployment_name]
model_name = var.model_name
model_version = var.model_version
token_limit = var.tokens_per_minute_per_caller
supported_endpoints = [
"/openai/v1/chat/completions",
"/openai/v1/responses",
]
}
}
content_safety_severity = var.content_safety_severity
alias_name = var.alias_name
alias_model = var.model_deployment_name
alias_users = concat(local.developer, var.model_users)
depends_on = [module.gateway]
}
# The Microsoft Learn MCP server, and who can call it.
module "tools" {
source = "./modules/tools"
gateway_id = module.gateway.id
gateway_hostname = module.gateway.hostname
tool_users = concat(local.developer, var.tool_users)
}
# One runtime API key per app. Read a key with az aigateway api-key list-secrets.
module "keys" {
source = "./modules/keys"
gateway_id = module.gateway.id
app_names = var.app_names
}
# The OpenTelemetry exporter to Application Insights. The Bicep track deploys it with scripts/connect-telemetry.sh,
# because Bicep can't wait for the OTLP endpoints. Terraform waits for them, assigns the role, and creates the
# exporter in the same apply.
module "telemetry" {
source = "./modules/telemetry"
gateway_id = module.gateway.id
gateway_principal_id = module.gateway.principal_id
app_insights_id = module.monitoring.app_insights_id
}On GitHub, open terraform/main.tf.
The gateway module declares the gateway itself and the Foundry User role for its identity, with a 60-minute
create timeout, because activation can take longer than the azapi default of 30 minutes. The other modules only
add child resources to it.
The models module declares the token limit and content safety right on the backend model, like models.bicep:
# Models, like modules/models.bicep: a Foundry provider that signs in with the gateway's managed identity, a
# backend model per deployment with a per-caller token limit and content safety, an alias that apps send in the
# "model" field, and the AI Gateway Models User role on the alias for Microsoft Entra principals.
terraform {
required_providers {
azapi = {
source = "Azure/azapi"
}
azurerm = {
source = "hashicorp/azurerm"
}
}
}
locals {
workspace_id = "${var.gateway_id}/workspaces/default"
models_user_role_id = "a6b68e3d-a1f0-4c60-8678-237c050cf4ea"
normalized_foundry_endpoint = endswith(var.foundry_endpoint, "/") ? var.foundry_endpoint : "${var.foundry_endpoint}/"
}
resource "azapi_resource" "provider" {
type = "Microsoft.ApiManagement/aigateways/workspaces/modelProviders@2026-11-01"
name = var.provider_name
parent_id = local.workspace_id
body = {
properties = {
modelProviderType = "Foundry"
displayName = var.provider_display_name
description = var.provider_description
foundry = {
endpoint = {
url = local.normalized_foundry_endpoint
}
resourceIds = var.foundry_resource_ids
}
authentication = {
managedIdentity = {
resource = "https://cognitiveservices.azure.com/"
}
}
}
}
schema_validation_enabled = false
}
resource "azapi_resource" "model" {
for_each = var.models
type = "Microsoft.ApiManagement/aigateways/workspaces/modelProviders/models@2026-11-01"
name = each.key
parent_id = azapi_resource.provider.id
body = {
properties = {
description = each.value.description
displayName = each.value.model_name
supportedEndpoints = each.value.supported_endpoints
deployment = {
resourceId = each.value.resource_id
modelName = reverse(split("/", each.value.resource_id))[0]
modelVersion = each.value.model_version
}
policies = [
{
type = "tokenLimit"
period = "minute"
count = each.value.token_limit
counterKey = ["identity"]
},
{
type = "contentSafety"
hateSeverity = var.content_safety_severity
violenceSeverity = var.content_safety_severity
sexualSeverity = var.content_safety_severity
selfHarmSeverity = var.content_safety_severity
},
]
}
}
schema_validation_enabled = false
}
resource "azapi_resource" "alias" {
type = "Microsoft.ApiManagement/aigateways/workspaces/aliases@2026-11-01"
name = var.alias_name
parent_id = local.workspace_id
body = {
properties = {
backendModels = [
{
resourceId = azapi_resource.model[var.alias_model].id
},
]
}
}
schema_validation_enabled = false
}
resource "azurerm_role_assignment" "alias_users" {
for_each = { for user in var.alias_users : user.principal_id => user }
name = uuidv5("url", "${azapi_resource.alias.id}/${each.key}/${local.models_user_role_id}")
scope = azapi_resource.alias.id
role_definition_id = "/subscriptions/${split("/", var.gateway_id)[2]}/providers/Microsoft.Authorization/roleDefinitions/${local.models_user_role_id}"
principal_id = each.key
principal_type = each.value.principal_type
skip_service_principal_aad_check = each.value.principal_type == "ServicePrincipal"
}On GitHub, open terraform/modules/models/main.tf.
The telemetry module replaces connect-telemetry.sh. A terraform_data resource runs the script's wait with the
Azure CLI, a GET reads the endpoints and the Data Collection Rule once, then the module assigns
Monitoring Metrics Publisher and creates the exporter:
# Telemetry, like scripts/connect-telemetry.sh and modules/telemetry.bicep: reads the OTLP endpoints and the Data
# Collection Rule of the Application Insights component, grants the gateway's identity Monitoring Metrics Publisher
# on the rule, then creates the gateway's OpenTelemetry exporter with the recommended token claims.
terraform {
required_providers {
azapi = {
source = "Azure/azapi"
}
azurerm = {
source = "hashicorp/azurerm"
}
}
}
locals {
metrics_publisher_role_id = "3913510d-42f4-4e42-8a64-420c390055eb"
}
# Like connect-telemetry.sh: register Microsoft.Monitor, which the managed Azure Monitor workspace needs, then wait
# up to 10 minutes for the three OTLP endpoints and the Data Collection Rule. Terraform runs it once, with the Azure
# CLI, after it creates the component.
resource "terraform_data" "otlp_endpoints" {
triggers_replace = [var.app_insights_id]
provisioner "local-exec" {
command = "az provider register --namespace Microsoft.Monitor --wait && az resource wait --ids ${var.app_insights_id} --api-version 2020-02-02 --custom \"properties.OTLPMetricsEndpoint && properties.OTLPLogsEndpoint && properties.OTLPTracesEndpoint && properties.DataCollectionRuleResourceId\" --interval 10 --timeout 600"
}
}
# Then read the endpoints and the rule once, with a GET that keeps its result in the state. A data source would read
# them again each time the gateway changes, and replace the role assignment.
resource "azapi_resource_action" "otlp" {
type = "Microsoft.Insights/components@2020-02-02"
resource_id = var.app_insights_id
method = "GET"
response_export_values = {
metrics_endpoint = "properties.OTLPMetricsEndpoint"
logs_endpoint = "properties.OTLPLogsEndpoint"
traces_endpoint = "properties.OTLPTracesEndpoint"
dcr_id = "properties.DataCollectionRuleResourceId"
}
depends_on = [terraform_data.otlp_endpoints]
lifecycle {
replace_triggered_by = [terraform_data.otlp_endpoints]
postcondition {
condition = alltrue([
for field in ["metrics_endpoint", "logs_endpoint", "traces_endpoint", "dcr_id"] :
try(length(self.output[field]) > 0, false)
])
error_message = "Application Insights has no OTLP endpoints or Data Collection Rule yet. Wait a few minutes, then run terraform apply -replace=module.telemetry.azapi_resource_action.otlp."
}
}
}
# A direct role assignment works on the rule, though its managed resource group's deny assignment blocks deployments.
resource "azurerm_role_assignment" "metrics_publisher" {
name = uuidv5("url", "${azapi_resource_action.otlp.output.dcr_id}/${var.gateway_id}/${local.metrics_publisher_role_id}")
scope = azapi_resource_action.otlp.output.dcr_id
role_definition_id = "/subscriptions/${split("/", var.gateway_id)[2]}/providers/Microsoft.Authorization/roleDefinitions/${local.metrics_publisher_role_id}"
principal_id = var.gateway_principal_id
principal_type = "ServicePrincipal"
skip_service_principal_aad_check = true
}
resource "azapi_resource" "exporter" {
type = "Microsoft.ApiManagement/aigateways/workspaces/telemetryExporters@2026-11-01"
name = var.exporter_name
parent_id = "${var.gateway_id}/workspaces/default"
body = {
properties = {
telemetryExporterType = "OpenTelemetry"
payloadCapture = false
openTelemetry = {
metricsEndpoint = {
url = azapi_resource_action.otlp.output.metrics_endpoint
}
logsEndpoint = {
url = azapi_resource_action.otlp.output.logs_endpoint
}
tracesEndpoint = {
url = azapi_resource_action.otlp.output.traces_endpoint
}
}
authentication = {
managedIdentity = {
resource = "https://monitor.azure.com"
}
}
telemetry = {
attributes = var.claim_attributes
}
}
}
schema_validation_enabled = false
depends_on = [azurerm_role_assignment.metrics_publisher]
}On GitHub, open terraform/modules/telemetry/main.tf.
The baseline stays a script on this track too, infra/scripts/apply-baselines.sh with infra/baselines.json,
because a resource that declares the default workspace's policies would replace the ones that are already there.
The other files are short. Open them on GitHub: variables.tf, outputs.tf, dev.tfvars, prod.tfvars, monitoring, foundry, gateway, tools, and keys.
Deploy the dev environment
Deploy from your machine first, so you see each stage before you automate it.
Preview the changes
# Preview what the deployment would create or change, with the dev parameters, without changing anything
echo "📊 What deployment $ENV_NAME would change:"
az deployment sub what-if -n "$ENV_NAME" -l "$LOCATION" \
-f infra/main.bicep -p infra/main.bicepparam# Preview what the deployment would create or change, with the dev parameters, without changing anything
Write-Host "📊 What deployment $env:ENV_NAME would change:"
az deployment sub what-if -n $env:ENV_NAME -l $env:LOCATION `
-f infra/main.bicep -p infra/main.bicepparamOn a first run, the preview lists the two resource groups and the resources in them as new. On later runs, it shows only what your change would modify. Paste it into the pull request so reviewers see the effect, not just the diff.
Deploy
# Deploy the subscription-scope template with the dev parameters, and keep only its provisioning state
STATE="$(az deployment sub create -n "$ENV_NAME" -l "$LOCATION" \
-f infra/main.bicep -p infra/main.bicepparam \
--query properties.provisioningState -o tsv)" &&
echo "🚀 Deployment $ENV_NAME finished: $STATE"# Deploy the subscription-scope template with the dev parameters, and keep only its provisioning state
$state = az deployment sub create -n $env:ENV_NAME -l $env:LOCATION `
-f infra/main.bicep -p infra/main.bicepparam `
--query properties.provisioningState -o tsv
if ($?) { Write-Host "🚀 Deployment $env:ENV_NAME finished: $state" }Provisioning the gateway and the model deployment takes about three minutes. The command prints
🚀 Deployment aigw-dev finished: Succeeded when everything is in place.
Load the outputs
Read the deployment outputs into the variables the scripts and the rest of the lab use:
# Read the deployment's outputs as JSON
OUTPUTS="$(az deployment sub show -n "$ENV_NAME" --query properties.outputs -o json)"
# Define a helper that reads one output: output names can come back with a different first-letter casing,
# so it matches them case-insensitively
out() { jq -r --arg k "$1" 'to_entries[] | select((.key | ascii_upcase) == $k) | .value.value' <<<"$OUTPUTS"; }
# Read the gateway, the alias, and the Foundry account into the variables that the scripts and later steps use
export RG="$(out AI_GATEWAY_RESOURCE_GROUP)"
export GATEWAY="$(out AI_GATEWAY_NAME)"
export AI_GATEWAY_URL="$(out AI_GATEWAY_URL)"
export AI_GATEWAY_MODEL="$(out AI_GATEWAY_MODEL)"
export AI_GATEWAY_LOCATION="$(out AI_GATEWAY_LOCATION)"
export AI_GATEWAY_PRINCIPAL_ID="$(out AI_GATEWAY_PRINCIPAL_ID)"
export FOUNDRY_RG="$(out FOUNDRY_MODELS_RESOURCE_GROUP)"
export FOUNDRY_NAME="$(out FOUNDRY_MODELS_NAME)"
export FOUNDRY_ID="$(out FOUNDRY_MODELS_RESOURCE_ID)"
echo "📝 Gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
echo "🏭 Foundry account $FOUNDRY_NAME in $FOUNDRY_RG"# Read the deployment's outputs
$outputs = az deployment sub show -n $env:ENV_NAME --query properties.outputs -o json | ConvertFrom-Json
# Read the gateway, the alias, and the Foundry account into the variables that the scripts and later steps use
# (PowerShell matches the output names case-insensitively)
$env:RG = $outputs.AI_GATEWAY_RESOURCE_GROUP.value
$env:GATEWAY = $outputs.AI_GATEWAY_NAME.value
$env:AI_GATEWAY_URL = $outputs.AI_GATEWAY_URL.value
$env:AI_GATEWAY_MODEL = $outputs.AI_GATEWAY_MODEL.value
$env:AI_GATEWAY_LOCATION = $outputs.AI_GATEWAY_LOCATION.value
$env:AI_GATEWAY_PRINCIPAL_ID = $outputs.AI_GATEWAY_PRINCIPAL_ID.value
$env:FOUNDRY_RG = $outputs.FOUNDRY_MODELS_RESOURCE_GROUP.value
$env:FOUNDRY_NAME = $outputs.FOUNDRY_MODELS_NAME.value
$env:FOUNDRY_ID = $outputs.FOUNDRY_MODELS_RESOURCE_ID.value
Write-Host "📝 Gateway $env:GATEWAY in $env:RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
Write-Host "🏭 Foundry account $env:FOUNDRY_NAME in $env:FOUNDRY_RG"Expected output:
📝 Gateway aigw-<token> in rg-aigw-dev-<token>-gateway
🌐 Runtime URL: https://aigw-abcde-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com
🔖 Model alias: chat
🏭 Foundry account aimodels-<token> in rg-aigw-dev-<token>-foundrymodelsConnect telemetry
Run the telemetry script. It waits until Azure has created the component's OTLP endpoints, grants the gateway's
identity Monitoring Metrics Publisher on the Data Collection Rule, then deploys modules/telemetry.bicep. Right
after the first deployment, the wait can take a few minutes.
# Wait for the OTLP endpoints, let the gateway's identity publish to them, then deploy the OpenTelemetry exporter
./infra/scripts/connect-telemetry.sh &&
echo "📊 Telemetry is connected for $GATEWAY"# Wait for the OTLP endpoints, let the gateway's identity publish to them, then deploy the OpenTelemetry exporter
./infra/scripts/connect-telemetry.ps1
if ($?) { Write-Host "📊 Telemetry is connected for $env:GATEWAY" }On GitHub, open infra/scripts/connect-telemetry.sh or infra/scripts/connect-telemetry.ps1.
Expected output:
⏳ Waiting for the OTLP endpoints of aigw-<token>-insights
🆔 Data Collection Rule: managed-aigw-<token>-insights-dcr
🔐 The gateway can publish to the Data Collection Rule
📊 The gateway sends metrics, logs, and traces to aigw-<token>-insights
📊 Telemetry is connected for aigw-<token>The role assignment needs the same rights as the deployment. Run the script after each deployment: running it again gives the same result.
What you need: Terraform 1.9 or later, as in Deploy a lab with Terraform,
and the variables from Before you begin. The configuration creates its own resource groups,
rg-aigw-dev-tf-gateway and rg-aigw-dev-tf-foundrymodels, so it doesn't touch a Bicep deployment.
Preview the changes
Move into the lab's Terraform folder, pass your region and your object ID to the configuration, then plan the dev
environment from dev.tfvars:
# Move into the lab's Terraform folder
cd ~/aigateway/labs/21-enterprise-reference/terraform &&
echo "📂 Terraform folder: $(pwd)"
# Deploy to the Azure CLI's current subscription, in the region from Before you begin, and give yourself
# data-plane access, like main.bicepparam does
export ARM_SUBSCRIPTION_ID="$(az account show --query id --output tsv)"
export TF_VAR_location="$LOCATION"
export TF_VAR_principal_id="$AZURE_PRINCIPAL_ID"
echo "📍 Subscription $ARM_SUBSCRIPTION_ID, region $TF_VAR_location"
# Download the providers, then plan the dev environment and save the plan, without changing anything
terraform init &&
terraform plan -var-file dev.tfvars -out tfplan &&
echo "📊 The plan for aigw-dev-tf is saved in tfplan"# Move into the lab's Terraform folder
Set-Location ~/aigateway/labs/21-enterprise-reference/terraform
if ($?) { Write-Host "📂 Terraform folder: $(Get-Location)" }
# Deploy to the Azure CLI's current subscription, in the region from Before you begin, and give yourself
# data-plane access, like main.bicepparam does
$env:ARM_SUBSCRIPTION_ID = az account show --query id --output tsv
$env:TF_VAR_location = $env:LOCATION
$env:TF_VAR_principal_id = $env:AZURE_PRINCIPAL_ID
Write-Host "📍 Subscription $env:ARM_SUBSCRIPTION_ID, region $env:TF_VAR_location"
# Download the providers, then plan the dev environment and save the plan, without changing anything
terraform init
if ($?) { terraform plan -var-file dev.tfvars -out tfplan }
if ($?) { Write-Host "📊 The plan for aigw-dev-tf is saved in tfplan" }On a first run, the plan ends with Plan: 23 to add, 0 to change, 0 to destroy. On later runs, like what-if, it
shows only what your change would modify. Paste it into the pull request so reviewers see the effect. The gateway
gets the same region as the other resources. To put it in another region, also set TF_VAR_ai_gateway_location.
Deploy with terraform apply
# Apply the saved plan: the resource groups, monitoring, Foundry, the gateway, models, tools, keys, and telemetry
terraform apply tfplan &&
echo "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)"# Apply the saved plan: the resource groups, monitoring, Foundry, the gateway, models, tools, keys, and telemetry
terraform apply tfplan
if ($?) { Write-Host "🚀 Gateway deployed: $(terraform output -raw ai_gateway_url)" }It usually takes a few minutes, most of it while the gateway activates and Terraform creates the role
assignments. Activation can take more than half an hour, and the gateway module waits up to an hour for it. The
telemetry module waits up to 10 minutes for the OTLP endpoints, but they're usually ready by the time the
gateway is. The apply ends with Apply complete! Resources: 23 added, 0 changed, 0 destroyed.
Load the outputs
Read the Terraform outputs into the variables the scripts and the rest of the lab use, then go back to the lab folder:
# Read the gateway, the alias, and the Foundry account into the variables that the scripts and later steps use
export RG="$(terraform output -raw ai_gateway_resource_group)"
export GATEWAY="$(terraform output -raw ai_gateway_name)"
export AI_GATEWAY_URL="$(terraform output -raw ai_gateway_url)"
export AI_GATEWAY_MODEL="$(terraform output -raw ai_gateway_model)"
export AI_GATEWAY_LOCATION="$(terraform output -raw ai_gateway_location)"
export AI_GATEWAY_PRINCIPAL_ID="$(terraform output -raw ai_gateway_principal_id)"
export FOUNDRY_RG="$(terraform output -raw foundry_models_resource_group)"
export FOUNDRY_NAME="$(terraform output -raw foundry_models_name)"
export FOUNDRY_ID="$(terraform output -raw foundry_models_resource_id)"
echo "📝 Gateway $GATEWAY in $RG"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"
echo "🔖 Model alias: $AI_GATEWAY_MODEL"
echo "🏭 Foundry account $FOUNDRY_NAME in $FOUNDRY_RG"
# Go back to the lab folder, where the scripts and the later steps run
cd .. && echo "📂 Lab folder: $(pwd)"# Read the gateway, the alias, and the Foundry account into the variables that the scripts and later steps use
$env:RG = terraform output -raw ai_gateway_resource_group
$env:GATEWAY = terraform output -raw ai_gateway_name
$env:AI_GATEWAY_URL = terraform output -raw ai_gateway_url
$env:AI_GATEWAY_MODEL = terraform output -raw ai_gateway_model
$env:AI_GATEWAY_LOCATION = terraform output -raw ai_gateway_location
$env:AI_GATEWAY_PRINCIPAL_ID = terraform output -raw ai_gateway_principal_id
$env:FOUNDRY_RG = terraform output -raw foundry_models_resource_group
$env:FOUNDRY_NAME = terraform output -raw foundry_models_name
$env:FOUNDRY_ID = terraform output -raw foundry_models_resource_id
Write-Host "📝 Gateway $env:GATEWAY in $env:RG"
Write-Host "🌐 Runtime URL: $env:AI_GATEWAY_URL"
Write-Host "🔖 Model alias: $env:AI_GATEWAY_MODEL"
Write-Host "🏭 Foundry account $env:FOUNDRY_NAME in $env:FOUNDRY_RG"
# Go back to the lab folder, where the scripts and the later steps run
Set-Location ..
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }Expected output:
📝 Gateway aigw-<token> in rg-aigw-dev-tf-gateway
🌐 Runtime URL: https://aigw-abcde-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com
🔖 Model alias: chat
🏭 Foundry account aimodels-<token> in rg-aigw-dev-tf-foundrymodels
📂 Lab folder: /home/<user>/aigateway/labs/21-enterprise-referenceTelemetry is already connected: the apply created the exporter, appinsights, after the role assignment. To check
for drift later, run terraform plan -var-file dev.tfvars from terraform/ again. It prints
No changes. Your infrastructure matches the configuration.
On both tracks, apply the gateway baseline from the lab folder with the same script.
Preview the gateway baseline
Print the merged policy array without changing anything:
# Print the merged policy array without changing anything: with DRY_RUN=1, the script stops before it writes
DRY_RUN=1 ./infra/scripts/apply-baselines.sh &&
echo "🔐 Baseline preview finished: the policies on $GATEWAY are unchanged"# Print the merged policy array without changing anything: with -DryRun, the script stops before it writes
./infra/scripts/apply-baselines.ps1 -DryRun
if ($?) { Write-Host "🔐 Baseline preview finished: the policies on $env:GATEWAY are unchanged" }The output shows a requestRateLimit entry and a costLimit entry with a newly generated id, plus any
Gateway scope policies the gateway already has.
Apply the gateway baseline
# Merge baselines.json into the gateway's Gateway scope policies, write them back, and list the policy types
./infra/scripts/apply-baselines.sh &&
echo "🔐 Gateway baseline is applied to $GATEWAY"# Merge baselines.json into the gateway's Gateway scope policies, write them back, and list the policy types
./infra/scripts/apply-baselines.ps1
if ($?) { Write-Host "🔐 Gateway baseline is applied to $env:GATEWAY" }#!/usr/bin/env bash
# Applies the gateway-wide baseline policies in infra/baselines.json to an AI Gateway.
#
# Gateway-scope policies live in the policies array of the gateway's default workspace. A PATCH replaces the whole
# array, so the script reads the current array, merges the baselines into it, and writes it back:
# - A baseline whose type already exists updates that policy in place and keeps its id, overrides, and any other
# fields you set in the portal. Updating a costLimit this way keeps its accumulated spend.
# - A baseline whose type doesn't exist is appended. A new costLimit gets a fresh, stable id.
# - Policies of other types (for example ones you added in the portal) are left untouched.
# Running it twice gives the same result. The az aigateway policy commands take each setting as a flag, not a
# policy array, so to write the array from a file, the script calls the management API with az rest.
#
# Usage:
# export RG="<gateway-resource-group>" GATEWAY="<gateway-name>"
# ./apply-baselines.sh [baselines.json]
# DRY_RUN=1 ./apply-baselines.sh # print the merged array without changing anything
#
# Requires Azure CLI and jq.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BASELINES="${1:-$SCRIPT_DIR/../baselines.json}"
API_VERSION="2026-11-01"
: "${RG:?Set RG to the gateway resource group}"
: "${GATEWAY:?Set GATEWAY to the gateway name}"
command -v jq >/dev/null || { echo "This script needs jq." >&2; exit 1; }
jq -e 'type == "array" and all(.[]; has("type"))' "$BASELINES" >/dev/null \
|| { echo "$BASELINES must be a JSON array of policy objects." >&2; exit 1; }
SUB="$(az account show --query id -o tsv)"
URL="https://management.azure.com/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.ApiManagement/aigateways/$GATEWAY/workspaces/default?api-version=$API_VERSION"
NEW_ID="$( (uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid) | tr 'A-Z' 'a-z')"
CURRENT="$(az rest --method get --url "$URL")"
POLICIES="$(printf '%s' "$CURRENT" | jq -c --slurpfile base "$BASELINES" --arg id "$NEW_ID" '
reduce $base[0][] as $b (.properties.policies // [];
if any(.[]; .type == $b.type)
then map(if .type == $b.type then . + $b else . end)
else . + [if $b.type == "costLimit" and ($b.id // "") == "" then $b + {id: $id} else $b end]
end)')"
echo "Gateway-scope policies after merge:"
printf '%s' "$POLICIES" | jq .
if [ "${DRY_RUN:-0}" = "1" ]; then
echo "DRY_RUN=1: nothing was changed."
exit 0
fi
az rest --method patch --url "$URL" --body "{\"properties\":{\"policies\":$POLICIES}}" --output none
echo "Applied. Current gateway-scope policy types:"
az rest --method get --url "$URL" --query "properties.policies[].type" -o tsvOn GitHub, open infra/scripts/apply-baselines.sh or infra/scripts/apply-baselines.ps1.
Expected output (end):
Applied. Current gateway-scope policy types:
requestRateLimit
costLimit
🔐 Gateway baseline is applied to aigw-<token>Run it again whenever baselines.json changes. It's idempotent: a second run gives the same array.
On the Bicep track, the deployment printed 🚀 Deployment aigw-dev finished: Succeeded and the telemetry script
printed 📊 Telemetry is connected. On the Terraform track, the apply printed
Apply complete! Resources: 23 added, 0 changed, 0 destroyed. On both, RG and GATEWAY point at the new
gateway, Monitoring shows the Overview, Models, Tools, Policies, and Traces tabs, and the
baseline script lists requestRateLimit and costLimit.
Automate deployments with GitHub Actions
Copy this lab's infra/ folder to the root of your own repository, so the workflow finds infra/main.bicep. On
the Terraform track, also copy the terraform/ folder. The workflow still runs the scripts and reads
baselines.json from infra/.
Then create an identity that GitHub Actions can sign in as, trust your repository's environments, and add the
workflow. Learn more in
Authenticate to Azure from GitHub Actions by OpenID Connect.
Set the repository once:
# Set the GitHub repository that holds the workflow, in the form org/repo
export GITHUB_REPO="<github-org>/<github-repo>"
echo "📝 GitHub repository: $GITHUB_REPO"# Set the GitHub repository that holds the workflow, in the form org/repo
$env:GITHUB_REPO = "<github-org>/<github-repo>"
Write-Host "📝 GitHub repository: $env:GITHUB_REPO"Create the deployment identity
Create an app registration and its service principal:
# Create the app registration that GitHub Actions signs in as, and keep its application (client) ID
export APP_ID="$(az ad app create --display-name "github-aigw-deploy" --query appId -o tsv)"
echo "👤 App registration github-aigw-deploy: $APP_ID"
# Create its service principal, the identity that gets the role assignments on your subscription
export SP_ID="$(az ad sp create --id "$APP_ID" --query id -o tsv)"
echo "👤 Service principal: $SP_ID"# Create the app registration that GitHub Actions signs in as, and keep its application (client) ID
$env:APP_ID = az ad app create --display-name "github-aigw-deploy" --query appId -o tsv
if ($?) { Write-Host "👤 App registration github-aigw-deploy: $env:APP_ID" }
# Create its service principal, the identity that gets the role assignments on your subscription
$env:SP_ID = az ad sp create --id $env:APP_ID --query id -o tsv
if ($?) { Write-Host "👤 Service principal: $env:SP_ID" }Trust your GitHub environments
Add one federated credential per GitHub environment. The workflow's job declares environment:, so GitHub's
token subject is the repository's subject prefix followed by :environment:<name>, and it must match exactly.
The prefix is repo:<github-org>/<github-repo> for older repositories. Repositories created after July 15, 2026
use immutable subject claims,
which add the owner and repository IDs, as in repo:<github-org>@<org-id>/<github-repo>@<repo-id>. GitHub
returns the prefix that your repository uses, so read it instead of typing it.
# Read the subject prefix of your repository's OIDC tokens, with the immutable IDs if the repository uses them
export SUBJECT_PREFIX="$(gh api "repos/$GITHUB_REPO/actions/oidc/customization/sub" --jq .sub_claim_prefix)"
echo "🔖 Token subject prefix: $SUBJECT_PREFIX"
# For each GitHub environment, add a federated credential that trusts GitHub's tokens for that environment only
for ENV in dev prod; do
# Write the credential to a file: its subject must match the workflow's environment exactly
cat > "credential-$ENV.json" <<EOF
{
"name": "github-$ENV",
"issuer": "https://token.actions.githubusercontent.com",
"subject": "$SUBJECT_PREFIX:environment:$ENV",
"description": "GitHub Actions deployments to the $ENV environment",
"audiences": [ "api://AzureADTokenExchange" ]
}
EOF
# Add the credential to the app registration, then delete the file
az ad app federated-credential create --id "$APP_ID" --parameters "credential-$ENV.json" -o none &&
echo "👤 Federated credential github-$ENV trusts the $ENV environment of $GITHUB_REPO"
rm "credential-$ENV.json"
done# Read the subject prefix of your repository's OIDC tokens, with the immutable IDs if the repository uses them
$env:SUBJECT_PREFIX = gh api "repos/$($env:GITHUB_REPO)/actions/oidc/customization/sub" --jq .sub_claim_prefix
Write-Host "🔖 Token subject prefix: $env:SUBJECT_PREFIX"
# For each GitHub environment, add a federated credential that trusts GitHub's tokens for that environment only
foreach ($envName in "dev", "prod") {
# Write the credential to a file: its subject must match the workflow's environment exactly
$file = "credential-$envName.json"
@{
name = "github-$envName"
issuer = "https://token.actions.githubusercontent.com"
subject = "$($env:SUBJECT_PREFIX):environment:$envName"
description = "GitHub Actions deployments to the $envName environment"
audiences = @("api://AzureADTokenExchange")
} | ConvertTo-Json | Set-Content -Path $file
# Add the credential to the app registration, then delete the file
az ad app federated-credential create --id $env:APP_ID --parameters $file -o none
if ($?) { Write-Host "👤 Federated credential github-$envName trusts the $envName environment of $env:GITHUB_REPO" }
Remove-Item $file
}Grant access to the subscription
# Read your subscription ID, the scope of the role assignments
export SUBSCRIPTION_ID="$(az account show --query id -o tsv)"
echo "🆔 Subscription: $SUBSCRIPTION_ID"
# Give the service principal Contributor, to deploy the resources, and Role Based Access Control Administrator,
# because the template creates role assignments
for ROLE in "Contributor" "Role Based Access Control Administrator"; do
az role assignment create --assignee-object-id "$SP_ID" --assignee-principal-type ServicePrincipal \
--role "$ROLE" --scope "/subscriptions/$SUBSCRIPTION_ID" -o none &&
echo "🔐 The deployment identity has the $ROLE role on the subscription"
done# Read your subscription ID, the scope of the role assignments
$env:SUBSCRIPTION_ID = az account show --query id -o tsv
Write-Host "🆔 Subscription: $env:SUBSCRIPTION_ID"
# Give the service principal Contributor, to deploy the resources, and Role Based Access Control Administrator,
# because the template creates role assignments
foreach ($role in "Contributor", "Role Based Access Control Administrator") {
az role assignment create --assignee-object-id $env:SP_ID --assignee-principal-type ServicePrincipal `
--role $role --scope "/subscriptions/$($env:SUBSCRIPTION_ID)" -o none
if ($?) { Write-Host "🔐 The deployment identity has the $role role on the subscription" }
}Create the GitHub environments
In your repository on GitHub, select SettingsEnvironmentsNew environment, and create dev
and prod. On prod, turn on Required reviewers and add the people who approve production deployments.
On the GitHub Free, Pro, and Team plans, required reviewers are
only available for public repositories.
Then store the sign-in details as environment variables. They identify the identity but can't authenticate anyone on their own, so they're variables, not secrets.
# Read your tenant ID, the Microsoft Entra tenant that the workflow signs in to
TENANT_ID="$(az account show --query tenantId -o tsv)"
echo "🆔 Tenant: $TENANT_ID"
# Store the app's client ID, the tenant ID, and the subscription ID as variables of each GitHub environment
for ENV in dev prod; do
gh variable set AZURE_CLIENT_ID --env "$ENV" --body "$APP_ID" -R "$GITHUB_REPO"
gh variable set AZURE_TENANT_ID --env "$ENV" --body "$TENANT_ID" -R "$GITHUB_REPO"
gh variable set AZURE_SUBSCRIPTION_ID --env "$ENV" --body "$SUBSCRIPTION_ID" -R "$GITHUB_REPO" &&
echo "📝 The $ENV environment has AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_SUBSCRIPTION_ID"
done# Read your tenant ID, the Microsoft Entra tenant that the workflow signs in to
$tenantId = az account show --query tenantId -o tsv
Write-Host "🆔 Tenant: $tenantId"
# Store the app's client ID, the tenant ID, and the subscription ID as variables of each GitHub environment
foreach ($envName in "dev", "prod") {
gh variable set AZURE_CLIENT_ID --env $envName --body $env:APP_ID -R $env:GITHUB_REPO
gh variable set AZURE_TENANT_ID --env $envName --body $tenantId -R $env:GITHUB_REPO
gh variable set AZURE_SUBSCRIPTION_ID --env $envName --body $env:SUBSCRIPTION_ID -R $env:GITHUB_REPO
if ($?) { Write-Host "📝 The $envName environment has AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_SUBSCRIPTION_ID" }
}Add the workflow
In your repository, copy the sample workflow into .github/workflows/, then commit and push both folders:
# Create the workflows folder, and copy the sample workflow into it
mkdir -p .github/workflows
cp infra/deploy-workflow.yml .github/workflows/deploy-ai-gateway.yml &&
echo "📄 Workflow file: .github/workflows/deploy-ai-gateway.yml"# Sample GitHub Actions workflow for the enterprise reference deployment.
#
# Copy this file to .github/workflows/deploy-ai-gateway.yml in your own repository, with the lab's infra folder at the
# repository root (infra/main.bicep, infra/main.bicepparam, infra/prod.bicepparam, infra/scripts/...).
#
# It signs in to Azure with OpenID Connect (no client secret is stored in GitHub), installs the aigateway CLI extension
# that the scripts use, previews the change with what-if, deploys the subscription-scope template, connects the
# gateway's telemetry to Application Insights, applies the gateway-wide baseline policies, and runs the smoke test.
#
# One-time setup (see the lab for the commands):
# - A Microsoft Entra app registration (or user-assigned managed identity) with a federated credential for each
# GitHub environment: issuer https://token.actions.githubusercontent.com, audience api://AzureADTokenExchange,
# subject <prefix>:environment:<dev|prod>. Read the prefix with
# gh api repos/<org>/<repo>/actions/oidc/customization/sub --jq .sub_claim_prefix: repositories created after
# July 15, 2026 add immutable IDs, as in repo:<org>@<org-id>/<repo>@<repo-id>.
# - That identity needs Contributor and Role Based Access Control Administrator on the target subscription,
# because the template creates resource groups and role assignments.
# - GitHub environments named dev and prod (add required reviewers to prod), each with the variables
# AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_SUBSCRIPTION_ID.
name: Deploy AI Gateway
on:
workflow_dispatch:
inputs:
environment:
description: Target environment
type: choice
options:
- dev
- prod
default: dev
push:
branches:
- main
paths:
- infra/**
permissions:
id-token: write
contents: read
concurrency:
group: deploy-ai-gateway-${{ inputs.environment || 'dev' }}
cancel-in-progress: false
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment || 'dev' }}
env:
DEPLOYMENT_LOCATION: eastus2
PARAMETERS_FILE: ${{ (inputs.environment || 'dev') == 'prod' && 'infra/prod.bicepparam' || 'infra/main.bicepparam' }}
steps:
- name: Check out the repository
uses: actions/checkout@v4
- name: Sign in to Azure with OpenID Connect
uses: azure/login@v2
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- name: Install the aigateway CLI extension
run: curl -fsSL https://aka.ms/aigateway-cli-install | sh
- name: Preview changes
run: |
az deployment sub what-if \
--name "aigw-${{ github.run_id }}" \
--location "$DEPLOYMENT_LOCATION" \
--template-file infra/main.bicep \
--parameters "$PARAMETERS_FILE"
- name: Deploy
id: deploy
run: |
OUTPUTS="$(az deployment sub create \
--name "aigw-${{ github.run_id }}" \
--location "$DEPLOYMENT_LOCATION" \
--template-file infra/main.bicep \
--parameters "$PARAMETERS_FILE" \
--query properties.outputs --output json)"
# Deployment output names can come back with a different first-letter casing, so match case-insensitively.
out() { jq -r --arg k "$1" 'to_entries[] | select((.key | ascii_upcase) == $k) | .value.value' <<<"$OUTPUTS"; }
{
echo "resource-group=$(out AI_GATEWAY_RESOURCE_GROUP)"
echo "gateway=$(out AI_GATEWAY_NAME)"
echo "gateway-url=$(out AI_GATEWAY_URL)"
echo "alias=$(out AI_GATEWAY_MODEL)"
} >> "$GITHUB_OUTPUT"
- name: Connect telemetry
env:
RG: ${{ steps.deploy.outputs.resource-group }}
GATEWAY: ${{ steps.deploy.outputs.gateway }}
run: bash infra/scripts/connect-telemetry.sh
- name: Apply gateway baseline policies
env:
RG: ${{ steps.deploy.outputs.resource-group }}
GATEWAY: ${{ steps.deploy.outputs.gateway }}
run: bash infra/scripts/apply-baselines.sh infra/baselines.json
- name: Smoke test
env:
RG: ${{ steps.deploy.outputs.resource-group }}
GATEWAY: ${{ steps.deploy.outputs.gateway }}
run: bash infra/scripts/smoke-test.sh "${{ steps.deploy.outputs.alias }}" web-chat learn
- name: Summary
run: |
{
echo "### AI Gateway deployed"
echo "- Environment: ${{ inputs.environment || 'dev' }}"
echo "- Gateway: ${{ steps.deploy.outputs.gateway }}"
echo "- Endpoint: ${{ steps.deploy.outputs.gateway-url }}"
echo "- Alias: ${{ steps.deploy.outputs.alias }}"
} >> "$GITHUB_STEP_SUMMARY"On GitHub, open infra/deploy-workflow.yml.
A push to main that changes infra/ deploys dev. Prod deploys only when you start the workflow manually and a
reviewer approves it.
Store the state in Azure
The pipeline runs on a new machine each time, so it can't read the terraform.tfstate file on yours. Keep the
state in an Azure Storage blob container instead, where you and the pipeline share it:
- Create a storage account and a blob container named
tfstate, as in Store Terraform state in Azure Storage. Your machine and GitHub's runners must both reach it, so check your organization's network policies for storage accounts first. - Give the deployment identity,
$SP_ID, and yourself the Storage Blob Data Contributor role on the container, so that both sign in to it with Microsoft Entra ID instead of an account key.
Then store the account's name and your object ID in GitHub, declare the backend, and move your local state into
the container. Run this in the lab's terraform/ folder, where your local state is:
# Store the state's storage account in both GitHub environments. Store your object ID in dev only: the pipeline
# passes it to the configuration, so that dev keeps your developer roles
export TFSTATE_ACCOUNT="<storage-account-name>"
for ENV in dev prod; do
gh variable set TFSTATE_STORAGE_ACCOUNT --env "$ENV" --body "$TFSTATE_ACCOUNT" -R "$GITHUB_REPO"
done
gh variable set DEVELOPER_PRINCIPAL_ID --env dev --body "$AZURE_PRINCIPAL_ID" -R "$GITHUB_REPO" &&
echo "📝 GitHub has TFSTATE_STORAGE_ACCOUNT in dev and prod, and DEVELOPER_PRINCIPAL_ID in dev"
# Declare the azurerm backend with Microsoft Entra sign-in. The account, the container, and the state's name come
# from -backend-config, so the same file works for dev and prod
cd ~/aigateway/labs/21-enterprise-reference/terraform &&
cat > backend.tf <<'EOF'
terraform {
backend "azurerm" {
use_azuread_auth = true
}
}
EOF
# Copy the local state into the container as dev.tfstate: answer yes when Terraform asks
terraform init -migrate-state \
-backend-config="storage_account_name=$TFSTATE_ACCOUNT" \
-backend-config="container_name=tfstate" \
-backend-config="key=dev.tfstate" &&
echo "🔒 The dev state is in the tfstate container of $TFSTATE_ACCOUNT"
# Move back to the lab's folder, where Verify runs the scripts
cd .. && echo "📂 Lab folder: $(pwd)"# Store the state's storage account in both GitHub environments. Store your object ID in dev only: the pipeline
# passes it to the configuration, so that dev keeps your developer roles
$env:TFSTATE_ACCOUNT = "<storage-account-name>"
foreach ($envName in "dev", "prod") {
gh variable set TFSTATE_STORAGE_ACCOUNT --env $envName --body $env:TFSTATE_ACCOUNT -R $env:GITHUB_REPO
}
gh variable set DEVELOPER_PRINCIPAL_ID --env dev --body $env:AZURE_PRINCIPAL_ID -R $env:GITHUB_REPO
if ($?) { Write-Host "📝 GitHub has TFSTATE_STORAGE_ACCOUNT in dev and prod, and DEVELOPER_PRINCIPAL_ID in dev" }
# Declare the azurerm backend with Microsoft Entra sign-in. The account, the container, and the state's name come
# from -backend-config, so the same file works for dev and prod
Set-Location ~/aigateway/labs/21-enterprise-reference/terraform
@'
terraform {
backend "azurerm" {
use_azuread_auth = true
}
}
'@ | Set-Content -Path backend.tf
# Copy the local state into the container as dev.tfstate: answer yes when Terraform asks
terraform init -migrate-state `
"-backend-config=storage_account_name=$($env:TFSTATE_ACCOUNT)" `
"-backend-config=container_name=tfstate" `
"-backend-config=key=dev.tfstate"
if ($?) { Write-Host "🔒 The dev state is in the tfstate container of $env:TFSTATE_ACCOUNT" }
# Move back to the lab's folder, where Verify runs the scripts
Set-Location ..
if ($?) { Write-Host "📂 Lab folder: $(Get-Location)" }Copy backend.tf into the terraform/ folder of your repository too. Don't commit the .terraform/ folder, the
terraform.tfstate files, or tfplan: they're your local working files, so add them to your .gitignore. Move the state before the pipeline's
first run. Otherwise, its terraform apply starts from an empty state and fails because the resources already
exist.
Add the workflow
In your repository, copy the sample workflow into .github/workflows/, as on the Bicep track:
# Create the workflows folder, and copy the sample workflow into it
mkdir -p .github/workflows
cp infra/deploy-workflow.yml .github/workflows/deploy-ai-gateway.yml &&
echo "📄 Workflow file: .github/workflows/deploy-ai-gateway.yml"Then make it run Terraform. The sign-in, the extension, the baseline, the smoke test, and the summary don't change, because the scripts still use the Azure CLI:
- Under
on.push.paths, addterraform/**, so that a change to the configuration deploys dev. - Replace the job's
envwith the one below.ARM_USE_OIDCmakes both providers and the backend sign in with GitHub's OpenID Connect token, likeazure/logindoes. SetTF_VAR_locationto the region you deployed from your machine: the token in the resource names depends on it. - Replace the Preview changes and Deploy steps with the Terraform steps below. Deploy keeps its
idand its four outputs, so the later steps read them as before. - Delete the Connect telemetry step: the
telemetrymodule connects it interraform apply.
env:
ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }}
ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }}
ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }}
ARM_USE_OIDC: true
TF_VAR_location: eastus2
TF_VAR_principal_id: ${{ vars.DEVELOPER_PRINCIPAL_ID }}
VAR_FILE: ${{ (inputs.environment || 'dev') == 'prod' && 'prod.tfvars' || 'dev.tfvars' }}
STATE_KEY: ${{ inputs.environment || 'dev' }}.tfstate
steps:
# Keep the checkout, sign-in, and extension steps here
- name: Install Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Connect to the state
working-directory: terraform
run: |
terraform init -input=false \
-backend-config="storage_account_name=${{ vars.TFSTATE_STORAGE_ACCOUNT }}" \
-backend-config="container_name=tfstate" \
-backend-config="key=$STATE_KEY"
- name: Preview changes
working-directory: terraform
run: terraform plan -input=false -var-file "$VAR_FILE" -out tfplan
- name: Deploy
id: deploy
working-directory: terraform
run: |
terraform apply -input=false tfplan
{
echo "resource-group=$(terraform output -raw ai_gateway_resource_group)"
echo "gateway=$(terraform output -raw ai_gateway_name)"
echo "gateway-url=$(terraform output -raw ai_gateway_url)"
echo "alias=$(terraform output -raw ai_gateway_model)"
} >> "$GITHUB_OUTPUT"
# Keep the baseline, smoke test, and summary steps hereUnlike an incremental Bicep deployment, terraform apply deletes what the configuration no longer declares. That's
why the job passes DEVELOPER_PRINCIPAL_ID as TF_VAR_principal_id: without it, the dev run would remove your
developer roles. prod.tfvars sets principal_id = "", and a value in a -var-file overrides an environment
variable, so prod never gets developer roles. The Deploy step applies the plan that Preview changes saved,
so it changes exactly what the log of the preview shows.
Commit and push infra/, terraform/, and the workflow. A push to main that changes infra/ or terraform/
deploys dev. Prod deploys only when you start the workflow manually and a reviewer approves it.
Run it
# Start the workflow for the dev environment
gh workflow run deploy-ai-gateway.yml -f environment=dev -R "$GITHUB_REPO" &&
echo "⏳ The Deploy AI Gateway workflow is queued for dev"
# Watch the run's jobs and steps until it finishes
echo "⏳ Watching the run (Ctrl+C stops watching, not the run):"
gh run watch -R "$GITHUB_REPO"Or, in GitHub, select ActionsDeploy AI GatewayRun workflow. The pipeline deploys to the same
resource groups you created from your machine, because the resource token depends only on the subscription,
the environment name, and the location. On the Bicep track, the pipeline doesn't set AZURE_PRINCIPAL_ID, so it
doesn't declare your developer roles, and incremental deployments leave the ones you already have in place. On the
Terraform track, the pipeline reads and updates the state you moved to the container, so its plan starts from
what you deployed.
The run's Smoke test step prints Smoke test passed., and the run summary lists the gateway, its
endpoint, and the alias chat.
Verify
Prove that both runtime paths work, that the policy layers are in place, and that the identity boundaries hold.
Run the smoke test
The same script runs at the end of the pipeline. It calls the alias with the web-chat key, then runs the MCP
handshake and tools/list on the learn MCP server. New role assignments can take a few minutes to apply, so
it retries the model call for about a minute.
# Call the alias with the web-chat key, then run the MCP handshake and tools/list on the learn MCP server
echo "📊 Smoke test of the $AI_GATEWAY_MODEL alias and the learn MCP server:"
./infra/scripts/smoke-test.sh "$AI_GATEWAY_MODEL" web-chat learn# Call the alias with the web-chat key, then run the MCP handshake and tools/list on the learn MCP server
Write-Host "📊 Smoke test of the $env:AI_GATEWAY_MODEL alias and the learn MCP server:"
./infra/scripts/smoke-test.ps1 -Alias $env:AI_GATEWAY_MODEL -KeyName web-chat -Server learnOn GitHub, open infra/scripts/smoke-test.sh or infra/scripts/smoke-test.ps1.
Expected output (the tool list can change as the Learn server evolves):
📊 Smoke test of the chat alias and the learn MCP server:
Gateway: https://aigw-abcde-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com
Model : HTTP 200, model=<model>, reply=ok, total_tokens=<n>
Tools : HTTP 200, 3 tools on learn
learn_microsoft_docs_search
learn_microsoft_code_sample_search
learn_microsoft_docs_fetch
Smoke test passed.That one model call passed every layer: the Gateway scope request rate and cost limits, then the model's token limit and content safety. The gateway called Foundry with its managed identity.
Review the policy layers in the portal
Select Policies. Request rate limit and Cost limit show Gateway in the Scope column.
Token rate limit and Content safety show the backend model instead. Select Models to see that
model and the alias chat, and MCP servers to see learn.
Call the alias with your Microsoft Entra identity
The dev parameters gave you AI Gateway Models User on the alias, so you can call it without a key:
# Get a Microsoft Entra token for AI Gateway, for your own identity
export AI_GATEWAY_TOKEN="$(az account get-access-token \
--scope https://ai.gateway.azure.com/Data.All --query accessToken -o tsv)"
echo "👤 Token acquired: in AI_GATEWAY_TOKEN (${#AI_GATEWAY_TOKEN} characters, not shown)"
# Call the alias with the token instead of a key: your AI Gateway Models User role on the alias lets you in
curl -sS "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
-H "Authorization: Bearer $AI_GATEWAY_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in five words.\"}]}" \
| jq -r 'if .choices then "🤖 \(.choices[0].message.content)" else "🚨 \(.error.message // .)" end'# Get a Microsoft Entra token for AI Gateway, for your own identity
$env:AI_GATEWAY_TOKEN = az account get-access-token `
--scope https://ai.gateway.azure.com/Data.All --query accessToken -o tsv
Write-Host "👤 Token acquired: in AI_GATEWAY_TOKEN ($($env:AI_GATEWAY_TOKEN.Length) characters, not shown)"
# Build the request: the alias in model, and one user message
$body = @{
model = $env:AI_GATEWAY_MODEL
messages = @(@{ role = "user"; content = "Say hello in five words." })
} | ConvertTo-Json -Depth 5
# Call the alias with the token instead of a key: your AI Gateway Models User role on the alias lets you in
$response = Invoke-RestMethod -Method Post `
-Uri "$($env:AI_GATEWAY_URL.TrimEnd('/'))/default/models/openai/v1/chat/completions" `
-Headers @{ Authorization = "Bearer $($env:AI_GATEWAY_TOKEN)" } `
-ContentType "application/json" -Body $body
if ($?) { Write-Host "🤖 $($response.choices[0].message.content)" }The reply is a short greeting. Your Entra identity has its own counters, separate from the web-chat key's.
Check the backend identity boundary
Confirm that the Foundry account rejects keys and that only the gateway's identity has Foundry User on it:
# Check that the Foundry account rejects keys: disableLocalAuth is true when key authentication is off
echo "🔑 disableLocalAuth on the Foundry account:"
az cognitiveservices account show -g "$FOUNDRY_RG" -n "$FOUNDRY_NAME" --query properties.disableLocalAuth
# List the roles that the gateway's managed identity has on the Foundry account
echo "🔐 Roles of the gateway's identity on the Foundry account:"
az role assignment list --scope "$FOUNDRY_ID" --assignee "$AI_GATEWAY_PRINCIPAL_ID" \
--query "[].roleDefinitionName" -o tsv# Check that the Foundry account rejects keys: disableLocalAuth is true when key authentication is off
Write-Host "🔑 disableLocalAuth on the Foundry account:"
az cognitiveservices account show -g $env:FOUNDRY_RG -n $env:FOUNDRY_NAME --query properties.disableLocalAuth
# List the roles that the gateway's managed identity has on the Foundry account
Write-Host "🔐 Roles of the gateway's identity on the Foundry account:"
az role assignment list --scope $env:FOUNDRY_ID --assignee $env:AI_GATEWAY_PRINCIPAL_ID `
--query "[].roleDefinitionName" -o tsvExpected output:
🔑 disableLocalAuth on the Foundry account:
true
🔐 Roles of the gateway's identity on the Foundry account:
Foundry UserWatch the traffic in Monitoring
Select Monitoring. After a few minutes, Overview shows the smoke test's calls under the web-chat key
and your Entra call under your identity, and Traces shows each request with its policy spans. Run the smoke
test a few more times if the charts are still empty.
The smoke test passed, Policies shows two Gateway scope policies plus the model's own, a keyless
call with your Entra token worked, the Foundry account has disableLocalAuth set to true, and
Monitoring shows the traffic.
Go further
- Add a content safety floor. Add a
contentSafetyentry tobaselines.jsonwith the shape from Set gateway-wide policy baselines, so MCP servers and future models get it too. - Go keyless everywhere. Remove entries from
appNames, move apps to Entra groups inmodelUsersandtoolUsers(app_names,model_users, andtool_usersin Terraform), and follow Go keyless with Microsoft Entra ID and RBAC. - Keep backend traffic private. Give the gateway outbound virtual network integration, and give the
Foundry account a private endpoint, as in
Lock down backend traffic with private networking. Pass the integration
subnet to
gateway.bicepasbackendSubnetIdor to the Terraformgatewaymodule asbackend_subnet_id, or add it to the running gateway withaz aigateway network update --subnet. To keep clients off the internet too, add an inbound private endpoint, which is in public preview and requires approval, then set Public network access to Disabled. See Reach the gateway through an inbound private endpoint and Configure inbound access. - Protect and govern tools. Add Agent 365 tool threat protection to sensitive MCP servers in the portal, and govern MCP servers with Agent 365.
- Charge back by team. Attribute tokens and cost to callers with Attribute tokens and cost with OpenTelemetry.
Clean up
The Foundry model deployment, Log Analytics, and Application Insights are billed normally, so delete what you deployed.
Delete the resource groups
Delete both resource groups. Each command waits until the group is gone, which can take several minutes:
# Delete the gateway's resource group, with the gateway, its keys, and the monitoring resources
az group delete -n "$RG" --yes &&
echo "🚮 Resource group $RG is deleted"
# Delete the resource group with the Foundry account and its model deployment
az group delete -n "$FOUNDRY_RG" --yes &&
echo "🚮 Resource group $FOUNDRY_RG is deleted"# Delete the gateway's resource group, with the gateway, its keys, and the monitoring resources
az group delete -n $env:RG --yes
if ($?) { Write-Host "🚮 Resource group $env:RG is deleted" }
# Delete the resource group with the Foundry account and its model deployment
az group delete -n $env:FOUNDRY_RG --yes
if ($?) { Write-Host "🚮 Resource group $env:FOUNDRY_RG is deleted" }Purge the Foundry account
Deleted Foundry accounts keep soft-delete records that hold their names. To reuse the name right away, purge it. Purging is permanent.
# Purge the deleted Foundry account, so that you can reuse its name right away
az cognitiveservices account purge -g "$FOUNDRY_RG" -n "$FOUNDRY_NAME" -l "$LOCATION" &&
echo "🚮 Foundry account $FOUNDRY_NAME is purged"# Purge the deleted Foundry account, so that you can reuse its name right away
az cognitiveservices account purge -g $env:FOUNDRY_RG -n $env:FOUNDRY_NAME -l $env:LOCATION
if ($?) { Write-Host "🚮 Foundry account $env:FOUNDRY_NAME is purged" }Delete everything Terraform created
In the lab's terraform/ folder, with the variables from Preview the changes still set, delete both resource
groups and everything in them. The azurerm provider also purges the
soft-deleted Foundry account. It takes about three minutes.
# Delete everything that terraform apply created for dev, and purge the soft-deleted Foundry account
cd ~/aigateway/labs/21-enterprise-reference/terraform &&
terraform destroy -auto-approve -var-file dev.tfvars &&
echo "🚮 Everything in aigw-dev-tf is deleted"# Delete everything that terraform apply created for dev, and purge the soft-deleted Foundry account
Set-Location ~/aigateway/labs/21-enterprise-reference/terraform
if ($?) { terraform destroy -auto-approve -var-file dev.tfvars }
if ($?) { Write-Host "🚮 Everything in aigw-dev-tf is deleted" }The command ends with Destroy complete! Resources: 23 destroyed. If you moved the state to Azure Storage,
terraform destroy reads and updates it there.
If you set up GitHub Actions, remove the deployment identity's role assignments and the app registration, then
delete the dev and prod environments in SettingsEnvironments:
# Remove the deployment identity's role assignments on the subscription
az role assignment delete --assignee "$SP_ID" --scope "/subscriptions/$SUBSCRIPTION_ID" &&
echo "🚮 The deployment identity's role assignments are deleted"
# Delete the app registration, with its federated credentials
az ad app delete --id "$APP_ID" &&
echo "🚮 App registration github-aigw-deploy is deleted"# Remove the deployment identity's role assignments on the subscription
az role assignment delete --assignee $env:SP_ID --scope "/subscriptions/$($env:SUBSCRIPTION_ID)"
if ($?) { Write-Host "🚮 The deployment identity's role assignments are deleted" }
# Delete the app registration, with its federated credentials
az ad app delete --id $env:APP_ID
if ($?) { Write-Host "🚮 App registration github-aigw-deploy is deleted" }If you deployed prod, repeat the clean-up with the aigw-prod deployment's outputs. On the Terraform track, run
terraform init -reconfigure with the same -backend-config options and key=prod.tfstate, then
terraform destroy -var-file prod.tfvars. Delete the state's storage account last, after you destroy every
environment whose state it holds.
Next steps
- Keep backend traffic off the public internet with Lock down backend traffic with private networking.
- Attribute usage to teams with Attribute tokens and cost with OpenTelemetry.
- Govern your MCP servers across the organization with Govern MCP servers with Agent 365.
- Read about every policy type and scope in the AI Gateway policies documentation.