Azure AI Gateway Labs
Documentation Open portal
Enterprise readiness ยท Lab 18

Lock down backend traffic with private networking

Create an Azure AI Gateway with private routing through your virtual network, reach a Foundry backend over a private endpoint with its public access turned off, change the routing of the running gateway, and give the gateway its own inbound private endpoint so you can turn off its public access.

Advanced 1 h 15 min PortalCLIBicepTerraform Updated
Source Edit

What you'll do

  • Explain which private networking options AI Gateway supports, and which ones need preview access
  • Create a gateway with private routing through a dedicated subnet delegated to Microsoft.App/environments
  • Give Foundry a private endpoint and private DNS, turn off its public access, and prove the gateway still reaches it
  • Remove and add back the private routing of a running gateway
  • Give the gateway an inbound private endpoint and its private DNS zone, and call it from a client in the virtual network
  • Turn off the gateway's public network access while clients in the virtual network still reach it
  • Clean up in the right order, so the subnets and the Foundry resource are released

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform

Your security team wants model traffic kept off the public internet wherever the platform allows it. In this lab you create a gateway with private routing, so its outbound calls leave through a subnet in your virtual network. Then you give the Foundry resource a private endpoint, turn off Foundry's public access, and prove that the gateway still reaches the model while the internet can't. You remove and add back the private routing of the running gateway. Finally, you give the gateway its own inbound private endpoint, call it from a client inside the virtual network, and turn off the gateway's public network access, so only private clients get through.

Important

Outbound virtual network integration (private routing) and inbound private endpoints for AI Gateway are in public preview. Inbound private endpoints also need access for your subscription: until Microsoft enables it, creating one fails with SubscriptionNotRegisteredForFeature. See Inbound private endpoints and Request access to private endpoints. Without access, skip Reach the gateway through an inbound private endpoint: the rest of the lab works without it. Setting the gateway's Public network access to Disabled is enforced: calls to its public endpoint return 403, and only clients that use a private endpoint get through. While the frontend is public, protect it with runtime keys or Microsoft Entra ID (Issue and rotate runtime API keys, Go keyless with Microsoft Entra ID and RBAC) and an ipFilter policy (Set gateway-wide policy baselines).

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own resource group, network, gateway, and a Foundry resource with gpt-5-mini that only this lab uses, so you can skip this section, including the variables. You need only the one-time setup above, plus Terraform for the Terraform track.

  • Complete Provision your AI Gateway and Call backends with managed identities. You need a Foundry resource with a chat model deployment, for example gpt-5-mini.
  • This lab creates a new gateway with private routing, so the gateway from the earlier labs stays as it is. You can also add private routing to a gateway that already exists: see Remove and add back private routing.
  • Every track uses the Azure CLI for the checks.
  • Permission to create a resource group, network resources, and a gateway, to change the Foundry resource, and to assign roles on it, for example Owner, or Contributor plus Role Based Access Control Administrator.
  • A region where AI Gateway and gpt-5-mini are available, as in Find the available regions. The virtual network must be in the gateway's subscription and region.
Warning

Turning off public access on the Foundry resource blocks every caller that reaches it over the internet, including the gateway from the earlier labs, which has no private routing, and tools on your workstation. Use a dedicated Foundry resource for this lab if others depend on it, and re-enable public access when you clean up.

Important

This lab creates billable resources: private endpoints, private DNS zones, and a container group that bills only while it runs. Follow Clean up when you finish.

Set these variables once. The portal and Azure CLI tracks and the Verify section reuse them; the Bicep + azd and Terraform tracks set the ones they need from their outputs. The lab puts the new gateway and the network in their own resource group, so clean-up is one command.

# Name the lab's resource group, region, and new gateway, and build the gateway's resource ID from your subscription
export RG="rg-ai-gateway-private"            # a new resource group for this lab
export LOCATION="eastus2"                    # a region where AI Gateway is available
export GATEWAY="<private-gateway>"           # a NEW, globally unique name, 24 characters or fewer
export SUBSCRIPTION_ID="$(az account show --query id -o tsv)"
export GATEWAY_ID="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RG/providers/Microsoft.ApiManagement/aigateways/$GATEWAY"
echo "๐Ÿ“ Gateway $GATEWAY in $RG ($LOCATION)"
echo "๐Ÿ†” Subscription: $SUBSCRIPTION_ID"

# Name the virtual network, its subnets, the NSG of the integration subnet, and the private endpoints
export VNET="vnet-aigw"
export INTEGRATION_SUBNET="snet-aigw-integration"
export PE_SUBNET="snet-private-endpoints"
export CLIENT_SUBNET="snet-clients"
export NSG="nsg-aigw-integration"
export FOUNDRY_PE="pe-foundry-$GATEWAY"
export GATEWAY_PE="pe-gateway-$GATEWAY"
echo "๐Ÿ“ Network $VNET with subnets $INTEGRATION_SUBNET, $PE_SUBNET, and $CLIENT_SUBNET, NSG $NSG"
echo "๐Ÿ“ Private endpoints $FOUNDRY_PE to Foundry and $GATEWAY_PE to the gateway"

# Point at the Foundry resource and deployment from the earlier labs, and read the account's resource ID and endpoint
export FOUNDRY_RG="<foundry-resource-group>"
export FOUNDRY_ACCOUNT="<foundry-account-name>"
export DEPLOYMENT="gpt-5-mini"
export FOUNDRY_ID="$(az cognitiveservices account show -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" --query id -o tsv)"
export FOUNDRY_ENDPOINT="$(az cognitiveservices account show -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" --query properties.endpoint -o tsv)"
echo "๐Ÿ†” Foundry account: $FOUNDRY_ID"
echo "๐ŸŒ Foundry endpoint: $FOUNDRY_ENDPOINT"

# Name what the lab publishes on the new gateway: a model provider, an alias, and a runtime API key
export PROVIDER="foundry-private"
export ALIAS="private-chat"
export KEY_NAME="private-net-lab"
echo "๐Ÿ“ Provider $PROVIDER, alias $ALIAS, and API key private-net-lab on the new gateway"

# Create the resource group that holds the new gateway and the network, so that clean-up is one command
az group create -n "$RG" -l "$LOCATION" -o none &&
  echo "๐Ÿ“ฆ Resource group $RG is ready in $LOCATION"
Tip

Single-line commands and the az network commands in the tracks work in both shells. In PowerShell, replace the Bash line continuation (\) with a backtick (`) or put the command on one line. Snippets that build JSON bodies or URLs have separate Bash and PowerShell tabs.

How it works

A gateway has two network directions. Inbound, clients call the gateway's frontend on <gateway-url>. Outbound, the gateway calls models, MCP servers, and APIs. With private routing, the gateway joins a dedicated, delegated subnet in your virtual network and sends its outbound calls through it, so it can reach backends on private IPs. When a backend's private DNS zones are linked to that virtual network, the gateway resolves the backend's usual host name to its private endpoint IP.

You can choose private routing when you create the gateway, or add, change, or remove it later. The create dialog calls it Private routing, and the gateway's Networking page calls it Virtual network integration. Each change takes one to three minutes. Most connectivity problems after that are DNS: the gateway must resolve each backend to its private IP.

PieceDirectionAvailableWhere you set itKey values
Integration subnetOutboundYesYour virtual networkAt least /27 (/24 recommended), same subscription and region as the gateway, dedicated to one gateway, delegated to Microsoft.App/environments. It needs no outbound NSG rules
Private routingOutboundPreview, at creation or laterCreate AI gatewayAdvancedPrivate routing, or NetworkingOutboundVirtual network integrationproperties.backend.subnet.id set to the integration subnet (az aigateway create or az aigateway network update with --subnet); backend.subnet set to null removes it
Backend private endpointsOutboundYesEach backend, plus its private DNS zones linked to the virtual networkFoundry uses the account group ID
Gateway private endpointInboundPreview, needs access for your subscriptionNetworkingInboundCreate a private endpointGroup ID Gateway, in the private DNS zone that follows the gateway's host name. Approved automatically. See Inbound private endpoints
Public network accessInboundYesNetworkingInboundPublic network access, or az aigateway network update --public-network-accessEnabled (default) or Disabled. Disabled makes the public endpoint return 403; clients that use a private endpoint still get through
IP filterInboundYesGateway-wide policy (lab 09)ipFilter with action: Allow and cidrRanges

Inbound private endpoints

An inbound private endpoint gives the gateway a private IP in a client virtual network. It's in preview and needs access for your subscription, which you ask for with an Azure support request (Service: Azure AI Gateway, Summary: Enable private endpoints for Azure AI Gateway). See Request access to private endpoints. Until access is granted, NetworkingInbound shows "Creation isn't enabled for this subscription", and creating a private endpoint with the group ID Gateway fails:

Text
(SubscriptionNotRegisteredForFeature) Subscription /subscriptions/<subscription-id>/resourceGroups//providers/
Microsoft.Network/subscriptions/ is not registered for feature Microsoft.Network/AllowPrivateEndpoints required to
carry out the requested operation.

Registering the feature yourself with az feature register --namespace Microsoft.Network --name AllowPrivateEndpoints doesn't skip the request: the feature stays Pending until Microsoft approves it. Once access is granted, it shows Registered:

Bash
# Show whether the subscription has private endpoint access: Registered once Microsoft approves the request
az feature show --namespace Microsoft.Network --name AllowPrivateEndpoints --query properties.state -o tsv

With access, the gateway approves its private endpoints by itself: the connection shows Approved with the description Automatically approved by AI Gateway.

The gateway's private link resource names the private DNS zone for its endpoint. It's region specific and follows the gateway's host name, for example privatelink.eastus2-01.ai.gateway.azure.com for a gateway in East US 2. The portal's private endpoint wizard proposes this zone. The product docs name privatelink.azure-api.net instead, which never gets the gateway's record, so create the zone that your gateway returns:

Bash
# Show the gateway's private link resource: its group ID and the private DNS zone that clients must resolve
az rest --method get --url "https://management.azure.com$GATEWAY_ID/privateLinkResources?api-version=2026-11-01" \
  --query "value[].{groupId:properties.groupId, zones:properties.requiredZoneNames}" -o json

A private endpoint doesn't turn off public access. Once every client resolves the gateway to its private IP, set Public network access to Disabled. You do both in Reach the gateway through an inbound private endpoint and Turn off the gateway's public network access.

Prepare the virtual network

You create one virtual network with two subnets. The integration subnet belongs to the new gateway alone, and private endpoints get their own subnet. Later, Reach the gateway through an inbound private endpoint adds a third subnet, snet-clients (10.20.2.0/24), for a test client.

SubnetPrefixUsed forSettings
snet-aigw-integration10.20.0.0/24Gateway private routing (outbound)NSG nsg-aigw-integration, delegated to Microsoft.App/environments, nothing else deployed in it
snet-private-endpoints10.20.1.0/24Private endpoints to backends such as Foundry, and the gateway's own private endpointNone

Pick one track. The Bicep + azd and Terraform tracks declare the network, the gateway, the Foundry resource and its private endpoint, and the model in one project, so you deploy everything once in the next section.

Choose your track
1

Create the network security group

In the Azure portal, create a network security group named nsg-aigw-integration in the lab's resource group and region. Keep its default rules: the gateway needs no custom outbound rules.

2

Create the virtual network and subnets

Create a virtual network named vnet-aigw in the same subscription and region as the new gateway, with the address space 10.20.0.0/16. Add the two subnets from the table above. On snet-aigw-integration, select the nsg-aigw-integration network security group and delegate the subnet to Microsoft.App/environments.

Checkpoint

The virtual network shows two subnets, and snet-aigw-integration lists the delegation Microsoft.App/environments and the NSG nsg-aigw-integration.

Create the gateway with private routing

This lab puts private routing in the create request, so the new gateway uses the integration subnet from its first call. To add private routing to a gateway that already exists, see Remove and add back private routing.

Choose your track
1

Enter the gateway details

  1. Open the AI Gateway portal. Open the gateway switcher in the top bar and select Create new gateway.
  2. In Create AI gateway, enter the new gateway's Name, then choose the subscription, the rg-ai-gateway-private Resource group (Existing), and the Region that matches the virtual network.
  3. Expand Advanced. Keep Enable system-assigned managed identity selected and Public network access set to Enabled: you test turning it off later in this lab.
2

Choose private routing

  1. Under Networking, select Private routing ("Allow access to private backends through secure outbound connections").
  2. In Virtual network, select vnet-aigw (rg-ai-gateway-private). In Subnet, select snet-aigw-integration (10.20.0.0/24). The lists show the networks in the gateway's subscription and region.
  3. Select Create gateway. All gateways opens with "Gateway creation is in progress", and the new gateway's Provisioning goes from Created to Succeeded within a few minutes.
Checkpoint

All gateways shows the new gateway with Provisioning Succeeded.

Whichever track you used, load the runtime URL and confirm that the gateway uses the integration subnet:

# Read the gateway's runtime URL
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "๐ŸŒ Runtime URL: $AI_GATEWAY_URL"

# Show the gateway's provisioning state, the subnet that its outbound calls go through, and its host name
echo "๐Ÿ”’ Gateway $GATEWAY and its private routing:"
az aigateway show -g "$RG" -n "$GATEWAY" \
  --query "{state:properties.provisioningState, backendSubnet:properties.backend.subnet.id, host:properties.frontend.defaultHostname}"

The commands print the runtime URL, for example ๐ŸŒ Runtime URL: https://aigw-x7k2q-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com, and then the gateway's settings:

JSON
{
  "state": "Succeeded",
  "backendSubnet": "/subscriptions/<subscription-id>/resourceGroups/rg-ai-gateway-private/providers/Microsoft.Network/virtualNetworks/vnet-aigw/subnets/snet-aigw-integration",
  "host": "aigw-x7k2q-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com"
}

Give Foundry a private endpoint

The gateway now sends outbound calls through your virtual network, but Foundry still resolves to its public IP. Give the Foundry resource a private endpoint in snet-private-endpoints and link its private DNS zones to the virtual network, so the gateway resolves Foundry to a private IP.

Choose your track
1

Add a private endpoint to Foundry

In the Azure portal, open your Foundry resource and add a private endpoint named pe-foundry-<private-gateway> in the lab's resource group, in vnet-aigw โ€บ snet-private-endpoints, for the account sub-resource, with private DNS integration turned on. The private DNS zones for Foundry are privatelink.cognitiveservices.azure.com, privatelink.openai.azure.com, and privatelink.services.ai.azure.com. For the full walkthrough, see How to configure network isolation for Microsoft Foundry.

Checkpoint

The private endpoint is approved, and the Foundry host has an A record in the 10.20.1.0/24 range:

Bash
# Show the status of the private endpoint's connection to Foundry
echo "๐Ÿ”’ Status of the connection from $FOUNDRY_PE to Foundry:"
az network private-endpoint show -g "$RG" -n "$FOUNDRY_PE" \
  --query "privateLinkServiceConnections[0].privateLinkServiceConnectionState.status" -o tsv

# List the A records in the Foundry private DNS zone: the account's name and its private IP
echo "๐Ÿ”’ A records in privatelink.cognitiveservices.azure.com:"
az network private-dns record-set a list -g "$RG" -z privatelink.cognitiveservices.azure.com \
  --query "[].{name:name, ip:aRecords[0].ipv4Address}" -o table

The connection status is Approved, and the table shows <foundry-account-name> with an IP like 10.20.1.4.

Publish a model on the new gateway

The new gateway starts empty. Publish your Foundry deployment on it, as in Make your first model call, and create a runtime API key. The gateway calls Foundry with its own system-assigned managed identity, which needs the Foundry User role on the account.

Choose your track
1

Import the deployment from Foundry

  1. In the AI Gateway portal, select the new gateway in the top bar.
  2. Select ModelsAddModels from Microsoft Foundry.
  3. Select the subscription and resource group of your Foundry account, then select your deployment and keep Managed identity as the Authentication method.
  4. Under Aliases, keep Create an alias for each model selected, then select Add models.
  5. Open the imported model on Models, open the Aliases tab, and copy the alias exactly as shown.
2

Create a runtime API key

Select AccessAPI keysCreate API key, enter private-net-lab in Key name, select Create, and copy the value. Then set both values in your shell without echoing the key:

# Set the alias that apps send in the model field
export AI_GATEWAY_MODEL="<model-name>"    # the alias from the Aliases tab
echo "๐Ÿ”– Model alias: $AI_GATEWAY_MODEL"

# Read the key without showing it, and export it for the next commands
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "๐Ÿ”‘ Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
Checkpoint

Call the model through the new gateway. Foundry's public access is still on, so a 200 proves that the gateway, the model, and the key work before you lock anything down:

# Call the model through the new gateway, and print only the HTTP status code
echo "๐Ÿ”’ Model call through the gateway, which reaches Foundry over the private endpoint:"
curl -sS -o /dev/null -w 'HTTP %{http_code}\n' "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly one word: ok\"}],\"max_completion_tokens\":128}"

If the first call returns 401 or 403 from the backend, the role assignment hasn't taken effect yet. Wait a few minutes and try again.

Turn off public access to Foundry

The gateway resolves Foundry through the private DNS zones, so its calls already use the private endpoint. Turn off Foundry's public access, so the private endpoint is the only way in.

Choose your track
1

Disable public network access on Foundry

After the private endpoint shows as approved, open the Foundry resource's networking settings in the Azure portal and disable public network access. See Configure virtual networks for Foundry Tools.

The change can take a few minutes to apply.

Remove and add back private routing

Private routing isn't fixed at creation: you can add a subnet to a running gateway, move it to another delegated subnet, or remove it. Remove it now, while Foundry accepts only private traffic, to see what the gateway's model calls depend on. Then add it back. Each change takes one to three minutes.

Choose your track
1

Turn off virtual network integration

  1. In the AI Gateway portal, open the new gateway and select NetworkingOutbound ("Choose how this gateway reaches backends.").
  2. Turn off Virtual network integration and select Save. The page shows "Applying networking changes... Networking edits are locked while this task runs." until the gateway finishes the change.

Wait until the change finishes, then call the model through the gateway again:

# Wait until the gateway has finished the change, then show its routing type
until [ "$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.provisioningState -o tsv)" = "Succeeded" ]; do
  sleep 10
done
echo "๐Ÿ”’ Routing of gateway ${GATEWAY}: $(az aigateway network show -g "$RG" -n "$GATEWAY" --query virtualNetworkType -o tsv)"

# Call the model through the gateway, and print the response and the HTTP status code
echo "๐Ÿšฆ Model call through the gateway:"
curl -sS -w '\nHTTP %{http_code}\n' "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly one word: ok\"}],\"max_completion_tokens\":128}"

The routing is None, and the call returns HTTP 403 with Foundry's error, which the gateway passes on:

JSON
{"error":{"code":"403","message": "Public access is disabled. Please configure private endpoint."}}

Without private routing, the gateway resolves Foundry's public address and calls it over the internet, which Foundry no longer accepts. Add the integration subnet back:

Choose your track
1

Turn virtual network integration back on

  1. On NetworkingOutbound, turn on Virtual network integration.
  2. In Virtual network, select vnet-aigw (rg-ai-gateway-private). In Subnet, select snet-aigw-integration (10.20.0.0/24). A subnet that isn't delegated shows "Not delegated to Microsoft.App/environments" and can't be used.
  3. Select Save, and wait until "Applying networking changes..." goes away.

To move the gateway to another subnet, select that subnet instead and save. It needs the same preparation as snet-aigw-integration.

Run the wait and the call from the previous step again. The routing is External again, and the call returns HTTP 200 with the model's answer.

Checkpoint

Without private routing, the gateway's model call fails, because the gateway reaches Foundry over the internet, and Foundry no longer accepts that. With private routing back, the same call returns HTTP 200.

Reach the gateway through an inbound private endpoint

So far, every client reaches the gateway over the internet. Give the gateway its own private endpoint in snet-private-endpoints, so that clients in the virtual network reach it on a private IP, and prove it with a test client that runs inside the network. This section needs private endpoint access for your subscription. Without it, skip to Turn off the gateway's public network access.

ResourceNameWhat it does
Private endpointpe-gateway-<private-gateway>Gives the gateway a private IP in snet-private-endpoints, with the group ID Gateway
Private DNS zoneprivatelink.<region>-01.ai.gateway.azure.comResolves the gateway's usual host name to that private IP inside the virtual network
Subnetsnet-clients (10.20.2.0/24)Holds the test client, delegated to Microsoft.ContainerInstance/containerGroups
Container groupaci-private-clientResolves the gateway's host name, calls the model once, prints the IP that answered, and stops
Choose your track
1

Create the gateway's private endpoint

  1. In the AI Gateway portal, open the new gateway's Networking page and select Inbound. Under Private endpoints, select Create a private endpoint. It opens the Create a private endpoint wizard in the Azure portal. If you use more than one directory, check that the Azure portal opened in the gateway's directory.
  2. On Basics, select the lab's subscription and the resource group rg-ai-gateway-private, enter the name pe-gateway-<private-gateway>, and select the gateway's Region, for example East US 2. The wizard doesn't default to the gateway's region. Network Interface Name fills in by itself.
  3. On Resource, check that Resource type is Microsoft.ApiManagement/aigateways, Resource is the new gateway, and Target sub-resource is Gateway.
  4. On Virtual Network, select vnet-aigw and the subnet snet-private-endpoints. The wizard selects the first virtual network in the region, so check it. Keep Dynamically allocate IP address.
  5. On DNS, keep Integrate with private DNS zone at Yes. The wizard proposes the gateway's zone, privatelink.<region>-01.ai.gateway.azure.com, in the lab's resource group.
  6. Select Review + create, wait for "Validation passed", and select Create. It takes a minute or two.
  7. Back on NetworkingInbound in the AI Gateway portal, select Refresh. The private endpoint shows Approved and Succeeded, with the group Gateway and the description "Automatically approved by AI Gateway."
Networking โ€บ Inbound lists the gateway's private endpoints
2

Add the client subnet

In the Azure portal, open vnet-aigw โ€บ Subnets, and add a subnet named snet-clients with the address range 10.20.2.0/24. Under Subnet delegation, select Microsoft.ContainerInstance/containerGroups.

1

Check the private endpoint and its DNS record

Whichever track you used, read the zone and the client subnet, then check the private endpoint and its DNS record:

# Name the gateway's private endpoint, and read its private DNS zone and the ID of the client subnet
export GATEWAY_PE="pe-gateway-$GATEWAY"
export GATEWAY_ZONE="$(az rest --method get \
  --url "https://management.azure.com$GATEWAY_ID/privateLinkResources?api-version=2026-11-01" \
  --query "value[0].properties.requiredZoneNames[0]" -o tsv)"
export CLIENT_SUBNET_ID="$(az network vnet subnet show -g "$RG" --vnet-name vnet-aigw -n snet-clients --query id -o tsv)"

# Show the status of the private endpoint's connection to the gateway
echo "๐Ÿ”’ Connection from $GATEWAY_PE to the gateway:"
az network private-endpoint show -g "$RG" -n "$GATEWAY_PE" \
  --query "privateLinkServiceConnections[0].privateLinkServiceConnectionState.{status:status, description:description}" -o table

# Show the same connection from the gateway's side
echo "๐Ÿ”’ Private endpoint connections of gateway ${GATEWAY}:"
az aigateway network private-endpoint-connection list -g "$RG" --gateway-name "$GATEWAY" \
  --query "[].{status:properties.privateLinkServiceConnectionState.status, state:properties.provisioningState}" -o table

# List the A records in the gateway's private DNS zone: the first label of the host name and its private IP
echo "๐Ÿ”’ A records in ${GATEWAY_ZONE}:"
az network private-dns record-set a list -g "$RG" -z "$GATEWAY_ZONE" \
  --query "[].{name:name, ip:aRecords[0].ipv4Address}" -o table

The connection is Approved, with the description Automatically approved by AI Gateway., the gateway lists it as Approved and Succeeded, and the zone has one A record, named after the first label of the gateway's host name, with an IP in 10.20.1.0/24.

2

Call the gateway from inside the virtual network

Your workstation still resolves the gateway's host name to its public IP, because the private DNS zone answers only inside the virtual network. To see the private path, run a test client in snet-clients: infra/private-client.bicep deploys an Azure Container Instances container group with the Azure CLI image, which resolves the gateway's host name, calls the model once, and prints the IP that answered. The key goes in as a secure environment variable, so it doesn't show in the deployment or in the container group's properties. The deployment returns once the call has finished, in about three minutes the first time.

# Register Azure Container Instances in the subscription, once
az provider register --namespace Microsoft.ContainerInstance --wait &&
  echo "๐Ÿ“ฆ Microsoft.ContainerInstance is registered"

# Take the gateway's host name out of the runtime URL
GATEWAY_HOST="${AI_GATEWAY_URL#https://}"; GATEWAY_HOST="${GATEWAY_HOST%%/*}"
echo "๐ŸŒ Gateway host: $GATEWAY_HOST"

# Deploy the test client in the client subnet: it resolves the host, calls the model once, and stops
cd ~/aigateway/labs/18-private-networking &&
  az deployment group create -g "$RG" -n private-client -f infra/private-client.bicep \
    -p subnetId="$CLIENT_SUBNET_ID" gatewayHostName="$GATEWAY_HOST" model="$AI_GATEWAY_MODEL" \
    apiKey="$AI_GATEWAY_API_KEY" -o none &&
  echo "๐Ÿš€ Test client aci-private-client ran in snet-clients"

# Show what the test client printed
az container logs -g "$RG" -n aci-private-client

The test client resolves the gateway's usual host name, through a CNAME to the privatelink zone, to the private endpoint's IP, and the call returns 200 from that IP:

Text
DNS: 10.20.1.5      aigw-x7k2q-k3v7qz2m9xw4p.privatelink.eastus2-01.ai.gateway.azure.com aigw-x7k2q-k3v7qz2m9xw4p.eastus2-01.ai.gateway.azure.com
HTTP 200 from 10.20.1.5
infra/private-client.bicepGitHub
// A throwaway client inside the virtual network, to test the gateway's inbound private endpoint: an Azure Container
// Instances container group in a subnet delegated to Microsoft.ContainerInstance/containerGroups. It resolves the
// gateway's host name, calls the model alias once through the gateway, prints the IP it reached and the HTTP status,
// and stops. Run it again with "az container start", and read its output with "az container logs".
// The API key is a secure environment variable: it never shows in the deployment or in the container's properties.
// Every track of lab 18 deploys it with "az deployment group create". Container Instances bills only while it runs.

@description('Name of the container group.')
param name string = 'aci-private-client'

@description('Region of the container group. It must be the region of its virtual network.')
param location string = resourceGroup().location

@description('Resource ID of a subnet delegated to Microsoft.ContainerInstance/containerGroups.')
param subnetId string

@description('Runtime host name of the gateway, without https://.')
param gatewayHostName string

@description('Model alias to call.')
param model string

@description('Runtime API key of the gateway.')
@secure()
param apiKey string

@description('Tags applied to the container group.')
param tags object = {}

// Resolve the host, then call the alias and print the HTTP status and the IP that answered.
var probe = '''
echo "DNS: $(getent hosts "$AI_GATEWAY_HOST")"
curl -sS -o /dev/null -w 'HTTP %{http_code} from %{remote_ip}\n' \
  "https://$AI_GATEWAY_HOST/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly one word: ok\"}],\"max_completion_tokens\":128}"
'''

resource client 'Microsoft.ContainerInstance/containerGroups@2023-05-01' = {
  name: name
  location: location
  tags: tags
  properties: {
    osType: 'Linux'
    restartPolicy: 'Never'
    subnetIds: [
      {
        id: subnetId
      }
    ]
    containers: [
      {
        name: 'probe'
        properties: {
          image: 'mcr.microsoft.com/azure-cli:latest'
          command: [ '/bin/bash', '-c', probe ]
          environmentVariables: [
            {
              name: 'AI_GATEWAY_HOST'
              value: gatewayHostName
            }
            {
              name: 'AI_GATEWAY_MODEL'
              value: model
            }
            {
              name: 'AI_GATEWAY_API_KEY'
              secureValue: apiKey
            }
          ]
          resources: {
            requests: {
              cpu: 1
              memoryInGB: 1
            }
          }
        }
      }
    ]
  }
}

output name string = client.name
Checkpoint

The gateway's private endpoint is Approved, its zone privatelink.<region>-01.ai.gateway.azure.com has an A record in 10.20.1.0/24, and the test client prints HTTP 200 from 10.20.1.x: it reached the gateway on its private IP.

Turn off the gateway's public network access

Public network access controls the gateway's own public endpoint. With Disabled, the gateway accepts calls only through its inbound private endpoints: your workstation is blocked, while the test client in snet-clients still gets through. If you skipped Reach the gateway through an inbound private endpoint, the gateway has no private endpoint, so turning public access off blocks every client. Turn it off briefly to see the effect, then turn it back on. Each change takes about a minute.

Choose your track
1

Disable public network access

  1. On the new gateway's Networking page, select Inbound ("Choose how clients reach this gateway through public networks or private endpoints.").
  2. Turn off Public network access ("Allow clients to reach this gateway from the public internet.") and select Save.
  3. In Disable public network access?, select Disable public access. Wait until "Applying networking changes..." goes away.

Call the model through the gateway again, with the same command as in Remove and add back private routing. It returns HTTP 403, from the gateway itself:

JSON
{"error":"Public network access is disabled for this resource."}

If you created the gateway's private endpoint, run the test client again. It takes about a minute, and these two commands work in Bash and PowerShell:

Bash
# Run the test client in snet-clients again, and show what it printed
az container start -g "$RG" -n aci-private-client -o none
az container logs -g "$RG" -n aci-private-client

It still prints HTTP 200 from 10.20.1.x: the private endpoint isn't affected by the public access setting.

Choose your track
1

Enable public network access again

On NetworkingInbound, turn on Public network access and select Save. Wait until "Applying networking changes..." goes away.

Checkpoint

With public network access Disabled, the call from your workstation returns HTTP 403 with Public network access is disabled for this resource., and the test client in the virtual network, if you created it, still gets HTTP 200 from the private IP. After you enable public access again, the call from your workstation returns HTTP 200. The change can take a minute after the command returns.

Verify

1

Check the gateway's private routing

Bash
# Show the gateway's network settings: the routing type and the subnet that its outbound calls go through
echo "๐Ÿ”’ Network settings of gateway ${GATEWAY}:"
az aigateway network show -g "$RG" -n "$GATEWAY" \
  --query "{routing:virtualNetworkType, subnet:virtualNetworkConfiguration.subnetResourceId}" -o json

The output shows "routing": "External", and subnet ends with /virtualNetworks/vnet-aigw/subnets/snet-aigw-integration.

2

Check name resolution for Foundry

Bash
# List the A records that the private endpoint registered in the zone
echo "๐Ÿ”’ A records in privatelink.cognitiveservices.azure.com:"
az network private-dns record-set a list -g "$RG" -z privatelink.cognitiveservices.azure.com \
  --query "[].{name:name, ip:aRecords[0].ipv4Address}" -o table

The Foundry account resolves to an IP in snet-private-endpoints (10.20.1.x) for anything that uses the virtual network's DNS, including the gateway.

3

Check name resolution for the gateway

If you created the gateway's private endpoint, check its record in the gateway's zone. The command works in Bash and PowerShell:

Bash
# List the A record that the gateway's private endpoint registered in its zone
echo "๐Ÿ”’ A records in ${GATEWAY_ZONE}:"
az network private-dns record-set a list -g "$RG" -z "$GATEWAY_ZONE" --query "[].{name:name, ip:aRecords[0].ipv4Address}" -o table

The gateway's host name resolves to an IP in snet-private-endpoints (10.20.1.x) for clients in the virtual network, such as the test client.

4

Call the model through the gateway

# Call the model through the gateway, and print only the HTTP status code
echo "๐Ÿ”’ Model call through the gateway, with Foundry's public access off:"
curl -sS -o /dev/null -w 'HTTP %{http_code}\n' "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly one word: ok\"}],\"max_completion_tokens\":128}"

The call still returns HTTP 200. Your workstation reaches the gateway on its public endpoint, and the gateway reaches Foundry through the integration subnet and the private endpoint.

5

Call Foundry directly from your workstation

# Get a Microsoft Entra token for Foundry with your own sign-in
TOKEN="$(az account get-access-token --resource https://cognitiveservices.azure.com --query accessToken -o tsv)"
echo "๐Ÿ‘ค Token acquired: in TOKEN (${#TOKEN} characters, not shown)"

# Call the deployment on Foundry's public endpoint directly, and print the response and the HTTP status code
echo "๐Ÿšฆ Direct call to Foundry from your workstation (expect HTTP 403, or 401 without a Foundry role):"
curl -sS -w '\nHTTP %{http_code}\n' "${FOUNDRY_ENDPOINT%/}/openai/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"model\":\"$DEPLOYMENT\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with exactly one word: ok\"}],\"max_completion_tokens\":128}"

The call returns HTTP 403, so the internet can't reach Foundry anymore, but the gateway can:

JSON
{"error":{"code":"403","message": "Public access is disabled. Please configure private endpoint."}}

Foundry checks your data-plane role before the network rule. Without a role such as Cognitive Services OpenAI User or Foundry User on the Foundry resource, the call returns HTTP 401 instead. The call is blocked either way:

JSON
{"error":{"code":"PermissionDenied","message":"Principal does not have access to API/Operation."}}

To see the network block itself, assign yourself Cognitive Services OpenAI User on the Foundry resource and call again later. A new data-plane role can take 30 minutes or more to apply, and until then calls can alternate between 401 and 403.

Checkpoint

The new gateway uses private routing through snet-aigw-integration, Foundry and, if you created its private endpoint, the gateway resolve to 10.20.1.x addresses inside the virtual network, model calls through the gateway return 200, and direct calls to Foundry from the internet return 403 (or 401 if you have no data-plane role on Foundry).

Go further

  • Filter who can call the gateway. While the frontend stays public, add a gateway-wide ipFilter policy (action: Allow with your egress cidrRanges) next to keys or Microsoft Entra ID. See Set gateway-wide policy baselines.
  • Hub-and-spoke networks. Keep backend private endpoints and their private DNS zones in a hub, peer the gateway's virtual network to it, and link the zones to the integrated virtual network. If you use your own DNS servers, forward the privatelink zones to Azure DNS. See Azure private endpoint DNS configuration.
  • Private MCP servers and APIs. Any backend with a private endpoint or a private IP that the integrated virtual network can reach works the same way, including the tool servers from Federate MCP tools behind one endpoint.
  • Data residency. Residency follows the whole request path, so account for where your backends, tools, logs, and clients live, not only the gateway region.

Clean up

On the portal and Azure CLI tracks, re-enable Foundry's public access first, because the gateway from the earlier labs and other callers reach it over the internet. Then remove the new gateway's role assignment on Foundry and delete the lab's resource group, which holds the gateway, the private endpoints, the private DNS zones, the virtual network, the NSG, and the test client if you created it.

Choose your track
1

Delete the lab's resources

In the Azure portal, re-enable public network access on the Foundry resource, remove the new gateway's Foundry User role assignment in its Access control (IAM), and delete rg-ai-gateway-private.

If you put the lab's resources in a resource group you want to keep, delete them in this order instead: the test client aci-private-client and the gateway's private endpoint, if you created them, then the gateway, with More actions (โ‹ฎ) โ€บ Delete on its row in All gateways of the AI Gateway portal, then Foundry's private endpoint, the private DNS zone links and zones, the virtual network, and the NSG.

Keep the gateway, model, and key from the earlier labs. The next labs reuse them. azd down and terraform destroy don't touch them, because the Bicep + azd and Terraform tracks created their own gateway and Foundry resource.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.