Azure AI Gateway Labs
Documentation Open portal
Models & routing · Lab 26New

Publish embedding models for search and RAG

Publish a Foundry embedding model through the gateway behind its own alias, generate vectors with curl, the OpenAI SDK, and the portal playground, build a mini RAG app on one gateway URL and key, and cap ingestion with a token limit.

Intermediate 30 min PortalCLIBicepTerraformCodeNotebook Updated
Notebook Source Edit

What you'll do

  • Publish a Foundry embedding deployment through the gateway behind its own alias
  • Generate embeddings with curl, the OpenAI Python SDK, and the portal's embeddings playground
  • Build a mini RAG app that embeds, retrieves, and answers through one gateway URL and one key
  • Cap an ingestion job with a per-caller token limit on the embedding model

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform, Code, Notebook

Your team is building a help desk assistant with retrieval-augmented generation (RAG): an ingestion job turns every article into a vector, and the app finds the articles closest to each question before a chat model answers. The chat model already goes through AI Gateway, but the embedding calls go straight to Foundry, with their own key and no limits or usage records. In this lab you publish text-embedding-3-small through the same gateway, call it with curl, the OpenAI SDK, and the portal, build a mini RAG app that uses one URL and one key for both models, and cap the ingestion job with a token limit.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, Foundry account with both models, aliases, token limit, and key, so you can skip this section. Run that track in Publish the embedding model first: it sets the variables that the next sections and Verify use. You need only the one-time setup above, and Python 3.10 or later.

  • Complete Publish a Foundry model and make your first call. You need the gateway, the gpt-5-mini alias, and a runtime API key. Lab 02 also gave the gateway the Foundry User role on your Foundry account, which covers the embedding deployment in this lab.
  • Permission to change the gateway, for example Contributor on its resource group, and to create deployments in the Foundry account.
  • Python 3.10 or later for the mini RAG app and Verify, and jq for the Bash steps.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/26-embeddings.

Set these variables once. Every track and the Verify section reuse them.

# Name your gateway, the Foundry account from lab 02, the provider, and the two aliases that apps call
export RG="<resource-group>"
export GATEWAY="<gateway>"
export FOUNDRY_RG="<foundry-resource-group>"     # resource group of the Foundry account from lab 02
export FOUNDRY_ACCOUNT="<foundry-account-name>"
export PROVIDER="foundry-models"                 # the gateway provider that holds gpt-5-mini
export EMBEDDINGS="text-embedding-3-small"       # the Foundry deployment and the backend model
export AI_GATEWAY_EMBEDDINGS_MODEL="text-embedding-3-small"   # the alias that apps send in "model"
export AI_GATEWAY_MODEL="gpt-5-mini"             # the chat alias from lab 02
echo "📝 Alias $AI_GATEWAY_EMBEDDINGS_MODEL will serve $EMBEDDINGS on gateway $GATEWAY in $RG"

# Read the resource IDs of the gateway and the Foundry account, and the gateway's runtime URL
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export FOUNDRY_ID="$(az cognitiveservices account show -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
echo "🆔 Gateway: $GATEWAY_ID"
echo "🆔 Foundry account: $FOUNDRY_ID"
echo "🌐 Runtime URL: $AI_GATEWAY_URL"

# Paste the gateway runtime key from lab 02 without showing it, and don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"

The CLI and Bicep tracks of lab 02 name the provider foundry-models. A portal import names it after your Foundry account. To check, list the backend models. The Provider column shows the provider of each one.

# List the backend models on the gateway, with the provider that holds each one
echo "✨ Backend models on $GATEWAY:"
az aigateway model list -g "$RG" --gateway-name "$GATEWAY" -o table

Deploy text-embedding-3-small in the same Foundry account as gpt-5-mini. Embedding inference is billed per use, at a small fraction of the price of chat tokens.

# Deploy text-embedding-3-small next to gpt-5-mini, with 20,000 tokens per minute of Global Standard capacity
az cognitiveservices account deployment create -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" \
  --deployment-name "$EMBEDDINGS" --model-name text-embedding-3-small --model-version 1 \
  --model-format OpenAI --sku-name GlobalStandard --sku-capacity 20 \
  --query "{name:name, state:properties.provisioningState}" &&
  echo "✨ Model deployment $EMBEDDINGS is ready in $FOUNDRY_ACCOUNT"

How it works

An embedding model turns text into a vector: a list of numbers in which texts with similar meanings sit close together. A RAG app embeds its documents once, when it ingests them, and embeds each question when a user asks it. Since the October 28, 2026 release, the gateway publishes embedding models like chat models: you register the deployment as a backend model, give it an alias, and apps send the alias in model. Apps get one URL and one key or Entra token for both models, ingestion jobs get the same token limits and usage records as chat, and you can change the backend behind the alias without touching the apps.

An embedding model supports only the embeddings endpoint. An alias exposes the endpoints that all its backend models share, and an embedding model shares none with a chat model, so each one gets its own alias.

WhatValue
EndpointPOST $AI_GATEWAY_URL/default/models/openai/v1/embeddings
OpenAI SDK base_url$AI_GATEWAY_URL/default/models/openai/v1, the same as for chat
Backend modelSupported endpoint /openai/v1/embeddings. The portal shows the API format Embeddings
inputA string, an array of strings, or arrays of token IDs. One request embeds a whole batch
dimensionsOptional. text-embedding-3-small returns 1,536 numbers by default, and fewer when you ask, such as 256
encoding_formatfloat (the default) or base64
usageprompt_tokens and total_tokens. Token limits and Monitoring count them

Publish the embedding model

Register the deployment as a backend model under your Foundry provider, then publish it under an alias with the same name, text-embedding-3-small, so that apps that called the model on Foundry keep the same model name.

Choose your track
1

Import the embedding model

  1. Open the AI Gateway portal and select your gateway in the top bar.
  2. Select ModelsAddModels from Microsoft Foundry.
  3. In Resource groups, select the resource group of your Foundry account. The wizard otherwise selects every Foundry resource in the subscription. Keep Managed identity as the Authentication method.
  4. Clear Create an alias for each model. When it's selected, the wizard names the alias <foundry-account-name>/text-embedding-3-small. You create a shorter alias in the next step.
  5. Under Models, select your Foundry account. Its row says This model provider already exists in your gateway. 1 new model is available.
  6. Select Add models. The portal confirms Models imported and 1 model(s) were created. No aliases were created.
Filter to your Foundry account's resource group, and clear Create an alias for each model.

Select ModelsBackend models. text-embedding-3-small shows Embeddings under API formats and -- under Aliases, and a banner says Some models don't have an alias yet.

The imported embedding model has the API format Embeddings and no alias yet.
2

Create the alias

  1. Select ModelsAddAlias for existing models.
  2. On Basics, set Alias name to text-embedding-3-small and Description to Embeddings for search and RAG. Select Next.
  3. On Auto model selection (preview), keep it off. Select Next.
  4. On Select models, select text-embedding-3-small, whose API formats column says Embeddings. The page notes that Models must share at least one API format, so don't add gpt-5-mini. Select Next.
  5. Keep the defaults on Session affinity, Priorities and weights, and Failure handling. On Review, select Create.
Select only the embedding model. A chat model shares no API format with it.

The alias's overview opens. Under Endpoints, Embeddings shows the Base URL for SDKs and the full Endpoint URL.

The alias's Embeddings endpoint. Test in catalog opens the embeddings playground.
Checkpoint

ModelsAliases lists text-embedding-3-small, and its overview shows an Embeddings endpoint that ends with /default/models/openai/v1/embeddings.

Generate embeddings

Send text to the alias and look at what comes back: one vector per input, and the tokens that the request used. A new alias reaches the runtime within a few seconds. If the first call returns model_not_found, wait a few seconds and try again.

Choose your track
1

Test in the playground

  1. On the alias's overview, select Test in catalog. The Generate embeddings playground opens in Catalog.
  2. Under Authentication, keep API key as the Method, and select your key from lab 02 in API key. The default, Built-in all-access API key, works too, but testing with your app's key tests its access.
  3. In Input text, enter How do I reset my password?. Set Dimensions to 256, and keep Float as the Encoding format.
  4. Select Generate embeddings.
The playground returns the full response, with a summary of the vectors, dimensions, input tokens, and time.

To embed a batch or token IDs, set Input format to JSON and enter an array, such as ["First document", "Second document"].

2

Continue in code

Select Continue in codeOpenAI SDK. The Python snippet sets base_url to the alias's base URL, which ends with /default/models/openai/v1, and calls client.embeddings.create. The cURL snippet calls the Embeddings API at the full endpoint URL.

Continue in code gives you the base URL and an embeddings call for the OpenAI SDK.
Checkpoint

The Response summary shows 1 vector · 256 dimensions · 7 input tokens. The first call to a new alias can take several seconds; the next ones take a few hundred milliseconds.

Build a mini RAG app

infra/rag.py is a whole RAG app in about 80 lines. It embeds eight help desk articles in one batch, embeds the question with the same alias, ranks the articles by cosine similarity, and asks gpt-5-mini to answer from the two best matches. Both models use the same OpenAI client, so the same gateway URL and key. A real app stores the vectors in a vector index, such as Azure AI Search, instead of a Python dictionary.

infra/rag.pyGitHub
"""A mini retrieval-augmented generation (RAG) app that runs entirely through AI Gateway: it embeds a few help desk
articles, finds the ones closest to a question with cosine similarity, and asks a chat model to answer from them.
Both models use the same gateway URL and the same runtime key.

Usage:
  pip install -r requirements.txt
  export AI_GATEWAY_URL="<gateway-url>"
  export AI_GATEWAY_MODEL="gpt-5-mini"                          # the chat alias
  export AI_GATEWAY_EMBEDDINGS_MODEL="text-embedding-3-small"   # the embeddings alias
  export AI_GATEWAY_API_KEY="..."      # never commit this value
  python rag.py ["Your question"]
"""

import math
import os
import sys

from openai import OpenAI

gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
chat_model = os.environ.get("AI_GATEWAY_MODEL", "gpt-5-mini")
embeddings_model = os.environ.get("AI_GATEWAY_EMBEDDINGS_MODEL", "text-embedding-3-small")
question = sys.argv[1] if len(sys.argv) > 1 else "My laptop screen cracked. How do I get it fixed, and how long does it take?"

client = OpenAI(
    base_url=f"{gateway_url}/default/models/openai/v1",
    api_key=os.environ["AI_GATEWAY_API_KEY"],
)

ARTICLES = {
    "password": "Reset a forgotten password in the self-service password portal. You need the authenticator app "
    "on your phone. A locked account unlocks by itself after 15 minutes.",
    "laptop-repair": "Report a cracked screen or other laptop damage in the IT portal under Hardware > Repair. "
    "Contoso ships a loaner laptop within two business days, and the repair takes about a week.",
    "vpn": "Connect to the corporate network with the Global Secure Access client. If the client shows "
    "Disconnected, sign out of Windows and sign in again.",
    "new-hire": "New employees get their laptop on their first day. Their manager requests it in the IT portal at "
    "least five business days before the start date.",
    "printers": "Add a printer from Settings > Printers. Floor printers are named after the building and the floor, "
    "for example RED-3.",
    "new-phone": "Before you wipe an old phone, register the authenticator app on the new phone from the security "
    "info page.",
    "software": "Install approved software from Company Portal. Software that isn't listed needs a request with a "
    "business justification.",
    "expenses": "Submit travel expenses within 30 days in the finance portal, with a receipt for every item over "
    "25 USD.",
}


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


# 1. Index: embed every article in one batch. A real app stores these vectors in a vector index.
ids = list(ARTICLES)
indexed = client.embeddings.create(model=embeddings_model, input=[ARTICLES[i] for i in ids])
index = {i: item.embedding for i, item in zip(ids, indexed.data)}
print(f"Indexed:  {len(index)} articles with {embeddings_model}, {indexed.usage.total_tokens} tokens")

# 2. Retrieve: embed the question with the same model, and rank the articles by cosine similarity.
query = client.embeddings.create(model=embeddings_model, input=question).data[0].embedding
ranked = sorted(((cosine(query, vector), i) for i, vector in index.items()), reverse=True)
top = ranked[:2]
print(f"Question: {question}")
for score, article in top:
    print(f"Match:    {score:.3f}  {article}")

# 3. Generate: answer with the chat model, from the retrieved articles only.
context = "\n".join(f"[{article}] {ARTICLES[article]}" for _, article in top)
completion = client.chat.completions.create(
    model=chat_model,
    messages=[
        {
            "role": "system",
            "content": "Answer the question in at most two sentences, using only the articles below. "
            "Cite the article ID in square brackets. If the articles don't answer it, say so.\n\n" + context,
        },
        {"role": "user", "content": question},
    ],
)
print(f"Answer:   {completion.choices[0].message.content}")
print(f"Tokens:   {completion.usage.total_tokens} with {chat_model}" if completion.usage else "")
1

Ask a question

Bash or PowerShell
# Install the openai package that the app uses, if you haven't on the Code track
pip install -r infra/requirements.txt &&
  echo "📦 The OpenAI SDK is installed"

# Run the app with its default question: it embeds the articles and the question, then answers with gpt-5-mini
echo "🤖 Mini RAG through the gateway:"
python infra/rag.py

Expected output (your answer differs):

Text
🤖 Mini RAG through the gateway:
Indexed:  8 articles with text-embedding-3-small, 216 tokens
Question: My laptop screen cracked. How do I get it fixed, and how long does it take?
Match:    0.701  laptop-repair
Match:    0.298  new-hire
Answer:   Report a cracked screen in the IT portal under Hardware > Repair; Contoso ships a loaner within two business days and the repair takes about a week [laptop-repair].
Tokens:   373 with gpt-5-mini
2

Ask your own question

Bash or PowerShell
# Ask a question that no article answers: the app should say so instead of making something up
echo "🤖 Mini RAG through the gateway:"
python infra/rag.py "Can I bring my dog to the office?"
Checkpoint

The best match for the laptop question is laptop-repair, the answer cites [laptop-repair], and the question about dogs gets an answer that says the articles don't cover it.

Cap ingestion with a token limit

An ingestion job can embed thousands of chunks in a few minutes and use up the deployment's quota that the RAG app needs for its questions. Give the embedding model its own token limit: 2,000 tokens per minute for each caller identity, low enough to watch it work. In production, size it from the deployment's capacity, 20,000 tokens per minute in this lab, and the number of callers.

Choose your track
1

Add the token limit

  1. Select PoliciesAdd policy.

  2. On Type, select Token rate limit. Select Next.

  3. On Scope, select Individual assets, then select text-embedding-3-small — foundry-models, the row whose Type is Model. Select Next.

    Scope the token limit to the embedding model, so that every alias that routes to it shares the limit.
  4. On Details, keep Per minute as the Time period, set Allowed tokens to 2000, and keep Identity in Counter dimensions. Select Create.

    2,000 tokens per minute, with one counter for each caller identity.
Checkpoint

Policies lists 2,000 tokens / minute per caller identity, of type Token rate limit, on Model: text-embedding-3-small.

Verify

1

Call each alias with the wrong API

Each alias accepts only the endpoints of its backend models. A chat request to the embeddings alias, or a request for a model that the gateway doesn't know, fails at the gateway:

# Send a chat request to the embeddings alias, which supports only the embeddings endpoint
curl -sS "${AI_GATEWAY_URL%/}/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_EMBEDDINGS_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
  | jq -r '"🚦 \(.error.code // .statusCode): \(.error.message // .message)"'

# Ask for an embedding model that the gateway doesn't publish
curl -sS "${AI_GATEWAY_URL%/}/default/models/openai/v1/embeddings" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"text-embedding-3-large\",\"input\":\"ping\"}" \
  | jq -r '"🚦 \(.error.code // .statusCode): \(.error.message // .message)"'

Expected output:

Text
🚦 unsupported_endpoint: Model 'text-embedding-3-small' does not support the requested endpoint '/default/models/openai/v1/chat/completions'.
🚦 model_not_found: 'text-embedding-3-large' is not a known model alias.
2

Run an ingestion burst

infra/ingest.py sends eight batches of 20 short chunks, about 800 tokens each, with the SDK's retries turned off so that you see every response. The gateway applies a new token limit within about 30 seconds, so if you added it a moment ago, wait half a minute first.

infra/ingest.pyGitHub
"""Simulates a bulk ingestion job: it embeds the same batch of document chunks again and again through AI Gateway,
without retries, and prints the HTTP status, the tokens of each batch, and Retry-After. Use it to watch a gateway
token limit stop a runaway ingestion job before it uses up the embedding deployment's capacity.

Usage:
  pip install -r requirements.txt
  export AI_GATEWAY_URL="<gateway-url>"
  export AI_GATEWAY_EMBEDDINGS_MODEL="text-embedding-3-small"
  export AI_GATEWAY_API_KEY="..."      # never commit this value
  python ingest.py [batches]
"""

import os
import sys

from openai import OpenAI, RateLimitError

gateway_url = os.environ["AI_GATEWAY_URL"].rstrip("/")
model = os.environ.get("AI_GATEWAY_EMBEDDINGS_MODEL", "text-embedding-3-small")
batches = int(sys.argv[1]) if len(sys.argv) > 1 else 8

# max_retries=0 shows every 429. In a real ingestion job, keep the SDK's retries: they wait for Retry-After.
client = OpenAI(
    base_url=f"{gateway_url}/default/models/openai/v1",
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    max_retries=0,
)

chunk = (
    "AI Gateway publishes models from Microsoft Foundry and other providers behind one endpoint, with API keys, "
    "token limits, cost limits, and usage telemetry for every call. Chunk {n} of the product handbook."
)
batch = [chunk.format(n=n) for n in range(1, 21)]

total = 0
print(f"{'Batch':<7}{'Status':<8}{'Tokens':<8}{'Total':<8}Retry-After")
for number in range(1, batches + 1):
    try:
        result = client.embeddings.create(model=model, input=batch)
        total += result.usage.total_tokens
        print(f"{number:<7}{200:<8}{result.usage.total_tokens:<8}{total:<8}-")
    except RateLimitError as error:
        retry_after = error.response.headers.get("retry-after", "-")
        print(f"{number:<7}{error.status_code:<8}{'-':<8}{total:<8}{retry_after} s")
Bash or PowerShell
# Send eight batches of 20 chunks to the embeddings alias, and print the status and tokens of each one
echo "🚦 Expect 429 Too Many Requests once the 2,000-token allowance runs out:"
python infra/ingest.py

Expected output:

Text
🚦 Expect 429 Too Many Requests once the 2,000-token allowance runs out:
Batch  Status  Tokens  Total   Retry-After
1      200     800     800     -
2      200     800     1600    -
3      200     800     2400    -
4      429     -       2400    11 s
5      429     -       2400    10 s
6      429     -       2400    10 s
7      429     -       2400    10 s
8      429     -       2400    10 s

The gateway checks the counter before it forwards a request. The third batch starts at 1,600 tokens, under the limit, so it passes and takes the counter to 2,400. Every batch after it gets 429 with a Retry-After header until the minute is over. Keep the SDK's retries on in a real ingestion job: the OpenAI SDKs wait for Retry-After and send the batch again.

While the counter is over the limit, every request to the alias gets 429, even the chat request from the previous step, because the gateway applies the token limit before it checks the endpoint.

Checkpoint

The chat request to the embeddings alias got unsupported_endpoint, the unknown model got model_not_found, and the burst got 429 after about 2,000 tokens.

Go further

  • Change the embedding model behind the alias. Apps keep sending text-embedding-3-small, but vectors from another model, or with other dimensions, can't be compared with the ones in your index. Re-embed your documents with the new model first, then switch the alias. See model aliases.
  • Spread ingestion across deployments. An alias can route among several deployments of the same embedding model by priority and weight, and each request or batch goes to one of them. See Route an alias across models by priority and weight.
  • Watch ingestion in Monitoring. Token usage per caller, model, and alias shows in Monitoring once the gateway sends telemetry to Application Insights. See Monitor usage, cost, and traces.
  • Sign in without keys. Apps can call both aliases with a Microsoft Entra token instead of a key. See Go keyless with Microsoft Entra ID and RBAC.
  • Shrink the payload. "encoding_format": "base64" returns each vector as a base64 string of 4-byte floats, 8,192 characters for 1,536 numbers, instead of a long JSON array.

Clean up

Delete the embeddings alias and its backend model. The token limit goes with the model, and gpt-5-mini, its alias, and your key from lab 02 stay.

Choose your track
1

Delete the alias and the model

  1. Select ModelsAliases. On the text-embedding-3-small row, select More actions (⋮) › Delete, then Delete alias. Requests that use the alias fail from now on, and the backend model stays.
  2. Select Backend models, where text-embedding-3-small now shows -- under Aliases. Select its checkbox, then Delete selected items (the trash can) in the bar that says 1 item selected, then Delete.
Checkpoint

ModelsBackend models lists only gpt-5-mini, with its alias gpt-5-mini.

On the portal and CLI tracks, the Foundry deployment stays. Delete it when no other app uses it:

# Delete the embedding deployment from the Foundry account
az cognitiveservices account deployment delete -g "$FOUNDRY_RG" -n "$FOUNDRY_ACCOUNT" --deployment-name "$EMBEDDINGS" &&
  echo "🚮 Model deployment $EMBEDDINGS is deleted"

Keep the gateway, gpt-5-mini, and the key from lab 02: the next labs reuse them. azd down and terraform destroy don't touch them, because the Bicep + azd and Terraform tracks created their own gateways.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.