Azure AI Gateway Labs
Documentation Open portal
Models & routing · Lab 31New

Publish Jev decision models for typed answers

Publish TypeSafe's Jev decision model through the gateway with a Custom provider, triage support tickets with typed answers and confidences from curl, PowerShell, the TypeSafe Python SDK, and the portal playground, and cap decision requests per caller with a request rate limit.

Intermediate 25 min PortalCLIBicepTerraformCodeNotebook Updated
Notebook Source Edit

What you'll do

  • Explain when a decision model fits better than a chat model
  • Publish Jev through a Custom provider, a backend model with the typesafe/v1/systemone endpoint, and an alias
  • Call the alias with curl, PowerShell, and the TypeSafe Python SDK, and read the typed answers
  • Limit decision requests per caller and read the 429 response with its Retry-After header

Before you start

Tracks: Portal, Azure CLI, Bicep + azd, Terraform, Code, Notebook

Your support desk receives hundreds of tickets an hour, and each one must be routed: is it urgent, which team owns it, and how upset is the customer? Today a chat model reads every ticket and writes JSON that the triage app parses, which takes seconds per ticket, costs chat tokens, and sometimes returns text that the app can't parse. TypeSafe's Jev is a decision model: you send the ticket and typed questions, and it returns typed answers with probabilities in about half a second. In this lab you publish Jev through AI Gateway with a Custom provider, so that the triage app calls it with a gateway key instead of the TypeSafe key, triage tickets with curl, PowerShell, the TypeSafe SDK, and the portal, and cap the decision requests of each caller.

Before you begin

One-time setupThe Setup guide tasks that this lab uses. Mark each one done once, and the labs stop listing it.

Tip

On the Bicep + azd and Terraform tracks, azd up or terraform apply creates its own gateway, the TypeSafe provider, the Jev model, the alias with its rate limit, and a key, so you can skip this section. Run that track in Publish Jev behind an alias first: it sets the variables that the next sections and Verify use. You need only the one-time setup above, a TypeSafe API key, and Python 3.10 or later.

  • A gateway from Provision your AI Gateway. This lab doesn't need a Foundry account. Publish a Foundry model and make your first call explains aliases and runtime keys, if they're new to you.
  • A TypeSafe API key. Sign in to the TypeSafe console and create one. TypeSafe bills the Jev calls that you make in this lab to your TypeSafe account, at the prices on its Models page.
  • Permission to change the gateway, for example Contributor on its resource group.
  • Python 3.10 or later for the code track and Verify, PowerShell 7.4 or later for the PowerShell steps, and jq for the Bash steps.
  • Run the scripts and samples from the lab's folder, ~/aigateway/labs/31-jev-decisions.

Set these variables once. Every track and the Verify section reuse them.

# Name your gateway from lab 01 and the alias that apps send in "model"
export RG="<resource-group>"
export GATEWAY="<gateway>"
export AI_GATEWAY_MODEL="ticket-triage"
echo "📝 Alias $AI_GATEWAY_MODEL will publish Jev on gateway $GATEWAY in $RG"

# Read the gateway's resource ID and runtime URL, and build the URL of its System One endpoint
export GATEWAY_ID="$(az aigateway show -g "$RG" -n "$GATEWAY" --query id -o tsv)"
export AI_GATEWAY_URL="https://$(az aigateway show -g "$RG" -n "$GATEWAY" --query properties.frontend.defaultHostname -o tsv)"
export JEV_URL="$AI_GATEWAY_URL/default/models/typesafe/v1/systemone"
echo "🆔 Gateway: $GATEWAY_ID"
echo "🌐 Decisions URL: $JEV_URL"

How it works

A chat model writes text. A decision model answers questions. Jev, TypeSafe's first System One model, reads a state, such as a support ticket as text or JSON, and answers every question in the request with a typed value and probabilities. Your code branches on answers.team.choice instead of parsing prose. Use a decision model for classification, routing, scoring, and yes-or-no checks that run on every request, where a chat model costs more and takes longer. Keep a chat model for writing replies and for reasoning.

Question typeWhat it asksWhat it returns
noulIs this statement true?noul: the probability of yes, from 0 to 1
choiceWhich option fits? criteria names and describes the optionschoice, its confidence, and the probabilities of every option
scoreWhere does it sit on a rubric? criteria lists the levels from low to highscore, which can fall between levels, confidence, a legend, and probabilities

The gateway publishes Jev like any other provider's model, with three resources:

ResourceWhat it holds
Model provider typesafeType Custom, the base URL https://api.typesafe.ai, and exactly one authentication header, Authorization: Bearer <TypeSafe key>. The gateway stores the key and never returns it
Backend model jev-latestThe supported endpoint typesafe/v1/systemone, without a leading slash, and the model that TypeSafe runs. The portal shows the API format Decisions
Alias ticket-triageThe name that apps send in model, and the alias's policies, such as a request rate limit
Runtime detailValue
Decisions URLPOST $AI_GATEWAY_URL/default/models/typesafe/v1/systemone
TypeSafe SDK base_url$AI_GATEWAY_URL/default/models/typesafe, with compression=False. The SDK adds /v1/systemone
CredentialsA gateway API key, in Authorization: Bearer or in api-key, or a Microsoft Entra token. Never the TypeSafe key
BodyUncompressed JSON with the alias in model, the state, and the questions. A compressed body gets 415 with SystemOneEncodingUnsupported
usageinput_tokens and output_tokens, which token limits count

An alias can route across several Jev models by priority and weight, but it can't mix them with chat or embedding models, because they share no endpoint. In this lab, a ticket with three questions used about 490 input and 70 output tokens, and a call took about half a second once the model was warm. The first call can take several seconds.

Publish Jev behind an alias

Add TypeSafe as a Custom provider with your TypeSafe key, publish jev-latest under the alias ticket-triage, and create a runtime key for the triage app.

Choose your track
1

Add the TypeSafe provider

  1. Open the AI Gateway portal and select your gateway in the top bar.

  2. Select ModelsAddModels from TypeSafe (Jev).

    The Add menu has its own entry for TypeSafe's Jev models.
  3. On Provider, keep Create new provider. Set Display name to TypeSafe, Provider name to typesafe, and Description to TypeSafe Jev decision models. Select Next.

    The provider name is part of the backend model's resource ID and can't change later.
  4. On Endpoint, keep https://api.typesafe.ai as the Base endpoint URL, and paste your TypeSafe key in API key. The gateway sends it to TypeSafe as Authorization: Bearer <key>. Select Next.

    Paste the TypeSafe key itself. The gateway adds the Bearer prefix.
2

Choose the Jev model and its alias

  1. On Models, the wizard includes jev-latest and jev-preview, each with an alias named typesafe/<model>. In the jev-preview row, select …Delete.

  2. In the jev-latest row, select …Edit. Keep Create an alias selected, set Alias to ticket-triage, and select Save.

    Give the alias the name that the triage app sends in model.
  3. Select Add models.

    One backend model, one alias.

Select ModelsBackend models. jev-latest shows the alias ticket-triage, the API format Decisions, and the provider TypeSafe.

Jev models have the API format Decisions.

Select ticket-triage. On its Overview, EndpointsDecisions shows the Base URL for the TypeSafe SDK and the full Endpoint URL.

The alias's Decisions endpoint: the Base URL for the SDK and the Endpoint URL for HTTP calls.
3

Create a runtime key

  1. Select AccessAPI keysCreate API key.

  2. Set Key name to app-ticket-triage, and select Create.

    One key for the triage app, so that its requests have their own rate-limit counter.
  3. API key created shows the key once. Select Copy & done.

    Copy the key now. You can read it again on the API keys tab.
  4. Paste the key into your shell without showing it:

# Paste the app-ticket-triage key without showing it, and don't commit it or write it to a file
printf "AI Gateway API key: " && read -rs AI_GATEWAY_API_KEY && export AI_GATEWAY_API_KEY && echo
echo "🔑 Key: in AI_GATEWAY_API_KEY (${#AI_GATEWAY_API_KEY} characters, not shown)"
The triage app's key next to the gateway's built-in key.
Checkpoint

ModelsAliases lists ticket-triage, its Decisions endpoint ends with /default/models/typesafe/v1/systemone, and the shell says that the key has 32 characters.

Send a decision request

Send the first ticket from infra/ticket.json with three questions: is it urgent (noul), which team owns it (choice), and how frustrated the customer is (score). Two things differ from a direct call to TypeSafe: the URL is the gateway's, and the credential is your gateway key. model is the alias. New resources reach the runtime within a minute. If the first call returns 404 with Resource not found or model_not_found, wait a few seconds and try again.

Choose your track
1

Run decisions in the playground

  1. Select Catalog, and open ticket-triage. The playground shows Run decisions. On the left, EndpointsAPI format is Decisions. AuthenticationMethod is API key with the Built-in all-access API key, or Microsoft Entra ID if you have a data role on the gateway, such as AI Gateway Models User.

  2. In State, paste the ticket:

    Text
    Subject: Charged twice for invoice INV-2291
    Message: I was billed twice this month for the same invoice. Please refund the duplicate charge today, our finance team closes the books tomorrow.
    Plan: Business
    Channel: email
  3. Under Questions, set Question 1: Question ID urgent, Answer type Noul, and Instructions Does the customer need an answer today?

    The playground calls the alias's Decisions endpoint with the state and the questions.
  4. Select Add question, and set Question 2: Question ID team, Answer type Choice, and Instructions Which support team should own this ticket? Add three options, billing, technical, and account, with Add option.

  5. Select Run decisions.

Each answer comes back typed: a probability for the noul, a selected option with its confidence for the choice.

To send the lab's ticket with all three questions, turn on JSON mode next to Run decisions, replace the request with the contents of infra/ticket.json, and select Run decisions. The response adds frustration with Score: 1.

Checkpoint

The Response shows Reported model: jev-1.14.0, the input and output tokens, urgent with a high Probability of yes, and team with Selected: billing. The first call to a new model can take several seconds.

Limit decision requests per caller

Every decision costs money at TypeSafe, and a triage job stuck in a loop can send thousands. Give the alias a request rate limit: 5 decision requests per minute for each caller identity, low enough to watch it work. Each API key or Entra identity gets its own counter. In production, size it from your ticket volume.

Choose your track
1

Add the request rate limit

  1. On the alias's page, select PoliciesAdd policy.

  2. On Type, under Cost and rate limits, select Request rate limit. Select Next.

    A request rate limit counts calls, whatever their size.
  3. On Policy details, set Calls to 5, keep 60 seconds as the Renewal period, and keep Identity in Counter dimensions. Select Create.

    Five calls per minute, with one counter for each caller identity.
Checkpoint

Policies lists 5 requests / 60 seconds per caller identity, of type Request rate limit, with the scope This alias.

The rate limit applies to every caller of the alias, each with its own counter.

Verify

1

Call the alias the wrong way

The alias serves only the System One endpoint, and apps must send the alias, not the backend model's name:

# Send a Chat Completions request to the decision alias, which serves only the System One endpoint
curl -sS "$AI_GATEWAY_URL/default/models/openai/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$AI_GATEWAY_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Is this ticket urgent?\"}]}" |
  jq -r '"🚦 \(.error.code // .statusCode): \(.error.message // .message)"'

# Send the decision request with the backend model's name instead of the alias
jq -c '. + {model: "jev-latest"}' infra/ticket.json |
  curl -sS "$JEV_URL" -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" --data @- |
  jq -r '"🚦 \(.error.code // .statusCode): \(.error.message // .message)"'

Expected output:

Text
🚦 SystemOneRequestNotAdmitted: System One requires the native typesafe/v1/systemone endpoint.
🚦 model_not_found: 'jev-latest' is not a known model alias.
2

Send a burst of decisions

Wait a minute, so that your earlier calls don't count, then send eight small decisions in a row with the same key. The Bash loop reads the Retry-After header with curl 7.84 or later.

# Wait a minute, so that the earlier calls don't count against this burst
echo "⏳ Waiting 60 seconds for a fresh rate-limit window" && sleep 60

# Build one small request: the same ticket, with only the urgent question
BODY="$(jq -c --arg model "$AI_GATEWAY_MODEL" '{model: $model, state: .state, questions: {urgent: .questions.urgent}}' infra/ticket.json)"

# Send eight decision requests in a row, and print each status, with the Retry-After header of a 429
for i in $(seq 1 8); do
  read -r CODE WAIT <<< "$(curl -sS -o /dev/null -w '%{http_code} %header{retry-after}' "$JEV_URL" \
    -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H "Content-Type: application/json" -d "$BODY")"
  [ "$CODE" = 429 ] && echo "🚦 #$i: 429, retry after $WAIT s" || echo "✅ #$i: $CODE"
done

On the code track, the sample sends the same burst with the TypeSafe SDK, with its retries off:

Python
# Wait a minute, then send eight decision requests in a row with the TypeSafe SDK
sleep 60 && python infra/triage.py --burst 8

Expected output of the Bash loop (your Retry-After values differ):

Text
⏳ Waiting 60 seconds for a fresh rate-limit window
✅ #1: 200
✅ #2: 200
✅ #3: 200
✅ #4: 200
✅ #5: 200
✅ #6: 200
🚦 #7: 429, retry after 6 s
🚦 #8: 429, retry after 5 s

The gateway answers the requests over the limit itself, with 429 and the body Rate limit is exceeded. Try again in 6 seconds., so they never reach TypeSafe. In this lab's runs, a sixth request passed before the first 429. The SDK prints the same statuses, and with its default retries it waits for Retry-After and sends the request again.

Checkpoint

The Chat Completions request got SystemOneRequestNotAdmitted, the backend model's name got model_not_found, and the burst got 429 with a Retry-After header after five or six decisions.

The gateway records each decision request, with its tokens, for Monitoring once it sends telemetry to Application Insights, as in Monitor usage, cost, and traces.

Go further

  • Limit tokens instead of calls. A tokenLimit on the alias counts Jev's input and output tokens, as for chat. With 1,000 tokens per minute, two full triage requests of about 570 tokens passed, and the third got 429 with Token limit is exceeded. See Throttle tokens per caller with token limits.
  • Go keyless. Apps can call the alias with a Microsoft Entra token for the scope https://ai.gateway.azure.com/Data.All instead of a key, with the AI Gateway Models User role. See Go keyless with Microsoft Entra ID and RBAC.
  • Pin and switch Jev versions. Publish a pinned Jev version as a second backend model, and route the alias across both by priority and weight, or move the alias when you're ready. Apps keep sending ticket-triage. See Route an alias across models by priority and weight.
  • Escalate low-confidence answers. Send a ticket to a person or to a chat model when confidence is low, and act on the rest automatically. TypeSafe's confidence guide explains the values.
  • Use a compatible host. Point the provider's base URL at another System One-compatible host, including any path prefix, and give it that host's credential.

Clean up

Delete the alias, the backend model, the provider, and the runtime key. The provider takes your TypeSafe key with it.

Choose your track
1

Delete the alias, the model, the provider, and the key

  1. Select ModelsAliases. In the ticket-triage row, select More actionsDelete, and confirm with Delete alias.
  2. Select Backend models › jev-latest, then More actionsDelete, and confirm with Delete.
  3. Select Model providersTypeSafe, then More actionsDelete, and confirm.
  4. Select AccessAPI keys, select app-ticket-triage, then Delete selected items, and confirm with Delete.

Deleting the provider doesn't revoke your TypeSafe key. If you created it for this lab, delete it in the TypeSafe console. Keep the gateway from lab 01: the next labs reuse it.

Next steps

Finished the lab?

Mark it complete to track your progress across paths. It's saved only in this browser.