This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Deploy Kubeflow with Password, Ingress and TLS

1: Least-privilege deployment role
2: Manage Kubeflow users

Background

Use the Azure CLI and the main.bicep template in this repository to deploy an Azure Kubernetes Service (AKS) cluster, then install Kubeflow on it. Kubeflow is installed from a pinned community distribution release on a hostname you choose, served over publicly trusted HTTPS, with a generated Dex password that can be rotated.

Prerequisites

An Azure Subscription (e.g. Free or Student account)
A GitHub Account
Bash 4 or later, on Linux, macOS, Windows Subsystem for Linux (WSL), Multipass, or Azure Cloud Shell. The deployment uses mapfile, so the Bash 3.2 that macOS ships is too old; install a newer one with Homebrew

Install the following into that shell:

Azure CLI
just, which runs every deployment step in this repository
Bicep 0.46.1 exactly, the version versions.env records and just validate-static checks
Kustomize v5.8.1 exactly, because later versions can remove Kustomize APIs used by the pinned Kubeflow release
Kubectl v1.35, v1.36 or v1.37, within one minor version of the 1.36 API server this deployment creates
Kubelogin, the exec plugin that signs you in to the cluster
Go 1.26.0 or greater. tools/password/go.mod requests the go1.26.6 toolchain, which Go downloads on demand; install 1.26.6 itself if GOTOOLCHAIN is set to local
git
Python 3.11 or later, with venv and pip. just e2e builds a virtual environment under .cache/ and runs the release gate in it. On Debian and Ubuntu venv is a separate python3-venv package
jq, used to find pods that started before Istio could give them a sidecar
OpenSSL, used to read the serving certificate’s serial number
sed, used to render the Dex configuration
curl, tar, sha256sum, base64, tr, grep and awk, used to fetch and verify the Kubeflow release and to render its secrets
Network access to codeload.github.com, the Go module proxy, and PyPI

The deployment checks the Kustomize version and refuses to run on a mismatch, so install it with Go rather than a package manager:

go install sigs.k8s.io/kustomize/kustomize/v5@v5.8.1
kustomize version

The Azure CLI ships and updates kubectl and kubelogin for you. If either is missing, install or update both with:

az aks install-cli

For a deployment identity with only the permissions this deployment needs, see the least-privilege custom role.

Deploy AKS

Deploy the cluster with the main.bicep template in this repository.

The cluster uses Microsoft Entra ID with Azure RBAC for Kubernetes authorization, so kubectl access follows from a role assignment rather than from a shared credential. AKS still issues a cluster-local admin credential to callers holding listClusterAdminCredential, which bypasses Entra entirely; the least-privilege custom role deliberately withholds it.

Sign in to the Azure CLI, if you are not signed in already. An existing non-interactive session, such as a service principal in a pipeline, is fine and needs nothing here.

az login

Note

If you have access to multiple subscriptions, you may need to run the following command to work with the appropriate subscription: az account set --subscription <NAME_OR_ID_OF_SUBSCRIPTION>.

Clone this repository.

git clone https://github.com/Azure/kubeflow-aks.git
cd kubeflow-aks

Set up your environment variables. Every recipe below reads these.

export RESOURCE_GROUP=kubeflow
export AKS_NAME=kubeflow-aks
export LOCATION=eastus
export SIGNEDINUSER=$(az ad signed-in-user show --query id --out tsv)

SIGNEDINUSER is the object ID of the signed-in user. The deployment grants it the Azure Kubernetes Service RBAC Cluster Admin role on the cluster, which kubectl needs because the cluster authenticates with Microsoft Entra ID.

LOCATION selects the region for the resource group created below. The cluster inherits the region from the resource group, so a group that already exists keeps its own region regardless of this value.

Note

az ad signed-in-user show works only for an interactive user sign-in. Signed in as a service principal it fails with /me request is only valid with delegated authentication flow. Use the principal’s object ID instead, and say so, because the deployment otherwise records the role assignment against the wrong principal type:

export SIGNEDINUSER=$(az ad sp show --id "$(az account show --query user.name --out tsv)" --query id --out tsv)
export SIGNEDINUSER_TYPE=ServicePrincipal

For a service-principal session, az account show --query user.name returns the application ID the principal signed in with, which is what az ad sp show resolves to its object ID. SIGNEDINUSER_TYPE accepts User, Group or ServicePrincipal and defaults to User.

Create the resource group, if it does not exist already. Against a group that exists this is a no-op that returns it.

az group create -n $RESOURCE_GROUP -l $LOCATION

Create the cluster. just validate previews the same deployment without changing anything, after running just validate-static, which needs neither an Azure account nor a cluster and checks the pinned tool versions, the Go password tool, the release gate, and the rendered manifests.

just deploy-aks

Note

just deploy-aks creates the cluster on the Free pricing tier, so cluster management costs nothing and you pay as you go for the nodes and other resources the cluster consumes. The Free tier carries no financially backed uptime SLA. The system node pool has two nodes, which you can size with the nodeCount and vmSize template parameters. Its Kubernetes version comes from versions.env. The cluster must not enforce Azure Policy or the AKS managed admission policies, which reject this workload. AKS also creates a second resource group of its own, named MC_<resource group>_<cluster>_<region>, which holds the nodes and their supporting resources.

Connect to the cluster

After the cluster is created, you can connect to it using the Azure CLI. The following command retrieves the credentials for your AKS cluster and configures kubectl to use them.

just credentials

Verify connectivity to the cluster. This should return a list of nodes.

kubectl get nodes

Note

The cluster authenticates to the Kubernetes API with Microsoft Entra ID, so kubectl signs you in through the kubelogin exec plugin and the first command prompts you to sign in. On Kubernetes 1.24 and later, az aks get-credentials writes that exec-plugin format for you, so there is no kubelogin convert-kubeconfig step for an interactive sign-in. In a non-interactive context such as CI, or when signed in as a service principal, run kubelogin convert-kubeconfig -l azurecli first. What authorizes you either way is the Azure Kubernetes Service RBAC Cluster Admin role assignment the deployment created for $SIGNEDINUSER.

Install Kubeflow

Run the deployment from the repository root, with the same environment variables exported above still set. No hostname needs choosing: the Bicep deployment generated one, and Kubeflow is served on it over publicly trusted HTTPS.

just deploy-kubeflow

The hostname is kubeflow-<unique>.<location>.cloudapp.azure.com, derived from the resource group, the cluster name and the location.

Note

Errors like failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": no endpoints available for service "kserve-webhook-server-service" can appear while the deployment runs. They mean a controller is not serving its admission webhook yet. The recipe prints Retrying to apply resources, waits twenty seconds and applies the manifests again until they stick, so a healthy install prints several of these. What tells you the deployment failed is just deploy-kubeflow exiting non-zero, not errors in its output.

Warning

Save the password just deploy-kubeflow prints. It is shown once and is never written to a file.

Wait for the deployment to settle, then check it:

just wait-ready
just e2e

just wait-ready waits for every pod, for the TLS certificate to be issued, and for Dex and OAuth2 Proxy to roll out. It also fails a pod that is Ready but has no Istio sidecar, which readiness alone cannot detect and which leaves large parts of the platform returning 503 through the ingress.

just e2e then exercises the deployment the way a user does. It reads the hostname from the certificate rather than taking one from the environment, signs in to Dex over publicly trusted HTTPS, loads the dashboard, creates a Notebook, waits for it to become ready, lists it through the Jupyter Web App API, opens JupyterLab, and deletes it again. It prints one PASS line per check and verifies certificates normally throughout, with no way to turn that off. It asks for the Dex password, or reads it from DEX_PASSWORD:

DEX_PASSWORD='the password just deploy-kubeflow printed' just e2e

It then forces the certificate to be reissued, to prove that ACME renewal still reaches the cluster through Istio. Nothing else can prove that: a renewal that has quietly stopped working looks identical to one that works until the certificate expires.

Warning

The reissuance deletes the TLS secret, so the site serves no certificate for the minute or two the new one takes to issue. Let’s Encrypt also issues at most five duplicate certificates per week, and each run uses one. Set E2E_SKIP_REISSUANCE=1 to run only the functional checks.

Note

just wait-ready allows fifteen minutes for the certificate. If it gives up there, look at the ACME challenge:

kubectl get challenge --namespace istio-system

A challenge still pending while the challenge URL answers from outside the cluster means cert-manager has stopped making progress rather than that anything is unreachable. Restarting its controller recovers it:

kubectl delete pod --namespace cert-manager \
    --selector app.kubernetes.io/component=controller

The certificate usually issues within a minute. Re-run just wait-ready.

Open the printed https:// URL and sign in as user@example.com.

Choosing the Azure hostname

To pick the Azure DNS label rather than accept the generated one, set DNS_LABEL before deploying. It has to be unique within the region.

export DNS_LABEL=my-unique-kubeflow-label

just deploy-kubeflow

The hostname becomes $DNS_LABEL.$LOCATION.cloudapp.azure.com.

Serving Kubeflow on your own domain

To serve Kubeflow on a domain you control, set DOMAIN to a lower-case FQDN. It becomes the certificate and login hostname, and DNS_LABEL selects the Azure name you point it at.

export DOMAIN=kubeflow.example.com

just deploy-kubeflow

Unlike the other two, this path does not complete in one command. just deploy-kubeflow stops and prints the unproxied DNS record to create. Create it, let it resolve publicly, then continue:

just wait-ready
just e2e

The record must send /.well-known/acme-challenge/ on port 80 straight to the Istio ingress. Do not put it behind a proxy or provider-side forced HTTPS: Let’s Encrypt renews a 90-day certificate after roughly 60 days, so an interception added later can leave a working deployment unable to renew. That path is the only one exempt from Kubeflow’s OAuth2 and JWT checks; all other unauthenticated HTTP traffic is denied.

Rotate the Dex password

just configure-dex

This generates a new password and cost-12 bcrypt hash, validates the hash before using it, replaces the dex-passwords Secret, restarts Dex, refreshes the RequestAuthentication JWKS URI so Istio picks up the new signing keys, and waits for the Dex and oauth2-proxy rollouts. Each run invalidates the previous password.

Warning

For production, integrate Dex with an external identity provider instead of relying on a shared static account. For password rotation and additional static users, see Manage Kubeflow users.

Clean up

Remove everything the deployment created, keeping the resource group and the role assignments scoped to it:

just group-empty

This deploys an empty template in Complete mode, which deletes every resource in the group. It prints the group it is about to empty and waits ten seconds first.

Warning

Check RESOURCE_GROUP before running this. It empties whichever group that variable names, and the resources are not recoverable.

Deleting the resource group itself also works, if you created it. It is worth knowing what that costs: deleting a group removes every role assignment scoped to it along with the resources. Where an administrator created the group and granted access to it, including through the least-privilege custom role, deleting the group destroys that grant. The custom role deliberately excludes resource-group deletion for this reason, so empty the group instead and leave removing it to whoever created it.

1 - Least-privilege deployment role

The deployment identity can use a custom role scoped to an existing target resource group. Creating that resource group, defining the custom role, and assigning it are bootstrap operations performed by an administrator. The deployment identity needs these management-plane actions:

[
  "Microsoft.Resources/subscriptions/resourceGroups/read",
  "Microsoft.Resources/deployments/read",
  "Microsoft.Resources/deployments/write",
  "Microsoft.Resources/deployments/validate/action",
  "Microsoft.Resources/deployments/whatIf/action",
  "Microsoft.Resources/deployments/operations/read",
  "Microsoft.ContainerService/managedClusters/read",
  "Microsoft.ContainerService/managedClusters/write",
  "Microsoft.ContainerService/managedClusters/listClusterUserCredential/action",
  "Microsoft.Authorization/roleAssignments/read",
  "Microsoft.Authorization/roleAssignments/write"
]

These permissions let just validate, just deploy-aks, and just credentials run, and let main.bicep grant the selected principal Azure Kubernetes Service RBAC Cluster Admin on the new cluster. No policy-assignment permission is required. Resource-group deletion remains an administrator cleanup operation and is deliberately outside this role.

Because Microsoft.Authorization/roleAssignments/write can otherwise assign any role at the resource-group scope, assign the custom role with an Azure RBAC condition that permits only the Azure Kubernetes Service RBAC Cluster Admin role used by main.bicep:

(
  (!(ActionMatches{'Microsoft.Authorization/roleAssignments/write'}))
  OR
  (
    @Request[Microsoft.Authorization/roleAssignments:RoleDefinitionId]
      ForAnyOfAnyValues:GuidEquals
      {b1ff04bb-8a4e-4dc4-8eb5-8693973ce19b}
  )
)

Set the condition version to 2.0. This constrained delegation prevents the deployment identity from using its role-assignment permission to grant Owner, User Access Administrator, or any other role.

2 - Manage Kubeflow users

Background

Change the Dex password and add more static users. Integrating Dex with Microsoft Entra ID is not covered here.

Change the Dex password

The deployment no longer ships a default password: rendered manifests carry an unusable sentinel, and just deploy-kubeflow generates a real one at install time. To replace it at any point, run:

just configure-dex

That generates a 32-character password and its cost-12 bcrypt hash, validates the hash, replaces the dex-passwords Secret, restarts Dex, refreshes the Istio RequestAuthentication JWKS URI, and prints the password once. To generate a password and hash without touching the cluster, run just password.

Warning

Do not paste a password into an online bcrypt generator. just password produces the hash locally from a cryptographically secure source.

To add more users, edit overlay/patches/dex-config.yaml.in, which patches Dex’s ConfigMap when the overlay is rendered. Keep using hashFromEnv so no hash is committed:

    staticPasswords:
    - email: user@example.com
      hashFromEnv: DEX_USER_PASSWORD
      username: user
      userID: "15841185641784"
    - email: user2@example.com
      hashFromEnv: DEX_USER2_PASSWORD
      username: user2
      userID: "15841185641785"

Generate the second user’s credentials, saving the printed password before continuing:

just password

Add its hash to the same Secret, then re-render and restart Dex:

kubectl patch secret dex-passwords -n auth --type=json \
  -p='[{"op":"add","path":"/data/DEX_USER2_PASSWORD","value":"'"$(printf %s '<paste-the-hash>' | base64 | tr -d '\n')"'"}]'
just deploy-kubeflow