Getting started on AKS

AKS is one supported way to reach the Kubernetes cluster that the Getting started on Kubernetes guide installs onto. This page compares the three ways to get there and links each one to its source runbook. Use it after deciding you want AKS specifically; if you already have a Kubernetes cluster of any kind, go straight back to step 2 of Getting started on Kubernetes.

Prerequisites

• An Azure subscription and the az CLI, logged in, with permission to create resource groups and AKS clusters (or an existing cluster you can already reach with kubectl).
• tau, kubectl, and helm on PATH; see Install the Tau CLI. tau cluster install shells out to helm internally.
• For the Terraform path: Terraform 1.9+ and Azure credentials usable by the AzureRM provider.
• For the GPU Terraform path: regional quota for the target GPU VM SKU.

Choose a path

1. Guided CPU quickstart (recommended for evaluation)

examples/aks-cpu-quickstart sequences resource-group creation, a CPU-only AKS cluster, tau cluster install, workspace creation, and a real PyTorch smoke workload, then tears everything down again. It is the fastest way to see TauGrid work end to end using only CPU quota.

Read its README.md first, then run:

./examples/aks-cpu-quickstart/run.sh

run.sh sequences only the commands documented in that README (az, tau, kubectl), keeping every step visible and reproducible. When you are done, tear the cluster and resource group down:

./examples/aks-cpu-quickstart/cleanup.sh

This path creates billable Azure resources (roughly $0.60–0.70/hour for the node pool). Budget for a 25–35 minute round trip and run cleanup.sh as soon as you finish; an idle cluster left running costs about $15/day. Confirm teardown with az group show --name taugrid-cpu-quickstart-rg, which should report ResourceGroupNotFound.

2. Reproducible GPU-capable Terraform

terraform/aks creates a GPU-enabled AKS environment (a system pool plus a GPU pool) and then invokes the same supported tau cluster install workflow. Use this path when your platform team wants to review, version, and re-apply the infrastructure instead of running a one-off script.

cd terraform/aks
terraform init
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars: subscription_id, resource_group_name, cluster_name
terraform plan -out=taugrid.tfplan
terraform show -no-color taugrid.tfplan
terraform apply taugrid.tfplan

The default GPU pool is one billable Standard_NC24ads_A100_v4 node. The tracked template also enables billable ADX, lifecycle recording, and Portal telemetry. Keep terraform.tfvars and taugrid.tfplan out of source control, and use an encrypted remote backend with locking for shared environments. PowerShell 7 is the default command_interpreter on every supported host OS; Linux, WSL, and macOS users can uncomment the Bash setting in the template to use Bash instead. See the full GPU-enabled AKS example for the complete workflow, including GPU stack modes, workspace bootstrap, ADX telemetry, and Portal. Destroy the environment with terraform destroy to stop GPU and ADX billing.

3. An existing AKS cluster

If your platform team already runs an AKS cluster, fetch its credentials and use it directly, skipping any provisioning step:

az aks get-credentials --resource-group <rg> --name <cluster-name>
export TAU_CONTEXT="<kubeconfig-context>"
kubectl --context "$TAU_CONTEXT" get nodes

Confirm the cluster meets the Kubernetes cluster prerequisites (Kubernetes 1.30+, GPU drivers and device plugins if workloads need GPUs, and a compatible CSI driver registered on every eligible node if workloads need PVCs), then continue with TauGrid installation.

AKS storage drivers and Flex nodes

Normal AKS agent pools receive AKS node metadata and bootstrap integration used by AKS-managed Azure CSI addon DaemonSets. TauGrid supports and expects those managed addons only on normal AKS-managed agent pools.

Externally registered nodes, including AKS Flex nodes, do not inherit that contract by default. TauGrid does not support relying on AKS-managed Azure CSI addons on Flex or other external nodes. This is not a claim that Azure storage can never run there. TauGrid helps platform operators set up a compatible deployment-managed CSI driver for Flex/external nodes; the operator still owns the storage account or backend, credentials, capacity, data lifecycle, and driver upgrades.

Use this setup path:

1. Select a CSI implementation whose vendor supports the external node’s operating system, container runtime, Kubernetes version, and storage backend.
2. Install its controller and node components using the driver’s documented installation method. Configure node selectors, affinities, and tolerations so the node component runs on every node eligible for the TauGrid workload; do not spoof AKS-managed node labels to schedule an AKS addon.
3. Create the driver’s StorageClass and PVC, then verify the PVC reaches Bound.
4. Verify the driver appears in CSIDriver and in the CSINode object for every eligible node.
5. Run a workload-profile-constrained mount/read/write canary using the same ServiceAccount and security settings as the customer workload.

TauGrid’s storage examples and validation commands provide the Kubernetes side of this setup. For Azure Files, start with the checked-in terraform/aks-flex-azurefile-csi root. It installs a distinctly named upstream driver on every Linux node by default and can be narrowed only to a label shared by every node eligible to mount the same PVC. It can also create a StorageClass that references a pre-created credential Secret without putting the account key in Terraform state. For managed RayJobs, eligibility includes head, CPU worker, and GPU worker node pools. Driver-specific identity, networking, and backend settings still require customer review.

A PVC can be Bound while a pod still fails before container startup with an event such as driver name file.csi.azure.com not found in the list of registered CSI drivers, or with file.csi.azure.com absent from the node’s CSINode object. See storage readiness for validation commands. Workloads that do not need persistence or offload can avoid this dependency by using pod-local ephemeral storage.

GPU software stack models

Whichever path creates or reaches your cluster, workload configs request standard Kubernetes nvidia.com/gpu resources in all three models. Platform configuration selects the owner of the driver, device plugin, and DCGM health exporter, plus the dcgmHealth source that TauGrid GPU monitoring scrapes:

The GPU Terraform root supports the first two rows through gpu_stack_mode; see the full GPU-enabled AKS example for the complete walkthrough of both. The third row applies when your platform team already runs GPU Operator on an existing AKS cluster. GPU Operator owns that software stack, and TauGrid consumes its DCGM exporter endpoint. See the GPU monitoring chart’s DCGM health sources documentation for the full dcgmHealth contract.

To use the AKS Managed GPU Experience preview, register the feature and confirm it before applying Terraform or creating the GPU pool:

az feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview
az feature show --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview --query properties.state --output tsv
az provider register --namespace Microsoft.ContainerService
az provider show --namespace Microsoft.ContainerService --query registrationState --output tsv

Validate

Whichever path you chose, confirm the cluster and control plane before creating a workspace:

kubectl --context "$TAU_CONTEXT" get nodes
tau cluster validate installation --context "$TAU_CONTEXT"

Continue with Create a workspace and Run a project workload in the Getting started on Kubernetes guide.