Examples on TauGrid

Live experiment evidence

Mon, 01 Jan 0001 00:00:00 +0000

This example runs the checked-in examples/portal-ray-stellar RayJob. It publishes immutable loss and accuracy chunks to durable storage while a metrics-offload sidecar projects the same scalar history to ADX/Kusto.

The run demonstrates three independent surfaces:

tau run status and logs for immediate execution.
tau run get for durable files on the workspace PVC.
Stellar for experiment comparison and scalar visualization.

Prerequisites

Prepare:

A Ready TauWorkspace named taugrid-default.
A writable blob-training PVC mounted at /data.
One schedulable NVIDIA GPU.
A digest-pinned taugrid-portal image supplied by the platform team.
Portal and the optional metrics ingestion path configured for the workspace.

The sidecar runs taugrid-portal experiment offload metrics, so use the Portal image rather than the Tau CLI image.

CPU queueing with Kueue and Ray

Mon, 01 Jan 0001 00:00:00 +0000

Use this example when you want to understand the scheduling layer beneath TauGrid using only the CPU capacity you already have. It submits raw RayJobs to Kueue and makes admission, pending work, and team borrowing visible.

This is a platform mechanics example that demonstrates Kueue/Ray scheduling directly, distinct from the TauGrid researcher workflow.

Prerequisites

a disposable or explicitly approved Kubernetes context;
Kueue and KubeRay installed;
kubectl access that can create namespaces, queues, and RayJobs; and
enough CPU capacity for two small Ray workloads.

Run the single-queue demonstration

git clone https://github.com/Azure/taugrid-examples.git
cd taugrid-examples/aks-blog/kueue-ray-cpu

./demo.sh single

The script applies the single-queue resources and enqueues two raw RayJobs. In a second terminal, watch admission:

Run GPU Ray Tune HPO on AKS

Mon, 01 Jan 0001 00:00:00 +0000

This guide runs the GPU HPO workflow for examples/ray-tune-smoke. It was verified on one Standard_NV12ads_A10_v5 node. A100 SKUs are also usable when the selected subscription and region have both sufficient quota and real-time allocation capacity.

The workflow creates billable Azure resources. Use a dedicated resource group. Record its owner, collect the required evidence, and delete the resources when the workflow is complete.

Ownership and completion gates

Complete these phases in order:

Provision a GPU-enabled TauGrid AKS environment

Mon, 01 Jan 0001 00:00:00 +0000

Use the repository’s terraform/aks root to create a GPU-enabled AKS environment. It provisions a system pool and a GPU pool, enables OIDC and Azure Workload Identity, and invokes the supported tau cluster install command.

tau cluster install installs the versioned TauGrid distribution that owns Kueue, KubeRay, the Tau controller, GPU monitoring, the baseline Kueue queue, and Portal; skip installing those components separately with Helm.

Prerequisites

an Azure subscription that passes the AKS cluster prerequisites, has GPU quota for the selected region and SKU, and an approved Terraform identity;
Azure CLI, Terraform 1.9 or later, kubectl, Helm, and local Python dependencies;
Azure credentials accepted by the AzureRM Terraform provider and permission to provision AKS, networking, storage, and identities; and
tau and PowerShell 7 on PATH. Linux and macOS users can configure the Terraform command interpreter to use Bash.

The default deployment creates one Standard_NC24ads_A100_v4 node. This node has one A100 80 GB GPU and is billable. Before applying, verify that the target region has capacity for the corresponding VM family. Change gpu_vm_size, gpu_count_per_node, and gpu_monitoring_sku_name together when selecting another GPU SKU.