TauGrid

CLI reference

Mon, 01 Jan 0001 00:00:00 +0000

The Tau CLI has exactly eight public command roots. Every example in this site uses one of these roots and its current nested subcommand path:

Root	Purpose
`tau cluster`	Install/uninstall TauGrid and validate an existing cluster
`tau workspace`	Create/adopt workspaces and inspect readiness, quota, and repo scaffolds
`tau run`	Config-first submit, lifecycle, and retry/resume for one run
`tau logs`	Find a run through configured Tau workspaces and stream its logs
`tau serve`	Render and manage serving deployments
`tau data`	Inspect dataset and model registries
`tau python`	Optionally author decorator-based `@tau.train` / `@tau.eval` workflows; standard `tau.yaml` projects do not need the Python SDK
`tau version`	Print the installed CLI version

Run evidence and the observability portal ship in a separate binary, taugrid-portal. See taugrid-portal experiment below.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

Start with the repository AGENTS.md and SDK_GUIDE.md.

Core principles:

Start from the user path and compatibility contract.
Keep commands thin and capability packages cohesive.
Reuse existing helpers and dependencies.
Keep durable local formats inspectable and versioned.
Preserve expstore as local truth.
Make retry, checkpoint, and partial-write behavior explicit.
Add tests at the owning layer.

External contract changes need external evidence. A manifest or telemetry change needs evidence beyond a utility unit test.

Make a compatible change

Mon, 01 Jan 0001 00:00:00 +0000

Start from the user path and compatibility contract.
Keep Cobra command files focused on wiring, flags, validation, and output.
Put reusable behavior in the owning internal/* capability package.
Preserve inspectable, versioned durable formats.
Keep expstore authoritative and hosted analytics downstream.
Test behavior at the layer that owns it.

Run focused package tests while iterating. Before TauGrid-internal closeout:

cd cli
go test ./...

Manifest, runtime, telemetry, or Kubernetes lifecycle changes also require the smallest safe end-to-end path that proves the external contract. Use the Kind smoke test when the changed contract belongs to the portable Kubernetes path.

Run your first target

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites:

The Tau CLI is installed.
The repository contains a workspace connection and a checked-in target, such as tau/train.yaml.
The platform workspace reports Ready.

tau run is the config-first entry point. Its optional positional TARGET argument resolves to a checked-in tau/<target>.yaml file. See the direct run config reference for the full tau.yaml field set. tau run automatically discovers the checked-in workspace connection.

Validate before submitting a real workload:

TauGrid object model

Mon, 01 Jan 0001 00:00:00 +0000

Full definitions for these terms live in the glossary. This page only explains how they relate and who owns what.

Concept	Owns
Repository	Code and history in one Git worktree
Project	One or more targets and a workspace connection
Workspace	Destination policy: cluster, namespace, queue, priority, output root, identity
Target	One checked-in runnable config
Run	One execution plus its lifecycle handle
Workload	Rendered Job or RayJob execution intent
Service	Online RayService or Deployment lifecycle target
Experiment	A comparison set over runs, metrics, and artifacts

The ownership split is deliberate:

What is Ray?

Mon, 01 Jan 0001 00:00:00 +0000

Ray is an open-source framework for running Python work across processes and machines. Teams use it for distributed training, hyperparameter tuning, data processing, and model serving.

KubeRay brings Ray to Kubernetes. Its operator manages resources such as RayJob and RayService, creates the Ray head and worker pods, and follows the Ray workload lifecycle.

How TauGrid uses Ray

TauGrid turns a repository target into either a Kubernetes Job or a KubeRay RayJob. A Ray workload follows this path:

What is TauGrid?

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid is an open, self-hosted AI compute platform for running AI workloads on Kubernetes. It brings the experience teams expect from a managed AI platform (simple submission, elastic distributed compute, shared GPU capacity, experiment visibility, and operational controls) to clusters they operate in their own cloud or datacenter.

Researchers work from a repository and the Tau CLI. Platform teams provide governed workspaces, queues, compute profiles, storage, identity, and observability. TauGrid connects those two experiences across training, fine-tuning, batch inference, and serving.

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid is a CLI, renderer, and local observer. Kueue, KubeRay, and the Kubernetes scheduler continue to own quota admission, Ray orchestration, and pod scheduling.

Layer	Responsibility
Research repository	Entrypoint, image, dependencies, resource intent, data, and outputs
TauGrid	Resolve, validate, render, submit, observe, and preserve workflow evidence
Workspace/platform	Cluster access, namespace, queue, priority, output root, identity, and shared policy
Kueue	Quota reservation, admission, priority, and preemption
Kubernetes/KubeRay	Pod scheduling and Job/Ray lifecycle
Project process	Training, evaluation, inference, and model-specific behavior

Code boundaries

cli/cmd: thin Cobra command wiring for the tau binary.
cli/internal: reusable capability packages behind tau.
core: the shared library module both binaries link.
portal: a separate module and binary, taugrid-portal, that hosts experiment tracking (Stellar) and the observability portal.
sdk/python: optional Python authoring APIs that delegate execution to the Go CLI.
controllers/tau-core: the separately deployed controller. One manager runs three reconcilers – TauCluster, TauWorkspace, and TauQuotaRequest.

The visual architecture guide maps the main packages and extension points.

Run the Kind smoke test

Mon, 01 Jan 0001 00:00:00 +0000

Use this contributor smoke test for changes to manifest rendering, submission, Kueue admission, KubeRay lifecycle handling, or the portable Kubernetes path. Kind supplies a disposable local Kubernetes API and Nodes for portable-path validation, distinct from an AKS or provider-integration acceptance test.

From the repository root, local dry-run verifies the checked-in fixture and rendering contracts entirely offline, with no cluster required:

make -C cli build
cli/bin/tau run validate --config examples/kind-smoke/tau.yaml
cli/bin/tau run --config examples/kind-smoke/tau.yaml --dry-run=client

The full smoke test requires Kind, kubectl, Helm, and a healthy Docker or Podman engine with at least 8 GiB of memory. Run:

Live experiment evidence

Mon, 01 Jan 0001 00:00:00 +0000

This example runs the checked-in examples/portal-ray-stellar RayJob. It publishes immutable loss and accuracy chunks to durable storage while a metrics-offload sidecar projects the same scalar history to ADX/Kusto.

The run demonstrates three independent surfaces:

tau run status and logs for immediate execution.
tau run get for durable files on the workspace PVC.
Stellar for experiment comparison and scalar visualization.

Prerequisites

Prepare:

A Ready TauWorkspace named taugrid-default.
A writable blob-training PVC mounted at /data.
One schedulable NVIDIA GPU.
A digest-pinned taugrid-portal image supplied by the platform team.
Portal and the optional metrics ingestion path configured for the workspace.

The sidecar runs taugrid-portal experiment offload metrics, so use the Portal image rather than the Tau CLI image.

Releasing TauGrid

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid release preparation updates:

SDK version.
Installation documentation.
cli/releases/vX.Y.Z.md.
The reviewed source commit on main.

Maintainers then create an annotated SemVer tag and manually dispatch the release workflow. The workflow builds twice, compares outputs, verifies checksums, publishes immutable assets, and proves clean bootstrap.

Never overwrite an existing release or asset. Do not update downstream minimum versions until post-publication Ubuntu and macOS verification succeeds.

Follow the exact RELEASING.md checklist.

Run and workload lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

A TauGrid run crosses independent control planes:

Resolution: TauGrid finds the project, workspace, and target.
Rendering: TauGrid validates intent and builds a Job or RayJob.
Admission: Kueue reserves quota and admits the workload.
Scheduling: Kubernetes places pods on matching nodes and devices.
Runtime: Ray or the Job controller starts the application.
Progress: The model or data process advances and writes evidence.
Completion: Results, checkpoints, metrics, and terminal state remain inspectable.

These states are not equivalent:

Run config reference

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the direct run config: the normal, hand-written tau.yaml that tau run --config reads to submit a workload. See direct run config vs. managed workflow manifest for how this differs from the SDK-generated, schema_version: 1 manifest that tau.train/tau.serve render. Both can use a tau.yaml filename; inspect the schema, not the filename. Use the managed workflow shape only when TauGrid needs to own workflow semantics such as staged train/eval lineage.

Share research with a teammate

Mon, 01 Jan 0001 00:00:00 +0000

Researchers working from different machines can use the same cluster and workspace to view and compare the same runs, logs, and charts.

Access model used in this guide

This guide describes the current interim admin-access workflow. The cluster owner gives each researcher their own cluster-admin kubeconfig. Each researcher keeps those credentials on their own machine and uses a separate port-forward to the shared Portal Service.

What is Kueue?

Mon, 01 Jan 0001 00:00:00 +0000

Kueue is a Kubernetes-native queue and admission system for batch and AI workloads. It decides when a workload may consume shared quota. After admission, the Kubernetes scheduler decides which nodes run the pods.

The main Kueue objects

A LocalQueue is the queue a workspace submits to.
A ClusterQueue combines quota across one or more LocalQueues.
A ResourceFlavor describes a class of capacity, such as CPU nodes or a particular GPU class.
A Workload records the resources a Job or RayJob requests and its admission state.

TauGrid resolves the workspace LocalQueue and adds it to the rendered workload. Kueue then evaluates quota, priority, queue order, and resource flavors.

Workspaces

Mon, 01 Jan 0001 00:00:00 +0000

A workspace connects repository intent to one cluster namespace, LocalQueue, workload ServiceAccount, and authorization contract.

Single active workspace

TauGrid v0 activates one workspace per cluster. The CLI resolves that workspace from the checked-in connection descriptor or the cluster’s primary-workspace marker, so researchers can omit the workspace name for normal runs.

The active workspace must be Ready before TauGrid submits a workload. Readiness confirms the namespace, queue, workload identity configuration, and authorization contract described in the TauWorkspace reference.

Cluster install values

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the Helm values accepted by tau cluster install. The distribution chart bundles Kueue, KubeRay, the Tau core controller, GPU monitoring, the taugrid-core services chart, and a portable baseline queue into a single versioned release. Portal is enabled by default for the operator quickstart; Stellar, lifecycle recorder, and image prewarm remain disabled until the platform opts in.

Print this reference from your terminal:

Configuration resolution

Mon, 01 Jan 0001 00:00:00 +0000

The normal researcher contract is a checked-in direct run config:

name: train
engine: rayjob
entrypoint: train.py

compute:
 workers: 2
 gpus_per_worker: 8

storage:
 data_pvc: training-data

TauGrid combines:

Explicit project and target selection.
Repository or monorepo discovery.
The project’s workspace connection descriptor.
Platform-owned workspace defaults.
Checked-in workload intent.
Temporary explicit operator overrides.

Ambiguity is an error. Dry-run output must keep default sources visible.

tau run validate --config tau/train.yaml
tau run train --dry-run=client
tau run explain-config

See direct run config vs. managed workflow manifest for how Python SDK-generated manifests relate to this default format.

Serve a trained model

Mon, 01 Jan 0001 00:00:00 +0000

tau serve turns a project-owned image and optional durable checkpoint into a service. Choose:

--kind=rayservice (default) for a Ray Serve application; KubeRay must be installed and the image must expose the specified Python import path.
--kind=deployment for a plain Kubernetes Deployment such as a raw vLLM, TGI, Triton, or custom HTTP server.

You need a platform-provided serving profile, a pinned image, the target namespace/context, and a checkpoint visible from the serving PVC when the endpoint loads model state.

Site and wiki parity

Mon, 01 Jan 0001 00:00:00 +0000

The TauGrid documentation site and GitHub Wiki are both maintained as complete surfaces. They are edited separately, so semantic parity is a human ownership responsibility rather than an automated synchronization claim.

site/data/wiki-parity.toml records:

The maintained site page and wiki counterpart.
The owning team.
Current maturity.
Last joint review date.

The documentation pipeline validates that mapped site pages exist and metadata is current whenever the site changes. Its consolidated Azure DevOps pipeline also runs the same parity and review-age checks on the Monday parity-only schedule.

TauWorkspace reference

Mon, 01 Jan 0001 00:00:00 +0000

TauWorkspace is a platform-authored Kubernetes resource in the TauGrid system namespace (tau-system by default). A custom tau cluster install --namespace <name> moves the controller and these objects together; administrative workspace commands use --system-namespace <name>, and repository descriptors persist the value as cluster.systemNamespace. It reconciles or verifies:

Target namespace and namespaced RBAC.
Kueue LocalQueue accessibility.
Workload identity ServiceAccount configuration.
Readiness and drift conditions.

Platform teams provision Azure resources and cluster-scoped quota policy, projects own their own code, and platform teams verify the durable storage PVC separately.

What is adx-mon?

Mon, 01 Jan 0001 00:00:00 +0000

adx-mon is Azure Data Explorer Monitor, an observability pipeline for Kubernetes clusters. It collects metrics, logs, and GPU telemetry, batches the data, and sends it to Azure Data Explorer (ADX/Kusto).

Its main components collect node and cluster signals, ingest batches into ADX, manage ADX schema commands, and evaluate configured alerts. Platform teams choose the signals, databases, retention, identities, and access rules.

How TauGrid uses adx-mon

adx-mon is an optional platform integration. A TauGrid environment can use it to:

Release contract

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid uses canonical annotated SemVer tags (vX.Y.Z) and a manually authorized GitHub Actions workflow. Publishing a release requires a manually authorized workflow run beyond the tag push itself.

The workflow:

Verifies the annotated tag and reviewed main commit.
Requires checked-in release notes.
Runs Go and Python release gates.
Compares two independent binary builds.
Builds the Python SDK wheel twice and requires byte-for-byte identical output.
Publishes raw binaries, the SDK wheel, install.sh, LICENSE, and SHA256SUMS as new assets, leaving any existing ones untouched.
Verifies every uploaded digest.
Proves CLI and SDK installation on clean Ubuntu and macOS runners.

The Python SDK keeps its own package version, but its source-aligned tau-*.whl is published in the same GitHub Release as the CLI.

What is Stellar?

Mon, 01 Jan 0001 00:00:00 +0000

Stellar is TauGrid’s experiment tracking and comparison experience. It ships in the taugrid-portal binary and presents run details, scalar metric history, and side-by-side comparisons in a browser or terminal.

What Stellar helps answer

Which code and settings produced this run?
How did loss, accuracy, throughput, or another metric change over time?
Which run performed best?
Where are the model, checkpoint, and other output files?
Did the run finish, retry, or resume?

Where the data comes from

TauGrid saves complete metric files and a small experiment index close to the run. Stellar can read that saved evidence directly. A platform can also provide an ADX/Kusto view for searching and comparing runs across workspaces.

Experiment evidence and artifacts

Mon, 01 Jan 0001 00:00:00 +0000

Every run should save enough information for you to answer four questions:

What code and settings produced this result?
Did the model improve during the run?
Where are the checkpoint and output files?
Can I compare, reproduce, or resume this run later?

What TauGrid saves

Saved data	Examples	Where it is saved
Run details	Project, run ID, config, and input references	TauGrid experiment index
Metrics over time	Loss, accuracy, throughput, and GPU use	Metric files
Metric summary	First and latest step, minimum, maximum, and point count	TauGrid experiment index
Output files	Checkpoints, models, images, tables, profiles, and reports	The directory set by `storage.output`
Run history	Attempts, retries, resumes, and final status	TauGrid experiment index and Kubernetes workload

The output files stay on the storage chosen by the platform team. TauGrid keeps an index so you can find and compare them.

TauGrid visual curriculum

Mon, 01 Jan 0001 00:00:00 +0000

Each diagram answers one reader question and preserves an editable source, SVG, and PNG. Solid elements represent GA capabilities. Orange dashed elements represent Alpha or Beta capabilities. Purple dashed elements represent Planned work.

Understand and evaluate

TauGrid in one picture What is TauGrid, and what is it not? SVG PNG Editable source

Why TauGrid What repeated friction does TauGrid remove? SVG PNG Editable source

Choosing the right front door When should I use TauGrid versus adjacent approaches? SVG PNG Editable source

TauGrid personas Who uses TauGrid? SVG PNG Editable source

Capability map What can TauGrid do today? SVG PNG Editable source

Adopt and configure

Adoption paths How can I try or deploy TauGrid? SVG PNG Editable source

TauGrid quickstart flow SVG PNG Editable source

Identity and security trust chains Who is the researcher and what may the workload access? SVG PNG Editable source

TauWorkspace reconciliation What does TauWorkspace reconcile? SVG PNG Editable source

Azure resource blueprint Which Azure resources support a TauGrid-enabled cluster? SVG PNG Editable source

Use TauGrid day to day

Researcher interaction journey How does a researcher submit, observe, retrieve, and iterate? SVG PNG Editable source

Workload chooser Which TauGrid path should I use? SVG PNG Editable source

Repository and config resolution How does repository intent become workload intent? SVG PNG Editable source

Data and artifact lifecycle Where do inputs and outputs live? SVG PNG Editable source

Experiment evidence flow How does Stellar remain local first? SVG PNG Editable source

Failure recovery What happens when a run is preempted or fails? SVG PNG Editable source

Training-to-serving direction How could a successful experiment become an endpoint? SVG PNG Editable source

Operate TauGrid

Queue and GPU placement Why is my run waiting, and where will it execute? SVG PNG Editable source

Troubleshooting layers Where is a stuck run blocked? SVG PNG Editable source

Observability and evidence Which signals prove platform and workload health? SVG PNG Editable source

Experimental multi-cluster execution How can TauGrid route work across clusters? SVG PNG Editable source

Extend and contribute

Code architecture and extension points Where should a contributor make a change? SVG PNG Editable source

Contribution-to-release path How is a contribution validated and shipped? SVG PNG Editable source

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This page is the canonical source for TauGrid terminology. Other concept and reference pages link to a definition here instead of redefining it. Where a term below conflicts with older prose elsewhere, this definition wins.

TauGrid

The CLI (and its companion Python SDK) that resolves a project’s checked-in workload contract, applies platform-owned policy, renders a Kubernetes Job or KubeRay RayJob, submits it, and gives the researcher one lifecycle surface for status, logs, results, retry, and resume. TauGrid resolves, renders, and observes workload lifecycle; Kueue and Kubernetes own scheduling and orchestration, and platform teams provision cloud infrastructure. See What is TauGrid and Architecture.

Getting started on Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid is a Kubernetes-native workflow layer for AI workloads. It installs on a Kubernetes cluster your platform provisions and operates. This is the canonical, provider-neutral installation path.

This page takes a platform operator from an empty workstation to a validated TauGrid workspace and a completed smoke workload. The portable path works on Kubernetes 1.30 or newer. Provider-specific identity, networking, storage, GPU drivers, and cluster lifecycle remain owned by your platform.

Enable a workspace

Mon, 01 Jan 0001 00:00:00 +0000

Before researchers can submit workloads, the platform team prepares the cluster and creates a TauWorkspace. TauGrid then keeps the workspace namespace, queue access, and Kubernetes permissions in the expected state.

Prepare the workspace

Prepare the Kubernetes cluster. Set up the network, node pools, identity, container registry, storage, GPU support, and CSI drivers required by your workloads.
Install TauGrid, then verify the installation:
```
tau cluster validate installation --context <context>
```
A successful validation exits with code 0.

Troubleshoot a run

Mon, 01 Jan 0001 00:00:00 +0000

Start with:

tau run status <run-name>

tau run status is the canonical lifecycle view for any run – read its phases top to bottom and stop at the first one that is not done. Then work through the layers below, in order, until you reach that phase:

Repository/connection resolution and cluster access – tau workspace connection (--offline for local configuration only).
TauWorkspace readiness and handoff validity – tau workspace status <name>.
Client-side config validation and rendering – tau run validate --config tau/train.yaml.
Queue admission and quota – the Kueue admission phase in tau run status.
Kubernetes scheduling, DRA, image pull, init, and readiness – the remaining phases in tau run status.
GPU/node/topology health – tau cluster validate nodes / tau cluster validate topology.
Ray/Job runtime progress and durable evidence – tau logs <run-name> and taugrid-portal experiment status <name>.
Recovery handoff – Retry and resume.

Confirm the queue admitted the workload (layer 4) before moving to GPU or node debugging (layer 6). Treat a Running pod phase (layer 5) as evidence of container start alone – confirm separately that the model or data process is making progress (layer 7).

CPU queueing with Kueue and Ray

Mon, 01 Jan 0001 00:00:00 +0000

Use this example when you want to understand the scheduling layer beneath TauGrid using only the CPU capacity you already have. It submits raw RayJobs to Kueue and makes admission, pending work, and team borrowing visible.

This is a platform mechanics example that demonstrates Kueue/Ray scheduling directly, distinct from the TauGrid researcher workflow.

Prerequisites

a disposable or explicitly approved Kubernetes context;
Kueue and KubeRay installed;
kubectl access that can create namespaces, queues, and RayJobs; and
enough CPU capacity for two small Ray workloads.

Run the single-queue demonstration

git clone https://github.com/Azure/taugrid-examples.git
cd taugrid-examples/aks-blog/kueue-ray-cpu

./demo.sh single

The script applies the single-queue resources and enqueues two raw RayJobs. In a second terminal, watch admission:

Run GPU Ray Tune HPO on AKS

Mon, 01 Jan 0001 00:00:00 +0000

This guide runs the GPU HPO workflow for examples/ray-tune-smoke. It was verified on one Standard_NV12ads_A10_v5 node. A100 SKUs are also usable when the selected subscription and region have both sufficient quota and real-time allocation capacity.

The workflow creates billable Azure resources. Use a dedicated resource group. Record its owner, collect the required evidence, and delete the resources when the workflow is complete.

Ownership and completion gates

Complete these phases in order:

Hand off a workspace

Mon, 01 Jan 0001 00:00:00 +0000

Complete this checklist after the TauWorkspace readiness gate reports Ready. A handoff is reproducible only when every step below succeeds from a clean checkout, not from a platform owner’s already-primed shell.

The non-secret connection descriptor

The only artifact a researcher needs is tau/workspace.connection.yaml, a non-secret workspace connection descriptor:

Field	Contains
`schema`	Fixed value `tau.workspace.connection.v1`.
`workspace`	The TauWorkspace name.
`cluster.contextName`	The Kubernetes context name Tau selects.
`cluster.systemNamespace`	The TauGrid system namespace; defaults to `tau-system`.
`access.method`	`kubeconfig` for an existing context or `aks` for automatic AKS credential acquisition.
`access.aks.resourceID`	Required only for `access.method: aks`; the AKS cluster’s ARM resource ID.
`access.aks.tenantID`	Required only for `access.method: aks`; the Microsoft Entra tenant ID.
`authorization.mode`	`cluster-wide` or `workspace-rbac`.
`authorization.requiredRole`	Required only in `workspace-rbac` mode; forbidden in `cluster-wide` mode.
`requirements.minTauVersion`	The minimum compatible `tau` CLI version.
`network.privateCluster`	Whether the Kubernetes API server requires private network access.
`network.instructions`	Required only when `privateCluster: true` (for example, VPN steps).

It must never contain a credential, kubeconfig, client secret, or cloud access token. With access.method: kubeconfig, tau loads the normal kubeconfig rules (including KUBECONFIG) and copies only the named context, cluster, and user into an isolated mode-0600 kubeconfig outside the repository. With access.method: aks, it obtains normal AKS cluster-user credentials through the caller’s Azure identity and isolates those instead.

Getting started on AKS

Mon, 01 Jan 0001 00:00:00 +0000

AKS is one supported way to reach the Kubernetes cluster that the Getting started on Kubernetes guide installs onto. This page compares the three ways to get there and links each one to its source runbook. Use it after deciding you want AKS specifically; if you already have a Kubernetes cluster of any kind, go straight back to step 2 of Getting started on Kubernetes.

Prerequisites

An Azure subscription and the az CLI, logged in, with permission to create resource groups and AKS clusters (or an existing cluster you can already reach with kubectl).
tau, kubectl, and helm on PATH; see Install the Tau CLI. tau cluster install shells out to helm internally.
For the Terraform path: Terraform 1.9+ and Azure credentials usable by the AzureRM provider.
For the GPU Terraform path: regional quota for the target GPU VM SKU.

Choose a path

Path	Use it when	Creates billable resources
Guided CPU quickstart	You want a guided, disposable, end-to-end evaluation with scripted cleanup, using only CPU quota	Yes
GPU Terraform root	You want reviewable, reproducible AKS infrastructure with GPU node pools that your platform team can adapt and own long-term	Yes
Existing AKS cluster	Your platform team already runs an AKS cluster that meets the prerequisites	Only if you scale it

1. Guided CPU quickstart (recommended for evaluation)

examples/aks-cpu-quickstart sequences resource-group creation, a CPU-only AKS cluster, tau cluster install, workspace creation, and a real PyTorch smoke workload, then tears everything down again. It is the fastest way to see TauGrid work end to end using only CPU quota.

Troubleshooting by lifecycle layer

Mon, 01 Jan 0001 00:00:00 +0000

This is the canonical diagnose-first path for a stuck or failed run. Work through the layers in order and stop at the first incomplete layer. Each layer names its own TauGrid command first; use raw kubectl only for the deep, operator-only inspection each layer calls out, never as a substitute for the TauGrid command.

Start every investigation the same way:

tau run status <run-name>

tau run status is the single canonical lifecycle view. It walks the same ordered phase tree for every run – Submitted, Kueue admission, (for a RayJob) RayCluster, pod scheduling, DRA allocation, image pull, init containers, container start, readiness, and (for a RayJob) RayJob status – and each phase reports pending, active, done, warning, or skipped. Read it top to bottom and treat the first phase that is not done as the layer to investigate, addressing it before any later phase. Add --watch to follow it live.

Provision a GPU-enabled TauGrid AKS environment

Mon, 01 Jan 0001 00:00:00 +0000

Use the repository’s terraform/aks root to create a GPU-enabled AKS environment. It provisions a system pool and a GPU pool, enables OIDC and Azure Workload Identity, and invokes the supported tau cluster install command.

tau cluster install installs the versioned TauGrid distribution that owns Kueue, KubeRay, the Tau controller, GPU monitoring, the baseline Kueue queue, and Portal; skip installing those components separately with Helm.

Prerequisites

an Azure subscription that passes the AKS cluster prerequisites, has GPU quota for the selected region and SKU, and an approved Terraform identity;
Azure CLI, Terraform 1.9 or later, kubectl, Helm, and local Python dependencies;
Azure credentials accepted by the AzureRM Terraform provider and permission to provision AKS, networking, storage, and identities; and
tau and PowerShell 7 on PATH. Linux and macOS users can configure the Terraform command interpreter to use Bash.

The default deployment creates one Standard_NC24ads_A100_v4 node. This node has one A100 80 GB GPU and is billable. Before applying, verify that the target region has capacity for the corresponding VM family. Change gpu_vm_size, gpu_count_per_node, and gpu_monitoring_sku_name together when selecting another GPU SKU.

Identity and security boundaries

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid platforms answer two different questions:

Who may submit and inspect Kubernetes workloads?
Which cloud resources may the workload pod access?

Human authentication leads to Kubernetes authorization in the target namespace and queue. Workload identity starts from a pod ServiceAccount and reaches only the external resources required by that workload.

TauWorkspace reconciles Kubernetes ServiceAccounts and reports readiness. Platform teams provision the Azure managed identity, federated credential, Key Vault, storage account, and cloud RBAC assignment that those ServiceAccounts federate with.

Retry and resume

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid has two recovery paths for a failed run: automatic retry, driven entirely by the resilience.* fields in your run config, and manual tau run resume. TauGrid performs retry as automatic behavior of tau run when resilience.max_retries > 0; there is no separate tau run retry subcommand to invoke.

Automatic retry (`resilience.*`)

resilience:
 max_retries: 2 # default 0 (disabled)
 retry_on: ["Preempted", "Evicted"] # default; OOMKilled is opt-in
 backoff_initial: 30s  # default
 backoff_max: 5m  # default
 checkpoint_path: /data/checkpoints/finetunes/<name>  # default, derived from run name

When max_retries > 0 and you did not pass --dry-run, tau run:

Prepare ADX/Kusto for TauGrid

Mon, 01 Jan 0001 00:00:00 +0000

ADX/Kusto is an optional platform data service. Platform teams provision its cluster, databases, Entra identities, federation, and database roles. Prepare those platform resources once; Portal, lifecycle recorder, and adx-mon are separate consumers. Their released charts manage the TauGrid/adx-mon schema objects that they own.

Provision the service

Choose an approved region, SKU, capacity, network path, retention policy, and cost owner. This CLI shape illustrates the required flags; choose SKU, capacity, and other values to match your own environment:

Queue, quota, topology, and GPU placement

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid resolves the selected ready TauCluster workload profile; upstream systems enforce the rendered queue, priority, resource, and placement contract. The controller’s resolved profile status is authoritative and stale status fails closed. See workload profile migration.

Stage	Owner	Decision
Target resolution	TauGrid	Requested workers, GPUs, placement, priority, and workspace defaults
Queue admission	Kueue	Whether shared quota may be consumed
Pod scheduling	Kubernetes	Which nodes satisfy resources, selectors, taints, and topology
Device allocation	Device plugin or DRA	Which concrete GPUs are assigned
Node scaling	Cluster infrastructure	Whether matching node capacity can appear

LocalQueues are tenant-facing entry points. ClusterQueues own quota and fairness. ResourceFlavors describe resource pools. Priority and preemption remain cluster-owned policy.

Enable lifecycle recorder

Mon, 01 Jan 0001 00:00:00 +0000

The lifecycle recorder is an optional, standalone ADX/Kusto producer; Portal is one optional consumer of its output. It runs one metadata-only tau run history record Deployment and records observations of Jobs, RayJobs, and Kueue Workloads in one workload namespace. Portal can optionally read the resulting rows as durable Ray history, but the recorder is useful independently of Portal.

Prepare the namespace, adx-mon, and identity

Before enabling the recorder:

Observability and evidence

Mon, 01 Jan 0001 00:00:00 +0000

Use each evidence plane for its intended scope:

Plane	Use
TauGrid status and logs	Immediate lifecycle of one submitted run
Kubernetes events and pod state	Admission, scheduling, startup, and termination
Ray and GPU metrics	Runtime and hardware behavior
Expstore and durable artifacts	Authoritative experiment, checkpoint, and recovery state
adx-mon and ADX/Kusto	Optional hosted scalar and fleet analysis
Dashboards and alerts	Consumer views over telemetry

A workload reaching Scheduled or Running status does not prove the model is making useful progress. Treat each as a separate claim and verify independently. Preserve raw logs, profiles, and artifacts when a diagnosis depends on them.

Configure Portal

Mon, 01 Jan 0001 00:00:00 +0000

Portal is the unified, read-only browser entry point. tau cluster install enables its operator-facing Kubernetes path by default in the system release namespace (tau-system unless --namespace selects another namespace); it ships as part of the one TauGrid umbrella release rather than a separate taugrid-core Helm release.

Understand the default boundary

The default distribution creates deployment/tau-portal, service/tau-portal, a dedicated ServiceAccount, and cluster-wide read-only Kubernetes RBAC. Portal remains ClusterIP-only and relies on network-level access control rather than application-level login, so kubectl port-forward is an operator diagnostic rather than a researcher endpoint.

Workload profile migration

Mon, 01 Jan 0001 00:00:00 +0000

Workload profiles are platform-owned declarations on the singleton TauCluster/cluster. They replace the removed file-based TopologyPolicy catalog. The controller resolves referenced queues and scheduling objects into status; TauGrid reads only that status as the single source of truth, rather than falling back to a compiled catalog or a local policy file.

Migrate the catalog

Translate each supported workload shape into spec.workloadProfiles. Keep live quota, capacity, ResourceFlavor selectors, and topology object names out of the declaration: those identities are observations in status.

Multi-cluster execution

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid supports deterministic dispatch from a manager to preconfigured MultiKueue workers in the environment below. The capability remains Alpha because its released environment and operational evidence are still narrow. It supports deterministic dispatch to a preconfigured, matched-version worker set rather than arbitrary worker discovery or unrestricted cross-cloud routing.

Supported manager environment

Component	Supported contract
TauGrid	Chart and images `0.3.x` from the same release
Kubernetes	`1.30` or newer, matching the TauGrid chart constraint
Kueue	TauGrid-pinned AKS chart and controller `0.18.2`
KubeRay	TauGrid-pinned operator `1.6.2` for RayJob dispatch
Platform	AKS manager and workers with operator-managed identities, networking, storage, GPU drivers, and compatible CRDs

Use the same Kubernetes, Kueue, KubeRay, and workload CRD minor versions on the manager and every worker. Other combinations are outside the supported contract.

TauGrid

CLI reference

Contributing

Make a compatible change

Run your first target

TauGrid object model

What is Ray?

How TauGrid uses Ray

What is TauGrid?

Architecture

Code boundaries

Run the Kind smoke test

Live experiment evidence

Prerequisites

Releasing TauGrid

Run and workload lifecycle

Run config reference

Share research with a teammate

Access model used in this guide

What is Kueue?

The main Kueue objects

Workspaces

Single active workspace

Cluster install values

Configuration resolution

Serve a trained model

Site and wiki parity

TauWorkspace reference

What is adx-mon?

How TauGrid uses adx-mon

Release contract

What is Stellar?

What Stellar helps answer

Where the data comes from

Experiment evidence and artifacts

What TauGrid saves

TauGrid visual curriculum

Understand and evaluate

Adopt and configure

Use TauGrid day to day

Operate TauGrid

Extend and contribute

Glossary

TauGrid

Getting started on Kubernetes

Enable a workspace

Prepare the workspace

Troubleshoot a run

CPU queueing with Kueue and Ray

Prerequisites

Run the single-queue demonstration

Run GPU Ray Tune HPO on AKS

Ownership and completion gates

Hand off a workspace

The non-secret connection descriptor

Getting started on AKS

Prerequisites

Choose a path

1. Guided CPU quickstart (recommended for evaluation)

Troubleshooting by lifecycle layer

Provision a GPU-enabled TauGrid AKS environment

Prerequisites

Identity and security boundaries

Retry and resume

Automatic retry (resilience.*)

Prepare ADX/Kusto for TauGrid

Provision the service

Queue, quota, topology, and GPU placement

Enable lifecycle recorder

Prepare the namespace, adx-mon, and identity

Observability and evidence

Configure Portal

Understand the default boundary

Workload profile migration

Migrate the catalog

Multi-cluster execution

Supported manager environment

Automatic retry (`resilience.*`)