[{"contents":" Feature state: GA Reviewed 2026-07-16 The Tau CLI has exactly eight public command roots. Every example in this site uses one of these roots and its current nested subcommand path:\nRoot Purpose tau cluster Install/uninstall TauGrid and validate an existing cluster tau workspace Create/adopt workspaces and inspect readiness, quota, and repo scaffolds tau run Config-first submit, lifecycle, and retry/resume for one run tau logs Find a run through configured Tau workspaces and stream its logs tau serve Render and manage serving deployments tau data Inspect dataset and model registries tau python Optionally author decorator-based @tau.train / @tau.eval workflows; standard tau.yaml projects do not need the Python SDK tau version Print the installed CLI version Run evidence and the observability portal ship in a separate binary, taugrid-portal. See taugrid-portal experiment below.\nThe old flat roots have been removed. Pre-v0.5 top-level aliases (ray, model, dataset, exp, queue, status, cancel, topology, lanes, storage, eval, shell, submit, and finetune) are gone; their capabilities live under the seven roots above. Job execution is owned directly by the config-first tau run path.\ntau cluster Install the versioned TauGrid distribution on a fresh Kubernetes cluster, or validate an already-onboarded cluster:\nSubcommand Purpose install Show the TauGrid plan, run the two-phase Helm install, and wait for control-plane readiness uninstall Drain the release-owned queue policy while Kueue still runs, then run helm uninstall for the TauGrid release (requires --yes) validate installation Re-run the read-only TauGrid control-plane readiness report validate nodes Run privileged GPU health probes (nvidia-smi, NVLink, IB, ECC) across nodes validate topology Check ResourceFlavor, node label, and IB readiness against ready TauCluster workload profiles profiles Inspect ready TauCluster workload profiles (subcommand: export) explain-values Print the TauGrid install values reference (human-readable Markdown) validate installation, validate nodes, and validate topology are platform/operator diagnostics: run them against a cluster you administer, not from inside a researcher project. Installation validation is read-only. Node validation creates short-lived privileged probes and remains explicitly opt-in. Pair these with per-run troubleshooting in recovery for workload-specific issues.\ninstall validates Kubernetes 1.30+, the three controller Deployments, TauCluster node readiness, the baseline ClusterQueue, and the narrow fail-closed quota guard before reporting success; provisioning AKS, mutating nodes, creating PVCs, and configuring Azure identities remain platform-owned steps outside this command. Readiness runs after Helm, so a failed readiness report leaves the release installed for inspection; uninstall still retains CRDs, custom resources, retained namespaces, storage, and workspace data. Existing ArgoCD-managed clusters should retain independent component Applications instead of installing the umbrella release.\nThe default install runs the Tau CLI\u0026rsquo;s bounded, component-aware readiness report after Helm applies the release, instead of Helm\u0026rsquo;s generic --wait or automatic rollback. Pass --wait to add Helm\u0026rsquo;s generic resource watcher as an earlier gate. Pass --atomic to roll back on a Helm failure; rollback also enables Helm\u0026rsquo;s watcher, and the Tau CLI translates this compatibility flag to Helm 4\u0026rsquo;s --rollback-on-failure. --timeout bounds each Helm operation and the Tau CLI\u0026rsquo;s readiness report.\nThe compiled chart default is the public oci://mcr.microsoft.com/aks/ai-runtime/helm/taugrid OCI reference. The requested version must be published. If it is unavailable, the Tau CLI reports the exact chart and version and asks the operator to pass an accessible --chart reference or local path.\nuninstall runs in two Helm phases. Helm\u0026rsquo;s own uninstall order deletes Deployments before custom resources, so removing the release in one pass would take the Kueue controller down while its ClusterQueue, ResourceFlavor, and Topology still carry the kueue.x-k8s.io/resource-in-use finalizer, leaving nothing able to clear it. The first phase therefore removes the queue policy while Kueue is still running. Pass --drain-queue=false to skip it. In the stranded-finalizer case, no controller remains to clear the finalizer, so raising --timeout only delays an expiring wait rather than fixing it.\nThe final uninstall waits for resource removal by default; pass --wait=false to return after Helm submits the deletions, and use --timeout to bound either mode.\nThe drain re-renders the release, so it needs the same --chart reference the release was installed from. Uninstall compares that reference against the chart name Helm recorded for the release and skips the drain when they disagree, rather than applying a different chart\u0026rsquo;s manifests on the way out. When the reference cannot be resolved, or the drained objects are still present afterwards (Kueue holds resource-in-use until an active run finishes), uninstall reports which objects may strand and proceeds; teardown still completes. Draining is only possible through tau, because Helm decides the deletion order of a release\u0026rsquo;s own custom resources; a plain helm uninstall of the TauGrid release still strands them.\nOn Helm 4, both verbs send --force-conflicts (paired with --server-side=true) because AKS\u0026rsquo;s admissionsenforcer addon co-owns .webhooks[*].namespaceSelector on Kueue\u0026rsquo;s webhook configurations. Helm 4 defaults to server-side apply, so every upgrade of the release, including the uninstall drain, requires --force-conflicts to avoid failing on those fields. Forcing settles field ownership only, leaving the AKS-managed selector values untouched since the chart declares no value for them. Helm 3 skips this entirely and receives neither flag. Leave the Kueue webhook configurations in place to avoid this conflict: recreating them can leave caBundle empty until the Kueue controller restarts.\ntau workspace Researcher-facing: list, status \u0026lt;name\u0026gt;, check \u0026lt;name\u0026gt;, connection [path], quota show \u0026lt;workspace\u0026gt;, quota request \u0026lt;workspace\u0026gt;, and init-repo NAME (scaffold a TauGrid-ready repo). Platform-operator: create [NAME] (create the single v0 researcher workspace) and adopt NAME (adopt an existing platform-provisioned namespace). Only quota request, create, and adopt write to the cluster; each previews by default and needs --apply to take effect. Workspace desired state is a native TauWorkspace delivered through Helm/Kustomize/GitOps and reconciled by the TauWorkspace controller.\ntau workspace create requires --principal-name \u0026lt;external-group-or-team\u0026gt;; --subject-name defaults to that principal when it is omitted. tau workspace connection verifies and pins the current project\u0026rsquo;s configured workspace access. Add --offline to validate only the repository mapping and descriptor; tau run also discovers and configures the connection automatically.\ntau run tau run [TARGET] [--config tau.yaml] is the config-first entry point. TARGET is an optional positional argument to the run root itself, rather than a subcommand: tau run train --dry-run=client runs the run root with TARGET=train and --dry-run=client, resolving tau/train.yaml. See run config for the field reference and first run for the full walkthrough.\nSubcommand Purpose validate [name] Validate --config (or the default tau.yaml); name overrides the config\u0026rsquo;s name:, which is used when it is omitted schema Print the JSON schema for the direct run config explain-config Print the direct run config field reference list List TauGrid-managed Jobs/RayJobs in a namespace status \u0026lt;job-name\u0026gt; Show lifecycle state and startup phases; --watch to poll logs \u0026lt;job-name\u0026gt; Compatibility path for tau logs \u0026lt;job-name\u0026gt; get \u0026lt;name\u0026gt; Fetch durable run results and artifacts cancel \u0026lt;job-name\u0026gt; Delete the underlying Job/RayJob and free its Kueue quota resume \u0026lt;name\u0026gt; --config tau.yaml Manually restart a failed run from its checkpoint There is no tau run retry subcommand; automatic retry is driven entirely by the resilience.* fields in your run config. See recovery for the full retry and resume contract.\ntau logs tau logs \u0026lt;run-name\u0026gt; searches the connected workspace first, then the locally configured Tau workspace connections. It streams Ray driver logs or batch Job pod logs without requiring Kubernetes context or namespace flags. Use --workspace, or --context and --namespace, when the same run name exists in more than one configured workspace. Use the native --tail N option instead of piping through the shell\u0026rsquo;s tail.\ntau serve deploy \u0026lt;name\u0026gt;, status \u0026lt;name\u0026gt;, scale \u0026lt;name\u0026gt;, and delete \u0026lt;name\u0026gt; render and manage a serving deployment from the same run-config-shaped intent as tau run. scale --kind=deployment is implemented; RayService scaling must currently be done by redeploying with --replicas or autoscaling flags. See serve a trained model.\ntau data Groups the dataset and model registries so researchers have one place to resolve durable inputs:\nSubcommand group Purpose dataset list|show|ref|alias|verify|register|rm|ingest|status The curated dataset registry model list|show|best|alias set|alias get|index rebuild Durable model checkpoints and aliases taugrid-portal experiment Run evidence and the Stellar dashboard ship in the taugrid-portal binary, not in tau. The command tree is unchanged from when it lived under tau experiment; only the binary name differs.\nSubcommand Purpose init NAME Initialize an experiment store track RUN Record run metadata, configs, artifact pointers, and metrics list List runs, groups, or experiments (--kind) search (alias runs) Full-text and filtered run search status NAME Show a run\u0026rsquo;s lifecycle status stellar NAME (alias dashboard) Render the local Stellar dashboard (-o html|json|tui) serve Serve Stellar over HTTP open NAME Serve and open a Stellar dashboard in the default browser compare NAME, plot NAME Compare or plot metrics across runs import jsonl Import JSONL scalar history into the store taugrid-portal portal serve serves the unified observability portal (Experiments, Jobs/Queue, Cluster Health) from the same binary.\ntaugrid-portal experiment and the hidden compatibility alias exp share the same command tree, so --help also surfaces offload, kusto, autocapture, capture, sql, export, and observe, plus a nested experiments search|tag-run group. Those are platform-internal and automation tools (metrics sidecar offload, query-builder internals); this reference covers only the researcher-facing commands.\nUse built-in help for the exact installed release:\ntau --help tau run --help tau logs --help tau run status --help The detailed current CLI guide is cli/README.md.\n","description":"Canonical Tau CLI command families","permalink":"/taugrid/docs/reference/cli/","title":"CLI reference"},{"contents":" Feature state: GA Reviewed 2026-07-16 Start with the repository AGENTS.md and SDK_GUIDE.md.\nCore principles:\nStart from the user path and compatibility contract. Keep commands thin and capability packages cohesive. Reuse existing helpers and dependencies. Keep durable local formats inspectable and versioned. Preserve expstore as local truth. Make retry, checkpoint, and partial-write behavior explicit. Add tests at the owning layer. External contract changes need external evidence. A manifest or telemetry change needs evidence beyond a utility unit test.\n","description":"Contributor expectations and validation depth","permalink":"/taugrid/docs/community/contributing/","title":"Contributing"},{"contents":" Feature state: GA Reviewed 2026-08-17 Start from the user path and compatibility contract. Keep Cobra command files focused on wiring, flags, validation, and output. Put reusable behavior in the owning internal/* capability package. Preserve inspectable, versioned durable formats. Keep expstore authoritative and hosted analytics downstream. Test behavior at the layer that owns it. Run focused package tests while iterating. Before TauGrid-internal closeout:\ncd cli go test ./... Manifest, runtime, telemetry, or Kubernetes lifecycle changes also require the smallest safe end-to-end path that proves the external contract. Use the Kind smoke test when the changed contract belongs to the portable Kubernetes path.\n","description":"Choose the owning package and prove the changed contract","permalink":"/taugrid/docs/community/change/","title":"Make a compatible change"},{"contents":" Feature state: GA Reviewed 2026-07-16 Prerequisites:\nThe Tau CLI is installed. The repository contains a workspace connection and a checked-in target, such as tau/train.yaml. The platform workspace reports Ready. tau run is the config-first entry point. Its optional positional TARGET argument resolves to a checked-in tau/\u0026lt;target\u0026gt;.yaml file. See the direct run config reference for the full tau.yaml field set. tau run automatically discovers the checked-in workspace connection.\nValidate before submitting a real workload:\ncd \u0026lt;research-repository\u0026gt; tau workspace connection tau run validate --config tau/train.yaml tau run train --dry-run=client tau run train tau workspace connection verifies the repository\u0026rsquo;s configured workspace, credentials, LocalQueue, and authorization without submitting a workload. tau run validate is the offline schema check. In a connected repository, --dry-run=client activates the workspace connection and reads the cluster\u0026rsquo;s workload-profile catalog, but does not submit the rendered workload. Use --dry-run=server to also validate the rendered object against the API server. The Tau CLI prints the submitted run name; record it for the rest of this walkthrough.\ntau run status \u0026lt;run-name\u0026gt; --watch tau logs \u0026lt;run-name\u0026gt; tau run get \u0026lt;run-name\u0026gt; status --watch renders the startup phase tree (Submitted, Kueue admission, scheduling, image pull, readiness) until the workload is ready, failed, or you interrupt it. For RayJobs, logs streams the Ray driver\u0026rsquo;s execution output rather than head-pod logs; for batch Jobs it streams the Job pod logs. get fetches durable run results and artifacts once the run has produced them.\nThe status and logs subcommands also work for a batch Job submitted outside TauGrid. Such Jobs are hidden from the default TauGrid-owned list; opt in when investigating a shared namespace:\ntau run list --include-external tau logs \u0026lt;job-name\u0026gt; -c \u0026lt;container\u0026gt; --previous --timestamps tau run status \u0026lt;job-name\u0026gt; --diagnostic-hints Batch logs also support --all-containers and --prefix. The diagnostic hints print correctly scoped kubectl top, detailed log, and exec commands for you to run directly; TauGrid\u0026rsquo;s own RBAC footprint is limited to generating those hints, not to proxying metrics or interactive exec itself. For an external or already deleted Job with no TauGrid result annotations, retrieve a known PVC path explicitly:\ntau run get \u0026lt;job-name\u0026gt; --path /data/\u0026lt;job-name\u0026gt;/results --pvc \u0026lt;pvc-name\u0026gt; If you need to stop a run before it finishes (for example, you spot a bad hyperparameter mid-training), cancel it instead of leaving it to fail on its own:\ntau run cancel \u0026lt;run-name\u0026gt; cancel deletes the underlying RayJob or Job and lets Kueue reclaim the workload\u0026rsquo;s quota. If a run instead fails on its own, do not resubmit blindly; see recovery for automatic retry and tau run resume semantics before you retry by hand.\nNext: look at your evidence A completed (or still-running) run\u0026rsquo;s metrics and artifacts are evidence. Once you have a run name, discover and open it in Stellar using the taugrid-portal binary:\ntaugrid-portal experiment search taugrid-portal experiment stellar \u0026lt;run-name\u0026gt; taugrid-portal experiment search (alias runs) lists indexed runs so you can find the one you just submitted if you did not keep the name. taugrid-portal experiment stellar renders the local Stellar dashboard for that run as static HTML by default; add -o tui for a terminal summary, or use taugrid-portal experiment open \u0026lt;run-name\u0026gt; to serve it and open your browser in one step.\nTo compare the same run from two machines, continue with Share research with a teammate.\nKeep namespace, queue, kubeconfig, and cloud credentials as workspace concerns rather than adding them to project config to work around platform readiness failures.\nWhen the run has produced a durable checkpoint and your project has a serving image, continue with Serve a trained model.\n","description":"Validate, submit, observe, and retrieve one repository target","permalink":"/taugrid/docs/developer-guide/first-run/","title":"Run your first target"},{"contents":" Feature state: GA Reviewed 2026-07-16 Full definitions for these terms live in the glossary. This page only explains how they relate and who owns what.\nConcept Owns Repository Code and history in one Git worktree Project One or more targets and a workspace connection Workspace Destination policy: cluster, namespace, queue, priority, output root, identity Target One checked-in runnable config Run One execution plus its lifecycle handle Workload Rendered Job or RayJob execution intent Service Online RayService or Deployment lifecycle target Experiment A comparison set over runs, metrics, and artifacts The ownership split is deliberate:\nProjects own code, images, runtime dependencies, resource intent, datasets, and artifacts. Workspaces own destination policy and shared platform defaults; projects keep owning their own code and workload shape. The TauWorkspace Kubernetes resource backs this policy, while platform teams provision the underlying Azure resources. TauGrid owns deterministic resolution and lifecycle handoff. ","description":"How repository, project, workspace, target, run, workload, and experiment relate","permalink":"/taugrid/docs/developer-guide/concepts/object-model/","title":"TauGrid object model"},{"contents":"Ray is an open-source framework for running Python work across processes and machines. Teams use it for distributed training, hyperparameter tuning, data processing, and model serving.\nKubeRay brings Ray to Kubernetes. Its operator manages resources such as RayJob and RayService, creates the Ray head and worker pods, and follows the Ray workload lifecycle.\nHow TauGrid uses Ray TauGrid turns a repository target into either a Kubernetes Job or a KubeRay RayJob. A Ray workload follows this path:\ntau run validates the target and creates a RayJob. Kueue waits for the required quota. Kubernetes places the Ray head and worker pods. Ray coordinates the application across those pods. TauGrid reports status, streams driver logs, and retrieves saved outputs. Ray owns application execution and communication between workers. TauGrid owns the submission and run experience around it.\nWhen Ray is useful Use Ray when a workload needs multiple workers, Ray Train, Ray Tune, or Ray Serve. A simple single-container task can run as a standard Kubernetes Job.\nThe Ray dashboard shows live cluster and task state while the Ray head is running. Stellar serves a different purpose: it shows saved experiment metrics and comparisons after the workload produces evidence.\nTry GPU Ray Tune HPO for a complete example.\n","description":"How Ray runs distributed AI workloads in TauGrid","permalink":"/taugrid/docs/getting-started/core-technologies/ray/","title":"What is Ray?"},{"contents":"TauGrid is an open, self-hosted AI compute platform for running AI workloads on Kubernetes. It brings the experience teams expect from a managed AI platform (simple submission, elastic distributed compute, shared GPU capacity, experiment visibility, and operational controls) to clusters they operate in their own cloud or datacenter.\nResearchers work from a repository and the Tau CLI. Platform teams provide governed workspaces, queues, compute profiles, storage, identity, and observability. TauGrid connects those two experiences across training, fine-tuning, batch inference, and serving.\nA complete path from code to compute repository -\u0026gt; target -\u0026gt; governed workspace -\u0026gt; queued compute -\u0026gt; results The checked-in target describes the workload. TauGrid resolves platform policy, renders a Kubernetes Job or KubeRay RayJob, submits it through shared queueing, and provides one lifecycle for status, logs, results, cancellation, retry, and resume.\nOne platform for researchers and operators Researchers move faster. The Tau CLI turns repository configuration into repeatable runs without requiring every project to set up its own cluster access, queueing, GPU placement, storage, logs, and recovery. Platform teams stay in control. Kubernetes remains the source of truth for capacity, policy, identity, and workload state. Teams deploy TauGrid into their own environment and integrate the storage, networking, and cloud services they already operate. Organizations keep their options open. TauGrid builds on Kubernetes, Kueue, KubeRay, and standard container images. Workloads, platform policy, and operational evidence remain inspectable and portable. Integrated where it matters TauGrid brings together:\nrepository-driven workload configuration and the Tau CLI; Kueue admission, quotas, and shared GPU queues; KubeRay orchestration for distributed Ray workloads; workspace identity, service accounts, and storage contracts; GPU topology, placement profiles, and node health; run status, logs, retry, resume, and experiment evidence; and Portal views for researchers and platform operators. Each layer keeps a clear operational owner: Kubernetes schedules pods, Kueue admits workloads, Ray and model frameworks execute training, and TauGrid ties them together into a single workflow.\nRead Architecture for component ownership or Getting started on Kubernetes for a runnable path.\n","description":"An open-source AI compute platform for Kubernetes infrastructure you control","permalink":"/taugrid/docs/getting-started/what-is-tau/","title":"What is TauGrid?"},{"contents":"TauGrid is a CLI, renderer, and local observer. Kueue, KubeRay, and the Kubernetes scheduler continue to own quota admission, Ray orchestration, and pod scheduling.\nLayer Responsibility Research repository Entrypoint, image, dependencies, resource intent, data, and outputs TauGrid Resolve, validate, render, submit, observe, and preserve workflow evidence Workspace/platform Cluster access, namespace, queue, priority, output root, identity, and shared policy Kueue Quota reservation, admission, priority, and preemption Kubernetes/KubeRay Pod scheduling and Job/Ray lifecycle Project process Training, evaluation, inference, and model-specific behavior Code boundaries cli/cmd: thin Cobra command wiring for the tau binary. cli/internal: reusable capability packages behind tau. core: the shared library module both binaries link. portal: a separate module and binary, taugrid-portal, that hosts experiment tracking (Stellar) and the observability portal. sdk/python: optional Python authoring APIs that delegate execution to the Go CLI. controllers/tau-core: the separately deployed controller. One manager runs three reconcilers \u0026ndash; TauCluster, TauWorkspace, and TauQuotaRequest. The visual architecture guide maps the main packages and extension points.\n","description":"How TauGrid connects repository intent to Kubernetes execution","permalink":"/taugrid/docs/getting-started/architecture/","title":"Architecture"},{"contents":" Feature state: GA Reviewed 2026-08-17 Use this contributor smoke test for changes to manifest rendering, submission, Kueue admission, KubeRay lifecycle handling, or the portable Kubernetes path. Kind supplies a disposable local Kubernetes API and Nodes for portable-path validation, distinct from an AKS or provider-integration acceptance test.\nFrom the repository root, local dry-run verifies the checked-in fixture and rendering contracts entirely offline, with no cluster required:\nmake -C cli build cli/bin/tau run validate --config examples/kind-smoke/tau.yaml cli/bin/tau run --config examples/kind-smoke/tau.yaml --dry-run=client The full smoke test requires Kind, kubectl, Helm, and a healthy Docker or Podman engine with at least 8 GiB of memory. Run:\ncd cli make test-kind-e2e The test creates or reuses the tau-kind cluster, installs the local TauGrid distribution with Kueue and KubeRay, submits Job and RayJob fixtures, and checks queue admission and lifecycle behavior. A successful run exits 0 after the required workloads and controllers reach their expected states.\nIt validates queue admission and lifecycle behavior on portable Kubernetes; validate GPU drivers, cloud identity, production storage, private networking, provider CSI integration, and live-cluster quota with an AKS-backed run instead.\nThe fixture, inspection commands, cleanup controls, and expected flow live in examples/kind-smoke.\n","description":"Prove portable TauGrid, Kueue, and KubeRay behavior on a disposable local cluster","permalink":"/taugrid/docs/community/kind-smoke-test/","title":"Run the Kind smoke test"},{"contents":" Feature state: Alpha Reviewed 2026-08-25 This example runs the checked-in examples/portal-ray-stellar RayJob. It publishes immutable loss and accuracy chunks to durable storage while a metrics-offload sidecar projects the same scalar history to ADX/Kusto.\nThe run demonstrates three independent surfaces:\ntau run status and logs for immediate execution. tau run get for durable files on the workspace PVC. Stellar for experiment comparison and scalar visualization. Prerequisites Prepare:\nA Ready TauWorkspace named taugrid-default. A writable blob-training PVC mounted at /data. One schedulable NVIDIA GPU. A digest-pinned taugrid-portal image supplied by the platform team. Portal and the optional metrics ingestion path configured for the workspace. The sidecar runs taugrid-portal experiment offload metrics, so use the Portal image rather than the Tau CLI image.\nInspect the evidence contract The checked-in run config declares:\nstorage: data_pvc: blob-training output: /data/projects/taugrid-default/runs/portal-ray-stellar metrics: history: - metrics-history-attempt-0/*.jsonl offload: enabled: true experiment: project: portal-demo name: ray-plus-stellar group: default The trainer writes one closed JSONL chunk per step. A temporary dotfile is flushed and atomically renamed, so readers only see complete rows:\n{\u0026#34;_step\u0026#34;:0,\u0026#34;_timestamp\u0026#34;:1760000000.125,\u0026#34;loss\u0026#34;:1.0,\u0026#34;accuracy\u0026#34;:0.0} {\u0026#34;_step\u0026#34;:1,\u0026#34;_timestamp\u0026#34;:1760000001.125,\u0026#34;loss\u0026#34;:0.5,\u0026#34;accuracy\u0026#34;:0.5} {\u0026#34;_step\u0026#34;:2,\u0026#34;_timestamp\u0026#34;:1760000002.125,\u0026#34;loss\u0026#34;:0.3333333333,\u0026#34;accuracy\u0026#34;:0.6666666667} _timestamp is a positive Unix-seconds number. The offloader uses _step and the timestamp to normalize each scalar into the metrics store.\nRender and submit From the repository root:\nmake install-tau-cli export TAU_METRICS_OFFLOAD_IMAGE=\u0026lt;taugrid-portal-image@sha256:digest\u0026gt; export TAU_METRICS_OFFLOAD_OUT=/var/run/tau/metrics-offload tau run --workspace taugrid-default \\ --config examples/portal-ray-stellar/tau.yaml \\ --dry-run=client Confirm that the rendered RayJob includes:\ntau.azure.com/stellar-experiment-id: ray-plus-stellar tau.azure.com/stellar-group-value: default The blob-training PVC A metrics-offload sidecar watching the immutable history glob Submit and follow the run:\ntau run --workspace taugrid-default \\ --config examples/portal-ray-stellar/tau.yaml tau run status portal-ray-stellar --watch tau logs portal-ray-stellar Inspect durable evidence List the directory recorded by storage.output:\ntau run get portal-ray-stellar -n taugrid-default The listing includes files shaped like:\nmetrics-history-attempt-0/ chunk-000000-\u0026lt;timestamp\u0026gt;.jsonl chunk-000001-\u0026lt;timestamp\u0026gt;.jsonl chunk-000002-\u0026lt;timestamp\u0026gt;.jsonl Copy one listed name and fetch its exact contents:\ntau run get portal-ray-stellar -n taugrid-default \\ --artifact metrics-history-attempt-0/\u0026lt;chunk-name\u0026gt;.jsonl \\ -o raw The durable file remains available through the workspace storage lifecycle. The offloaded scalar rows become available in Stellar after the terminal tau/run_status marker is ingested.\nOpen the live views Open the Portal run page:\n/portal/runs/taugrid-default/\u0026lt;run-name\u0026gt; While the Ray head is Ready, Ray dashboard opens runtime state. After the metrics marker arrives, Open in Experiments opens the durable scalar history. The two links answer different questions and share the same run identity.\nWhat this example proves Check Evidence GPU workload ran tau run status, logs, Ray dashboard Loss improved Immutable JSONL chunks and Stellar series Durable files exist tau run get listing and file retrieval Fleet projection completed Terminal metrics marker and Stellar link Run identity stayed aligned Project, experiment, group, and run annotations Read Experiment evidence and artifacts for the store methodology, normalized schema, summaries, artifact publication, and ownership model.\n","description":"Publish scalar history, inspect durable artifacts, and open the same run in Stellar","permalink":"/taugrid/docs/examples/experiment-evidence/","title":"Live experiment evidence"},{"contents":" Feature state: GA Reviewed 2026-07-16 TauGrid release preparation updates:\nSDK version. Installation documentation. cli/releases/vX.Y.Z.md. The reviewed source commit on main. Maintainers then create an annotated SemVer tag and manually dispatch the release workflow. The workflow builds twice, compares outputs, verifies checksums, publishes immutable assets, and proves clean bootstrap.\nNever overwrite an existing release or asset. Do not update downstream minimum versions until post-publication Ubuntu and macOS verification succeeds.\nFollow the exact RELEASING.md checklist.\n","description":"Human-authorized and reproducible release publication","permalink":"/taugrid/docs/community/releasing/","title":"Releasing TauGrid"},{"contents":" Feature state: GA Reviewed 2026-07-16 A TauGrid run crosses independent control planes:\nResolution: TauGrid finds the project, workspace, and target. Rendering: TauGrid validates intent and builds a Job or RayJob. Admission: Kueue reserves quota and admits the workload. Scheduling: Kubernetes places pods on matching nodes and devices. Runtime: Ray or the Job controller starts the application. Progress: The model or data process advances and writes evidence. Completion: Results, checkpoints, metrics, and terminal state remain inspectable. These states are not equivalent:\nAdmitted does not mean scheduled. Scheduled does not mean the GPU is healthy. Running does not mean the model is progressing. Completed does not prove required artifacts were preserved. Use the layered troubleshooting guide to find the first failed transition.\n","description":"Independent transitions from repository resolution to useful progress","permalink":"/taugrid/docs/developer-guide/concepts/lifecycle/","title":"Run and workload lifecycle"},{"contents":" Feature state: GA Reviewed 2026-07-16 This page documents the direct run config: the normal, hand-written tau.yaml that tau run --config reads to submit a workload. See direct run config vs. managed workflow manifest for how this differs from the SDK-generated, schema_version: 1 manifest that tau.train/tau.serve render. Both can use a tau.yaml filename; inspect the schema, not the filename. Use the managed workflow shape only when TauGrid needs to own workflow semantics such as staged train/eval lineage.\nNormal research projects check in direct config:\nname: train engine: rayjob entrypoint: train.py runtime: image: \u0026lt;pinned-image\u0026gt; compute: workers: 2 gpus_per_worker: 8 storage: data_pvc: training-data output: /data/checkpoints/workflows/train storage.data_pvc names an existing Bound PVC in the resolved workload namespace. TauGrid references and mounts that claim; the platform owner provisions and owns the PVC, StorageClass, CSI configuration, and backing storage, choosing the backend and managing its lifecycle.\nDirect batch Jobs can stage an immutable reference directory from another image, skipping ConfigMap creation:\nengine: job storage: image_assets: - name: pinned-reference-assets image: \u0026lt;registry\u0026gt;/\u0026lt;repository\u0026gt;@sha256:\u0026lt;64-hex-digest\u0026gt; source_path: /opt/source-assets mount_path: /opt/reference TauGrid renders each asset as a pinned init-container image that runs /bin/cp -a into an emptyDir, then mounts that volume read-only in the main container. Source images must provide /bin/cp. Names and paths are validated, TauGrid-reserved paths cannot be replaced, and mutable image tags are rejected. storage.image_assets is intentionally limited to direct engine: job configs; managed workflows and RayJob configs reject it.\nDirect Jobs can also stage their complete source tree from an immutable OCI image:\nname: source-backed-job engine: job entrypoint: experiments/train.py run: ttl_seconds_after_finished: 3600 source: image: \u0026lt;registry\u0026gt;/\u0026lt;repository\u0026gt;@sha256:\u0026lt;64-hex-digest\u0026gt; path: /workspace TauGrid copies run.source.path into an emptyDir with an init container, mounts the per-pod working copy at /tau/source, and runs entrypoint relative to that directory, keeping source bytes out of environment variables and ConfigMaps entirely. The source image must provide /bin/sh, cp, and chmod; mutable tags, absolute entrypoints, RayJob dispatch, managed workflows, and combining run.source with run.working_dir are rejected before rendering. The init container normalizes working-copy permissions and sets the executable bit on the entrypoint before a potentially non-root workload container starts. Build and push the source image once, then reuse its digest in every run config. Kubernetes uses the workload\u0026rsquo;s configured private-registry authentication; do not put registry credentials or signed download URLs in the config.\nDirect Jobs can set the main container\u0026rsquo;s initial working directory directly, skipping shell-specific cd logic in the entrypoint:\nengine: job entrypoint: train.py runtime: image: \u0026lt;pinned-image\u0026gt; working_dir: /workspace/project runtime.working_dir must be a clean absolute path inside the image. TauGrid passes it straight through as Kubernetes container.workingDir, relaying the configured path unparsed; the image owns creating that directory and providing its own local files. It is Job-only and cannot be combined with run.source. Ray\u0026rsquo;s run.working_dir remains a different, host-relative field that packages a local project into Ray runtime_env.\nFor example, build a source-only image whose /workspace contains the checked out tree, push it to the platform\u0026rsquo;s private registry, and resolve the immutable digest before generating run configs:\ndocker buildx build --push -t \u0026#34;$SOURCE_REPOSITORY:$GIT_SHA\u0026#34; -f Dockerfile.source . SOURCE_DIGEST=\u0026#34;$(docker buildx imagetools inspect \\ \u0026#34;$SOURCE_REPOSITORY:$GIT_SHA\u0026#34; --format \u0026#39;{{.Manifest.Digest}}\u0026#39;)\u0026#34; printf \u0026#39;%s@%s\\n\u0026#39; \u0026#34;$SOURCE_REPOSITORY\u0026#34; \u0026#34;$SOURCE_DIGEST\u0026#34; run.ttl_seconds_after_finished is an optional retention period from 1 through 2,147,483,647 seconds for a completed or failed direct Job. It maps to Kubernetes spec.ttlSecondsAfterFinished; omission keeps TauGrid\u0026rsquo;s 28800-second default. The TTL starts only after every regular container has finished running.\nLiteral environment values are limited to 64 KiB each and 128 KiB in aggregate before workload creation. TauGrid\u0026rsquo;s generated embedded-payload environment entries are also capped at 64 KiB. Use run.source or storage.image_assets for content instead of embedding archives in runtime.env.\nMain field groups:\nGroup Purpose name, engine, entrypoint Workload identity and execution mode; script aliases entrypoint, and the nested run.* block adds immutable Job source staging and Ray project-directory shipping runtime Image, direct-Job container working directory, packages, environment, and optional restricted pod security; top-level image is a lowest-priority alias for runtime.image compute Worker, GPU, CPU, and memory intent execution Launcher, node/process topology, launcher configs, and Ray Tune search settings policy Explicit operator/accounting overrides storage Durable data, output, checkpoint, and extra mounts metrics Published JSONL metric paths and the opt-in offload sidecar resilience Automatic retry filters, backoff, and checkpoint path; see recovery profiler Bounded rank-scoped profiling experiment Project, experiment name, and group workflow Delegation to an SDK-generated managed workflow manifest Validate the installed contract. --config always names an explicit file; tau run itself instead takes an optional positional TARGET (for example tau run train) that resolves tau/train.yaml; the two serve different purposes, so keep validating against an explicit path:\ntau run validate --config tau.yaml tau run schema -o json tau run explain-config For the full submit-to-evidence loop against a named target, see first run.\nThe installed CLI is the final schema authority: use tau run schema.\nRestricted pod security Set runtime.security.mode: restricted when the workload images can run as non-root:\nruntime: image: \u0026lt;pinned-image\u0026gt; security: mode: restricted TauGrid applies the Kubernetes Restricted Pod Security fields to the Job or RayJob pod and every generated main, sidecar, and init container. Missing container user and group IDs default to numeric 65532; explicit nonzero IDs are preserved. Rendering fails if a profile requests root, privileged mode, privilege escalation, or added capabilities, and the image must support the selected non-root identity.\nConfig identity TauGrid records one hash of the validated direct config on submitted Jobs and RayJobs. The hash covers runtime, resources, environment, storage, and other run behavior, so resume can warn when the config changed. Script and packaged-source content use separate payload-digest annotations.\n","description":"Direct config-first workload intent","permalink":"/taugrid/docs/reference/run-config/","title":"Run config reference"},{"contents":" Feature state: Alpha Reviewed 2026-08-24 Researchers working from different machines can use the same cluster and workspace to view and compare the same runs, logs, and charts.\nAccess model used in this guide This guide describes the current interim admin-access workflow. The cluster owner gives each researcher their own cluster-admin kubeconfig. Each researcher keeps those credentials on their own machine and uses a separate port-forward to the shared Portal Service.\nThis workflow does not use workspace-directory browser authentication. That mode requires an external sign-in proxy that is not part of the current setup.\nBefore you start Both researchers need:\ntheir own cluster-admin kubeconfig for the same TauGrid cluster; access to the same workspace; the same research project; and the kubeconfig path, context, Portal namespace, and Portal Service name supplied by the cluster owner. Do not send kubeconfig files, tokens, passwords, or secrets to each other.\nShare a run Researcher 1 starts a run:\ntau run train Tau prints the run name. Researcher 1 sends that run name to Researcher 2.\nBoth researchers can now check the run:\ntau run status \u0026lt;run-name\u0026gt; tau logs \u0026lt;run-name\u0026gt; Use --watch to keep the status open while the run starts:\ntau run status \u0026lt;run-name\u0026gt; --watch Both researchers should see the same queue state, start time, Pods, and final result.\nSee shared and waiting runs List Tau runs in the workspace:\ntau run list If the team also uses Jobs created outside Tau, include them:\ntau run list --include-external Open one waiting run with:\ntau run status \u0026lt;run-name\u0026gt; --diagnostic-hints Both researchers should see the same reason when a run is waiting for space in the queue.\nShare a Portal view Each researcher starts a separate port-forward on their own machine. This opens the current admin Portal view; it does not sign researchers in separately. Use the kubeconfig, context, namespace, and Service name supplied by the cluster owner.\nOn Researcher 1\u0026rsquo;s machine:\nkubectl --kubeconfig \u0026lt;cluster-kubeconfig\u0026gt; \\ --context \u0026lt;cluster-context\u0026gt; \\ --namespace \u0026lt;portal-namespace\u0026gt; \\ port-forward service/\u0026lt;portal-service\u0026gt; 8080:80 On Researcher 2\u0026rsquo;s machine, run the same command with that machine\u0026rsquo;s own cluster credentials:\nkubectl --kubeconfig \u0026lt;cluster-kubeconfig\u0026gt; \\ --context \u0026lt;cluster-context\u0026gt; \\ --namespace \u0026lt;portal-namespace\u0026gt; \\ port-forward service/\u0026lt;portal-service\u0026gt; 8080:80 Keep each port-forward terminal open. Each researcher opens this address in a browser on their own machine:\nhttp://127.0.0.1:8080/portal The addresses look the same, but each one uses the port-forward running on that researcher\u0026rsquo;s machine. Both port-forwards connect to the same Portal Service in the cluster.\nResearcher 1 opens a run and sends its full Stellar link to Researcher 2. A shared link has this form:\nhttp://127.0.0.1:8080/stellar?target=\u0026lt;run-name\u0026gt;\u0026amp;project=\u0026lt;project\u0026gt;\u0026amp;workspace=\u0026lt;workspace\u0026gt; Researcher 2 opens the link while their own port-forward is running. The Portal uses the data source configured for the workspace.\nBoth researchers should check:\nthe run name; the project name; the run state; the metric names; the number of chart points; and the run group. If these match, both researchers are reading the same shared result.\nExample: share the published Ray Tune run This example uses the published GPU Ray Tune HPO walkthrough and its examples/ray-tune-smoke files.\nResearcher 1 follows the walkthrough and starts the run:\ntau run --config tau.yaml --dry-run=client tau run --config tau.yaml The checked-in example names the run tune-smoke. Researcher 1 sends these sample values to Researcher 2:\nRun: tune-smoke Project: ray-tune-demo Workspace: shared-research Researcher 2 checks the same run from their own machine:\ntau run status tune-smoke tau logs tune-smoke The names ray-tune-demo and shared-research are sample names. A team can replace them with its own project and workspace names.\nResearcher 1 opens tune-smoke through their port-forward and sends the full browser link to Researcher 2. Researcher 2 opens it through their own port-forward. Both researchers should see the same run name and state. Use the next example to publish metrics and compare charts in Stellar.\nExample: compare several runs This example uses the published examples/portal-ray-stellar files. It puts three runs in one experiment so both researchers can compare them on one page.\nBefore starting, ask the cluster owner to confirm that:\nthe workspace is Ready and has a writable blob-training PVC; the workspace has GPU quota and allocatable GPU capacity; the Portal has a Kusto query source for the workspace; and you have the platform-supplied taugrid-portal image pinned by digest. Each run uses one GPU. One available GPU can run them one at a time; three available GPUs can run all three at the same time.\nSet the image and offloader working directory in the terminal that starts the runs:\nexport TAU_METRICS_OFFLOAD_IMAGE=\u0026lt;platform-supplied-taugrid-portal@sha256:digest\u0026gt; export TAU_METRICS_OFFLOAD_OUT=/var/run/tau/metrics-offload Copy the example so train.py stays beside the Tau config files:\ncp -R examples/portal-ray-stellar ray-metric-study cp ray-metric-study/tau.yaml ray-metric-study/baseline.yaml cp ray-metric-study/tau.yaml ray-metric-study/short-run.yaml cp ray-metric-study/tau.yaml ray-metric-study/long-run.yaml Keep the same project and experiment name in all three files:\nexperiment: project: ray-tune-demo name: ray-metric-study Give each file its own run name, group, output folder, and step count:\nConfig Run name Group Output folder Steps ray-metric-study/baseline.yaml ray-baseline baseline /data/projects/shared-research/runs/ray-metric-study/baseline 20 ray-metric-study/short-run.yaml ray-short-run short-run /data/projects/shared-research/runs/ray-metric-study/short-run 10 ray-metric-study/long-run.yaml ray-long-run long-run /data/projects/shared-research/runs/ray-metric-study/long-run 40 For example, the changing part of ray-metric-study/baseline.yaml is:\nname: ray-baseline runtime: env: PORTAL_DEMO_STEPS: \u0026#34;20\u0026#34; storage: data_pvc: blob-training output: /data/projects/shared-research/runs/ray-metric-study/baseline policy: workspace: shared-research namespace: shared-research experiment: project: ray-tune-demo name: ray-metric-study group: baseline Use the matching values from the table in the other two files. Keep the metrics settings from the published example in every file. If the workspace uses a different Kubernetes namespace or PVC name, use the values supplied by the cluster owner.\nResearcher 1 starts the three runs:\ntau run --config ray-metric-study/baseline.yaml tau run --config ray-metric-study/short-run.yaml tau run --config ray-metric-study/long-run.yaml Researcher 1 sends Researcher 2 the shared experiment name:\nray-metric-study Researcher 1 opens the experiment through their port-forward and sends its full link to Researcher 2:\nhttp://127.0.0.1:8080/stellar?target=ray-metric-study\u0026amp;project=ray-tune-demo\u0026amp;workspace=shared-research Researcher 2 opens the link while their own port-forward is running. The experiment page should show all three runs together. Both researchers can compare the loss and accuracy charts for baseline, short-run, and long-run.\nWhat is shared Researchers in the same workspace can see:\nrun names and state; queue and waiting state; Job and Pod state; logs they are allowed to read; and metrics sent to the shared metrics store. If the views do not match Check these items in order:\nBoth researchers are using the same cluster. Both researchers are using the same workspace. Both researchers opened the same run name and project. Each researcher\u0026rsquo;s port-forward is still running. Both browser pages were refreshed. If a chart is still missing, send the run name, project, workspace, and missing metric name to the cluster owner.\n","description":"View the same runs, queue state, logs, and charts from two machines","permalink":"/taugrid/docs/developer-guide/share-research/","title":"Share research with a teammate"},{"contents":"Kueue is a Kubernetes-native queue and admission system for batch and AI workloads. It decides when a workload may consume shared quota. After admission, the Kubernetes scheduler decides which nodes run the pods.\nThe main Kueue objects A LocalQueue is the queue a workspace submits to. A ClusterQueue combines quota across one or more LocalQueues. A ResourceFlavor describes a class of capacity, such as CPU nodes or a particular GPU class. A Workload records the resources a Job or RayJob requests and its admission state. TauGrid resolves the workspace LocalQueue and adds it to the rendered workload. Kueue then evaluates quota, priority, queue order, and resource flavors.\nWhat waiting means A queued workload can be healthy even when no pods are running. Kueue keeps it pending until the required quota becomes available. TauGrid shows that state in tau run status so developers can distinguish queue waiting from pod startup or application failure.\nThe default TauGrid queue uses Kueue BestEffortFIFO. Older eligible work usually proceeds first, while priority and quota rules can change the order. Platform owners can replace the baseline with explicit per-team quotas and GPU flavors.\nSee Queue, quota, topology, and GPU placement or try the CPU queueing example.\n","description":"How Kueue shares quota and admits TauGrid workloads","permalink":"/taugrid/docs/getting-started/core-technologies/kueue/","title":"What is Kueue?"},{"contents":"A workspace connects repository intent to one cluster namespace, LocalQueue, workload ServiceAccount, and authorization contract.\nSingle active workspace Feature state: GA Reviewed 2026-08-12 TauGrid v0 activates one workspace per cluster. The CLI resolves that workspace from the checked-in connection descriptor or the cluster\u0026rsquo;s primary-workspace marker, so researchers can omit the workspace name for normal runs.\nThe active workspace must be Ready before TauGrid submits a workload. Readiness confirms the namespace, queue, workload identity configuration, and authorization contract described in the TauWorkspace reference.\nMultiple workspaces Feature state: Alpha Reviewed 2026-08-12 A cluster can contain multiple TauWorkspace objects, but only one is active. The controller marks additional workspaces Degraded with reason AdditionalWorkspaceBlocked and removes their LocalQueue, researcher RoleBinding, and ServiceAccount, leaving the namespace without workspace metadata until promotion.\nAfter the active workspace is deleted and its owned resources are cleaned up, the controller can promote a remaining workspace. This Alpha lifecycle supports controlled, one-at-a-time workspace replacement; concurrent active workspaces and concurrent tenant isolation remain out of scope for this stage.\nPlatform operators can inspect all workspace objects:\ntau workspace list tau workspace status \u0026lt;name\u0026gt; Add capacity to the active workspace\u0026rsquo;s queue, or use a separate cluster, instead of creating a second workspace for concurrent capacity.\n","description":"Understand workspace routing, readiness, and the Alpha multi-workspace lifecycle","permalink":"/taugrid/docs/developer-guide/concepts/workspaces/","title":"Workspaces"},{"contents":" Feature state: GA Reviewed 2026-08-18 This page documents the Helm values accepted by tau cluster install. The distribution chart bundles Kueue, KubeRay, the Tau core controller, GPU monitoring, the taugrid-core services chart, and a portable baseline queue into a single versioned release. Portal is enabled by default for the operator quickstart; Stellar, lifecycle recorder, and image prewarm remain disabled until the platform opts in.\nPrint this reference from your terminal:\ntau cluster explain-values The Helm release namespace is the only namespace setting for TauGrid system workloads and Services. tau cluster install defaults it to tau-system; --namespace \u0026lt;name\u0026gt; moves the Kueue, KubeRay, Tau controller, Portal, GPU monitoring, and other enabled first-party workloads together. The first-party charts follow their Helm release namespace, and the deprecated gpu-monitoring.namespace override must remain empty. Cluster-scoped resources remain cluster-scoped, and Kueue keeps its Kubernetes API aggregation binding in kube-system.\nMultiKueue capability Key Type Default Description kueue.aksExtension.enableMultiKueue bool true Install the pinned Kueue MultiKueue controller capability The standard installation includes MultiKueue API/controller support and read-only prerequisite observation. Operators separately create worker credentials, AdmissionChecks, MultiKueueConfigs, MultiKueueClusters, dedicated queues, and profiles. TauCluster.status.conditions[MultiKueueReady] reports the health of those actual operator-owned prerequisites. See Multi-cluster execution before publishing a MultiKueue profile.\nComponents Toggle sub-charts with components.\u0026lt;key\u0026gt;.enabled:\nKey Default Description components.kueue.enabled true Kueue job scheduler components.kuberayOperator.enabled true KubeRay operator components.tauCoreController.enabled true Tau core controller (TauWorkspace, TauCluster) components.taugridCore.enabled true Include the services chart, including the default Portal components.gpuMonitoring.enabled unset GPU monitoring follows components.tauCoreController.enabled until explicitly set Baseline Queue A portable Kueue queue bootstrapped on first install. These quotas bound concurrent Kueue admission; Kubernetes scheduling still enforces real capacity. Production operators should replace them with deliberate capacity policy.\nKey Type Default Description baselineQueue.enabled bool true Create the ClusterQueue, LocalQueue, ResourceFlavors, and Topology baselineQueue.name string jobqueue LocalQueue name (DNS label) baselineQueue.namespaceSelector object matchExpressions: [{key: tau.azure.com/workspace, operator: Exists}] Namespaces that receive the LocalQueue baselineQueue.topology.enabled bool true Create a Topology object for hostname-level scheduling baselineQueue.topology.name string default-node-topology Topology object name baselineQueue.topology.requiredLevel string kubernetes.io/hostname Required topology level copied from managed GPU flavors to generated pod templates; custom levels are rendered above the always-present hostname leaf baselineQueue.flavor.* object taugrid-default-cpu, Linux, no tolerations CPU/memory flavor; keep GPU labels and tolerations out baselineQueue.resources list cpu: 100000, memory: 100Ti CPU/memory admission quota baselineQueue.gpu.enabled bool true Add GPU resources and flavors to the node-resource group baselineQueue.gpu.coveredResources list nvidia.com/gpu GPU resources covered by the node-resource group baselineQueue.gpu.flavors list generic taugrid-default-gpu GPU flavors and per-flavor quotas CPU, memory, and GPU share one Kueue resource group so each GPU pod set receives one node flavor across all of its requested resources. taugrid-default-cpu has zero GPU quota, while the generic taugrid-default-gpu has CPU/memory plus GPU quota and supports gpu_class: any on a fresh install. When hardware is known, replace the GPU flavor list with class-specific flavors and label matching nodes with the canonical A10, A100, H100, H200, GB200, or GB300 class from policy.gpu_class. Only GPU flavors carry topologyName and the managed kueue.x-k8s.io/podset-required-topology metadata annotation. Connected TauGrid submission copies that requirement onto generated GPU pod templates when no explicit placement policy is present. Raw Kubernetes manifests remain expert-controlled. The CPU/memory flavor remains non-TAS. For upgrades with saved legacy values, remove GPU resources from baselineQueue.resources and move all GPU class/series labels and GPU-node tolerations out of baselineQueue.flavor before adding their replacements under baselineQueue.gpu.flavors. Declare GPU-node taints under each flavor\u0026rsquo;s nodeTaints. TauGrid fails template rendering if the old mixed values would duplicate GPU coverage or constrain CPU-only admission. Replace the generic GPU flavor with class-specific flavors rather than keeping both: exact class quota must not fall back to an unlabeled ResourceFlavor.\nPortal The following are defaults of the TauGrid umbrella distribution used by tau cluster install. The standalone taugrid-core chart keeps Portal disabled, so platforms that install that child chart directly must opt in explicitly. Portal follows the Helm release namespace selected by tau cluster install --namespace; the CLI default is tau-system.\nKey Type Default Description taugrid-core.portal.enabled bool true Install the operator Portal taugrid-core.portal.serviceAccount.create bool true Create the dedicated Portal ServiceAccount taugrid-core.portal.serviceAccount.name string tau-portal Portal ServiceAccount name taugrid-core.portal.rbac.create bool true Create cluster-wide read-only Kubernetes RBAC for Portal These defaults make the Portal shell, Runs and run-detail views, Cluster Nodes view, and live Ray discovery available to an operator through the ClusterIP Service. See Configure Portal to separately configure Kusto-backed boards, the scoped computed Jobs board, KueueViz, an authenticated researcher endpoint, and a durable experiment store.\nbaselineQueue.gpu.flavors Declare GPU-node taints (for example sku=gpu:NoSchedule) in each GPU flavor\u0026rsquo;s nodeTaints. This makes the flavor ineligible for CPU-only pods if generic CPU quota is exhausted. TauGrid injects sku=gpu and nvidia.com/gpu tolerations into GPU workloads, so those workloads remain eligible. Keep a matching taint out of the flavor\u0026rsquo;s tolerations: repeating it there would make Kueue automatically tolerate it for every pod and remove the CPU-isolation guard.\n# taugrid-values.yaml baselineQueue: gpu: flavors: - name: a100-pool nodeLabels: kubernetes.io/os: linux tau.azure.com/gpu-class: a100-80gb nodeTaints: - key: sku value: gpu effect: NoSchedule tolerations: [] resources: - name: nvidia.com/gpu nominalQuota: \u0026#34;1\u0026#34; Sub-Chart Pass-Through The remaining top-level keys pass values directly to embedded sub-charts:\nPrefix Sub-chart Common overrides kueue.* Kueue v0.18 controllerManager.manager.image, managerConfig kuberay-operator.* KubeRay v1.6 image, configuration, podAnnotations tau-core-controller.* Tau controller image, tauCluster.nodeLabelRules taugrid-core.* Services chart prewarm.enabled, stellar.enabled, portal.enabled gpu-monitoring.* GPU monitoring gpuSkus, daemonset, metricsCollector, namespace Example: GPU Cluster TauGrid\u0026rsquo;s controller derives the node contract from node.kubernetes.io/instance-type and repairs drift:\nAKS VM size GPU series GPU class Standard_NC24ads_A100_v4 nc24ads-a100-v4 a100-80gb Standard_ND96amsr_A100_v4 ndm-a100-v4 a100-80gb Standard_NC40ads_H100_v5 nc-h100-v5 h100-95gb Standard_ND96isr_H200_v5 nd-h200-v5 h200-141gb Use tau-core-controller.tauCluster.extraNodeLabelRules for another reviewed VM size. Setting nodeLabelRules replaces the built-in catalog. CPU-only clusters and GPU pools scaled to zero remain ready when no catalog entry currently matches.\n# taugrid-values.yaml: H200 cluster with GPU taints baselineQueue: gpu: flavors: - name: h200-pool nodeLabels: kubernetes.io/os: linux kueue.azure.com/gpu-series: nd-h200-v5 tau.azure.com/gpu-class: h200-141gb nodeTaints: - key: sku value: gpu effect: NoSchedule tolerations: [] resources: - name: nvidia.com/gpu nominalQuota: \u0026#34;8\u0026#34; See Also CLI reference: tau cluster Install TauGrid Source: charts/taugrid/values.yaml ","description":"TauGrid distribution chart configurable values","permalink":"/taugrid/docs/reference/cluster-install-values/","title":"Cluster install values"},{"contents":" Feature state: GA Reviewed 2026-07-16 The normal researcher contract is a checked-in direct run config:\nname: train engine: rayjob entrypoint: train.py compute: workers: 2 gpus_per_worker: 8 storage: data_pvc: training-data TauGrid combines:\nExplicit project and target selection. Repository or monorepo discovery. The project\u0026rsquo;s workspace connection descriptor. Platform-owned workspace defaults. Checked-in workload intent. Temporary explicit operator overrides. Ambiguity is an error. Dry-run output must keep default sources visible.\ntau run validate --config tau/train.yaml tau run train --dry-run=client tau run explain-config See direct run config vs. managed workflow manifest for how Python SDK-generated manifests relate to this default format.\n","description":"How repository and workspace intent become a workload","permalink":"/taugrid/docs/developer-guide/concepts/config-resolution/","title":"Configuration resolution"},{"contents":" Feature state: GA Reviewed 2026-07-16 tau serve turns a project-owned image and optional durable checkpoint into a service. Choose:\n--kind=rayservice (default) for a Ray Serve application; KubeRay must be installed and the image must expose the specified Python import path. --kind=deployment for a plain Kubernetes Deployment such as a raw vLLM, TGI, Triton, or custom HTTP server. You need a platform-provided serving profile, a pinned image, the target namespace/context, and a checkpoint visible from the serving PVC when the endpoint loads model state.\nServed workloads are admitted through Kueue. tau serve resolves the platform-managed default LocalQueue from the target namespace and stamps it on the workload automatically, handling queue selection on the researcher\u0026rsquo;s behalf. If the namespace has no usable default LocalQueue, deployment fails with an onboarding error rather than creating pods Kueue never admits.\nRender before deployment Use a resolved checkpoint path for an offline client dry-run:\ntau serve deploy \u0026lt;service-name\u0026gt; \\ --kind=rayservice \\ --profile \u0026lt;serve-profile\u0026gt; \\ --image \u0026lt;pinned-image\u0026gt; \\ --import-path serve:app \\ --checkpoint \u0026lt;checkpoint-path\u0026gt; \\ --checkpoint-pvc \u0026lt;pvc\u0026gt; \\ --namespace \u0026lt;namespace\u0026gt; \\ --context \u0026lt;context\u0026gt; \\ --dry-run=client --checkpoint mounts the selected PVC at /data, resolves relative paths under /data/checkpoints, and sets TAU_MODEL_PATH. Your application still owns how it loads the model and handles requests.\nDeploy and inspect Drop --dry-run=client from the same command, then inspect the endpoint:\ntau serve status \u0026lt;service-name\u0026gt; \\ --kind=rayservice \\ --namespace \u0026lt;namespace\u0026gt; \\ --context \u0026lt;context\u0026gt; Preview the inference experience The strongest inference examples render real model behavior instead of replaying a canned response. This browser-only specimen loads the artifact produced by the market-policy TauGrid workload: an 8-input, 24-hidden-unit policy and value network trained for a synthetic market-making environment, then exported for module Worker inference.\nWatch an actor-critic policy run live The exact 316-parameter model exported by the checked-in TauGrid training workload.\nLoading market policy Price 100.00 Signal 0.000 Volatility 0.000 Inventory 0.00 Step reward 0.000 Total reward 0.000 Run policy New market Synthetic environment only. No live market data and no financial advice.\nActor and critic Loading Preparing model The Worker is loading trained weights.\nPredicted step reward - Action probabilities Available after the first inference. Observation 8 environment values Worker inference 24-unit hidden layer Policy + value 3 actions and reward Environment step Price, inventory, reward Training device - Parameters 316 Held-out accuracy - Last inference - Synthetic research workload. The repository example trains this exact PyTorch architecture on a TauGrid GPU worker. Its exported Short / Flat / Long actor and reward critic run here without a server. This interactive example requires JavaScript. Run the exact trainer on TauGrid:\ntau run --config examples/market-policy/tau.yaml --dry-run=client tau run --config examples/market-policy/tau.yaml Before submitting, set storage.data_pvc in the example to the writable PVC from your platform handoff. The RayJob resolves namespace and queue policy from the configured TauWorkspace, dispatches training to one h200-141gb worker, and writes the same tau-market-policy.json format loaded above to durable workspace storage. The manifest pins TauGrid\u0026rsquo;s public MCR Ray/CUDA image by digest and installs exact PyTorch and NumPy versions through runtime.pip, so the workspace needs package-index access during startup. For deterministic site verification, make train-market-policy from site/ invokes the same train.py twice with CPU explicitly selected and compares the exports while leaving the checked-in H200 artifact untouched. The environment is synthetic and is not financial advice. To adapt the pattern to another endpoint, keep the same proof structure: accept real input, render domain-native output, expose useful model state, and identify the exact layer that measured latency.\nIf a completed managed finetune or model-registry entry already records the checkpoint, use --from-finetune \u0026lt;run\u0026gt; or --from-model \u0026lt;model-ref\u0026gt; instead of --checkpoint. Those forms read Kubernetes metadata and therefore cannot use client dry-run; use --dry-run=server or a live deployment.\nFor a non-Ray server, switch to --kind=deployment. Add --deployment-port when the container listens on a port, and --readiness-path / --service-port when the platform should render those contracts: a probe path needs a port from --service-port, --service-target-port, or --deployment-port.\nScale or remove Plain Deployments can be scaled directly:\ntau serve scale \u0026lt;service-name\u0026gt; \\ --kind=deployment \\ --replicas 3 \\ --namespace \u0026lt;namespace\u0026gt; \\ --context \u0026lt;context\u0026gt; RayService scaling goes through redeploy: because its Serve config is a serialized field, redeploy a RayService with --replicas to change the count, or set --min-replicas and --max-replicas when creating it.\nRemove either kind explicitly:\ntau serve delete \u0026lt;service-name\u0026gt; \\ --kind=rayservice \\ --namespace \u0026lt;namespace\u0026gt; \\ --context \u0026lt;context\u0026gt; Serving reads the namespace and context you pass explicitly, sourced from the platform handoff, rather than from tau/workspace.connection.yaml. The project image must provide the configured import path and all runtime dependencies.\n","description":"Render, deploy, inspect, and remove an online endpoint","permalink":"/taugrid/docs/developer-guide/serve-model/","title":"Serve a trained model"},{"contents":" Feature state: GA Reviewed 2026-07-16 The TauGrid documentation site and GitHub Wiki are both maintained as complete surfaces. They are edited separately, so semantic parity is a human ownership responsibility rather than an automated synchronization claim.\nsite/data/wiki-parity.toml records:\nThe maintained site page and wiki counterpart. The owning team. Current maturity. Last joint review date. The documentation pipeline validates that mapped site pages exist and metadata is current whenever the site changes. Its consolidated Azure DevOps pipeline also runs the same parity and review-age checks on the Monday parity-only schedule.\nWhen user-facing behavior or maturity changes:\nUpdate the versioned site page. Update the corresponding wiki page. Review both as one documentation change. Refresh the parity manifest review date. The checks detect stale ownership and missing mappings; confirm that two independently edited pages say the same thing by reviewing both together, as in step 3 above.\n","description":"Governance for two separately maintained documentation surfaces","permalink":"/taugrid/docs/community/wiki-parity/","title":"Site and wiki parity"},{"contents":" Feature state: Alpha Reviewed 2026-07-16 TauWorkspace is a platform-authored Kubernetes resource in the TauGrid system namespace (tau-system by default). A custom tau cluster install --namespace \u0026lt;name\u0026gt; moves the controller and these objects together; administrative workspace commands use --system-namespace \u0026lt;name\u0026gt;, and repository descriptors persist the value as cluster.systemNamespace. It reconciles or verifies:\nTarget namespace and namespaced RBAC. Kueue LocalQueue accessibility. Workload identity ServiceAccount configuration. Readiness and drift conditions. Platform teams provision Azure resources and cluster-scoped quota policy, projects own their own code, and platform teams verify the durable storage PVC separately.\nFor tau workspace create, --system-namespace selects where the TauWorkspace object lives and defaults to the installed TauGrid system namespace; --namespace selects the researcher workload Namespace and defaults to the workspace name. These are intentionally separate because system components share one namespace while workload isolation remains workspace-scoped.\nWorkspace status is intentionally agent-readable:\ntau workspace list tau workspace status \u0026lt;name\u0026gt; Workspace RBAC is the API default and what tau workspace create writes; the controller binds the researcher subject in the workspace namespace. Cluster-wide authorization is an explicit opt-out: the platform grants researcher access separately. Some existing clusters use this mode. The multiple-workspace lifecycle is Alpha: v0 activates one workspace and blocks additional workspace objects until the active workspace is removed.\nThe controller source and raw Kustomize manifests live in the Azure/taugrid repository. Versioned MCR release artifacts include the controller image, the standalone controller OCI chart, the TauGrid umbrella OCI chart, and the CRDs packaged by both charts.\nFresh clusters consume the umbrella chart through tau cluster install. The standalone chart is only for a separately managed controller and must not be installed alongside an umbrella release that already enables components.tauCoreController. See Install TauGrid for the supported installation entry point and readiness gate. This page defines the workspace API and readiness contract only; installing the cluster control plane is covered there.\nWorkspace readiness and recovery Start with the condition reason and message, rather than a guessed repair:\ntau workspace status \u0026lt;name\u0026gt; --context \u0026lt;context\u0026gt; tau workspace status \u0026lt;name\u0026gt; --context \u0026lt;context\u0026gt; -o json The overall phase currently requires RBACReady and QueueReady; DriftDetected=True also makes it Degraded. Storage remains platform-owned desired state, tracked outside TauWorkspace\u0026rsquo;s own conditions.\nCondition Platform action RBACReady=False In workspace-rbac mode, correct the declared subject/role and let the controller reconcile namespaced RBAC. In cluster-wide mode, repair the pre-existing authorization that the platform manages directly. QueueReady=False Restore the named LocalQueue and its accessible backing ClusterQueue; fix that queue directly rather than creating a second queue with a different name to bypass the workspace spec. DriftDetected=True Fix the dependency or workspace spec named in the message, then let the controller restore its owned namespace/RBAC objects. Ready proves RBAC and queue reconciliation only, not storage. TauGrid 0.1 has no StorageReady condition on TauWorkspace or TauCluster, so a workspace can reach Ready while a workload\u0026rsquo;s configured PVC is missing or unbound. Verify the platform-managed claim directly before handing the workspace to a researcher:\nkubectl get pvc blob-training -n \u0026lt;workspace-namespace\u0026gt; --context \u0026lt;context\u0026gt; Even Bound proves existence only; write validation happens at mount time on the workload\u0026rsquo;s own pod, so failures such as wrong BlobFuse credentials or a read-only mount surface there rather than in workspace status.\nWorkloadIdentityReady is diagnostic; the overall phase gate currently excludes it. Resolve it before handing off workloads that rely on Azure Workload Identity.\nThe controller reconciles again periodically. After repairing the named dependency, wait for:\ntau workspace check \u0026lt;name\u0026gt; --context \u0026lt;context\u0026gt; TauGrid\u0026rsquo;s normal submission path always honors workspace readiness, with no bypass flag. Raw Kubernetes clients are governed by ordinary RBAC and Kueue rather than a custom TauGrid admission policy.\n","description":"Kubernetes workspace desired state and readiness","permalink":"/taugrid/docs/reference/workspace/","title":"TauWorkspace reference"},{"contents":"adx-mon is Azure Data Explorer Monitor, an observability pipeline for Kubernetes clusters. It collects metrics, logs, and GPU telemetry, batches the data, and sends it to Azure Data Explorer (ADX/Kusto).\nIts main components collect node and cluster signals, ingest batches into ADX, manage ADX schema commands, and evaluate configured alerts. Platform teams choose the signals, databases, retention, identities, and access rules.\nHow TauGrid uses adx-mon adx-mon is an optional platform integration. A TauGrid environment can use it to:\ncollect Kubernetes, Kueue, and GPU signals for fleet views; receive scalar experiment metrics from a TauGrid metrics sidecar; create approved tables, mappings, and functions through its ManagementCommand resource; and send data to ADX for dashboards, searches, and alerts across workspaces. adx-mon moves data into ADX. The TauGrid Portal and Stellar query ADX through their own read identities.\nTauGrid also saves experiment evidence alongside each run\u0026rsquo;s output files. That local evidence supports retrieval and comparison, while the optional ADX copy supports platform-wide analysis.\nPlatform owners can continue with Prepare ADX/Kusto for TauGrid and Observability and evidence.\n","description":"How adx-mon sends Kubernetes and experiment telemetry to Azure Data Explorer","permalink":"/taugrid/docs/getting-started/core-technologies/adx-mon/","title":"What is adx-mon?"},{"contents":" Feature state: GA Reviewed 2026-08-17 TauGrid uses canonical annotated SemVer tags (vX.Y.Z) and a manually authorized GitHub Actions workflow. Publishing a release requires a manually authorized workflow run beyond the tag push itself.\nThe workflow:\nVerifies the annotated tag and reviewed main commit. Requires checked-in release notes. Runs Go and Python release gates. Compares two independent binary builds. Builds the Python SDK wheel twice and requires byte-for-byte identical output. Publishes raw binaries, the SDK wheel, install.sh, LICENSE, and SHA256SUMS as new assets, leaving any existing ones untouched. Verifies every uploaded digest. Proves CLI and SDK installation on clean Ubuntu and macOS runners. The Python SDK keeps its own package version, but its source-aligned tau-*.whl is published in the same GitHub Release as the CLI.\nSee cli/RELEASING.md.\n","description":"How Tau CLI binaries and source-aligned SDK releases are published","permalink":"/taugrid/docs/reference/releases/","title":"Release contract"},{"contents":"Stellar is TauGrid\u0026rsquo;s experiment tracking and comparison experience. It ships in the taugrid-portal binary and presents run details, scalar metric history, and side-by-side comparisons in a browser or terminal.\nWhat Stellar helps answer Which code and settings produced this run? How did loss, accuracy, throughput, or another metric change over time? Which run performed best? Where are the model, checkpoint, and other output files? Did the run finish, retry, or resume? Where the data comes from TauGrid saves complete metric files and a small experiment index close to the run. Stellar can read that saved evidence directly. A platform can also provide an ADX/Kusto view for searching and comparing runs across workspaces.\nadx-mon can move metric samples into ADX, while Stellar reads and presents the result. The saved run files remain available for retrieval and detailed inspection.\nStellar and the Ray dashboard The Ray dashboard shows live tasks, actors, and workers while a Ray cluster is running. Stellar shows experiment evidence that remains useful after the runtime pods have finished.\nOpen a local dashboard with:\ntaugrid-portal experiment stellar \u0026lt;run-name\u0026gt; Try Live experiment evidence to publish loss and accuracy, retrieve output files, and open the run in Stellar.\n","description":"How Stellar turns saved run data into experiment views and comparisons","permalink":"/taugrid/docs/getting-started/core-technologies/stellar/","title":"What is Stellar?"},{"contents":" Feature state: GA Reviewed 2026-08-25 Every run should save enough information for you to answer four questions:\nWhat code and settings produced this result? Did the model improve during the run? Where are the checkpoint and output files? Can I compare, reproduce, or resume this run later? What TauGrid saves Saved data Examples Where it is saved Run details Project, run ID, config, and input references TauGrid experiment index Metrics over time Loss, accuracy, throughput, and GPU use Metric files Metric summary First and latest step, minimum, maximum, and point count TauGrid experiment index Output files Checkpoints, models, images, tables, profiles, and reports The directory set by storage.output Run history Attempts, retries, resumes, and final status TauGrid experiment index and Kubernetes workload The output files stay on the storage chosen by the platform team. TauGrid keeps an index so you can find and compare them.\nHow metric storage works TauGrid saves the complete metric history close to the run. This keeps the data available even when a dashboard or external data service is offline.\nThe run writes complete metric files. A file becomes visible only after the write finishes. Readers never receive a half-written file. TauGrid stores the points in a consistent format. Each run uses the same fields for names, steps, times, and values. The saved files use Parquet, which works well for large metric histories. A retry does not create another copy. TauGrid recognizes a repeated write and reuses the saved record. If the content changed, TauGrid reports the conflict instead of replacing the old data. TauGrid builds a short summary. For each metric, it records the number of points, the step range, the minimum and maximum values, and the latest valid value. Invalid numbers such as NaN and infinity are counted separately. Dashboards can use an optional copy. Platform teams can send metric points to ADX/Kusto for dashboards and searches across many workspaces. The files saved with the run remain the source of truth. Each metric point includes:\nField What it tells you project, run_group_id, run_id Which experiment and run produced the point metric_name What was measured, such as loss or accuracy step, wall_time When the point occurred in training and on the clock value, unit The measured number and its unit, when one applies source, split, tags Where the point came from and useful labels such as dataset split or workspace You can review the same metric history in the TauGrid Portal. The saved run data remains the source of truth.\nSave output files Set storage.output to a directory on the workspace storage:\nstorage: data_pvc: blob-training output: /data/projects/vision-lab/runs/train-042 The run can write directly to that directory. Use staged publishing when output files should appear only after all writes finish:\nstorage: data_pvc: blob-training output: /data/projects/vision-lab/runs/train-042 publish: staged With staged publishing, the run writes files to TAU_OUTPUT_STAGING_DIR. TauGrid checks the files, copies them into the durable output directory, and marks the set as complete. Commands only return a completed set.\nRetrieve output files List the files saved by a run:\ntau run get \u0026lt;run-name\u0026gt; -n \u0026lt;workspace-namespace\u0026gt; Fetch one file:\ntau run get \u0026lt;run-name\u0026gt; -n \u0026lt;workspace-namespace\u0026gt; \\ --artifact reports/evaluation.json -o raw Set storage.checkpoint when a run produces a model that you plan to serve. The value points to one file or directory inside the run\u0026rsquo;s output. TauGrid records its location so this command can find it later:\ntau serve deploy --from-finetune \u0026lt;run-name\u0026gt; Try a live example Run the live experiment evidence example to publish loss and accuracy, inspect the saved files with tau run get, and view the metric history in the TauGrid Portal.\nSee Observability for platform telemetry, Prepare ADX/Kusto for optional dashboards across many workspaces, and Retry and resume for recovery state.\n","description":"What TauGrid saves from a run, where it goes, and how to retrieve it","permalink":"/taugrid/docs/developer-guide/concepts/evidence/","title":"Experiment evidence and artifacts"},{"contents":"Each diagram answers one reader question and preserves an editable source, SVG, and PNG. Solid elements represent GA capabilities. Orange dashed elements represent Alpha or Beta capabilities. Purple dashed elements represent Planned work.\nUnderstand and evaluate TauGrid in one picture What is TauGrid, and what is it not? SVG PNG Editable source Why TauGrid What repeated friction does TauGrid remove? SVG PNG Editable source Choosing the right front door When should I use TauGrid versus adjacent approaches? SVG PNG Editable source TauGrid personas Who uses TauGrid? SVG PNG Editable source Capability map What can TauGrid do today? SVG PNG Editable source Adopt and configure Adoption paths How can I try or deploy TauGrid? SVG PNG Editable source TauGrid quickstart flow SVG PNG Editable source Identity and security trust chains Who is the researcher and what may the workload access? SVG PNG Editable source TauWorkspace reconciliation What does TauWorkspace reconcile? SVG PNG Editable source Azure resource blueprint Which Azure resources support a TauGrid-enabled cluster? SVG PNG Editable source Use TauGrid day to day Researcher interaction journey How does a researcher submit, observe, retrieve, and iterate? SVG PNG Editable source Workload chooser Which TauGrid path should I use? SVG PNG Editable source Repository and config resolution How does repository intent become workload intent? SVG PNG Editable source Data and artifact lifecycle Where do inputs and outputs live? SVG PNG Editable source Experiment evidence flow How does Stellar remain local first? SVG PNG Editable source Failure recovery What happens when a run is preempted or fails? SVG PNG Editable source Training-to-serving direction How could a successful experiment become an endpoint? SVG PNG Editable source Operate TauGrid Queue and GPU placement Why is my run waiting, and where will it execute? SVG PNG Editable source Troubleshooting layers Where is a stuck run blocked? SVG PNG Editable source Observability and evidence Which signals prove platform and workload health? SVG PNG Editable source Experimental multi-cluster execution How can TauGrid route work across clusters? SVG PNG Editable source Extend and contribute Code architecture and extension points Where should a contributor make a change? SVG PNG Editable source Contribution-to-release path How is a contribution validated and shipped? SVG PNG Editable source ","description":"Twenty-three diagrams from evaluation through contribution","permalink":"/taugrid/docs/overview/visual-curriculum/","title":"TauGrid visual curriculum"},{"contents":" Feature state: GA Reviewed 2026-07-16 This page is the canonical source for TauGrid terminology. Other concept and reference pages link to a definition here instead of redefining it. Where a term below conflicts with older prose elsewhere, this definition wins.\nTauGrid The CLI (and its companion Python SDK) that resolves a project\u0026rsquo;s checked-in workload contract, applies platform-owned policy, renders a Kubernetes Job or KubeRay RayJob, submits it, and gives the researcher one lifecycle surface for status, logs, results, retry, and resume. TauGrid resolves, renders, and observes workload lifecycle; Kueue and Kubernetes own scheduling and orchestration, and platform teams provision cloud infrastructure. See What is TauGrid and Architecture.\nRepository / research repository One Git worktree containing a TauGrid-enabled research project, or a monorepo catalog of several. The repository owns entrypoint code, images, runtime dependencies, and data/output contracts; TauGrid only resolves and renders against it.\nProject The unit inside a repository that owns one or more targets and exactly one workspace connection descriptor at a time. A monorepo can hold multiple projects.\nTarget A checked-in, named, runnable config \u0026ndash; for example tau/train.yaml, or a direct run config passed with tau run --config. \u0026ldquo;Target\u0026rdquo; means \u0026ldquo;which checked-in workload to run.\u0026rdquo; It is unrelated to cluster context; keep the two distinct.\nCluster context The kubectl context TauGrid operates against, selected with --context or the TAU_CONTEXT environment variable. Cluster context answers \u0026ldquo;which cluster,\u0026rdquo; never \u0026ldquo;which checked-in workload.\u0026rdquo; See target for that meaning.\nWorkspace / workspace environment Shorthand for the platform-owned destination and shared defaults a project resolves against: cluster, namespace, queue, priority, output root, and identity. This shorthand is safe when the surrounding text describes policy rather than a specific object. When precision matters, say TauWorkspace for the Kubernetes resource, or workspace connection descriptor for the client-side file a repository holds.\nTauWorkspace The Kubernetes custom resource (tau.azure.com/v1alpha1, kind TauWorkspace) that the Tau workspace controller reconciles in the tau-system namespace. It is a Kubernetes API for onboarding, policy intent, and status: its spec declares target namespace, Kueue queue, and defaults for workload identity, output root, and priority; its controller reconciles or verifies namespace/RBAC, Kueue LocalQueue accessibility, and workload-identity ServiceAccount configuration, then reports status conditions and phase. The spec has no storage field: durable storage is external platform desired state.\nTauWorkspace is a Kubernetes onboarding and policy intent and status API scoped to namespace, queue, and workload-identity wiring. The platform operator and Azure/provider tooling own the AKS cluster, network, node pools, Kueue/KubeRay installation, and cloud RBAC outside TauWorkspace. See TauWorkspace reference and Identity and security boundaries.\nWorkspace connection descriptor The non-secret tau/workspace.connection.yaml file a platform operator hands to a repository once its TauWorkspace is Ready. It names the Kubernetes context, access method, TauGrid system namespace, and workspace contract a project resolves against and must never contain a credential or kubeconfig. This file is client-side project configuration only: the TauWorkspace object it describes is reconciled independently by the controller and is unaffected by edits to this file. cluster.systemNamespace defaults to tau-system.\nChecking in the descriptor is explicit repository/platform preconfiguration. tau run discovers it automatically; tau workspace connection verifies and pins the configured access before the first run, while --offline validates only the repository configuration. On first cluster-backed use, TauGrid isolates the named context from the user\u0026rsquo;s kubeconfig or obtains AKS cluster-user credentials, verifies the live workspace contract, and records a durable configuration pin separately from short-lived readiness evidence. An unchanged pinned connection can refresh Ready, LocalQueue, and authorization checks noninteractively after the readiness cache expires. Descriptor, trust, or live workspace-contract drift requires interactive review.\nworkspace-rbac is the API default and what tau workspace create writes. In that mode the controller binds the researcher subject in the workspace namespace. cluster-wide is an explicit opt-out: the workspace supplies only routing and policy defaults, and the platform grants researcher access separately, which is how some existing clusters are configured. The multiple-workspace lifecycle is Alpha: v0 activates one workspace and blocks additional workspace objects until the active workspace is removed. Researcher isolation still requires its negative-access gate.\nStatus condition / Ready A TauWorkspace reports Kubernetes-style status conditions and an overall status.phase of Pending, Ready, or Degraded. \u0026ldquo;Ready\u0026rdquo; currently means RBACReady and QueueReady are true and no drift is detected. WorkloadIdentityReady is diagnostic; the overall phase gate currently excludes it. TauGrid 0.1 has no StorageReady condition on TauWorkspace or TauCluster, so Ready confirms only the conditions above: a platform-managed durable PVC may still be missing or unbound, Azure infrastructure may be older than the current reconcile, and researcher-scoped isolation needs its own check; see Identity and security boundaries.\nProfile / resource profile The render-time resource contract (name, lane, and spec) a workload builder consumes to size compute \u0026ndash; GPU, CPU, and memory intent \u0026ndash; for a run. A profile describes shape only. It is narrower than a topology preset, which additionally decides queue, priority, and placement routing.\nTopology and placement The platform-owned mapping from a researcher-facing preset (for example azure.research.training.l) to Kueue-facing queue, priority, and topology metadata: which queue admits the workload, its priority class, and any required or preferred node topology for pod placement. TauGrid resolves topology intent; Kueue and Kubernetes still own admission and scheduling. See Queue, quota, topology, and GPU placement.\nQueue / LocalQueue / ClusterQueue Kueue objects, not TauGrid objects. A LocalQueue is the tenant-facing entry point a workload references. A ClusterQueue owns shared quota and fairness across the LocalQueues bound to it. TauGrid resolves which LocalQueue a run should target; Kueue decides admission. See Queue, quota, topology, and GPU placement.\nDirect run config vs. managed workflow manifest The normal researcher contract is a direct run config: a checked-in tau run --config YAML file (name, engine, compute, storage, and so on) that TauGrid validates and renders directly. Its conventional filename is tau.yaml.\nA managed workflow manifest carries schema_version: 1 and is normally generated by the Python SDK for staged train/eval or renderer-level workflows. It may also be named tau.yaml, so the schema, not the filename, identifies the format. It is machine-generated output rather than a hand-authored default a researcher writes directly. See Configuration resolution and the run config reference.\nRun One execution of a target plus its lifecycle handle \u0026ndash; the object tau run status, tau logs, tau run get, tau run cancel, and tau run resume operate on.\nWorkload (Job / RayJob) The rendered Kubernetes object TauGrid submits: a batch/v1 Job for single-pod work, or a KubeRay RayJob for multi-node Ray execution. The workload is what Kueue admits and Kubernetes schedules: a run\u0026rsquo;s rendered execution artifact, downstream of the run\u0026rsquo;s own lifecycle handle.\nService / endpoint An online lifecycle target rendered by tau serve as either a KubeRay RayService or a Kubernetes Deployment. A service consumes a project-owned image and optional durable checkpoint, and it keeps its own lifecycle surface via tau serve status and tau serve delete, separate from run lifecycle commands.\nExperiment A comparison set over runs, their metrics, and their artifacts, scoped by experiment.project. Two fields define that identity inside a direct run config\u0026rsquo;s experiment block:\nname \u0026ndash; the stable experiment identifier (experiment.name). group \u0026ndash; a named arm of runs within that experiment (experiment.group). Evidence / metrics / artifacts / checkpoints Files and records saved by a run: metric history, summaries, checkpoints, model outputs, images, tables, reports, and retry or resume state. TauGrid keeps the durable copy with the experiment. The TauGrid Portal and optional ADX/Kusto dashboards provide additional ways to view and compare it. See Experiment evidence and artifacts.\n","description":"Canonical TauGrid terms, so overlapping words mean one thing everywhere","permalink":"/taugrid/docs/reference/glossary/","title":"Glossary"},{"contents":"TauGrid is a Kubernetes-native workflow layer for AI workloads. It installs on a Kubernetes cluster your platform provisions and operates. This is the canonical, provider-neutral installation path.\nThis page takes a platform operator from an empty workstation to a validated TauGrid workspace and a completed smoke workload. The portable path works on Kubernetes 1.30 or newer. Provider-specific identity, networking, storage, GPU drivers, and cluster lifecycle remain owned by your platform.\nWhat you will set up Install the Tau CLI and local Kubernetes tools. Select or provision a Kubernetes cluster. Install and validate the TauGrid control plane. Create a workspace for workloads. Submit a smoke workload. The first four steps require cluster-administrator access. Researchers need only the Tau CLI, access to the finished workspace, and the repository or configuration they will run.\n1. Install the Tau CLI Install these tools on the operator workstation:\nkubectl Helm 3 or 4 Git the released tau CLI Install the latest Tau CLI release with standard user permissions:\ncurl -fsSL https://github.com/Azure/taugrid/releases/latest/download/install.sh | sh export PATH=\u0026#34;$HOME/.local/bin:$PATH\u0026#34; tau version --short kubectl version --client helm version --short Add $HOME/.local/bin to your shell startup file so future terminals can find tau. The installer supports Linux and macOS on amd64 and arm64, verifies the release checksum, and installs only the CLI.\nFor a pinned version, upgrades, source builds, or the optional Python SDK, see the Tau CLI release assets and the repository installation instructions.\n2. Prepare a Kubernetes cluster TauGrid needs a reachable Kubernetes 1.30+ cluster and a kubeconfig context with cluster-administrator access for installation. Start with an existing cluster if you have one:\nexport TAU_CONTEXT=\u0026#34;\u0026lt;kubeconfig-context\u0026gt;\u0026#34; kubectl --context \u0026#34;$TAU_CONTEXT\u0026#34; version kubectl --context \u0026#34;$TAU_CONTEXT\u0026#34; get nodes Confirm that the server is Kubernetes 1.30 or newer and that the nodes required for your first workload are Ready.\nBefore TauGrid is installed, the cluster owner is responsible for:\nAPI-server network access and authentication; worker node pools and autoscaling; container registry access; GPU drivers and device plugins when workloads request GPUs; CSI drivers, StorageClasses, and persistent volumes when workloads use storage; and cloud identities and permissions when workloads call provider services. TauGrid consumes these capabilities through Kubernetes APIs. See Architecture for component ownership.\n3. Install TauGrid TauGrid installs a version-aligned OCI chart from Microsoft Container Registry. The released Tau CLI is the supported installation entry point.\nFirst preview the exact release as a dry run:\ntau cluster explain-values tau cluster install --context \u0026#34;$TAU_CONTEXT\u0026#34; --dry-run The dry run should end with TauGrid render summary (nothing was applied). Review the rendered plan before continuing.\nInstall TauGrid:\ntau cluster install --context \u0026#34;$TAU_CONTEXT\u0026#34; The default distribution installs Kueue, KubeRay, the Tau controller and CRDs, a portable baseline queue, GPU monitoring profiles, and the TauGrid Portal. The chart excludes provider GPU drivers and storage drivers.\nIf the platform needs different queue quotas, GPU labels, or component settings, keep the complete reviewed configuration in one values file:\ntau cluster install \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --values taugrid-values.yaml \\ --dry-run tau cluster install \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --values taugrid-values.yaml Pass the same complete values file on later upgrades because installation resets omitted settings to chart defaults. The cluster install values reference documents every supported setting.\nRe-run the control-plane readiness gate:\ntau cluster validate installation --context \u0026#34;$TAU_CONTEXT\u0026#34; Continue only when the command exits successfully and every enabled core check is PASS. Only a Helm release that passes this gate is ready for a workspace.\n4. Create a workspace TauGrid v0 supports one active workspace per cluster. A workspace reconciles the workload Namespace, LocalQueue, ServiceAccount selection, and researcher RBAC. Platform teams provision cloud storage, a PVC, and cloud identity resources separately.\nFor a single-operator evaluation, create the default workspace:\nexport TAU_WORKSPACE=\u0026#34;taugrid-default\u0026#34; export TAU_SYSTEM_NAMESPACE=\u0026#34;tau-system\u0026#34; tau workspace create \u0026#34;$TAU_WORKSPACE\u0026#34; \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --apply When --principal-name is omitted, TauGrid creates an intentionally inert researcher group binding. The cluster administrator can still complete this guide, but no researcher receives access. For a shared environment, create the workspace with the real Kubernetes subject from the start instead:\ntau workspace create \u0026#34;$TAU_WORKSPACE\u0026#34; \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --principal-name \u0026#34;\u0026lt;kubernetes-group-or-user\u0026gt;\u0026#34; \\ --subject-kind Group \\ --apply Use --subject-kind User for an individual user or ServiceAccount for an automation identity. The value must match the subject presented to Kubernetes by your provider\u0026rsquo;s authentication layer.\nWait for reconciliation and inspect the result:\nkubectl --context \u0026#34;$TAU_CONTEXT\u0026#34; wait \\ --for=jsonpath=\u0026#39;{.status.phase}\u0026#39;=Ready \\ \u0026#34;workspaces.tau.azure.com/$TAU_WORKSPACE\u0026#34; \\ --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; \\ --timeout=5m tau workspace status \u0026#34;$TAU_WORKSPACE\u0026#34; --context \u0026#34;$TAU_CONTEXT\u0026#34; tau workspace check \u0026#34;$TAU_WORKSPACE\u0026#34; --context \u0026#34;$TAU_CONTEXT\u0026#34; The workspace is ready when tau workspace check exits successfully, RBACReady=True, QueueReady=True, and no drift is reported.\nIf the first project needs persistent storage, provision the provider\u0026rsquo;s StorageClass and PVC separately, then include the claim in the workspace check:\ntau workspace check \u0026#34;$TAU_WORKSPACE\u0026#34; \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --data-pvc \u0026#34;\u0026lt;pvc-name\u0026gt;\u0026#34; A bound claim proves provisioning; application-level read/write access still needs a separate check. Test the mount with the same ServiceAccount and security settings the workload will use before handing the workspace to a researcher.\n5. Run a project workload For a larger provider-neutral CPU workload, clone the repository and run the distributed CPU Ray example:\ngit clone https://github.com/Azure/taugrid.git cd taugrid tau run \\ --context \u0026#34;$TAU_CONTEXT\u0026#34; \\ --config examples/cpu-multi-interest-ray/tau.yaml Hand off to researchers Before a shared workspace is ready for researchers:\nBind the workspace to the real Kubernetes user, group, or ServiceAccount. Build and publish the project image with an immutable tag or digest. Provision and test any required PVC and workload identity. Generate or configure the research repository, keeping kubeconfigs, tokens, client secrets, and registry credentials out of it. Complete a checked-in project target from a clean researcher checkout and identity. Use the platform admin guide to enable a workspace and hand it off. The AKS handoff path can additionally generate a non-secret connection descriptor that acquires managed-Entra cluster-user credentials; other Kubernetes platforms can supply a kubeconfig through their normal access process.\nNext steps Run the TauGrid examples for training, tuning, and serving. Configure deliberate quotas and profiles with cluster install values. Add provider integrations only when workloads need them; use the platform admin guide for identity, storage, and observability. Use the platform admin guide for production identity, storage, observability, and lifecycle ownership. Need an AKS cluster to run this guide against? See Getting started on AKS. ","description":"Install TauGrid on Kubernetes, create a workspace, and run your first workload","permalink":"/taugrid/docs/platform-admin-guide/kubernetes/","title":"Getting started on Kubernetes"},{"contents":" Feature state: GA Reviewed 2026-08-25 Before researchers can submit workloads, the platform team prepares the cluster and creates a TauWorkspace. TauGrid then keeps the workspace namespace, queue access, and Kubernetes permissions in the expected state.\nPrepare the workspace Prepare the Kubernetes cluster. Set up the network, node pools, identity, container registry, storage, GPU support, and CSI drivers required by your workloads.\nInstall TauGrid, then verify the installation:\ntau cluster validate installation --context \u0026lt;context\u0026gt; A successful validation exits with code 0.\nThe standard installation uses the released TauGrid images and charts from Microsoft Container Registry. Platform teams with existing Kueue, KubeRay, and monitoring installations can connect TauGrid to those components when they meet the same requirements.\nCreate durable storage for the workspace. This includes the cloud storage service, Kubernetes StorageClass, and PVC used for datasets, checkpoints, and run output.\nCreate the TauWorkspace through your usual Helm, Kustomize, or GitOps process. TauGrid prepares the namespace, Kubernetes permissions, LocalQueue, and optional workload identity ServiceAccount:\nkubectl apply -f workspace.yaml See the example TauWorkspace CR for the expected fields and structure.\nFor GitOps, commit the same Kubernetes object to the repository managed by your deployment system.\nWait for the workspace to become Ready:\ntau workspace status \u0026lt;workspace\u0026gt; --context \u0026lt;context\u0026gt; tau workspace check \u0026lt;workspace\u0026gt; --context \u0026lt;context\u0026gt; See Workspace readiness and recovery if either command reports a problem.\nCreate tau/workspace.connection.yaml and run a checked-in project target in the handoff checklist. This file contains workspace details and can be safely committed to the research repository.\ntau workspace init generates the connection file as part of a new repo scaffold when the cluster flags are provided:\ntau workspace init \u0026lt;name\u0026gt; \\ --workspace \u0026lt;workspace\u0026gt; \\ --azure-subscription-id \u0026lt;subscription-id\u0026gt; \\ --azure-tenant-id \u0026lt;tenant-id\u0026gt; \\ --aks-resource-group \u0026lt;resource-group\u0026gt; \\ --aks-cluster \u0026lt;cluster-name\u0026gt; For an existing repository, create the file manually using the template as a reference.\nAfter these checks pass, send the researcher the repository URL and access instructions. TauWorkspace works with Kubernetes clusters across providers. The current automatic connection setup also includes an AKS-specific path.\n","description":"Prepare a Kubernetes workspace for researchers","permalink":"/taugrid/docs/platform-admin-guide/enable-workspace/","title":"Enable a workspace"},{"contents":" Feature state: GA Reviewed 2026-07-16 Start with:\ntau run status \u0026lt;run-name\u0026gt; tau run status is the canonical lifecycle view for any run \u0026ndash; read its phases top to bottom and stop at the first one that is not done. Then work through the layers below, in order, until you reach that phase:\nRepository/connection resolution and cluster access \u0026ndash; tau workspace connection (--offline for local configuration only). TauWorkspace readiness and handoff validity \u0026ndash; tau workspace status \u0026lt;name\u0026gt;. Client-side config validation and rendering \u0026ndash; tau run validate --config tau/train.yaml. Queue admission and quota \u0026ndash; the Kueue admission phase in tau run status. Kubernetes scheduling, DRA, image pull, init, and readiness \u0026ndash; the remaining phases in tau run status. GPU/node/topology health \u0026ndash; tau cluster validate nodes / tau cluster validate topology. Ray/Job runtime progress and durable evidence \u0026ndash; tau logs \u0026lt;run-name\u0026gt; and taugrid-portal experiment status \u0026lt;name\u0026gt;. Recovery handoff \u0026ndash; Retry and resume. Confirm the queue admitted the workload (layer 4) before moving to GPU or node debugging (layer 6). Treat a Running pod phase (layer 5) as evidence of container start alone \u0026ndash; confirm separately that the model or data process is making progress (layer 7).\nThe full decision path, with what each command\u0026rsquo;s success and failure prove and who owns the fix at each layer, is in Troubleshooting by lifecycle layer.\n","description":"Locate the first failed lifecycle transition","permalink":"/taugrid/docs/platform-admin-guide/troubleshoot/","title":"Troubleshoot a run"},{"contents":" Feature state: Alpha Reviewed 2026-07-16 Use this example when you want to understand the scheduling layer beneath TauGrid using only the CPU capacity you already have. It submits raw RayJobs to Kueue and makes admission, pending work, and team borrowing visible.\nThis is a platform mechanics example that demonstrates Kueue/Ray scheduling directly, distinct from the TauGrid researcher workflow.\nPrerequisites a disposable or explicitly approved Kubernetes context; Kueue and KubeRay installed; kubectl access that can create namespaces, queues, and RayJobs; and enough CPU capacity for two small Ray workloads. Run the single-queue demonstration git clone https://github.com/Azure/taugrid-examples.git cd taugrid-examples/aks-blog/kueue-ray-cpu ./demo.sh single The script applies the single-queue resources and enqueues two raw RayJobs. In a second terminal, watch admission:\nkubectl get workloads -A --watch kubectl get rayjobs -A kubectl get pods -A The demo includes a shared-context safety guard. Do not bypass it with --allow-shared-context unless the target cluster is intentionally approved for these resources.\nWhat to carry back to TauGrid TauGrid adds reviewed configuration, workspace policy, queue resolution, and lifecycle commands on top of the objects shown here. Use this example to understand why a TauGrid run can be submitted but still wait for admission.\nContinue with queue, quota, and GPU placement or the operator troubleshooting path.\nSee the complete example README for the team-borrowing mode and manual manifest path.\n","description":"Observe Kueue admission and quota behavior with raw RayJobs and no GPU quota.","permalink":"/taugrid/docs/examples/cpu-queueing/","title":"CPU queueing with Kueue and Ray"},{"contents":" Feature state: GA Reviewed 2026-08-17 This guide runs the GPU HPO workflow for examples/ray-tune-smoke. It was verified on one Standard_NV12ads_A10_v5 node. A100 SKUs are also usable when the selected subscription and region have both sufficient quota and real-time allocation capacity.\nThe workflow creates billable Azure resources. Use a dedicated resource group. Record its owner, collect the required evidence, and delete the resources when the workflow is complete.\nOwnership and completion gates Complete these phases in order:\nPhase Owner Completion gate AKS setup (Azure/provider) Azure platform operator Managed-Entra/Azure-RBAC AKS is reachable through normal cluster-user credentials; the CPU system pool and GPU pool are Ready; the provider GPU driver and device plugin expose a usable device. TauGrid setup Kubernetes/TauGrid platform operator The checked-in TauGrid chart passes installation and node validation. Workspace setup Kubernetes/TauGrid platform operator A TauWorkspace for the actual researcher subject is Ready, and the repository connection descriptor is generated. Researcher workflow Researcher The repository-first dry-run, optional smoke, six-trial Tune run, logs, result grid, and terminal lifecycle gates all pass with context, namespace, and queue resolved automatically. See Install TauGrid for the general provider/platform boundary. Azure platform operators provision AKS, Azure RBAC, the GPU driver, and the device plugin; TauGrid builds on top of them.\n1. Record local state and choose the target Run from a TauGrid checkout with az, kubectl, kubelogin, Helm, Git, and the Go version declared in cli/go.mod. Install tau from this checkout to use the same version for the CLI, chart, and example:\nmake install-tau-cli TAU_BIN_DIR=\u0026#34;$(go env GOBIN)\u0026#34; test -n \u0026#34;$TAU_BIN_DIR\u0026#34; || TAU_BIN_DIR=\u0026#34;$(go env GOPATH)/bin\u0026#34; export PATH=\u0026#34;$TAU_BIN_DIR:$PATH\u0026#34; command -v tau tau version --short tau --help \u0026gt;/dev/null Persist TAU_BIN_DIR on PATH for later terminals. The Tau CLI installation guide explains upgrades, the recommended GitHub Release installation, and the optional Python SDK.\nRecord the original local state and select the target:\nTAUGRID_ROOT=\u0026#34;$PWD\u0026#34; ORIGINAL_SUBSCRIPTION_ID=\u0026#34;$(az account show --query id -o tsv)\u0026#34; ORIGINAL_KUBECONFIG=\u0026#34;${KUBECONFIG:-$HOME/.kube/config}\u0026#34; ORIGINAL_CONTEXT=\u0026#34;$( KUBECONFIG=\u0026#34;$ORIGINAL_KUBECONFIG\u0026#34; kubectl config current-context )\u0026#34; TARGET_SUBSCRIPTION_ID=\u0026#34;\u0026lt;subscription-id\u0026gt;\u0026#34; TENANT_ID=\u0026#34;\u0026lt;tenant-id\u0026gt;\u0026#34; LOCATION=\u0026#34;westus2\u0026#34; RESOURCE_GROUP=\u0026#34;tau-ray-tune-a10-rg\u0026#34; AKS_CLUSTER=\u0026#34;tau-ray-tune-a10\u0026#34; GPU_POOL=\u0026#34;gpua10\u0026#34; GPU_SKU=\u0026#34;Standard_NV12ads_A10_v5\u0026#34; GPU_FAMILY=\u0026#34;StandardNVADSA10v5Family\u0026#34; RESEARCHER_UPN=\u0026#34;\u0026lt;researcher-upn\u0026gt;\u0026#34; RESEARCHER_OBJECT_ID=\u0026#34;\u0026lt;researcher-object-id\u0026gt;\u0026#34; WORK_DIR=\u0026#34;$(mktemp -d)\u0026#34; export KUBECONFIG=\u0026#34;$WORK_DIR/kubeconfig\u0026#34; Each Azure command below specifies --subscription \u0026quot;$TARGET_SUBSCRIPTION_ID\u0026quot;. Resolve identity object IDs through the approved identity-management path. Do not export access tokens or create credentials to bypass Conditional Access.\n2. Check cost, SKU visibility, quota, and allocation risk Check the exact SKU and its regional restrictions:\naz vm list-skus \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --resource-type virtualMachines \\ --size \u0026#34;$GPU_SKU\u0026#34; \\ --all \\ --query \u0026#34;[0].{name:name,locations:locations,restrictions:restrictions}\u0026#34; \\ --output jsonc az vm list-usage \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --query \u0026#34;[?name.value==\u0026#39;$GPU_FAMILY\u0026#39;].[name.localizedValue,currentValue,limit]\u0026#34; \\ --output table Do not continue unless the SKU has no blocking restriction and the family has enough unused vCPU quota for the planned node count. Quota permits capacity requests; actual allocation still depends on regional, zonal, or subscription-specific capacity, which can be exhausted.\nThis workflow requires one Standard_NV12ads_A10_v5 node, which exposes one 8 GiB A10 partition. A100 shapes cost more and may require a different regional family quota. Select an A100 only where both quota and a real create request succeed. Do not scale past the budgeted node count to work around a capacity failure.\n3. Provision managed-Entra/Azure-RBAC AKS Create one small CPU system node and one GPU worker node:\naz group create \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --name \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --tags taugrid-e2e=true taugrid-disposable=true az aks create \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --enable-managed-identity \\ --enable-aad \\ --enable-azure-rbac \\ --nodepool-name system \\ --node-count 1 \\ --node-vm-size Standard_D4_v5 \\ --node-osdisk-type Managed \\ --no-ssh-key az aks nodepool add \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --cluster-name \u0026#34;$AKS_CLUSTER\u0026#34; \\ --name \u0026#34;$GPU_POOL\u0026#34; \\ --mode User \\ --node-count 1 \\ --node-vm-size \u0026#34;$GPU_SKU\u0026#34; \\ --node-osdisk-type Managed \\ --labels sku=gpu \\ --node-taints sku=gpu:NoSchedule AKS owns reserved labels such as accelerator; do not set them yourself. The GPU node should receive accelerator=nvidia from AKS.\nGrant the researcher normal AKS cluster-user credential access unless the platform has already done so:\nAKS_ID=\u0026#34;$( az aks show \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER\u0026#34; \\ --query id -o tsv )\u0026#34; MANAGED_RESOURCE_GROUP=\u0026#34;$( az aks show \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER\u0026#34; \\ --query nodeResourceGroup -o tsv )\u0026#34; az role assignment create \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --assignee-object-id \u0026#34;$RESEARCHER_OBJECT_ID\u0026#34; \\ --assignee-principal-type User \\ --role \u0026#34;Azure Kubernetes Service Cluster User Role\u0026#34; \\ --scope \u0026#34;$AKS_ID\u0026#34; The platform operator still needs an approved AKS administration role to install cluster components. Do not grant cluster-admin to the researcher. The TauWorkspace provides the required namespaced lifecycle permissions.\nFetch credentials only into the isolated file and record the exact target before the first Kubernetes operation:\naz aks get-credentials \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --name \u0026#34;$AKS_CLUSTER\u0026#34; \\ --file \u0026#34;$KUBECONFIG\u0026#34; \\ --overwrite-existing kubectl config current-context kubectl config view --minify \\ -o jsonpath=\u0026#39;{.contexts[0].context.cluster}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl get nodes -o wide Both nodes must be Ready.\n4. Verify Kubernetes can allocate the GPU For the default AKS GPU-driver mode, AKS installs the host driver. This walkthrough installs the checked-in, digest-pinned NVIDIA device plugin:\nkubectl apply \\ -f \u0026#34;$TAUGRID_ROOT/examples/aks-gpu-quickstart/nvidia-device-plugin.yaml\u0026#34; kubectl -n kube-system rollout status \\ daemonset/nvidia-device-plugin-daemonset \\ --timeout=5m kubectl get nodes -l \u0026#34;agentpool=$GPU_POOL\u0026#34; \\ -o custom-columns=\u0026#39;NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\u0026#39; If the platform selected an AKS mode that skips the host driver, use the corresponding supported Azure/NVIDIA GPU Operator path instead. Do not combine two driver managers. In either mode, nvidia.com/gpu must be nonzero before TauGrid submission.\nVerify that an ordinary pod can consume the resource and run nvidia-smi:\nkubectl apply -f - \u0026lt;\u0026lt;\u0026#39;YAML\u0026#39; apiVersion: v1 kind: Pod metadata: name: tau-gpu-readiness spec: restartPolicy: Never nodeSelector: sku: gpu tolerations: - key: sku operator: Equal value: gpu effect: NoSchedule containers: - name: probe image: mcr.microsoft.com/aks/ai-runtime/ray:py3.12-ray2.56.0-cuda13.0 command: [\u0026#34;/bin/sh\u0026#34;, \u0026#34;-c\u0026#34;] args: - nvidia-smi -L \u0026amp;\u0026amp; nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader resources: requests: nvidia.com/gpu: \u0026#34;1\u0026#34; limits: nvidia.com/gpu: \u0026#34;1\u0026#34; YAML kubectl wait \\ --for=jsonpath=\u0026#39;{.status.phase}\u0026#39;=Succeeded \\ pod/tau-gpu-readiness \\ --timeout=10m kubectl logs tau-gpu-readiness kubectl delete pod tau-gpu-readiness Do not proceed unless the log identifies the expected NVIDIA device.\n5. Install TauGrid and enable the workspace Install and validate the checked-in chart:\ntau cluster install \\ --chart \u0026#34;$TAUGRID_ROOT/charts/taugrid\u0026#34; \\ --wait \\ --atomic \\ --timeout 15m tau cluster validate installation tau cluster validate nodes The required gates are READY: 8/8 checks passed and at least one healthy GPU node.\nCreate the workspace for the subject asserted by AKS:\ntau workspace create taugrid-default \\ --queue jobqueue \\ --principal-name \u0026#34;$RESEARCHER_UPN\u0026#34; \\ --subject-kind User \\ --subject-name \u0026#34;$RESEARCHER_UPN\u0026#34; \\ --apply kubectl wait \\ --for=jsonpath=\u0026#39;{.status.phase}\u0026#39;=Ready \\ workspace/taugrid-default \\ -n tau-system \\ --timeout=5m tau workspace check taugrid-default For a group handoff, use --subject-kind Group and the group value AKS places in the token claim, since a display name never appears there.\n6. Generate the research repository Generate the non-secret connection descriptor, then copy the example into that repository:\nRESEARCH_REPO=\u0026#34;$WORK_DIR/ray-tune-research\u0026#34; tau workspace init-repo ray-tune-research \\ --output \u0026#34;$RESEARCH_REPO\u0026#34; \\ --image mcr.microsoft.com/aks/ai-runtime/ray:py3.12-ray2.56.0-cuda13.0 \\ --workspace taugrid-default \\ --azure-subscription-id \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --azure-tenant-id \u0026#34;$TENANT_ID\u0026#34; \\ --aks-resource-group \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --aks-cluster \u0026#34;$AKS_CLUSTER\u0026#34; cp \u0026#34;$TAUGRID_ROOT/examples/ray-tune-smoke/tau.yaml\u0026#34; \u0026#34;$RESEARCH_REPO/\u0026#34; cp \u0026#34;$TAUGRID_ROOT/examples/ray-tune-smoke/train.py\u0026#34; \u0026#34;$RESEARCH_REPO/\u0026#34; git -C \u0026#34;$RESEARCH_REPO\u0026#34; init git -C \u0026#34;$RESEARCH_REPO\u0026#34; add . cd \u0026#34;$RESEARCH_REPO\u0026#34; tau workspace connection The descriptor must contain authorization.mode: workspace-rbac and requiredRole: tau-researcher-v1. It contains no kubeconfig, token, or client secret.\n7. Run the six-trial sweep using the resolved workspace connection The example defines three learning rates and two batch sizes:\n3 learning rates x 2 batch sizes x 1 sample = 6 trials It permits two concurrent trials, but each trial needs one GPU. With the one-GPU pool above, Ray keeps the second trial pending and executes the sweep sequentially. Adding more GPUs can increase concurrency; it is optional capacity rather than a correctness requirement.\nFrom the generated repository:\ntau run --config tau.yaml --dry-run=client tau run --config tau.yaml TauGrid resolves the cluster, workspace namespace, and LocalQueue automatically from tau/workspace.connection.yaml, so skip passing --context, --namespace, or --queue.\nImmediately capture the Ray Jobs application stream in one terminal:\ntau logs tune-smoke -f | tee tune-smoke.log Watch lifecycle state in another:\ntau run status tune-smoke --watch The generated Tune driver stages the researcher source on remote TorchTrainer workers and forwards arbitrary Ray Train metrics and checkpoint paths into the outer Tune result grid, treating loss the same as any other metric.\n8. Require every success gate Do not call the run successful until all of these are true:\nGate Expected evidence Search space The application log says Number of trials 6. Metrics All six configurations reach five training iterations and report their loss series. Best result The final log prints Best config and Best loss. For the checked-in deterministic example, the best config is {'batch_size': 32, 'lr': 0.1} and the final loss is 5.587555555555555. RayJob tau run status reports Complete / SUCCEEDED and Job finished successfully. Kueue The Workload is Finished, admitted true, reason Succeeded. Compute release tau run status reports quota released and no active Ray pods. ClusterQueue admittedWorkloads, pendingWorkloads, and reservingWorkloads are zero; every flavor usage and reservation total is zero. The platform operator can inspect the final queue counters with:\nkubectl get clusterqueue jobqueue \\ -o custom-columns=\u0026#39;NAME:.metadata.name,ADMITTED:.status.admittedWorkloads,PENDING:.status.pendingWorkloads,RESERVING:.status.reservingWorkloads\u0026#39; kubectl get clusterqueue jobqueue -o yaml KubeRay removes the RayCluster pods shortly after completion. Post-run local logs and /home/nonroot/ray_results persist only as long as the head pod unless central log offload is configured. Keep tau logs -f attached through completion, or configure the platform\u0026rsquo;s supported central log backend before the run. The checked-in example prints its result grid and best result to the application stream, which is the durable record; the head-local result directory is ephemeral only.\n9. Clean up and verify restoration Capture logs and status first, then delete in dependency order:\nkubectl delete rayjob tune-smoke \\ -n taugrid-default \\ --ignore-not-found # If the optional smoke ran, delete its run with `tau run cancel \u0026lt;smoke-run\u0026gt;`. kubectl delete workspace taugrid-default \\ -n tau-system \\ --ignore-not-found # TauWorkspace cleanup intentionally retains its target namespace. kubectl delete namespace taugrid-default --ignore-not-found tau cluster uninstall \\ --chart \u0026#34;$TAUGRID_ROOT/charts/taugrid\u0026#34; \\ --yes \\ --wait kubectl delete daemonset nvidia-device-plugin-daemonset \\ -n kube-system \\ --ignore-not-found az group delete \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --name \u0026#34;$RESOURCE_GROUP\u0026#34; \\ --yes \\ --no-wait Wait until both the primary and AKS-managed resource groups are gone, then confirm the GPU-family usage returned to its pre-run value:\naz group exists \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --name \u0026#34;$RESOURCE_GROUP\u0026#34; az group exists \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --name \u0026#34;$MANAGED_RESOURCE_GROUP\u0026#34; az vm list-usage \\ --subscription \u0026#34;$TARGET_SUBSCRIPTION_ID\u0026#34; \\ --location \u0026#34;$LOCATION\u0026#34; \\ --query \u0026#34;[?name.value==\u0026#39;$GPU_FAMILY\u0026#39;].[name.localizedValue,currentValue,limit]\u0026#34; \\ --output table Remove the disposable research repository and isolated kubeconfig only after checking that WORK_DIR is the expected temporary directory. If Helm generated an untracked charts/taugrid/Chart.lock or charts/taugrid/charts/, inspect git status and remove only those generated dependency artifacts.\nFinally restore and verify the original local targets:\naz account set --subscription \u0026#34;$ORIGINAL_SUBSCRIPTION_ID\u0026#34; export KUBECONFIG=\u0026#34;$ORIGINAL_KUBECONFIG\u0026#34; az account show --query \u0026#39;[name,id]\u0026#39; -o table test \u0026#34;$(kubectl config current-context)\u0026#34; = \u0026#34;$ORIGINAL_CONTEXT\u0026#34; The cleanup is complete only when no disposable AKS resource group remains, GPU quota usage has returned, and the original subscription and Kubernetes context are active.\n","description":"Provision GPU-capable AKS, enable TauGrid, and run the six-trial Ray Tune example through a repository workspace.","permalink":"/taugrid/docs/examples/gpu-ray-tune/","title":"Run GPU Ray Tune HPO on AKS"},{"contents":" Feature state: GA Reviewed 2026-08-18 Complete this checklist after the TauWorkspace readiness gate reports Ready. A handoff is reproducible only when every step below succeeds from a clean checkout, not from a platform owner\u0026rsquo;s already-primed shell.\nThe non-secret connection descriptor The only artifact a researcher needs is tau/workspace.connection.yaml, a non-secret workspace connection descriptor:\nField Contains schema Fixed value tau.workspace.connection.v1. workspace The TauWorkspace name. cluster.contextName The Kubernetes context name Tau selects. cluster.systemNamespace The TauGrid system namespace; defaults to tau-system. access.method kubeconfig for an existing context or aks for automatic AKS credential acquisition. access.aks.resourceID Required only for access.method: aks; the AKS cluster\u0026rsquo;s ARM resource ID. access.aks.tenantID Required only for access.method: aks; the Microsoft Entra tenant ID. authorization.mode cluster-wide or workspace-rbac. authorization.requiredRole Required only in workspace-rbac mode; forbidden in cluster-wide mode. requirements.minTauVersion The minimum compatible tau CLI version. network.privateCluster Whether the Kubernetes API server requires private network access. network.instructions Required only when privateCluster: true (for example, VPN steps). It must never contain a credential, kubeconfig, client secret, or cloud access token. With access.method: kubeconfig, tau loads the normal kubeconfig rules (including KUBECONFIG) and copies only the named context, cluster, and user into an isolated mode-0600 kubeconfig outside the repository. With access.method: aks, it obtains normal AKS cluster-user credentials through the caller\u0026rsquo;s Azure identity and isolates those instead.\nA provider-neutral descriptor uses an existing Kubernetes context:\nschema: tau.workspace.connection.v1 workspace: research cluster: contextName: research-cluster systemNamespace: tau-system access: method: kubeconfig authorization: mode: workspace-rbac requiredRole: tau-researcher-v1 requirements: minTauVersion: 0.3.0 network: privateCluster: false An AKS platform can replace only the access block:\naccess: method: aks aks: resourceID: /subscriptions/\u0026lt;subscription\u0026gt;/resourceGroups/\u0026lt;group\u0026gt;/providers/Microsoft.ContainerService/managedClusters/\u0026lt;cluster\u0026gt; tenantID: \u0026lt;tenant-uuid\u0026gt; Repository placement Commit tau/workspace.connection.yaml at the repository root, alongside the target configs it governs (for example tau/smoke.yaml, tau/train.yaml). Two ways to produce it:\nAuthor it directly from the table above. Generate a provider-neutral descriptor with tau workspace init-repo \u0026lt;name\u0026gt; --workspace \u0026lt;workspace\u0026gt; --kube-context \u0026lt;context\u0026gt; --image \u0026lt;build-tag\u0026gt;. For automatic AKS access, add --azure-subscription-id \u0026lt;id\u0026gt; --azure-tenant-id \u0026lt;id\u0026gt; --aks-resource-group \u0026lt;group\u0026gt; --aks-cluster \u0026lt;cluster\u0026gt;. --kube-context then defaults to the AKS cluster name. The generated targets are ready only after the project image is built and pushed, its immutable tag or digest is written back, and config validation succeeds:\n./scripts/configure.sh --image \u0026#34;\u0026lt;registry\u0026gt;/\u0026lt;repository\u0026gt;:\u0026lt;immutable-tag\u0026gt;\u0026#34; tau run validate --config tau/train.yaml Verify the repository connection tau workspace connection This resolves credentials, contacts Kubernetes, and verifies the descriptor\u0026rsquo;s workspace, LocalQueue, and authorization contract without submitting a workload. Use tau workspace connection --offline when only local descriptor validation is appropriate. Before handoff, platform operators can also inspect the named workspace directly:\ntau workspace check \u0026lt;workspace\u0026gt; --context \u0026lt;context\u0026gt; First project run From the same clean checkout, in order:\ntau run validate --config tau/train.yaml tau run train --dry-run=client tau run train The first command validates the config entirely offline. In a connected repository, the client dry-run activates the descriptor, resolves credentials through its configured access method, verifies and pins the workspace contract, and reads the live workload-profile catalog without submitting the rendered workload. The final command submits the project target and exercises its declared image and resources. Validate the workspace PVC mount and external cloud identity or data service with separate readiness checks.\nWhat \u0026ldquo;handed off\u0026rdquo; means A handoff is done only when all of the following hold, reproduced from a clean checkout:\ntau workspace check \u0026lt;workspace\u0026gt; exits 0. A checked-in project target completes. Only then send the researcher the repository URL and, if network.privateCluster is true, the connection instructions. See Hand off to researchers for the matching completion checklist.\n","description":"The concrete, non-secret artifact and proof that ends platform Day 0","permalink":"/taugrid/docs/platform-admin-guide/handoff/","title":"Hand off a workspace"},{"contents":"AKS is one supported way to reach the Kubernetes cluster that the Getting started on Kubernetes guide installs onto. This page compares the three ways to get there and links each one to its source runbook. Use it after deciding you want AKS specifically; if you already have a Kubernetes cluster of any kind, go straight back to step 2 of Getting started on Kubernetes.\nPrerequisites An Azure subscription and the az CLI, logged in, with permission to create resource groups and AKS clusters (or an existing cluster you can already reach with kubectl). tau, kubectl, and helm on PATH; see Install the Tau CLI. tau cluster install shells out to helm internally. For the Terraform path: Terraform 1.9+ and Azure credentials usable by the AzureRM provider. For the GPU Terraform path: regional quota for the target GPU VM SKU. Choose a path Path Use it when Creates billable resources Guided CPU quickstart You want a guided, disposable, end-to-end evaluation with scripted cleanup, using only CPU quota Yes GPU Terraform root You want reviewable, reproducible AKS infrastructure with GPU node pools that your platform team can adapt and own long-term Yes Existing AKS cluster Your platform team already runs an AKS cluster that meets the prerequisites Only if you scale it 1. Guided CPU quickstart (recommended for evaluation) examples/aks-cpu-quickstart sequences resource-group creation, a CPU-only AKS cluster, tau cluster install, workspace creation, and a real PyTorch smoke workload, then tears everything down again. It is the fastest way to see TauGrid work end to end using only CPU quota.\nRead its README.md first, then run:\n./examples/aks-cpu-quickstart/run.sh run.sh sequences only the commands documented in that README (az, tau, kubectl), keeping every step visible and reproducible. When you are done, tear the cluster and resource group down:\n./examples/aks-cpu-quickstart/cleanup.sh This path creates billable Azure resources (roughly $0.60–0.70/hour for the node pool). Budget for a 25–35 minute round trip and run cleanup.sh as soon as you finish; an idle cluster left running costs about $15/day. Confirm teardown with az group show --name taugrid-cpu-quickstart-rg, which should report ResourceGroupNotFound.\n2. Reproducible GPU-capable Terraform terraform/aks creates a GPU-enabled AKS environment (a system pool plus a GPU pool) and then invokes the same supported tau cluster install workflow. Use this path when your platform team wants to review, version, and re-apply the infrastructure instead of running a one-off script.\ncd terraform/aks terraform init cp terraform.tfvars.example terraform.tfvars # edit terraform.tfvars: subscription_id, resource_group_name, cluster_name terraform apply The default GPU pool is one billable Standard_NC24ads_A100_v4 node; keep terraform.tfvars out of source control since it can hold subscription and naming details. See the full GPU-enabled AKS example for the complete workflow, including GPU stack modes, ADX telemetry, and Portal. Destroy the environment with terraform destroy to stop GPU billing.\n3. An existing AKS cluster If your platform team already runs an AKS cluster, fetch its credentials and use it directly, skipping any provisioning step:\naz aks get-credentials --resource-group \u0026lt;rg\u0026gt; --name \u0026lt;cluster-name\u0026gt; export TAU_CONTEXT=\u0026#34;\u0026lt;kubeconfig-context\u0026gt;\u0026#34; kubectl --context \u0026#34;$TAU_CONTEXT\u0026#34; get nodes Confirm the cluster meets the Kubernetes cluster prerequisites (Kubernetes 1.30+, GPU drivers and device plugins if workloads need GPUs, storage classes if workloads need PVCs), then continue with TauGrid installation.\nGPU software stack models Whichever path creates or reaches your cluster, workload configs request standard Kubernetes nvidia.com/gpu resources in all three models. Platform configuration selects the owner of the driver, device plugin, and DCGM health exporter, plus the dcgmHealth source that TauGrid GPU monitoring scrapes:\nModel Stack owner TauGrid DCGM source Autoscaling / support note Terraform gpu_stack_mode = \u0026quot;self_managed\u0026quot; (default) This repository\u0026rsquo;s Terraform: a standalone NVIDIA device plugin plus the upstream DCGM exporter; NVIDIA GPU Operator is a separate existing-cluster model exporter at http://dcgm-exporter.dcgm-exporter.svc:9400/metrics Standard GPU autoscaling is supported Terraform gpu_stack_mode = \u0026quot;aks_managed_preview\u0026quot; AKS Managed GPU Experience: the NVIDIA driver, device plugin, and a node-local DCGM exporter at port 19400 host-dcgmi at the default http://localhost:19400/metrics GPU autoscaling is unsupported during the preview; requires the Microsoft.ContainerService/ManagedGPUExperiencePreview feature Existing cluster with an externally managed NVIDIA GPU Operator GPU Operator, per its own ClusterPolicy exporter with an explicit non-loopback Service URL, commonly port 9400 Autoscaling and lifecycle follow the cluster\u0026rsquo;s GPU Operator configuration The GPU Terraform root supports the first two rows through gpu_stack_mode; see the full GPU-enabled AKS example for the complete walkthrough of both. The third row applies when your platform team already runs GPU Operator on an existing AKS cluster. GPU Operator owns that software stack, and TauGrid consumes its DCGM exporter endpoint. See the GPU monitoring chart\u0026rsquo;s DCGM health sources documentation for the full dcgmHealth contract.\nTo use the AKS Managed GPU Experience preview, register the feature and confirm it before applying Terraform or creating the GPU pool:\naz feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview az feature show --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview --query properties.state --output tsv az provider register --namespace Microsoft.ContainerService az provider show --namespace Microsoft.ContainerService --query registrationState --output tsv Validate Whichever path you chose, confirm the cluster and control plane before creating a workspace:\nkubectl --context \u0026#34;$TAU_CONTEXT\u0026#34; get nodes tau cluster validate installation --context \u0026#34;$TAU_CONTEXT\u0026#34; Continue with Create a workspace and Run a project workload in the Getting started on Kubernetes guide.\n","description":"Choose a disposable CPU runbook, a reproducible GPU Terraform root, or an existing AKS cluster, then continue the standard install","permalink":"/taugrid/docs/platform-admin-guide/aks-setup/","title":"Getting started on AKS"},{"contents":" Feature state: GA Reviewed 2026-07-16 This is the canonical diagnose-first path for a stuck or failed run. Work through the layers in order and stop at the first incomplete layer. Each layer names its own TauGrid command first; use raw kubectl only for the deep, operator-only inspection each layer calls out, never as a substitute for the TauGrid command.\nStart every investigation the same way:\ntau run status \u0026lt;run-name\u0026gt; tau run status is the single canonical lifecycle view. It walks the same ordered phase tree for every run \u0026ndash; Submitted, Kueue admission, (for a RayJob) RayCluster, pod scheduling, DRA allocation, image pull, init containers, container start, readiness, and (for a RayJob) RayJob status \u0026ndash; and each phase reports pending, active, done, warning, or skipped. Read it top to bottom and treat the first phase that is not done as the layer to investigate, addressing it before any later phase. Add --watch to follow it live.\nTwo things are never proof of training progress by themselves:\nQueue admission (layer 4) means quota was reserved, not that pods are running. A Running pod phase (layer 5) means containers started, not that the model or data process is making useful progress (layer 7). For current GPU runtime evidence, add --run-profile:\ntau run status \u0026lt;run-name\u0026gt; --run-profile When NVIDIA dcgm-exporter is installed with its Kubernetes pod-resource labels, the run profile reads the exporter through the Kubernetes API and reports:\ngpu_allocation: DRA ResourceClaim allocation identity, when the run uses DRA. gpu_devices: the matching workload pod/container, GPU index, and GPU UUID. gpu_utilization: current average and maximum DCGM_FI_DEV_GPU_UTIL. gpu_memory: current average and maximum DCGM_FI_DEV_FB_USED, converted from MiB to GiB. gpu_activity: active when matching device utilization is above zero, idle now when utilization is zero but framebuffer use remains observed, or idle when both are zero. This is an instantaneous device-level signal, distinct from proof of useful model progress. cuda_compute_process: explicitly unavailable because dcgm-exporter\u0026rsquo;s supported scalar metrics expose only aggregate utilization, excluding compute PIDs and CUDA contexts, so utilization alone cannot identify a process or context. The command performs only read-only Kubernetes API calls against unmodified, uninstrumented workloads. It performs at most two targeted discovery requests (the official dcgm-exporter label and TauGrid\u0026rsquo;s GPU-monitoring label), then at most one current metrics request per distinct workload node for each profile refresh. TauGrid\u0026rsquo;s GPU-monitoring DaemonSet proxies the existing node-local DCGM endpoint rather than starting another collector. --watch --run-profile repeats that bounded snapshot at the selected watch interval.\nAn active run shows observed values only when samples match both its namespace and pod names. An allocated-but-idle run shows 0% as a real observation. A completed run keeps any allocation identity still present in Kubernetes but reports live GPU telemetry unavailable; missing samples always report as unavailable, never as a false zero. Missing exporter pods, Kubernetes enrichment, pod-proxy RBAC, or required metrics are also reported as unavailable. Listing exporter pods requires cluster-wide pod read access, and reading metrics requires the pods/proxy subresource; base tau run status output still works when those permissions are absent.\nAt a glance # Layer Primary command Owner if it fails 1 Repository/connection resolution and cluster access tau workspace connection (--offline for local configuration only) Researcher (descriptor) or platform (access) 2 TauWorkspace readiness and handoff validity tau workspace status \u0026lt;name\u0026gt; Platform operator 3 Client-side config validation and rendering tau run validate --config tau/train.yaml Researcher 4 Queue admission and quota tau run status \u0026lt;run-name\u0026gt; (Kueue admission phase) Platform/queue owner 5 Kubernetes scheduling, DRA, image pull, init, readiness tau run status \u0026lt;run-name\u0026gt; (remaining phases) Platform (scheduling/DRA/image) or researcher (crash) 6 GPU/node/topology health tau cluster validate nodes / tau cluster validate topology Platform/node-pool operator 7 Ray/Job runtime progress and durable evidence tau logs \u0026lt;run-name\u0026gt; and taugrid-portal experiment status \u0026lt;name\u0026gt; Researcher (app logic) or platform (evidence pipeline) 8 Recovery handoff \u0026ndash; See Retry and resume 1. Repository/connection resolution and cluster access Suspected layer: TauGrid client-side resolution of the project\u0026rsquo;s workspace connection descriptor, and whether the resolved cluster context is actually reachable.\nConnection check:\ntau workspace connection What success proves: TauGrid found the current project\u0026rsquo;s connection, resolved credentials, reached Kubernetes, and verified the TauWorkspace, LocalQueue, and authorization contract. Add --offline to prove only local project and descriptor resolution.\nWhat failure means: No descriptor found, more than one candidate found, or the descriptor failed schema validation. This is a repository configuration problem rather than a workload problem, and every later layer stays unreachable until it is fixed.\ntau run discovers the descriptor automatically, so this command is a preflight rather than an activation prerequisite. If live connection fails but tau workspace connection --offline succeeds, the problem is credential resolution, VPN/DNS reachability, Kubernetes availability, or RBAC rather than descriptor parsing.\nNext owner/action: Missing or invalid descriptor \u0026ndash; researcher action required; get a valid descriptor from the platform operator who owns the TauWorkspace. Reachability or permission failure on the first live call \u0026ndash; platform action required (cluster access/RBAC), or a transient network condition if it clears on retry.\n2. TauWorkspace readiness and handoff validity Suspected layer: The TauWorkspace object\u0026rsquo;s reconciled status conditions and phase \u0026ndash; the platform-owned onboarding contract behind the descriptor from layer 1.\nPrimary command:\ntau workspace status \u0026lt;name\u0026gt; Use tau workspace check \u0026lt;name\u0026gt; in scripts; it exits non-zero unless the workspace is Ready.\nWhat success proves: status.phase is Ready: RBACReady and QueueReady are true and no drift is detected. The diagnostic WorkloadIdentityReady condition remains important for workloads that use Azure Workload Identity, but today\u0026rsquo;s overall phase gate excludes it. Storage is separate platform desired state. Ready proves the core workspace handoff is valid for submitting workloads; confirming Azure infrastructure freshness and queue admission for a specific run (layer 4) still require their own checks.\nWhat Ready leaves unconfirmed: TauGrid 0.1 has no StorageReady condition on TauWorkspace or TauCluster, so a Ready workspace can still have a missing or unbound platform-managed PVC. If the symptom is storage-shaped, check the claim directly instead of trusting the phase:\nkubectl get pvc blob-training -n \u0026lt;namespace\u0026gt; --context \u0026lt;context\u0026gt; What failure means: Pending or Degraded phase means one or more platform-owned conditions are unmet \u0026ndash; namespace/RBAC, queue accessibility, or workload identity. This is an onboarding/policy problem for the platform to fix, not something a researcher can resolve by resubmitting the run.\nNext owner/action: Platform action required. TauWorkspace is reconciled entirely by the platform\u0026rsquo;s workspace controller; let it reconcile RBAC, queues, and storage rather than hand-editing them to work around a Degraded condition. See the TauWorkspace reference for the condition-driven recovery path.\n3. Client-side config validation and rendering Suspected layer: TauGrid\u0026rsquo;s local schema validation and render step, before anything is submitted to the cluster.\nPrimary command:\ntau run validate --config tau/train.yaml Add --dry-run=client to a normal tau run invocation for the same check inline, or --dry-run=server to render and dry-run against the API server while skipping admission.\nWhat success proves: The checked-in direct run config parses, passes schema validation, and resolves to a renderable Workload (Job or RayJob). tau run validate runs entirely offline. In a connected repository, tau run --dry-run=client activates the workspace connection and reads the live workload-profile catalog without submitting the rendered workload. Use --dry-run=server for API-server validation or submit a checked-in target to exercise workspace readiness and admission.\nWhat failure means: A schema or field error, an ambiguous target, or a manifest that looks like an SDK-generated managed workflow being run through the wrong command. These are authoring problems in the repository, not cluster or scheduling problems.\nNext owner/action: Researcher action required. Fix the run config or target in the repository; nothing here touches the cluster, a queue, or a node.\n4. Queue admission and quota Suspected layer: Kueue admission of the rendered Workload.\nPrimary command:\ntau run status \u0026lt;run-name\u0026gt; Read the Kueue admission phase line: N/M admitted queue=\u0026lt;names\u0026gt;, plus a reason= hint while a workload awaits admission.\nWhat success proves: All workloads show Admitted. This proves quota was reserved \u0026ndash; it does not prove pods were scheduled or the process is running; continue to layer 5.\nWhat failure means: 0/N admitted with a quota-related reason means the LocalQueue/ClusterQueue currently lacks capacity, including borrowing limits. A reason mentioning preemption or eviction means Kueue reclaimed capacity for higher-priority work. Confirm this phase reports done before moving on to GPU or node debugging (layer 6).\nNext owner/action: Platform/queue owner for capacity or priority changes. For read-only, operator-only deep inspection of the queue objects themselves:\nkubectl get workload -n \u0026lt;namespace\u0026gt; kubectl describe clusterqueue \u0026lt;cluster-queue-name\u0026gt; kubectl describe localqueue \u0026lt;local-queue-name\u0026gt; -n \u0026lt;namespace\u0026gt; These kubectl commands confirm what tau run status already reported; run tau run status first. See Queue, quota, topology, and GPU placement.\n5. Kubernetes scheduling, DRA, image pull, init, and readiness Suspected layer: Kubernetes pod scheduling, DRA device-claim allocation (where the workload uses it), image pull, init containers, and container start \u0026ndash; everything between \u0026ldquo;admitted\u0026rdquo; and \u0026ldquo;ready.\u0026rdquo;\nPrimary command:\ntau run status \u0026lt;run-name\u0026gt; --watch Read the remaining phases in order: Pod scheduling, DRA allocation, Image pull, Init containers, Container start, Ready.\nWhat success proves: Pods were assigned to nodes; any referenced ResourceClaims were allocated (this phase is skipped when a workload requests GPUs through the device plugin instead of DRA \u0026ndash; that is expected behavior rather than a failure); the image was pulled; init containers exited 0; the main container started and passed readiness.\nWhat failure means:\nPod scheduling stuck: node selector/taint mismatch, or admission from layer 4 may still be in progress \u0026ndash; re-check layer 4 first. DRA allocation stuck past roughly 30 seconds: usually no matching GPU device; check ResourceSlice availability for the pool. Image pull failing (ErrImagePull/ImagePullBackOff): verify the image name, tag, registry credentials, and node egress. Init containers/Container start failing: the application or its init step is crashing before it becomes ready \u0026ndash; an application problem, rather than an infrastructure one. Next owner/action: Scheduling and DRA capacity \u0026ndash; platform/node-pool owner. Image pull \u0026ndash; researcher (wrong pinned tag) or the image\u0026rsquo;s owning team. Init/container crash \u0026ndash; researcher action required.\nOperator-only deep inspection, after tau run status has identified the stuck phase:\nkubectl describe pod \u0026lt;pod-name\u0026gt; -n \u0026lt;namespace\u0026gt; kubectl get resourceclaim -n \u0026lt;namespace\u0026gt; kubectl get events -n \u0026lt;namespace\u0026gt; --sort-by=.lastTimestamp Feature state: Alpha Reviewed unknown If tau run status shows only a MultiKueue placement phase and no pod phases progress locally, the workload dispatched to a worker cluster. Inspect it from the worker context, or see Multi-cluster execution (Alpha) before assuming the run is stuck.\n6. GPU/node/topology health Suspected layer: The physical or virtual health of the assigned GPU nodes, and whether the live cluster topology still matches the topology preset\u0026rsquo;s expected ResourceFlavor chain.\nPrimary commands:\ntau cluster validate nodes --gpu-class \u0026lt;class\u0026gt; --min-healthy \u0026lt;N\u0026gt; tau cluster validate topology --profile \u0026lt;profile-name\u0026gt; cluster validate nodes accepts --context, --gpu-class, --selector (alternative to --gpu-class), --min-healthy (fail if fewer than N nodes are healthy), and --timeout (default 2m, per-pod). It runs privileged validation pods on the selected GPU nodes and checks nvidia-smi, NVLink, IB, and ECC health.\ncluster validate topology accepts --context and --profile (validate one ready TauCluster workload profile; when omitted, validates every ready profile in the current TauCluster profile-set revision).\nWhat success proves: The nodes backing the workload pass hardware health checks, and/or the preset\u0026rsquo;s full Kueue-facing chain has matching, Ready nodes.\nWhat failure means: A node reported DEGRADED/UNHEALTHY for a specific reason (NVLink down, uncorrectable ECC, IB down) \u0026ndash; a hardware/node problem, rather than a TauGrid or application bug. cluster validate topology reporting zero matching nodes for a ResourceFlavor means the node pool, instance type, or GPU device plugin diverges from what the preset expects.\nNext owner/action: Platform/node-pool operator. cluster validate nodes requires cluster-admin-level RBAC to create privileged validation pods; cluster validate topology is read-only (kubectl get on Kueue objects and Nodes) but still needs cluster-scoped read access. Both are deliberately operator-only diagnostics.\n7. Ray/Job runtime progress and durable evidence Suspected layer: Whether the application process itself is making progress, and whether that progress is durably recorded as evidence independent of pod lifecycle.\nPrimary commands:\ntau logs \u0026lt;run-name\u0026gt; taugrid-portal experiment status \u0026lt;name\u0026gt; tau logs fetches the actual Ray Job driver execution log for a RayJob (not head-pod container logs), or the batch Job\u0026rsquo;s pod logs for a plain job. taugrid-portal experiment status/taugrid-portal experiment list show durable experiment records \u0026ndash; metric history, summaries, and artifacts \u0026ndash; that outlive the pod. tau run get \u0026lt;run-name\u0026gt; fetches a specific durable result artifact (--artifact NAME) when the run\u0026rsquo;s config declared a storage.output path.\nAfter KubeRay deletes a terminal RayJob\u0026rsquo;s head pod, read the driver output from the central log offload by supplying the explicit ADX identity:\ntau logs \u0026lt;run-name\u0026gt; \\ --kusto-cluster \u0026lt;Logs.ContainerLogs Cluster value\u0026gt; \\ --kusto-endpoint \u0026lt;adx-endpoint\u0026gt; \\ --kusto-database \u0026lt;logs-database\u0026gt; TauGrid requires the exact Cluster value instead of guessing from the kube context, which lacks a stable observability identity.\nWhat success proves: The driver log shows expected progress (loss decreasing, steps advancing, checkpoints written), and that progress is mirrored into the durable experiment record. Neither queue admission (layer 4) nor a Running container (layer 5) proves this \u0026ndash; always check logs and evidence directly.\nWhat failure means: Logs show no progress despite a Ready/Running phase: an application-level hang (data loader stall, deadlock), rather than an infrastructure fault. An empty or missing experiment record despite healthy logs: the metrics-offload path or checkpoint contract is missing for this run, rather than a training failure.\nNext owner/action: No progress in logs \u0026ndash; researcher action required (application logic). Missing durable evidence with otherwise healthy logs \u0026ndash; platform/TauGrid owner for the metrics-offload or expstore configuration. See Experiment evidence and artifacts.\n8. Recovery handoff Once you have identified the first failed layer above and classified whether it is transient, apply the matching recovery action from Retry and resume. Do not resume or retry before you have located the first failed layer \u0026ndash; retrying past an unresolved layer 1-3 problem, a Degraded TauWorkspace, or an actual quota/node problem only reproduces the same failure.\n","description":"Diagnose the first failed transition instead of guessing","permalink":"/taugrid/docs/platform-admin-guide/troubleshooting/","title":"Troubleshooting by lifecycle layer"},{"contents":" Feature state: Alpha Reviewed 2026-08-26 Use the repository\u0026rsquo;s terraform/aks root to create a GPU-enabled AKS environment. It provisions a system pool and a GPU pool, enables OIDC and Azure Workload Identity, and invokes the supported tau cluster install command.\ntau cluster install installs the versioned TauGrid distribution that owns Kueue, KubeRay, the Tau controller, GPU monitoring, the baseline Kueue queue, and Portal; skip installing those components separately with Helm.\nPrerequisites an Azure subscription that passes the AKS cluster prerequisites, has GPU quota for the selected region and SKU, and an approved Terraform identity; Azure CLI, Terraform 1.9 or later, kubectl, Helm, and local Python dependencies; Azure credentials accepted by the AzureRM Terraform provider and permission to provision AKS, networking, storage, and identities; and tau and PowerShell 7 on PATH. Linux and macOS users can configure the Terraform command interpreter to use Bash. The default deployment creates one Standard_NC24ads_A100_v4 node. This node has one A100 80 GB GPU and is billable. Before applying, verify that the target region has capacity for the corresponding VM family. Change gpu_vm_size, gpu_count_per_node, and gpu_monitoring_sku_name together when selecting another GPU SKU.\nDeploy From a checkout of this repository:\ncd terraform/aks terraform init terraform apply -var=\u0026#34;subscription_id=\u0026lt;your-subscription-id\u0026gt;\u0026#34; By default, Terraform uses gpu_stack_mode = \u0026quot;self_managed\u0026quot;: it installs a standalone NVIDIA device plugin plus the upstream NVIDIA DCGM exporter. NVIDIA GPU Operator remains a separate existing-cluster model, while the cluster platform retains ownership of the underlying NVIDIA driver. Terraform also creates a local ignored admin kubeconfig and values file under terraform/aks/generated/. For the default A100 pool, it then normalizes MIG mode, restarts the GPU VM scale set, and waits for allocatable GPUs before running:\ntau cluster install --values generated/taugrid-values.yaml --version 0.4.0 In this mode, the generated values configure TauGrid GPU monitoring with dcgmHealth.source: exporter and exporterUrl: http://dcgm-exporter.dcgm-exporter.svc:9400/metrics, pointing at the standalone DCGM exporter\u0026rsquo;s own Service rather than a host-local endpoint.\nSet gpu_stack_mode = \u0026quot;aks_managed_preview\u0026quot; to use AKS Managed GPU Experience instead. This preview mode uses EnableManagedGPUExperience=true at GPU pool creation; AKS then owns the NVIDIA driver, device plugin, and a node-local DCGM exporter host service at port 19400. TauGrid GPU monitoring uses dcgmHealth.source: host-dcgmi with the default http://localhost:19400/metrics in this mode. Before applying, register the feature, wait for Registered, and refresh the AKS resource provider:\naz feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview az feature show --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview --query properties.state --output tsv az provider register --namespace Microsoft.ContainerService az provider show --namespace Microsoft.ContainerService --query registrationState --output tsv GPU cluster autoscaling is not supported in the managed preview. The current AzureRM provider does not expose gpuProfile.nvidia.managementMode, so the tag is a temporary workaround. When AzureRM exposes that field, replace the tag with the provider setting; retain the feature registration until AKS makes the feature generally available.\nIf the local installation step fails after AKS has been created, correct the local prerequisite and rerun terraform apply. Terraform replaces that step when its cluster, generated values, or TauGrid version changes.\nAn existing cluster with an externally managed GPU Operator is a third, distinct operational model alongside Terraform\u0026rsquo;s standalone and AKS-managed modes. GPU Operator owns the GPU software stack according to its own ClusterPolicy, and TauGrid consumes the Operator\u0026rsquo;s DCGM exporter endpoint with dcgmHealth.source: exporter and an explicit non-loopback Service URL (commonly port 9400). See the GPU software stack models comparison and the GPU monitoring chart\u0026rsquo;s DCGM health sources documentation. Regardless of which of these three models owns the stack, workload configs always request standard Kubernetes nvidia.com/gpu resources unchanged.\nVerify and provision a workspace Fetch an operator kubeconfig:\nterraform output -raw get_credentials_command Run the command printed above to fetch the operator kubeconfig.\nVerify both TauGrid and GPU capacity:\ntau cluster validate installation --timeout 10m kubectl get nodes -l accelerator=nvidia kubectl get nodes -o jsonpath=\u0026#39;{range .items[*]}{.metadata.name}{\u0026#34; \u0026#34;}{.status.allocatable.nvidia\\.com/gpu}{\u0026#34;\\n\u0026#34;}{end}\u0026#39; The default path is an operator sandbox using local administrator credentials. It creates platform infrastructure only, so a platform owner provisions the researcher workspace with an explicit Entra group object ID before submitting workloads:\ntau workspace create taugrid-default \\ --namespace taugrid-default \\ --principal-name \u0026lt;entra-group-object-id\u0026gt; \\ --apply tau workspace check taugrid-default To provision the Entra-backed workspace in the same apply, set the following before the first terraform apply. Terraform applies a native TauWorkspace after tau cluster install; the controller reconciles the workload Namespace, jobqueue LocalQueue, and namespace-scoped researcher RBAC.\nworkspace_namespace = \u0026#34;taugrid-default\u0026#34; bootstrap_workspace = { name = \u0026#34;taugrid-default\u0026#34; entra_group_object_id = \u0026#34;\u0026lt;entra-group-object-id\u0026gt;\u0026#34; } When bootstrap is enabled, Terraform configures Portal\u0026rsquo;s computed Jobs board with an explicit operator scope limited to this workspace namespace and jobqueue. This is still an operator-only ClusterIP diagnostic path, not a researcher-facing authenticated Portal endpoint.\nOn a retained cluster, removing bootstrap_workspace stops Terraform from applying the CR but intentionally does not delete an existing workspace or its workloads. Remove a workspace through the workspace administration workflow after reviewing the impact.\nOptional ADX and lifecycle history ADX observability is opt-in because it creates additional billable resources. Both ADX and lifecycle history can be enabled in the initial plan and apply:\nenable_adx = true adx_cluster_name = \u0026#34;\u0026#34; enable_lifecycle_recorder = true workspace_namespace = \u0026#34;taugrid-default\u0026#34; An empty adx_cluster_name generates a stable 20-character taugrid\u0026lt;13-hex-characters\u0026gt; candidate from the subscription, resource group, and AKS cluster names. Set an explicit globally unique name only if Azure reports that the candidate is in use. Terraform preserves the deployed name on later applies, including a legacy 15-character automatic name, so upgrading does not replace the cluster. After apply, inspect the deployed name with terraform output -raw adx_cluster_name. An explicit name is used unchanged for a new cluster; changing the name of an existing cluster requires an explicit data-preserving migration.\nTerraform installs adx-mon alongside TauGrid. In self-managed mode it discovers the upstream DCGM exporter Pod. TauGrid GPU monitoring uses that exporter\u0026rsquo;s node-local Service in self-managed mode; in AKS managed preview mode it collects from the GPU node host service.\nWhen lifecycle history is enabled, Terraform creates workspace_namespace before installing TauGrid so the chart can render the recorder\u0026rsquo;s namespace scoped RBAC in the same apply. If bootstrap_workspace is configured, Terraform applies the TauWorkspace after installing TauGrid. Otherwise, the tau workspace create command above adopts the bootstrapped namespace. Both paths let the controller reconcile its labels, LocalQueue, RBAC, and other workspace settings.\nFor a real GPU workload, use the A100 GPU quickstart after the GPU allocatable-resource check succeeds. It verifies CUDA execution, not only scheduling, and can incur additional GPU cost.\nPortal Terraform uses the TauGrid distribution default and installs Portal as tau-portal in the tau-system namespace. Its Service is ClusterIP-only, keeping it reachable only from inside the cluster network. An operator can inspect it with:\nkubectl port-forward service/tau-portal 18080:80 --namespace=tau-system This is an operator diagnostic path; researcher access requires a platform-owned authenticated HTTPS proxy in front of Portal. The default Portal serves Kubernetes-backed boards. Experiment, cluster-health, and cost boards require separately configured ADX and Azure Workload Identity.\nDestroy Destroy the environment when it is no longer needed to stop GPU billing:\nterraform destroy -var=\u0026#34;subscription_id=\u0026lt;your-subscription-id\u0026gt;\u0026#34; ","description":"Build AKS, GPU capacity, TauGrid, and Portal with the repository Terraform root.","permalink":"/taugrid/docs/examples/full-cluster/","title":"Provision a GPU-enabled TauGrid AKS environment"},{"contents":" Feature state: GA Reviewed 2026-07-16 TauGrid platforms answer two different questions:\nWho may submit and inspect Kubernetes workloads? Which cloud resources may the workload pod access? Human authentication leads to Kubernetes authorization in the target namespace and queue. Workload identity starts from a pod ServiceAccount and reaches only the external resources required by that workload.\nTauWorkspace reconciles Kubernetes ServiceAccounts and reports readiness. Platform teams provision the Azure managed identity, federated credential, Key Vault, storage account, and cloud RBAC assignment that those ServiceAccounts federate with.\nworkspace-rbac is the API default and what tau workspace create writes: the controller binds the researcher subject in the workspace namespace. cluster-wide is an explicit opt-out: the workspace supplies routing defaults only, and the platform grants researcher access separately; some existing clusters run that way. The multiple-workspace lifecycle is Alpha: v0 activates one workspace and blocks additional workspace objects until the active workspace is removed. Researcher isolation still requires tests proving that one workspace cannot access another\u0026rsquo;s resources, plus production rollout gates.\n","description":"Separate human Kubernetes authorization from workload cloud identity","permalink":"/taugrid/docs/platform-admin-guide/identity/","title":"Identity and security boundaries"},{"contents":" Feature state: GA Reviewed 2026-07-16 TauGrid has two recovery paths for a failed run: automatic retry, driven entirely by the resilience.* fields in your run config, and manual tau run resume. TauGrid performs retry as automatic behavior of tau run when resilience.max_retries \u0026gt; 0; there is no separate tau run retry subcommand to invoke.\nAutomatic retry (resilience.*) resilience: max_retries: 2 # default 0 (disabled) retry_on: [\u0026#34;Preempted\u0026#34;, \u0026#34;Evicted\u0026#34;] # default; OOMKilled is opt-in backoff_initial: 30s # default backoff_max: 5m # default checkpoint_path: /data/checkpoints/finetunes/\u0026lt;name\u0026gt; # default, derived from run name When max_retries \u0026gt; 0 and you did not pass --dry-run, tau run:\nWaits for terminal workload state. Classifies the failure into one of: OOMKilled, Preempted, Evicted, Completed, Running, or Unknown. Checks the failure reason against retry_on (case-insensitive) and the remaining attempt budget. If the failure reason is not in retry_on, TauGrid does not retry: it exits with an error naming the reason and the configured list, so an unexpected failure surfaces instead of looping silently. Applies bounded exponential backoff: doubles from backoff_initial, capped at backoff_max. Injects the checkpoint directory, attempt number, and failure reason into the next attempt\u0026rsquo;s environment. Deletes the failed workload and resubmits the same config. If every attempt is exhausted, TauGrid exits with an error rather than leaving the workload retrying indefinitely.\nOOM, preemption, and eviction are not interchangeable retry_on defaults to [\u0026quot;Preempted\u0026quot;, \u0026quot;Evicted\u0026quot;]; both are transient, infrastructure-driven states where the same config commonly succeeds on resubmission. OOMKilled is deliberately not in the default list: the same resources requesting the same memory usually reproduce the same OOM, so automatically retrying it unchanged is opt-in. Add it to retry_on only after you have adjusted compute (or the workload\u0026rsquo;s memory footprint), otherwise you are just spending queue time to fail the same way again.\nUnknown is never retryable, automatically or manually. If TauGrid cannot classify the failure, that means the signal you\u0026rsquo;d need to decide \u0026ldquo;retry\u0026rdquo; or \u0026ldquo;fix and resubmit\u0026rdquo; is missing: inspect tau run status \u0026lt;name\u0026gt; and tau run logs \u0026lt;name\u0026gt; first rather than looping on an unclassified failure.\nIf retries keep exhausting for the same reason, check whether the cluster itself is unhealthy before resubmitting again: tau cluster validate nodes runs privileged GPU health probes across nodes. See the CLI reference for its flags and cluster-level troubleshooting.\nManual resume tau run resume \u0026lt;run-name\u0026gt; --config tau/train.yaml --config is required: resume re-resolves the same direct run config used for the original submission. TauGrid discovers the checkpoint directory (or --from to override it), injects it into the new attempt, deletes the old workload, and resubmits. If the original failure was OOMKilled, resume requires --force, the same opt-in-only-after-you\u0026rsquo;ve-changed-something reasoning as automatic retry applies here too, just gated by an explicit flag instead of a config list.\nResume requires a durable checkpoint contract (a storage mount under /data, not node-local scratch). If your workload only ever wrote checkpoints to ephemeral storage, there is nothing to resume from: that state vanishes once the workload is deleted.\n","description":"Recover transient failure while preserving root cause","permalink":"/taugrid/docs/platform-admin-guide/recovery/","title":"Retry and resume"},{"contents":" Feature state: Alpha Reviewed 2026-08-17 ADX/Kusto is an optional platform data service. Platform teams provision its cluster, databases, Entra identities, federation, and database roles. Prepare those platform resources once; Portal, lifecycle recorder, and adx-mon are separate consumers. Their released charts manage the TauGrid/adx-mon schema objects that they own.\nProvision the service Choose an approved region, SKU, capacity, network path, retention policy, and cost owner. This CLI shape illustrates the required flags; choose SKU, capacity, and other values to match your own environment:\naz extension add --name kusto az kusto cluster create --resource-group \u0026lt;resource-group\u0026gt; \\ --name \u0026lt;adx-cluster-name\u0026gt; --location \u0026lt;region\u0026gt; \\ --sku name=\u0026lt;approved-sku\u0026gt; tier=\u0026lt;approved-tier\u0026gt; capacity=\u0026lt;instance-count\u0026gt; \\ --enable-streaming-ingest false for db in Metrics Logs CostTracking Audit; do az kusto database create --resource-group \u0026lt;resource-group\u0026gt; \\ --cluster-name \u0026lt;adx-cluster-name\u0026gt; --database-name \u0026#34;$db\u0026#34; \\ --read-write-database location=\u0026lt;region\u0026gt; \\ soft-delete-period=P31D hot-cache-period=P7D done Current Azure CLI Kusto extensions reject kind=ReadWrite; a read-write database is implied instead by --read-write-database.\nMetrics is needed by Portal Kusto views and lifecycle history. Logs, CostTracking, and Audit are required only for the corresponding adx-mon pipelines.\nCreate identities and grant roles For AKS, federate each exact ServiceAccount to a managed identity. Workload Identity obtains an Entra token; granting ADX database access is a separate role-assignment step below.\naz identity federated-credential create --resource-group \u0026lt;resource-group\u0026gt; \\ --identity-name \u0026lt;identity-name\u0026gt; --name \u0026lt;credential-name\u0026gt; \\ --issuer \u0026lt;aks-oidc-issuer\u0026gt; \\ --subject system:serviceaccount:\u0026lt;namespace\u0026gt;:\u0026lt;service-account\u0026gt; \\ --audiences api://AzureADTokenExchange An authorized ADX administrator must separately grant database roles. In production, use distinct principals: Portal reader (Viewer), lifecycle recorder writer (Ingestor), and adx-mon identity (currently ADX database Admin for the databases whose schema it reconciles). Azure RBAC, Kubernetes RBAC, and ADX database roles are independent.\nFor example, an ADX administrator can grant the two least-privilege consumer roles in PowerShell or another approved Azure CLI environment:\n$tenant = az account show --query tenantId -o tsv az kusto database-principal-assignment create ` --resource-group \u0026lt;resource-group\u0026gt; --cluster-name \u0026lt;adx-cluster-name\u0026gt; ` --database-name Metrics --principal-assignment-name taugrid-portal-viewer ` --principal-id \u0026lt;portal-managed-identity-client-id\u0026gt; --principal-type App ` --role Viewer --tenant-id $tenant az kusto database-principal-assignment create ` --resource-group \u0026lt;resource-group\u0026gt; --cluster-name \u0026lt;adx-cluster-name\u0026gt; ` --database-name Metrics --principal-assignment-name taugrid-recorder-ingestor ` --principal-id \u0026lt;recorder-managed-identity-client-id\u0026gt; --principal-type App ` --role Ingestor --tenant-id $tenant Use a stable, unique assignment name per database/principal/role. These commands grant ADX data-plane access only; create the federated credential and Kubernetes RBAC through the separate steps above.\nLet the platform charts manage producer schemas Let the platform charts manage TauGrid lifecycle tables, mappings, and functions automatically; treat taugrid-portal exp kusto schema --ingestion lifecycle as a development/release artifact generator rather than a platform deployment prerequisite. It produces KQL text for review, not for direct execution against ADX.\nFor a release that includes lifecycle schema management, enable adx-mon first and grant its identity the ADX database Admin role on Metrics. Explicitly enabling lifecycleRecorder.schemaManagement then creates an adx-mon ManagementCommand that idempotently creates or updates Metrics.TauExpRunLifecycle, its named JSON mapping, and TauExpRunLifecycleDashboardRows(). The recorder itself retains only the Ingestor role.\nCheck that automation before enabling a Kusto-backed Portal capability:\nkubectl -n \u0026lt;adx-mon-namespace\u0026gt; get managementcommand \\ \u0026lt;lifecycle-schema-resource-name\u0026gt; \\ -o jsonpath=\u0026#39;{.status.conditions[0].status}{\u0026#34; \u0026#34;}{.status.conditions[0].reason}{\u0026#34;\\n\u0026#34;}\u0026#39; Wait for a successful status. adx-mon v0.3.0 reconciles commands every 10 minutes. The recorder starts immediately and retries ingestion until the schema is available; use kubectl rollout status deploy/tau-lifecycle-recorder --timeout=25m to wait for it to become Ready. Set lifecycleRecorder.schemaManagement.enabled=false only when an existing platform-owned automation already manages the identical lifecycle contract.\nStellar uses scalar metric tables supplied by its selected ingestion path (commonly adx-mon remote-write Metrics.ExperimentMetrics). For adx-mon, follow its published chart guide and enable Metrics/Logs table precreation before broad collection to avoid ADX control-plane throttling.\nOnly hand consumers a tested endpoint, database, ServiceAccount subject, and non-secret identity client ID. Then configure Portal or adx-mon.\n","description":"Provision and authorize the optional Azure Data Explorer data plane used by TauGrid integrations","permalink":"/taugrid/docs/platform-admin-guide/prepare-adx-kusto/","title":"Prepare ADX/Kusto for TauGrid"},{"contents":" Feature state: GA Reviewed 2026-07-16 TauGrid resolves the selected ready TauCluster workload profile; upstream systems enforce the rendered queue, priority, resource, and placement contract. The controller\u0026rsquo;s resolved profile status is authoritative and stale status fails closed. See workload profile migration.\nStage Owner Decision Target resolution TauGrid Requested workers, GPUs, placement, priority, and workspace defaults Queue admission Kueue Whether shared quota may be consumed Pod scheduling Kubernetes Which nodes satisfy resources, selectors, taints, and topology Device allocation Device plugin or DRA Which concrete GPUs are assigned Node scaling Cluster infrastructure Whether matching node capacity can appear LocalQueues are tenant-facing entry points. ClusterQueues own quota and fairness. ResourceFlavors describe resource pools. Priority and preemption remain cluster-owned policy.\nGPU class contract policy.gpu_class is hardware-only and maps exactly to the node label tau.azure.com/gpu-class:\nResearcher value Meaning any No class selector; any compatible GPU ResourceFlavor may be admitted a10-4gb, a10-8gb, a10-12gb, a10-24gb NVIDIA A10, including Azure fractional GPU sizes a100-40gb NVIDIA A100 with 40 GB memory a100-80gb NVIDIA A100 with 80 GB memory h100-80gb NVIDIA H100 with 80 GB memory h100-95gb NVIDIA H100 with 95 GB memory h200-141gb NVIDIA H200 with 141 GB memory gb200-192gb NVIDIA GB200 with 192 GB memory gb300-288gb NVIDIA GB300 with 288 GB memory For a specific class, TauGrid renders the canonical label as both workload metadata and a pod node selector. Queue preflight accepts a ResourceFlavor only when spec.nodeLabels[\u0026quot;tau.azure.com/gpu-class\u0026quot;] equals the requested class, matching solely on that label: ndm-a100-v4, nd-h200-v5, and taugrid-default are platform identifiers, distinct from researcher API values.\nPlacement stays in policy.topology: independent, single-node-nvlink, multi-node-nccl, or elastic-workers. GPU class values encode hardware only; NVLink, InfiniBand, NCCL, and same-host placement are expressed separately through policy.topology.\nThe legacy inputs a100-nvlink-80gb, h100-standalone-95gb, and h200-nvlink-141gb are accepted for one compatibility window, normalized before validation/rendering, and produce a CLI deprecation warning. New configs and platform assets must use canonical values.\nExisting-cluster migration Update downstream run configs/examples to canonical values first. Before changing live queue or flavor objects, stop submitters, hold admission, and drain the queue. HoldAndDrain evicts admitted/reserving workloads while leaving already-pending workloads in place; inspect those workloads and delete or deactivate their owning Job, RayJob, or workflow before waiting for zero. Keep namespaces and PVCs in place throughout.\nexport CLUSTER_QUEUE=jobqueue kubectl patch clusterqueue \u0026#34;$CLUSTER_QUEUE\u0026#34; --type=merge \\ -p \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;stopPolicy\u0026#34;:\u0026#34;HoldAndDrain\u0026#34;}}\u0026#39; # Inspect pending owners and cancel them through their owning controller. kubectl get workloads -A -o \\ custom-columns=NAMESPACE:.metadata.namespace,WORKLOAD:.metadata.name,OWNER_KIND:.metadata.ownerReferences[0].kind,OWNER:.metadata.ownerReferences[0].name,ADMITTED:.status.conditions[?(@.type==\\\u0026#34;Admitted\\\u0026#34;)].status kubectl wait --for=jsonpath=\u0026#39;{.status.reservingWorkloads}\u0026#39;=0 \\ \u0026#34;clusterqueue/$CLUSTER_QUEUE\u0026#34; --timeout=10m kubectl wait --for=jsonpath=\u0026#39;{.status.admittedWorkloads}\u0026#39;=0 \\ \u0026#34;clusterqueue/$CLUSTER_QUEUE\u0026#34; --timeout=10m kubectl wait --for=jsonpath=\u0026#39;{.status.pendingWorkloads}\u0026#39;=0 \\ \u0026#34;clusterqueue/$CLUSTER_QUEUE\u0026#34; --timeout=10m After the queue is empty, upgrade the Tau controller chart so its reviewed VM-size catalog reconciles both labels. Add custom hardware through extraNodeLabelRules, which defines those labels directly rather than relying on labels a node might already carry:\nCPU-only clusters and GPU pools scaled to zero remain ready when no catalog entry currently matches.\ntau-core-controller: tauCluster: extraNodeLabelRules: - match: vmSizes: [Standard_Custom_H200_v5] labels: kueue.azure.com/gpu-series: custom-h200-v5 tau.azure.com/gpu-class: h200-141gb Wait for the singleton TauCluster to report NodesReady, then create replacement ResourceFlavors with new names rather than patching referenced objects: Kueue may protect or reject changes to immutable/in-use flavor fields.\nVerify the exact contract before submitting specific-class work:\nkubectl get resourceflavor -o \\ custom-columns=NAME:.metadata.name,GPU_CLASS:.spec.nodeLabels.tau\\\\.azure\\\\.com/gpu-class kubectl get nodes -L tau.azure.com/gpu-class tau cluster validate nodes --gpu-class a100-80gb --min-healthy 1 tau cluster validate nodes --gpu-class h200-141gb --min-healthy 1 Fresh installs use a generic CPU flavor, taugrid-default-cpu, with zero GPU quota and a generic GPU flavor, taugrid-default-gpu, with CPU, memory, and GPU quota in the same node-resource group. The GPU flavor remains valid for gpu_class: any and must not be advertised as a specific hardware class.\nKeep the sole ResourceFlavor in a resource group that covers CPU, memory, and GPU free of any GPU class label. Kueue injects that flavor\u0026rsquo;s node labels for every admitted workload, which pins CPU-only work to GPU nodes. Use a zero-GPU CPU flavor plus class-labeled GPU flavors in the same group. Upgrade older single-flavor installs by draining admission and splitting the flavor:\nStop new submissions and set stopPolicy: HoldAndDrain. Kueue drains reserving/admitted workloads, but pending workloads remain pending. Inspect them with kubectl get workloads -A, then cancel or delete each pending workload\u0026rsquo;s owning Job, RayJob, or workflow before waiting for all three counts to reach zero. Keep PVCs and namespaces in place. Create a generic non-TAS CPU ResourceFlavor that carries no GPU labels or GPU admission taints. Create separate GPU ResourceFlavors with exact tau.azure.com/gpu-class labels, GPU nodeTaints, topologyName, and the managed resource annotation kueue.x-k8s.io/podset-required-topology=\u0026lt;level\u0026gt;. Connected TauGrid submission copies that platform requirement onto generated GPU pod templates when no explicit placement policy is present. CPU-only jobs remain admissible through the non-TAS CPU flavor and cannot consume GPU quota. Replace spec.resourceGroups with one group covering CPU, memory, and GPU. Give the CPU flavor CPU/memory quota and zero GPU quota. Give each GPU flavor CPU, memory, and GPU quota. Kueue then assigns one node flavor across every resource requested by a pod set. Restore admission with kubectl patch clusterqueue \u0026quot;$CLUSTER_QUEUE\u0026quot; --type=merge -p '{\u0026quot;spec\u0026quot;:{\u0026quot;stopPolicy\u0026quot;:\u0026quot;None\u0026quot;}}', submit CPU and GPU smoke targets, and remove the old mixed flavor only after it is no longer referenced, keeping namespaces, PVCs, and workspace data in place throughout this migration. For a one-GPU A100 cluster the resulting queue shape is:\nspec: resourceGroups: - coveredResources: [cpu, memory, nvidia.com/gpu] flavors: - name: taugrid-system resources: - {name: cpu, nominalQuota: \u0026#34;100000\u0026#34;} - {name: memory, nominalQuota: 100Ti} - {name: nvidia.com/gpu, nominalQuota: \u0026#34;0\u0026#34;} - name: taugrid-a100-80gb resources: - {name: cpu, nominalQuota: \u0026#34;100000\u0026#34;} - {name: memory, nominalQuota: 100Ti} - {name: nvidia.com/gpu, nominalQuota: \u0026#34;1\u0026#34;} A successful preflight confirms eligibility alone; actual capacity is reserved only at admission or scheduling, and can still change before then.\n","description":"How policy intent becomes admitted and scheduled pods","permalink":"/taugrid/docs/platform-admin-guide/policy-and-placement/","title":"Queue, quota, topology, and GPU placement"},{"contents":" Feature state: Alpha Reviewed 2026-08-26 The lifecycle recorder is an optional, standalone ADX/Kusto producer; Portal is one optional consumer of its output. It runs one metadata-only tau run history record Deployment and records observations of Jobs, RayJobs, and Kueue Workloads in one workload namespace. Portal can optionally read the resulting rows as durable Ray history, but the recorder is useful independently of Portal.\nPrepare the namespace, adx-mon, and identity Before enabling the recorder:\nFor a manual Helm or tau cluster install workflow, create the workload namespace first, usually by creating its TauWorkspace; the recorder chart intentionally expects targetNamespace to already exist. The repository\u0026rsquo;s full-cluster Terraform workflow instead bootstraps that namespace before installing TauGrid. It can apply a configured bootstrap_workspace later in the same apply, or let a platform owner create a TauWorkspace that adopts and reconciles the namespace after the apply.\nPrepare ADX/Kusto, deploy adx-mon, and grant its identity the ADX database Admin role on Metrics. The TauGrid chart creates an adx-mon ManagementCommand for the lifecycle table, mapping, and function, so skip running taugrid-portal exp kusto schema manually for this deployment path.\nCreate a dedicated managed identity and federate its exact ServiceAccount subject:\nsystem:serviceaccount:\u0026lt;system-namespace\u0026gt;:tau-lifecycle-recorder Grant that identity the ADX database Ingestor role. This role is separate from Azure RBAC and Kubernetes RBAC. It is enough for queued ingestion; adx-mon, rather than the recorder, manages the table, mapping, and function.\nThe recorder\u0026rsquo;s Kubernetes Role is namespace-scoped, permitting get/list/watch only for Jobs, RayJobs, and Kueue Workloads in targetNamespace: metadata-only access that leaves workload storage, logs, and Secrets untouched. When schemaManagement.enabled=true, the chart additionally creates a Role in schemaManagement.namespace which can only get/list/watch the one named ManagementCommand. That read-only permission is used by the init container to wait for schema readiness; it is strictly limited to that one object, well short of write access to adx-mon objects or ADX database-management access.\nMerge recorder settings into the canonical platform values This is an intentionally non-standalone merge fragment. Merge it into the same complete, reviewed \u0026lt;platform-values.yaml\u0026gt; that configures the TauGrid umbrella release. Keep it there rather than saving it as a small independent values file: tau cluster install resets Helm release values during upgrades, so supplying only an overlay can remove resources from previously enabled components.\nThe chart accepts only the released Tau CLI image on MCR. Pin the released image by tag, or set tag: \u0026quot;\u0026quot; and provide its immutable digest.\nUse a release that includes the recorder\u0026rsquo;s current queued-ingestion contract and lifecycle schema management. Schema management is deliberately opt-in: the configuration below enables it explicitly. It creates the named TauExpRunLifecycleMapping through adx-mon, and the recorder sends an idempotent ADX extent tag with each batch. Chart and image must be released together, so keep the released image and Deployment as shipped rather than substituting an arbitrary private image or hand-editing the Deployment as a production workaround.\n# Merge into the existing \u0026lt;platform-values.yaml\u0026gt; as a partial fragment. # Preserve components, baselineQueue, Kueue/KubeRay/controller settings, and # every existing taugrid-core service configuration. taugrid-core: lifecycleRecorder: enabled: true targetNamespace: \u0026lt;workspace-namespace\u0026gt; cluster: \u0026lt;aks-name\u0026gt; workspaceId: \u0026lt;workspace-name\u0026gt; kusto: endpoint: https://\u0026lt;adx\u0026gt;.\u0026lt;region\u0026gt;.kusto.windows.net database: Metrics table: TauExpRunLifecycle schemaManagement: enabled: true namespace: \u0026lt;adx-mon-namespace\u0026gt; resourceName: taugrid-lifecycle-schema image: repository: mcr.microsoft.com/aks/ai-runtime/tau tag: \u0026lt;released-tag\u0026gt; # Alternatively: tag: \u0026#34;\u0026#34; and digest: sha256:\u0026lt;released-digest\u0026gt; workloadIdentity: enabled: true serviceAccount: create: true name: tau-lifecycle-recorder annotations: azure.workload.identity/client-id: \u0026lt;recorder-writer-client-id\u0026gt; rbac: create: true The chart adds the required azure.workload.identity/use: \u0026quot;true\u0026quot; Pod label. The federated credential subject must match both namespaces and the ServiceAccount name exactly.\nInstall and verify Upgrade the single umbrella release with the complete canonical values file:\ntau cluster install --context \u0026lt;context\u0026gt; --version \u0026lt;taugrid-release-version\u0026gt; \\ --values \u0026lt;platform-values.yaml\u0026gt; # adx-mon reconciles ManagementCommands asynchronously, exposing no # completion condition, so the recorder starts immediately and retries its # first ingestion until the schema is available. Wait for the Deployment to # become Ready, then inspect adx-mon if it remains NotReady. kubectl -n \u0026lt;system-namespace\u0026gt; rollout status \\ deploy/tau-lifecycle-recorder --timeout=25m kubectl -n \u0026lt;adx-mon-namespace\u0026gt; describe managementcommand \\ taugrid-lifecycle-schema kubectl -n \u0026lt;system-namespace\u0026gt; logs deploy/tau-lifecycle-recorder --tail=100 If the recorder remains NotReady, inspect the command and adx-mon logs, then correct its ADX role, database configuration, or network path. Automatic schema management currently supports only the adx-mon configured Metrics database and the canonical TauExpRunLifecycle table.\nKnown limitation: adx-mon does not currently publish a reliable success or failure status for ManagementCommand. The recorder therefore cannot wait for schema creation before it starts. It retries ingestion until the schema is available, but a short-lived workload that is deleted during this initial window can be absent from lifecycle history. This chart will restore a schema readiness gate after adx-mon publishes that completion contract.\nSubmit a workload in targetNamespace, then query the configured ADX database:\nTauExpRunLifecycle | where cluster == \u0026#34;\u0026lt;aks-name\u0026gt;\u0026#34; | order by observed_at desc | take 20 Rows appear after the recorder\u0026rsquo;s polling interval and ADX queued-ingestion latency. Check the ManagementCommand, recorder logs, and managed identities' database roles if no rows appear. To expose these rows in Portal, follow Configure Portal and set its portal.runHistory.enabled only after recorder writes succeed.\nFor an ADX-side diagnosis that distinguishes ingestion from Kubernetes discovery, a schema administrator can inspect recent ingestion failures:\n.show ingestion failures | where FailedOn \u0026gt; ago(1h) | where Database == \u0026#34;Metrics\u0026#34; and Table == \u0026#34;TauExpRunLifecycle\u0026#34; | project FailedOn, ErrorCode, Details | order by FailedOn desc ","description":"Record TauGrid workload lifecycle observations to a prepared ADX/Kusto database","permalink":"/taugrid/docs/platform-admin-guide/enable-lifecycle-recorder/","title":"Enable lifecycle recorder"},{"contents":" Feature state: GA Reviewed 2026-08-25 Use each evidence plane for its intended scope:\nPlane Use TauGrid status and logs Immediate lifecycle of one submitted run Kubernetes events and pod state Admission, scheduling, startup, and termination Ray and GPU metrics Runtime and hardware behavior Expstore and durable artifacts Authoritative experiment, checkpoint, and recovery state adx-mon and ADX/Kusto Optional hosted scalar and fleet analysis Dashboards and alerts Consumer views over telemetry A workload reaching Scheduled or Running status does not prove the model is making useful progress. Treat each as a separate claim and verify independently. Preserve raw logs, profiles, and artifacts when a diagnosis depends on them.\nMetrics-store methodology TauGrid separates durable experiment ownership from fleet analytics:\nLayer Responsibility Immutable metric files Preserve normalized scalar history in Parquet Expstore SQLite index Track runs, metric files, artifacts, idempotency keys, and summaries Metric summaries Provide finite counts, step ranges, min/max values, and the latest finite point per run and metric ADX/Kusto projection Support cross-workspace dashboards and fleet queries Expstore is authoritative. ADX ingestion uses checkpoints and query-time deduplication so retries preserve one logical scalar point. Configure ADX retention, hot-cache duration, identities, and database roles as platform policy. The workspace\u0026rsquo;s durable storage lifecycle governs local metric files and artifacts.\nUse project, run_group_id, and run_id for experiment identity. Use metric_name, step, and wall_time to align and compare scalar histories. Tags add workspace and workload dimensions without changing those core keys.\nFor a runnable workload that publishes immutable loss and accuracy chunks, see Live experiment evidence. For the full local evidence and artifact contract, see Experiment evidence and artifacts.\nFor the optional ADX-backed data plane, see Prepare ADX/Kusto for TauGrid. For Portal deployment and its available boards, see Configure Portal.\n","description":"Immediate lifecycle, durable experiment state, and fleet telemetry","permalink":"/taugrid/docs/platform-admin-guide/observability/","title":"Observability and evidence"},{"contents":" Feature state: Alpha Reviewed 2026-08-13 Portal is the unified, read-only browser entry point. tau cluster install enables its operator-facing Kubernetes path by default in the system release namespace (tau-system unless --namespace selects another namespace); it ships as part of the one TauGrid umbrella release rather than a separate taugrid-core Helm release.\nUnderstand the default boundary The default distribution creates deployment/tau-portal, service/tau-portal, a dedicated ServiceAccount, and cluster-wide read-only Kubernetes RBAC. Portal remains ClusterIP-only and relies on network-level access control rather than application-level login, so kubectl port-forward is an operator diagnostic rather than a researcher endpoint.\nCapability Default state Additional requirement Portal shell, Runs, run detail, Cluster Nodes, live Ray discovery Available through the default Kubernetes client and RBAC A matching live workload or Ray head Service; Events remain a separate RBAC capability Jobs / Queue computed board Disabled portal.jobs.scopeMode plus reviewed workspace-directory or operator scopes; workload profiles are read-only from ready TauCluster status Kueue (Live) Disabled KueueViz Deployments/Services and portal.kueueviz.enabled=true Experiments, Cluster Health, Cost Degraded ADX/Kusto endpoint, database, and a query identity Durable Ray history Disabled Lifecycle recorder, successful schema management, and portal.runHistory.enabled=true Researcher browser access Not installed Platform-owned authenticated HTTPS proxy, DNS, certificate, and reviewed network path Services and Observability pages Planned No shipped backend yet The live Ray dashboard reflects present state only. Durable history remains available after KubeRay removes runtime Pods only when the lifecycle recorder and Kusto path are configured.\nVerify the default Portal Install or upgrade TauGrid with the cluster\u0026rsquo;s complete values file, then verify the Portal resource and health endpoint:\nexport TAU_SYSTEM_NAMESPACE=tau-system tau cluster install --context \u0026lt;context\u0026gt; --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; --values \u0026lt;platform-values.yaml\u0026gt; kubectl --context \u0026lt;context\u0026gt; --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; rollout status deployment/tau-portal --timeout=180s kubectl --context \u0026lt;context\u0026gt; --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; get service/tau-portal serviceaccount/tau-portal kubectl --context \u0026lt;context\u0026gt; --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; port-forward service/tau-portal 18080:80 # In another terminal: curl -fsS http://127.0.0.1:18080/healthz The installation validation should include PASS Portal, the rollout should complete, the Service should be ClusterIP, and the health request should succeed. A default install can open /portal and Kubernetes-backed run details, but the capability table above remains the acceptance boundary.\nConfirm release capability before enabling ADX boards Use a published TauGrid release whose chart and images include native Kusto Portal support. Confirm the template contract for the exact release you use. The matching taugrid-core chart must accept portal.runHistory.enabled with portal.kusto.endpoint and no portal.kusto.queryCommand, and it must derive the azure.workload.identity/use: \u0026quot;true\u0026quot; Pod label from the Portal ServiceAccount\u0026rsquo;s azure.workload.identity/client-id annotation.\nBefore changing a production release, render the exact published chart and review these conditions together with its pinned Portal image. If the release lacks them, use a newer published release instead of working around the gate with a nonexistent queryCommand or a manual Deployment patch.\nKeep one canonical TauGrid values file The capability sections below describe keys in one desired-state document rather than independent Helm overlays. tau cluster install resets Helm release values during its upgrade path, so a later invocation containing only a small Portal/Kusto fragment can reset other customized TauGrid components to chart defaults and cause Helm to remove resources that are no longer rendered. Keep the complete reviewed configuration for the cluster in \u0026lt;platform-values.yaml\u0026gt;, merge every enabled component into that file, and pass that full file on every upgrade. Portal has no separate namespace setting; it follows the TauGrid Helm release namespace.\nKeep taugrid-core and the umbrella release mutually exclusive: installing both would let each try to own Portal or recorder resources, creating ambiguous Helm ownership.\nScope the Kubernetes-backed boards Create the target workspace first. The following is an intentionally non-standalone merge fragment for the canonical reviewed values file. The Portal enablement, ServiceAccount, and RBAC keys repeat distribution defaults so the desired state is explicit; preserve every existing top-level setting and every other enabled component in that file.\n# Merge into the existing \u0026lt;platform-values.yaml\u0026gt; as a partial fragment. # Preserve components, baselineQueue, Kueue/KubeRay/controller settings, and # every existing taugrid-core service configuration. taugrid-core: portal: enabled: true cluster: \u0026lt;aks-name\u0026gt; workspace: \u0026lt;workspace-name\u0026gt; workloadNamespace: \u0026lt;workspace-namespace\u0026gt; serviceAccount: create: true name: tau-portal rbac: create: true tau cluster install --context \u0026lt;context\u0026gt; --version \u0026lt;taugrid-release-version\u0026gt; \\ --namespace \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; \\ --values \u0026lt;platform-values.yaml\u0026gt; kubectl -n \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; rollout status deploy/tau-portal --timeout=180s portal.jobs.scopeMode is disabled by default. Configure workspace-directory or explicit operator scopes before enabling it, so the Jobs board exposes only the intended workspace scope. workloadNamespace scopes the legacy Runs and Ray views only; portal.jobs.scopeMode separately authorizes the computed Jobs board.\nAdd Kusto-backed boards to the same Portal block First prepare ADX/Kusto. Merge the following keys into the existing taugrid-core.portal map above, keeping them there instead of a second values file passed alone to tau cluster install. A bare endpoint uses Portal\u0026rsquo;s native DefaultAzureCredential path; portal.kusto.queryCommand is only an explicit adapter override and must exist in the image.\n# Fields to merge into the portal object already shown above. source: kusto kusto: endpoint: https://\u0026lt;adx\u0026gt;.\u0026lt;region\u0026gt;.kusto.windows.net database: Metrics serviceAccount: annotations: azure.workload.identity/client-id: \u0026lt;portal-query-identity-client-id\u0026gt; Durable Ray history is optional. Follow Enable lifecycle recorder after adx-mon has successfully reconciled its lifecycle schema command and the writer identity is ready. Then merge the following field into the existing portal map:\nrunHistory: enabled: true Do this only after the recorder is successfully writing rows. This is an additional Portal capability layered on top of its core definition.\nAfter the rollout, verify both the workload-identity injection and the durable history API before exposing Portal through an ingress:\nkubectl -n \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; get pod -l app=tau-portal \\ -o jsonpath=\u0026#39;{.items[0].metadata.labels.azure\\.workload\\.identity/use}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl -n \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; get pod -l app=tau-portal \\ -o jsonpath=\u0026#39;{.items[0].spec.containers[0].env[?(@.name==\u0026#34;AZURE_FEDERATED_TOKEN_FILE\u0026#34;)].value}{\u0026#34;\\n\u0026#34;}\u0026#39; kubectl -n \u0026#34;$TAU_SYSTEM_NAMESPACE\u0026#34; port-forward svc/tau-portal 18080:80 # In another terminal: curl -fsS \u0026#39;http://127.0.0.1:18080/api/portal/runs?workspace=\u0026lt;workspace-name\u0026gt;\u0026#39; The API must report \u0026quot;historyState\u0026quot;:\u0026quot;available\u0026quot; after the lifecycle recorder has ingested records. history-unavailable normally signals an ADX identity, network, schema, or release-template problem rather than evidence that Portal is reading durable history.\nPortal Service is intentionally ClusterIP. Production access needs a platform-owned authenticated HTTPS proxy, DNS, certificate, and reviewed network path; kubectl port-forward is an operator diagnostic only.\n","description":"Verify the default TauGrid Portal and configure additional Kubernetes and Kusto-backed capabilities","permalink":"/taugrid/docs/platform-admin-guide/enable-portal/","title":"Configure Portal"},{"contents":"Workload profiles are platform-owned declarations on the singleton TauCluster/cluster. They replace the removed file-based TopologyPolicy catalog. The controller resolves referenced queues and scheduling objects into status; TauGrid reads only that status as the single source of truth, rather than falling back to a compiled catalog or a local policy file.\nMigrate the catalog Translate each supported workload shape into spec.workloadProfiles. Keep live quota, capacity, ResourceFlavor selectors, and topology object names out of the declaration: those identities are observations in status.\napiVersion: tau.azure.com/v1alpha1 kind: TauCluster metadata: name: cluster spec: workloadProfiles: - name: research.training.1x description: One-GPU training through the shared queue. applicability: teams: [research] lanes: [training] namespaces: [research-workloads] gpusPerWorker: 1 workerCount: 1 mode: fixed placement: independent defaultLocalQueue: jobqueue executionTarget: singleCluster priorities: workloadPriorityClassName: taugrid-default podPriorityClassName: taugrid-default The controller chart\u0026rsquo;s tauCluster.workloadProfiles value is the canonical default catalog. Helm lists replace rather than merge, so a site override must supply the complete reviewed catalog. The chart synchronization test keeps the checked-in Helm, Kustomize, and controller sample declarations identical.\nDelete old policy ConfigMaps, volume mounts, environment variables, and files only after every submitter uses a ready TauCluster profile. There is no compatibility fallback.\nReadiness is fail closed TauGrid accepts a profile set only when all of the following are true:\nstatus.workloadProfiles.observedGeneration equals metadata.generation. the WorkloadProfilesReady condition is True for that generation; the published profileSetHash is non-empty and matches the normalized resolved profiles; the selected profile\u0026rsquo;s Ready condition is True for that generation; and namespace, team, and lane applicability authorize the caller. Missing, stale, drifted, forbidden, ambiguous, or unready data stops rendering. TauGrid stamps successful output with tau.azure.com/tau-cluster-generation, tau.azure.com/workload-profile-set-hash, and tau.azure.com/workload-profile. These annotations identify the observed revision only; confirm quota or capacity availability with a separate check.\nConnected and offline rendering A normal run, server dry-run, or apply reads the connected cluster:\ntau run --config tau.yaml --dry-run=server Client rendering is still connected unless an explicit snapshot is configured. Export only a ready revision:\ntau cluster profiles export --context my-cluster \\ --output profiles.snapshot.yaml Then set policy.workload_profile_snapshot: profiles.snapshot.yaml and provide explicit policy.namespace, policy.team, and policy.lane. Snapshot input is accepted only with --dry-run=client; it cannot authorize server dry-run or apply. Treat snapshots as immutable review artifacts and re-export after a TauCluster revision.\nMultiKueue readiness and ownership The multiKueue execution target has one deterministic profile contract:\nthe standard distribution installs MultiKueue controller support; TauCluster.status.conditions[MultiKueueReady] reports a current, active AdmissionCheck, referenced MultiKueueConfig, and active worker; the catalog includes a ready profile with executionTarget: multiKueue, a dedicated LocalQueue, and the ordinary team, namespace, and lane applicability required by the operator; and profile selection resolves that profile explicitly through policy.profile or implicitly as the unique ready, applicable profile. Failure of readiness or profile resolution stops dispatch. A MultiKueue profile uses the same fail-closed applicability and ambiguity rules as every other execution target. The supported boundary is TauGrid-rendered workloads whose kind and dependencies are configured on the manager and every eligible worker. Direct kubectl apply, hand-written Workloads, and objects mutated after TauGrid renders them are outside this contract.\nThe platform owner owns worker credentials, least-privilege access, distribution and rotation, queue isolation, namespace and ServiceAccount parity, image pull identity, storage reachability, and revocation. This constrained supported capability remains Alpha until release evidence covers an environment/version matrix, manager-to-worker E2E, negative authorization and credential tests, operational enablement, credential rotation/revocation, drain, and rollback.\nDrain and roll back Before changing queue bindings, profile scope, or execution target:\nstop new submissions and set affected ClusterQueues to HoldAndDrain; cancel pending owners and wait for pending, reserving, and admitted counts to reach zero; export the current TauCluster object and ready profile snapshot; apply the new catalog and wait for the new generation and every selected profile to become Ready; then run connected server-dry-run checks; and restore admission. Rollback by draining again, restoring the previous catalog, waiting for its new generation to become Ready, removing the MultiKueue profile and routing objects, then revoking unused worker credentials. Running pods stay pinned to their original placement when a profile changes. Removing profile authorization blocks new TauGrid submissions only. Continue status, log, cancellation, and cleanup operations until every manager and worker object is terminal, since active remote workloads still require active management. Controller status is an observation rather than a reservation: queues, credentials, storage, nodes, and capacity can change between readiness, render, admission, and scheduling.\n","description":"Move placement catalogs into TauCluster and operate profile revisions safely","permalink":"/taugrid/docs/platform-admin-guide/workload-profiles/","title":"Workload profile migration"},{"contents":" Feature state: Alpha Reviewed 2026-08-20 TauGrid supports deterministic dispatch from a manager to preconfigured MultiKueue workers in the environment below. The capability remains Alpha because its released environment and operational evidence are still narrow. It supports deterministic dispatch to a preconfigured, matched-version worker set rather than arbitrary worker discovery or unrestricted cross-cloud routing.\nSupported manager environment Component Supported contract TauGrid Chart and images 0.3.x from the same release Kubernetes 1.30 or newer, matching the TauGrid chart constraint Kueue TauGrid-pinned AKS chart and controller 0.18.2 KubeRay TauGrid-pinned operator 1.6.2 for RayJob dispatch Platform AKS manager and workers with operator-managed identities, networking, storage, GPU drivers, and compatible CRDs Use the same Kubernetes, Kueue, KubeRay, and workload CRD minor versions on the manager and every worker. Other combinations are outside the supported contract.\nPlatform-owner configuration The standard taugrid installation enables the pinned Kueue MultiKueue controller capability and gives the Tau controller read-only access to its prerequisites:\nhelm upgrade --install taugrid \\ oci://mcr.microsoft.com/aks/ai-runtime/helm/taugrid \\ --version 0.4.0 \\ --namespace tau-system --create-namespace \\ --wait --atomic There is no separate values file or approval list. The standard installation supplies controller support only; the platform owner creates worker identities and routing resources through the steps below. Before publishing a MultiKueue profile, the platform owner must:\nProvision a least-privilege worker kubeconfig Secret and protect it from tenants; platform owners alone create and copy worker credentials. Create explicit MultiKueueCluster and MultiKueueConfig objects. Create a dedicated AdmissionCheck whose spec.controllerName is exactly kueue.x-k8s.io/multikueue, and wait for it and at least one referenced worker to report Active=True. Create a dedicated, non-global MultiKueue queue/profile. The default jobqueue remains single-cluster and must never receive a MultiKueue AdmissionCheck. Verify workload dependencies on every eligible worker before authorizing tenants. TauCluster.status.conditions[MultiKueueReady] reports actual manager-visible prerequisite health: a correctly controlled and active AdmissionCheck, its referenced MultiKueueConfig, and at least one active MultiKueueCluster must all resolve. AdmissionCheck names and installed controller support alone fall short of readiness.\nTenant and workload boundary Dispatch requires all of the following:\nthe standard distribution\u0026rsquo;s MultiKueue controller support; a current MultiKueueReady=True condition proving manager prerequisites; a ready, applicable multiKueue profile backed by a dedicated queue; and explicit selection through policy.profile or unambiguous implicit selection as the unique ready, applicable profile. See workload profile migration for the profile contract. Client dry-run, server dry-run, and apply use the same profile selection rules. Direct kubectl submissions and objects mutated after TauGrid renders them are operator-controlled, out-of-contract paths that rely on Kueue\u0026rsquo;s own admission control rather than a TauGrid-installed name-based CEL policy or admission webhook.\nWorker dependency and security contract Every eligible worker needs compatible Kubernetes, Kueue, KubeRay, and workload CRDs plus matching queue, ServiceAccount, Secret, StorageClass, PVC, image, dataset, checkpoint, GPU, topology, network, and storage capabilities. Grant worker credentials only the verbs and namespaces Kueue needs, rotate them through the platform identity system, and never expose them in researcher namespaces.\nAutomatic dataset replication is Planned. Running pods and node-local scratch stay pinned to their original cluster. Do not remove cluster pins from stateful workloads until common storage and identity contracts are activated and verified on every worker.\nFailure modes Symptom Meaning and action MultiKueueReady=False with PrerequisitesNotReady Inspect the named AdmissionCheck, config, worker, credentials, and Kueue controller status. MultiKueue profile fails to reach Ready Inspect its ExecutionReady condition and verify that every referenced ClusterQueue uses the ready AdmissionCheck. Placement remains pending or retries Check worker connectivity, quota, dependency parity, image pulls, and worker events. Manager has no local pods This is normal after remote dispatch. Continue manager-side status, logs, cancellation, and cleanup. Drain and rollback Remove tenant authorization and stop new submissions to every dedicated profile/queue. Do not repoint the default queue. Hold and drain the dedicated ClusterQueues and explicitly cancel unwanted pending owners. Wait for every manager Workload to become terminal and for remote cleanup and finalizers to complete. Verify worker Jobs/RayJobs and pods directly. Remove the MultiKueue profiles, then remove dedicated AdmissionCheck/queue wiring after no workload references it. Revoke and delete worker credentials only after no active or retained remote workload needs them. Disabling controllers or abandoning credentials while remote workloads are active can orphan expensive pods, leave finalizers stuck, lose centralized log resolution, and prevent manager-side cancellation. Removing profile authorization blocks new TauGrid submissions while leaving status, logs, cancellation, resume, and cleanup available for already-existing MultiKueue workloads.\nPromotion blockers Promotion to Beta is blocked on: a supported environment/version matrix, released manager-to-worker E2E coverage, negative authorization and credential tests, credential rotation/revocation evidence, and repeatable enablement, drain, and rollback exercises.\n","description":"Deterministic dispatch to preconfigured MultiKueue workers","permalink":"/taugrid/docs/platform-admin-guide/multicluster/","title":"Multi-cluster execution"}]