Platform admin guide on TauGrid

Identity and security boundaries

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid platforms answer two different questions:

Who may submit and inspect Kubernetes workloads?
Which cloud resources may the workload pod access?

Human authentication leads to Kubernetes authorization in the target namespace and queue. Workload identity starts from a pod ServiceAccount and reaches only the external resources required by that workload.

TauWorkspace reconciles Kubernetes ServiceAccounts and reports readiness. Platform teams provision the Azure managed identity, federated credential, Key Vault, storage account, and cloud RBAC assignment that those ServiceAccounts federate with.

Queue, quota, topology, and GPU placement

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid resolves the selected ready TauCluster workload profile; upstream systems enforce the rendered queue, priority, resource, and placement contract. The controller’s resolved profile status is authoritative and stale status fails closed. See workload profile migration.

Stage	Owner	Decision
Target resolution	TauGrid	Requested workers, GPUs, placement, priority, and workspace defaults
Queue admission	Kueue	Whether shared quota may be consumed
Pod scheduling	Kubernetes	Which nodes satisfy resources, selectors, taints, and topology
Device allocation	Device plugin or DRA	Which concrete GPUs are assigned
Node scaling	Cluster infrastructure	Whether matching node capacity can appear

LocalQueues are tenant-facing entry points. ClusterQueues own quota and fairness. ResourceFlavors describe resource pools. Priority and preemption remain cluster-owned policy.

Observability and evidence

Mon, 01 Jan 0001 00:00:00 +0000

Use each evidence plane for its intended scope:

Plane	Use
TauGrid status and logs	Immediate lifecycle of one submitted run
Kubernetes events and pod state	Admission, scheduling, startup, and termination
Ray and GPU metrics	Runtime and hardware behavior
Expstore and durable artifacts	Authoritative experiment, checkpoint, and recovery state
adx-mon and ADX/Kusto	Optional hosted scalar and fleet analysis
Dashboards and alerts	Consumer views over telemetry

A workload reaching Scheduled or Running status does not prove the model is making useful progress. Treat each as a separate claim and verify independently. Preserve raw logs, profiles, and artifacts when a diagnosis depends on them.

Workload profile migration

Mon, 01 Jan 0001 00:00:00 +0000

Workload profiles are platform-owned declarations on the singleton TauCluster/cluster. They replace the removed file-based TopologyPolicy catalog. The controller resolves referenced queues and scheduling objects into status; TauGrid reads only that status as the single source of truth, rather than falling back to a compiled catalog or a local policy file.

Migrate the catalog

Translate each supported workload shape into spec.workloadProfiles. Keep live quota, capacity, ResourceFlavor selectors, and topology object names out of the declaration: those identities are observations in status.

Multi-cluster execution

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid supports deterministic dispatch from a manager to preconfigured MultiKueue workers in the environment below. The capability remains Alpha because its released environment and operational evidence are still narrow. It supports deterministic dispatch to a preconfigured, matched-version worker set rather than arbitrary worker discovery or unrestricted cross-cloud routing.

Supported manager environment

Component	Supported contract
TauGrid	Chart and images `0.3.x` from the same release
Kubernetes	`1.30` or newer, matching the TauGrid chart constraint
Kueue	TauGrid-pinned AKS chart and controller `0.18.2`
KubeRay	TauGrid-pinned operator `1.6.2` for RayJob dispatch
Platform	AKS manager and workers with operator-managed identities, networking, storage, GPU drivers, and compatible CRDs

Use the same Kubernetes, Kueue, KubeRay, and workload CRD minor versions on the manager and every worker. Other combinations are outside the supported contract.