Workload profile migration
Workload profiles are platform-owned declarations on the singleton
TauCluster/cluster. They replace the removed file-based TopologyPolicy
catalog. The controller resolves referenced queues and scheduling objects into
status; TauGrid reads only that status as the single source of truth, rather than
falling back to a compiled catalog or a local policy file.
Migrate the catalog
Translate each supported workload shape into spec.workloadProfiles. Keep live
quota, capacity, ResourceFlavor selectors, and topology object names out of the
declaration: those identities are observations in status.
apiVersion: tau.azure.com/v1alpha1
kind: TauCluster
metadata:
name: cluster
spec:
workloadProfiles:
- name: research.training.1x
description: One-GPU training through the shared queue.
applicability:
teams: [research]
lanes: [training]
namespaces: [research-workloads]
gpusPerWorker: 1
workerCount: 1
mode: fixed
placement: unconstrained
defaultLocalQueue: jobqueue
executionTarget: singleCluster
priorities:
workloadPriorityClassName: taugrid-default
podPriorityClassName: taugrid-default
The controller chart’s tauCluster.workloadProfiles value is the canonical
default catalog. Helm lists replace rather than merge, so a site override must
supply the complete reviewed catalog. The chart synchronization test keeps the
checked-in Helm, Kustomize, and controller sample declarations identical.
placement accepts unconstrained, same-host,
same-accelerator-domain, same-network-domain, or same-site.
Accelerator-domain placement selects one provider-declared NVLink/NVL72 island;
network-domain placement selects one scale-out InfiniBand or RoCE fabric and
may span accelerator domains. Matching GPU models never imply a shared island.
Neither placement guarantees distinct hosts, and an unavailable shared domain
remains pending instead of falling back. Multi-node direct Job torchrun adds a
separate one-per-GPU-Node host-slot request: its ranks use distinct hosts while
the selected placement keeps them in one fabric domain. That slot is exclusive
across all multi-node torchrun ranks, including ranks from different runs.
An optional platform-owned nodeSelector map on each workload profile pins
hardware or site labels, using Kubernetes label keys and values. Tau preserves
it in the observed profile and applies it to every direct or managed Job pod and
Ray worker, including selectors whose label value is the valid empty string;
Ray heads retain their system-node placement. For dedicated nightly profiles,
declare the target series and site here and activate the profiles through the
platform’s normal review process. Run-config policy.node_selector overrides
remain prohibited when selecting a workload profile. Selectors constrain eligible
nodes but do not reserve capacity or guarantee distinct hosts.
Delete old policy ConfigMaps, volume mounts, environment variables, and files only after every submitter uses a ready TauCluster profile. There is no compatibility fallback.
Readiness is fail closed
TauGrid accepts a profile set only when all of the following are true:
status.workloadProfiles.observedGenerationequalsmetadata.generation.- the
WorkloadProfilesReadycondition isTruefor that generation; - the published
profileSetHashis non-empty and matches the normalized resolved profiles; - the selected profile’s
Readycondition isTruefor that generation; and - namespace, team, and lane applicability authorize the caller.
When a run doesn’t name policy.profile, TauGrid selects the unique profile that
passes these checks. If the run sets an explicit GPU count (Job compute.gpus
or RayJob gpus_per_worker), implicit selection also skips profiles with a
different gpusPerWorker. That way a CPU-only run and a GPU run can’t resolve
to each other’s profile. The default catalog includes azure.workspace.cpu,
which has no team or lane restriction, so the CPU-only smoke and train
targets from tau workspace init-repo resolve it without naming a profile.
Missing, stale, drifted, forbidden, ambiguous, or unready data stops rendering.
TauGrid stamps successful output with
tau.azure.com/tau-cluster-generation,
tau.azure.com/workload-profile-set-hash, and
tau.azure.com/workload-profile. These annotations identify the observed
revision only; confirm quota or capacity availability with a separate check.
Profile readiness is not storage readiness. When a workload using a profile
mounts a PVC, the platform must provide an equivalent compatible StorageClass,
claim, and CSI implementation across the profile’s full eligible node set. The
driver’s node component must register on every eligible node; otherwise Kueue
and Kubernetes can place an admitted pod on a node where it fails before
container startup. Validate each node’s CSINode registration and run a
profile-constrained mount canary before advertising the profile for
PVC-backed workloads. For AKS Flex or other external nodes, do not rely on
AKS-managed Azure CSI addons; follow TauGrid’s customer setup path for a
compatible deployment-managed driver as described in
the AKS storage boundary.
Node-local hot volumes
A profile can declare a node-local NVMe scratch volume with hotVolume:
- name: research.training.8x
# ...
hotVolume:
storageClassName: tau-hot
defaultSize: 1Ti
maxSize: 2Ti
storageClassName names a StorageClass whose CSI driver publishes
CSIStorageCapacity. Because the StorageClass uses
volumeBindingMode: WaitForFirstConsumer, kube-scheduler places each pod on a
node with enough free local capacity. Kueue doesn’t account for hot storage.
Both sizes are Kubernetes quantities, and defaultSize can’t exceed maxSize.
The controller publishes sizes in canonical form, so 1024Gi appears as 1Ti.
Choose maxSize so it fits: it must not exceed a node’s usable NVMe divided
by the number of hot-volume pods that node runs at once.
Runs that use the profile get /mnt from this StorageClass on every direct Job
pod and every Ray worker. Each pod gets its own claim of the run’s
storage.hot_size, or defaultSize when the run doesn’t set one, and the
claim is deleted with the pod. The Ray head keeps an emptyDir. TauGrid
rejects a run whose hot_size exceeds maxSize. Pods with a hot claim set
fsGroup: 65532 so non-root workloads can write to it. Kubernetes also applies
that group to other volumes whose CSI driver supports fsGroup, such as a
ReadWriteOnce disk mounted at /data. A claim is released only when its
pod is deleted, so direct Jobs with a hot claim keep finished pods for 600
seconds unless the run sets run.ttl_seconds_after_finished. A multiKueue
profile can’t declare hotVolume, because the controller observes storage on
the manager cluster rather than on the workers.
Each pod with a hot claim checks /mnt before the workload or ray start
runs, and logs the mount source, filesystem type, size, free space, and
backing. The pod fails rather than falling back to /data when /mnt is on the
container root filesystem or on the same device as the kubelet’s pod
directory. That failure means the StorageClass didn’t give the pod a dedicated
volume. The pod also fails when /mnt fails the free-space or write check. See
the run config reference for the full list of
checks.
A profile with hotVolume gets a HotVolumeReady condition, and the profile
is Ready only when that condition is True. The condition is False in
these cases:
- the StorageClass doesn’t exist;
- the StorageClass doesn’t use
WaitForFirstConsumer; - no node is eligible for the profile’s ClusterQueues;
- no
CSIStorageCapacityfor the StorageClass reports free capacity on any of those nodes.
A node is eligible for a ClusterQueue when, in every resource group the
profile requests from, at least one flavor’s nodeLabels match the node. A
GPU-only resource group applies only when gpusPerWorker is greater than
zero. An eligible node counts only when a matching CSIStorageCapacity object
reports capacity greater than zero. Some drivers publish zero-capacity objects for
nodes without local disks, and those nodes don’t count. Capacity is free
space, so the condition also turns False while every profile node’s local
disks are full. That case has the reason HotVolumeCapacityUnavailable; the
other cases use HotVolumeNotReady. The condition message reports how many
profile nodes have free capacity. The condition doesn’t check the fit rule,
and it doesn’t replace the CSINode registration and mount canary checks
described above. Profiles without hotVolume don’t get the condition.
A failed run’s own hot claim can use the last free capacity. So automatic
retries and tau run resume accept a profile that is unready only for
HotVolumeCapacityUnavailable: they still check authorization and every other
dependency, delete the failed workload, and then wait up to 10 minutes for the
profile to become Ready before submitting the replacement. New submissions
still require Ready.
Connected and offline rendering
A normal run, server dry-run, or apply reads the connected cluster:
tau run --config tau.yaml --dry-run=server
Client rendering is still connected unless an explicit snapshot is configured. Export only a ready revision:
tau cluster profiles export --context my-cluster \
--output profiles.snapshot.yaml
Then set policy.workload_profile_snapshot: profiles.snapshot.yaml and provide
explicit policy.namespace, policy.team, and policy.lane. Snapshot input is
accepted only with --dry-run=client; it cannot authorize server dry-run or
apply. Treat snapshots as immutable review artifacts and re-export after a
TauCluster revision.
MultiKueue readiness and ownership
The multiKueue execution target has one deterministic profile contract:
- the standard distribution installs MultiKueue controller support;
TauCluster.status.conditions[MultiKueueReady]reports a current, active AdmissionCheck, referenced MultiKueueConfig, and active worker;- the catalog includes a ready profile with
executionTarget: multiKueue, a dedicated LocalQueue, and the ordinary team, namespace, and lane applicability required by the operator; and - profile selection resolves that profile explicitly through
policy.profileor implicitly as the unique ready, applicable profile.
Failure of readiness or profile resolution stops dispatch. A MultiKueue profile
uses the same fail-closed applicability and ambiguity rules as every other
execution target. The supported boundary is TauGrid-rendered workloads whose kind
and dependencies are configured on the manager and every eligible worker.
Direct kubectl apply, hand-written Workloads, and objects mutated after TauGrid
renders them are outside this contract.
The platform owner owns worker credentials, least-privilege access, distribution and rotation, queue isolation, namespace and ServiceAccount parity, image pull identity, storage reachability, and revocation. This constrained supported capability remains Alpha until release evidence covers an environment/version matrix, manager-to-worker E2E, negative authorization and credential tests, operational enablement, credential rotation/revocation, drain, and rollback.
Drain and roll back
Before changing queue bindings, profile scope, or execution target:
- stop new submissions and set affected ClusterQueues to
HoldAndDrain; - cancel pending owners and wait for pending, reserving, and admitted counts to reach zero;
- export the current TauCluster object and ready profile snapshot;
- apply the new catalog and wait for the new generation and every selected profile to become Ready; then run connected server-dry-run checks; and
- restore admission.
Rollback by draining again, restoring the previous catalog, waiting for its new generation to become Ready, removing the MultiKueue profile and routing objects, then revoking unused worker credentials. Running pods stay pinned to their original placement when a profile changes. Removing profile authorization blocks new TauGrid submissions only. Continue status, log, cancellation, and cleanup operations until every manager and worker object is terminal, since active remote workloads still require active management. Controller status is an observation rather than a reservation: queues, credentials, storage, nodes, and capacity can change between readiness, render, admission, and scheduling.