TauWorkspace reference

Kubernetes workspace desired state and readiness

TauWorkspace is a platform-authored Kubernetes resource in the TauGrid system namespace (tau-system by default). A custom tau cluster install --namespace <name> moves the controller and these objects together; administrative workspace commands use --system-namespace <name>, and repository descriptors persist the value as cluster.systemNamespace. It reconciles or verifies:

  • Target namespace and namespaced RBAC.
  • Kueue LocalQueue accessibility.
  • Workload identity ServiceAccount configuration.
  • Readiness and drift conditions.

Platform teams provision Azure resources and cluster-scoped quota policy, projects own their own code, and platform teams verify the durable storage PVC separately.

For tau workspace create, --system-namespace selects where the TauWorkspace object lives and defaults to the installed TauGrid system namespace; --namespace selects the researcher workload Namespace and defaults to the workspace name. These are intentionally separate because system components share one namespace while workload isolation remains workspace-scoped.

Workspace status is intentionally agent-readable:

tau workspace list
tau workspace status <name>

Workspace RBAC is the API default and what tau workspace create writes; the controller binds the researcher subject in the workspace namespace. Cluster-wide authorization is an explicit opt-out: the platform grants researcher access separately. Some existing clusters use this mode. The multiple-workspace lifecycle is Alpha: v0 activates one workspace and blocks additional workspace objects until the active workspace is removed.

The controller source and raw Kustomize manifests live in the Azure/taugrid repository. Versioned MCR release artifacts include the controller image, the standalone controller OCI chart, the TauGrid umbrella OCI chart, and the CRDs packaged by both charts.

Fresh clusters consume the umbrella chart through tau cluster install. The standalone chart is only for a separately managed controller and must not be installed alongside an umbrella release that already enables components.tauCoreController. See Install TauGrid for the supported installation entry point and readiness gate. This page defines the workspace API and readiness contract only; installing the cluster control plane is covered there.

Fixed team quotas with native borrowing

Set TauCluster.spec.queues.shareTeamQuotas: true (Helm: tauCluster.queues.shareTeamQuotas) to share unused capacity between teams. This opt-in is mutually exclusive with capacityTeamRef; both fields are unset by default. Existing explicit-quota and exclusive capacity modes keep their previous behavior.

Each TauTeam in the controller’s system namespace must declare its guarantee using spec.quota, by exact ResourceFlavor and resource. Workspaces set spec.teamRef.name and omit spec.quota: they share their team’s entitlement without fixed workspace splits. Existing workspace names, workload namespaces, and storage do not need to change. Reassigning an existing workspace to another team preserves its namespace only when its authoritative workspace UID still matches; this is not permission to adopt another workspace’s namespace. Owned LocalQueue team labels follow the same workspace UID; external queues are not relabeled, and existing queue policies are preserved. Deleting teams cannot acquire new workspaces. Existing workspaces can retain their applied membership or move to another team so deletion can finish. Direct Jobs in shared mode retain the workspace’s LocalQueue, including the default jobqueue, while verifying its canonical workspace ClusterQueue. No LocalQueue rename or workload profile preset is required.

The platform owner must provide the Kueue Cohort named tau-shared-teams, with the label tau.azure.com/shared-team-quotas: cluster matching the singleton TauCluster’s name. It must not be labeled as TauGrid-managed and must not have a parent. Its spec.resourceGroups declares explicit, static spare quota by ResourceFlavor and resource, using the existing Kueue API. A parent with no resource groups contributes no spare. An odd spare GPU belongs here rather than being silently dropped or assigned as an extra team guarantee.

TauGrid never creates, updates or deletes this parent. Child team Cohorts contain their fixed nominal guarantees; workspace ClusterQueues have zero nominal quota. The sum of the explicit parent spare and team guarantees is validated against discovered supply, without counting workspace nominal quota again. Inventory growth, Team creation or Team deletion never causes a parent quota increase. Deleting a Team removes its guarantee, not a transfer to the parent; reusing that quota requires operator maintenance.

Only explicitly configured flavor/resource pairs become queue quotas. As required by Kueue, flavors that share a resource must cover identical resource sets across the combined parent and Team declarations; disjoint sets are also valid. Incompatible combinations are rejected before creating quotas, not filled with invented budgets. Established zero-only entries are retained when a Team disappears, without retaining its positive guarantee. CPU, memory and other resources may be declared explicitly as static shared spare, but TauGrid does not invent ancillary guarantees or copy inventory into the parent. Resources not managed by the queue remain subject to Kubernetes placement and resource requests. GPU classes are separate flavors, not interchangeable equivalents. Capacity is allocatable supply, not instantaneously free devices or a guarantee that a particular image, architecture or placement can run on every node.

A team’s nominal quota is an eventual entitlement, not a hard usage cap. Omit team borrowingLimit and lendingLimit to borrow available spare capacity from the whole tree and lend unused guarantees. An explicit zero disables that direction; a finite borrowing limit caps additional usage above nominal. Workspaces receive borrowing ceilings derived from the fixed configured tree, not live inventory, and do not receive additional nominal budgets. Cross-queue reclaim and borrow-preemption are disabled: borrowed jobs finish normally, so an entitled team may wait for borrowers to complete. Guarantees are not immediately reclaimable and are not preserved through physical supply loss.

There are no automatic root/team transfers or inventory-driven rebalancing. Native Kueue borrowing uses unused quota without rewriting anyone’s guarantee. Overlapping external budgets, legacy full-inventory discovery queues, duplicate workspace nominal quotas, and retained exclusive capacity allocations block reconciliation rather than being silently adopted. Missing resources and disappeared flavors have zero capacity. If fixed guarantees plus static spare or active reservations exceed supply, or inventory or ownership cannot be validated, owned workspace queues use Kueue Hold. Hold preserves admitted jobs but cancels quota-only reservations; it is not HoldAndDrain. New admissions resume after the problem clears; operator-owned queue holds are preserved, including a stop-policy change made while a controller-owned Hold was active. Recovery clears only the controller’s Hold. User-authored guarantees are never silently scaled down.

Recovery also clears controller-owned Holds on existing deleting-workspace queues so pending work can finish before cleanup. It does not recreate missing queues or change their routing, quotas or operator-owned stop policies. Workspace cleanup and finalizer removal share the quota-reconciliation lock; both paths refresh Workspace identity inside that boundary before mutating queue state.

Eligible Linux GPU nodes must have tau.azure.com/gpu-class; an unclassified Ready, schedulable, non-deleting GPU node blocks this mode rather than silently disappearing from whole-pool accounting. When instance-type labels are absent, platform-owned label rules can match exact authoritative hardware labels:

tauCluster:
  extraNodeLabelRules:
    - match:
        labels:
          nvidia.com/gpu.product: NVIDIA-GB10
          nvidia.com/gpu.machine: NVIDIA_DGX_Spark
          kubernetes.io/arch: arm64
      labels:
        tau.azure.com/gpu-class: gb10

This declares a distinct class without inferring dedicated GPU memory from unified host RAM. Every match label must be present and equal. Existing match.vmSizes rules remain supported; if both selectors are supplied, both must match. Conflicting rules on an actual node fail reconciliation.

Bootstrap and quota maintenance

Upgrade the controller and CRDs together before enabling the mode. Declare the static parent and every Team guarantee before bootstrap. TauGrid creates the Team Cohorts before creating zero-nominal workspace queues.

Established Team Cohort quota specifications are not rewritten. Adding or recreating a Team Cohort while shared workspace queues exist is rejected with an operator-maintenance diagnostic. A live TauTeam.spec.quota edit therefore does not silently move quota or claim an atomic cross-Cohort change. Parent quota edits are also operator maintenance, not a live redistribution API.

For budget changes or migration, pause submissions and TauGrid reconciliation, wait for admitted jobs and reservations to finish, and remove shared workspace ClusterQueues while their scheduler is still running so its cleanup can finish. Coordinate stopping the scheduler through its owner. During that stopped boundary, update the explicit static parent and Team declarations and remove changed Team Cohorts. Resume TauGrid to create the complete fixed hierarchy before the scheduler and submissions resume. Keep TauWorkspace objects, namespace UIDs and storage intact. A managed scheduler requires its owner’s supported maintenance process; do not install a second controller or uninstall the scheduler as a shortcut.

To migrate from exclusive capacity, drain and remove its retained budget under the same maintenance boundary; the shared reconciler does not take over a nonzero exclusive budget or an automatically managed shared root. To leave shared mode, drain the hierarchy and remove its static root through the platform owner. A retained shared root blocks other capacity modes. Published releases are never rewritten by this migration.

Automatically shared exclusive team capacity

Automatic capacity is explicit and independent of queue ownership. Select one team in the singleton TauCluster; omit spec.quota on that team and its team-linked workspaces:

apiVersion: tau.azure.com/v1alpha1
kind: TauCluster
metadata:
  name: cluster
spec:
  managementMode: Reconcile
  queues:
    capacityTeamRef:
      name: research

For Helm, set tauCluster.queues.capacityTeamRef.name. The referenced TauTeam and its TauWorkspace objects live in the controller’s system namespace. Each workspace must set spec.teamRef.name to that team. No flavor/resource allocations or per-workspace splits are required. Omitting capacityTeamRef retains the existing explicit-quota and external-queue behavior; an empty quota does not become unlimited.

The pool is the total allocatable inventory of Linux GPU nodes with tau.azure.com/gpu-class, filtered by TauCluster.spec.nodes.selector. Only Ready, schedulable, non-deleting nodes contribute. Each node is counted once in its GPU-class flavor; all advertised allocatable resources, including CPU, memory, GPU, RDMA and host slots, are accounted in their native quantities. Missing resources have zero quota, not unlimited quota. This is not instantaneously free capacity: running Pods are not subtracted. Kubernetes placement, taints and available devices still constrain actual execution.

The team’s root Kueue Cohort owns the inventory budget exactly once. Its workspace ClusterQueues have zero nominal quota and can each borrow the available team pool. Other workspace reservations therefore consume the same budget rather than independent copies. Admission checks on existing, authoritatively owned workspace queues are retained. External queue ownership and routing are never adopted as a fallback.

Overlapping budgets or retained external queue reservations are rejected, including differently named ResourceFlavors whose selectors match pool nodes. Legacy GPU discovery cannot run alongside this mode. Remove conflicting declarations through their owners; TauGrid does not divide the pool or rewrite external budgets. Node, flavor, queue and team changes trigger reconciliation; a periodic retry also recovers missed events and transient dependency errors. If an established automatic pool cannot be validated, its root budget is zeroed to block new admissions until reconciliation succeeds.

Inventory shrink reduces the root budget even below in-flight reservations; it never fabricates replacement capacity or evicts running work. A workspace’s borrowing limit can temporarily retain its existing reservations, but the shared root remains bounded by current inventory and prevents new aggregate over-admission. Disappeared flavors remain with zero capacity so existing contracts can drain. To leave this mode, drain workloads and remove the team’s workspaces before removing or reassigning capacityTeamRef. A retained automatic Cohort budget blocks reallocation until its owner has reconciled the old allocation away.

Workspace readiness and recovery

Start with the condition reason and message, rather than a guessed repair:

tau workspace status <name> --context <context>
tau workspace status <name> --context <context> -o json

The overall phase currently requires RBACReady and QueueReady; DriftDetected=True also makes it Degraded. Storage remains platform-owned desired state, tracked outside TauWorkspace’s own conditions.

ConditionPlatform action
RBACReady=FalseIn workspace-rbac mode, correct the declared subject/role and let the controller reconcile namespaced RBAC. In cluster-wide mode, repair the pre-existing authorization that the platform manages directly.
QueueReady=FalseRestore the named LocalQueue and its accessible backing ClusterQueue; fix that queue directly rather than creating a second queue with a different name to bypass the workspace spec.
DriftDetected=TrueFix the dependency or workspace spec named in the message, then let the controller restore its owned namespace/RBAC objects.

During namespace retargeting, unfinished Workloads on retained external queues can delay cleanup of the old namespace’s routing metadata. Submitted, unfinished Jobs, RayJobs, and RayServices that have not received a derived Workload yet retain routing metadata on deletion or retargeting, using their queue label and the live queue selector. Completed or failed batch sources do not block cleanup. A RayJob is finished only when status.jobDeploymentStatus is Complete or Failed, matching Kueue; a failed driver with a retry still pending retains its route. RayJob discovery is skipped when the optional RayJob API is not installed; RayService discovery is checked independently and also tolerates an absent API. Other discovery errors block cleanup until the source state can be read. RayServices have no terminal job state: a queued service retains required routing metadata and unchanged execution identity until the service is deleted, even if a derived Workload reports Finished. Delete the service before completing workspace deletion or retargeting. Both Helm and Kustomize grant the controller read-only discovery of Jobs, RayJobs, and RayServices. Kueue-managed Deployments submitted with tau serve deploy --kind=deployment also retain routing before their Pod Workloads exist. Discovery uses the Pod template’s managed and queue labels, not just the Deployment’s own labels. A Deployment has no terminal job state: a present managed Deployment retains its required route even at zero replicas or after a Pod Workload finishes. Delete the Deployment before completing workspace deletion or retargeting. Both controller RBAC manifests include read-only Deployment discovery; unreadable Deployment lists block cleanup. An unchanged workload identity ServiceAccount is retained until those Workloads finish, while researcher authorization is reconciled independently. Changing the ServiceAccount name or client ID, or removing workloadIdentity, revokes the obsolete ServiceAccount before granting the current researcher access to the old namespace, even during the drain. Pending Jobs that use the revoked account can no longer create Pods; the operator must cancel or replace affected work before the drain can complete. Keep the identity unchanged until draining finishes when uninterrupted execution is required.

If identity cleanup fails or deletion is delayed by a finalizer, researcher bindings are removed until obsolete identity revocation is confirmed. After cleanup recovers, the current researcher’s binding is restored in the retained resolved namespace even while its Workloads are unfinished, so the researcher can inspect or cancel the remaining work. Cluster-wide authorization does not create a researcher binding.

Storage readiness is platform-owned

Ready proves RBAC and queue reconciliation only, not storage. TauGrid 0.1 has no StorageReady condition on TauWorkspace or TauCluster, so a workspace can reach Ready while a workload’s configured PVC is missing or unbound. Verify the platform-managed claim directly before handing the workspace to a researcher:

kubectl get pvc blob-training -n <workspace-namespace> --context <context>

Even Bound proves existence only; write validation happens at mount time on the workload’s own pod. The platform must also verify that the selected CSI driver is registered on every node eligible for the workload profile:

kubectl get csidrivers
kubectl get csinode <eligible-node> \
  -o jsonpath='{range .spec.drivers[*]}{.name}{"\n"}{end}'

Repeat the CSINode check for every eligible node, then run a mount/read/write canary with the workload’s profile, ServiceAccount, and security settings. Wrong credentials or read-only mounts surface in the canary or workload. Missing node registration fails earlier, before container startup, with pod events such as driver name file.csi.azure.com not found in the list of registered CSI drivers; file.csi.azure.com will also be absent from that node’s CSINode object.

AKS-managed Azure CSI addons are supported and expected on normal AKS agent pools, but TauGrid does not support relying on those managed addons on Flex or other externally registered nodes. TauGrid helps operators set up and validate a compatible deployment-managed CSI driver; operators retain ownership of the storage backend, credentials, capacity, data lifecycle, and driver upgrades. See AKS storage drivers and Flex nodes.

WorkloadIdentityReady is diagnostic; the overall phase gate currently excludes it. Resolve it before handing off workloads that rely on Azure Workload Identity.

The controller reconciles again periodically. After repairing the named dependency, wait for:

tau workspace check <name> --context <context>

TauGrid’s normal submission path always honors workspace readiness, with no bypass flag. Raw Kubernetes clients are governed by ordinary RBAC and Kueue rather than a custom TauGrid admission policy.