Troubleshooting on TauGrid

Troubleshoot a run

Mon, 01 Jan 0001 00:00:00 +0000

Start with:

tau run status <run-name>

tau run status is the canonical lifecycle view for any run – read its phases top to bottom and stop at the first one that is not done. Then work through the layers below, in order, until you reach that phase:

Repository/connection resolution and cluster access – tau workspace connection (--offline for local configuration only).
TauWorkspace readiness and handoff validity – tau workspace status <name>.
Client-side config validation and rendering – tau run validate --config tau/train.yaml.
Queue admission and quota – the Kueue admission phase in tau run status.
Kubernetes scheduling, DRA, image pull, init, and readiness – the remaining phases in tau run status.
GPU/node/topology health – tau cluster validate nodes / tau cluster validate topology.
Ray/Job runtime progress and durable evidence – tau logs <run-name> and taugrid-portal experiment status <name>.
Recovery handoff – Retry and resume.

Confirm the queue admitted the workload (layer 4) before moving to GPU or node debugging (layer 6). Treat a Running pod phase (layer 5) as evidence of container start alone – confirm separately that the model or data process is making progress (layer 7).

Troubleshooting by lifecycle layer

Mon, 01 Jan 0001 00:00:00 +0000

This is the canonical diagnose-first path for a stuck or failed run. Work through the layers in order and stop at the first incomplete layer. Each layer names its own TauGrid command first; use raw kubectl only for the deep, operator-only inspection each layer calls out, never as a substitute for the TauGrid command.

Start every investigation the same way:

tau run status <run-name>

tau run status is the single canonical lifecycle view. It walks the same ordered phase tree for every run – Submitted, Kueue admission, (for a RayJob) RayCluster, pod scheduling, DRA allocation, image pull, init containers, container start, readiness, and (for a RayJob) RayJob status – and each phase reports pending, active, done, warning, or skipped. Read it top to bottom and treat the first phase that is not done as the layer to investigate, addressing it before any later phase. Add --watch to follow it live.

Retry and resume

Mon, 01 Jan 0001 00:00:00 +0000

TauGrid has two recovery paths for a failed run: automatic retry, driven entirely by the resilience.* fields in your run config, and manual tau run resume. TauGrid performs retry as automatic behavior of tau run when resilience.max_retries > 0; there is no separate tau run retry subcommand to invoke.

Automatic retry (`resilience.*`)

resilience:
 max_retries: 2 # default 0 (disabled)
 retry_on: ["Preempted", "Evicted"] # default; OOMKilled is opt-in
 backoff_initial: 30s  # default
 backoff_max: 5m  # default
 checkpoint_path: /data/checkpoints/finetunes/<name>  # default, derived from run name

When max_retries > 0 and you did not pass --dry-run, tau run:

Troubleshooting on TauGrid

Troubleshoot a run

Troubleshooting by lifecycle layer

Retry and resume

Automatic retry (resilience.*)

Automatic retry (`resilience.*`)