Observability and evidence
Use each evidence plane for its intended scope:
| Plane | Use |
|---|---|
| TauGrid status and logs | Immediate lifecycle of one submitted run |
| Kubernetes events and pod state | Admission, scheduling, startup, and termination |
| Ray and GPU metrics | Runtime and hardware behavior |
| Expstore and durable artifacts | Authoritative experiment, checkpoint, and recovery state |
| adx-mon and ADX/Kusto | Optional hosted scalar and fleet analysis |
| Dashboards and alerts | Consumer views over telemetry |
A workload reaching Scheduled or Running status does not prove the model is making useful progress. Treat each as a separate claim and verify independently. Preserve raw logs, profiles, and artifacts when a diagnosis depends on them.
Metrics-store methodology
TauGrid separates durable experiment ownership from fleet analytics:
| Layer | Responsibility |
|---|---|
| Immutable metric files | Preserve normalized scalar history in Parquet |
| Expstore SQLite index | Track runs, metric files, artifacts, idempotency keys, and summaries |
| Metric summaries | Provide finite counts, step ranges, min/max values, and the latest finite point per run and metric |
| ADX/Kusto projection | Support cross-workspace dashboards and fleet queries |
Expstore is authoritative. ADX ingestion uses checkpoints and query-time deduplication so retries preserve one logical scalar point. Configure ADX retention, hot-cache duration, identities, and database roles as platform policy. The workspace’s durable storage lifecycle governs local metric files and artifacts.
PVC-backed metric history and artifacts depend on the platform-provided CSI driver, not on TauGrid. The driver must register on every node eligible for the workload profile. This includes metrics-offload workloads: ADX is an optional projection and does not repair a PVC mount that fails before the trainer or sidecar starts. On AKS Flex or other external nodes, the deployment must use a compatible deployment-managed CSI implementation rather than rely on AKS-managed Azure CSI addons. TauGrid provides the customer setup and validation path; see the AKS storage boundary.
For disposable runs that need neither durable history nor offload, write only
to pod-local ephemeral storage. Those files disappear with the pod and are not
available for recovery, tau run get, or later experiment comparison.
Use project, run_group_id, and run_id for experiment identity. Use
metric_name, step, and wall_time to align and compare scalar histories.
Tags add workspace and workload dimensions without changing those core keys.
For a runnable workload that publishes immutable loss and accuracy chunks, see Live experiment evidence. For the full local evidence and artifact contract, see Experiment evidence and artifacts.
For the optional ADX-backed data plane, see Prepare ADX/Kusto for TauGrid. For Portal deployment and its available boards, see Configure Portal.