NVIDIA has open-sourced the internal tool it uses to explain why distributed AI training jobs fail. Called Attribution, the system traces a cascade of errors back to the single event that started it, instead of reporting where the job happened to stop. The company published the tool and a technical walkthrough on November 2, after running it for years across its own infrastructure.
How it turns noise into one diagnosis
When a distributed job fails, the symptom list misleads by design. One GPU or fabric event can surface as communication errors on other ranks, scheduler state changes, and thousands of downstream messages, each pointing somewhere different. Engineers then chase the loudest error, which is often just where the workload noticed the problem.
Attribution is workload-centric: it analyzes a failure in the context of one workload attempt rather than as isolated node events. It correlates workload-scoped logs, metrics, telemetry, and scheduler state, then returns a structured diagnosis covering the outcome, the root cause, the affected nodes, the supporting symptoms, and an evidence trail.
The pipeline runs in four stages. First, the Blueprint identifies the workload attempt from scheduler identifiers or an explicit execution window and node allocation. Then connectors collect observations from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and network metrics. The Service then extracts events and separates downstream symptoms from the cause.
The worked example: one faulty link
NVIDIA walks through a synthetic workload in which one GPU hits a hard fault, XID 149, and communication errors follow across other ranks. Application logs on three nodes show three different NCCL errors, while system logs on one node contain the XID 149 message, decoding to an NVLink signal integrity issue. Attribution follows that evidence to a single verdict.
Filtering does the decisive work. An early rule decodes the XID burst evidence into a signal integrity symptom on the faulty node, while the three NCCL events pass through unchanged. Later refinement keeps that symptom and demotes the others, so the final diagnosis names the faulty node and the signal issue, with the NCCL reports preserved as evidence.
The system is also designed to admit uncertainty. When several causes fit the evidence, the attributed domain is reported as ambiguous, and when evidence is too thin, the result can come back unattributed. That honesty matters: a tool that always names a culprit would eventually name the wrong one, and a confident wrong answer is worse than a flagged gap.
Service and Blueprint
Attribution ships as two components around a core rules engine. The Attribution Service is a self-hosted HTTP wrapper that receives normalized observations and returns one verdict per workload attempt. The Attribution Blueprint is a Python orchestrator that discovers workloads, runs collectors, calls the Service, and writes the result with a read-only dashboard.
The Blueprint already connects to Slurm, Kueue, and static inputs for execution context, and to Loki, local files, Kubernetes pod logs, and PromQL metrics for evidence, with stdout and SQLite outputs. Teams can add rules for new failure modes, build connectors for their own schedulers, and route results into incident, recovery, and fleet-management systems.
Why It Matters
Training failures are a tax on the entire AI buildout, and the tax grows with cluster size. A single bad GPU or flaky fabric link can idle thousands of accelerators while engineers argue about which error matters, and at today’s hardware prices that idle time is expensive. Automating triage turns a forensic exercise into a routine verdict.
The release fits a wider shift in AI infrastructure, where the bottleneck is moving from raw compute to the connections between machines. The optical networking at the heart of the recent Marvell-Lumentum deal shows how much money now rides on keeping thousands of GPUs talking reliably. Attribution attacks the same problem from the operations side.
Our read
The most interesting claim is not the open-sourcing itself but the scale behind it. NVIDIA says Attribution analyzes millions of workloads each week across hundreds of thousands of GPUs on its own infrastructure, and in one large cluster it traced port errors to patterns around specific host and switch ports. That production history separates this release from a demo project.
There is a natural limit to celebrate, too. Attribution’s verdict is only as good as the telemetry underneath it, and NVIDIA says so explicitly. Clusters with patchy logging will get ambiguous or unattributed results, which is correct behavior but a real adoption hurdle. The tool rewards teams that already instrument well, and exposes the gaps in everyone else’s observability.
That dynamic echoes NVIDIA’s broader cloud push, where its cloud-customer rules show how tightly the company now manages who runs what on its hardware. In both cases, control over infrastructure is becoming a product in its own right, not just a support function for selling chips.
Background
Attribution sits inside NVIDIA’s health and resilience software, alongside readiness checks, continuous health monitoring, and the systems that repair infrastructure and restart workloads. It complements those systems rather than replacing them: readiness checks catch problems before a job starts, monitoring watches jobs in flight, and Attribution explains what went wrong when a job fails.
Source: NVIDIA Technical Blog
0 Comments