In this note
- Uptime, availability and goodput
- Published GPU failure rates
- What fails inside a GPU node
- Interruptions by cluster size
- Why per-node uptime overstates cluster availability
- Checkpoints: what each interruption costs
- Training goodput for a 32 to 128-node cluster
- Spares, hot standby and replacement time
- How to write a GPU cluster SLA
- Questions to ask a GPU provider
- What Lightyard's SLA covers
A GPU cluster can meet a 99.9% uptime SLA on every node and still lose a tenth of its training time. A synchronous training job needs every node it was launched on, and each failure costs the whole cluster a restart plus the work done since the last checkpoint. Sixty-four nodes that are each up 99.9% of the time are all up together only about 94% of the time.
The number that matters for training is goodput: the share of the hours you pay for that turns into training progress you keep. Large published runs report 85% to 97%. On our estimates, a 64-node HGX cluster lands between about 85% and 98%, depending mostly on two things an SLA rarely mentions: how fast a failed node is replaced by a healthy one, and how long a checkpoint blocks the job.
The failure rates are published research. The goodput model is our estimate, with the assumptions in tables.
Uptime, availability and goodput
| Term | What it measures | Who controls it |
|---|---|---|
| Node availability | Share of time one node is healthy and usable | The provider |
| Cluster availability | Share of time at least the nodes your job needs are healthy together | The provider, and spare capacity |
| Effective training time ratio (ETTR) | Share of wall-clock time spent on training steps that are kept | Provider and your training stack |
| Goodput | Productive training time over reserved time; used here the same way as ETTR | Both |
Google's TPU fleet paper splits goodput into scheduling goodput (everything the job needs is available at once), runtime goodput (counted only up to the last saved checkpoint) and program goodput (how close the code runs to ideal) (Google (opens in a new tab)). A provider's SLA can only speak to the first. Your checkpoint and restart tooling decide most of the second.
Published GPU failure rates
None of the public data covers Blackwell yet.
| Source | Fleet | What it reports |
|---|---|---|
| Meta, Llama 3 405B pre-training (opens in a new tab) | 16,384 H100, 54 days | 419 unexpected and 47 planned interruptions; about 78% of unexpected ones hardware; over 90% effective training time |
| Meta FAIR research clusters (opens in a new tab) | 16k and 8k A100, 11 months | 6.50 and 2.34 failures per 1,000 node-days; mean time to failure 47.7 days for 8-GPU jobs, 7.9 hours for 1,024-GPU jobs; ETTR 0.85 to 0.9 on the largest jobs |
| NCSA Delta, UIUC study (opens in a new tab) | 448 A100, 608 H100 (GH200) | Node availability about 99.4% (A100) and 99.3% (H100); H100 memory errors 3.2 times as frequent per GPU as A100 |
| ByteDance (opens in a new tab) | 9,600-GPU jobs | 97% ETTR on a three-month job, with warm standby machines |
| Google, Gemini training (opens in a new tab) | TPU pods | Goodput on the largest job raised from 85% to 97%; silent data corruption expected "every week or two" |
The Llama 3 run is the best benchmark for a dedicated training cluster: one job, one cluster, mature tooling. It works out to about 3.8 unexpected interruptions per 1,000 node-days, or about 1.4 per 8-GPU node a year, 1.1 of them hardware, the rate our cost model uses. We use Meta's research-cluster rate of 6.50 as the pessimistic case. Its 7.9-hour figure for 1,024-GPU jobs is worse than hardware rates predict, because real jobs also fail on filesystems, drivers and their own code.
What fails inside a GPU node
The counts are Meta's, from the Llama 3 run; the fix column is our summary.
| Component | Llama 3 interruptions (of 419) | Usual fix |
|---|---|---|
| Faulty GPU | 148 | Reset; if it repeats, swap the GPU or the node |
| GPU HBM3 memory | 72 | Row remapping or reset; swap the GPU if it keeps failing |
| Software bug and dependency | 56 | Your stack or a driver |
| Network switch or cable | 35 | Reseat or replace optic or cable; replace switch |
| Unplanned host maintenance | 32 | Drain and fix the host |
| GPU SRAM, system processor, thermal | 42 | Firmware or reset; swap if it repeats |
| NIC and NCCL watchdog timeouts | 14 | Replace NIC; find the straggler or bad link |
| Silent data corruption | 6 | Replay the step to find the GPU; swap it |
| SSD, power supply, chassis, CPU, memory, IO board | 14 | Field-replaceable part |
Faulty GPUs and HBM alone were 220 of 419. Memory needs watching on newer parts: the Delta study found the 512 spare rows per GPU for remapping didn't grow with H100's larger memory, and only about 59% of remapping attempts succeeded (Delta study (opens in a new tab)).
Meta's research-cluster paper lists backend network links, filesystem mounts, GPU memory and PCIe errors as the leading attributed causes, and saw 50% to 75% bandwidth loss from link problems during bring-up (Meta FAIR (opens in a new tab)). A 64-node cluster with eight 800 Gb/s NICs per node has 512 compute links with an optic or cable at each end, and a flapping link usually shows up as a slow job, not a failed one.
The same paper found 40 "lemon" nodes, 1.2% and 1.7% of the two clusters, that kept failing jobs. Removing them cut the failure rate of jobs over 512 GPUs from 14% to 4%. A node that fails twice should be replaced, not repaired again, and the SLA should say so.
Blackwell adds a dedicated RAS (reliability, availability and serviceability) engine for early fault detection (NVIDIA (opens in a new tab)). Nobody has published fleet data on how much it helps.
Interruptions by cluster size
Mean time between interruptions (MTBI) is one over the job's node count times the per-node rate. Our projections from the two published rates:
| Cluster | GPUs | MTBI at Llama 3 rate | Per month | MTBI at Meta research rate | Per month |
|---|---|---|---|---|---|
| 16 HGX nodes | 128 | 16.5 days | 1.8 | 9.6 days | 3.2 |
| 32 HGX nodes | 256 | 8.2 days | 3.7 | 4.8 days | 6.3 |
| 4 NVL72 racks | 288 | 7.3 days | 4.1 | 4.3 days | 7.1 |
| 64 HGX nodes | 512 | 4.1 days | 7.4 | 2.4 days | 12.7 |
| 8 NVL72 racks | 576 | 3.7 days | 8.3 | 2.1 days | 14.2 |
| 128 HGX nodes | 1,024 | 2.1 days | 14.8 | 1.2 days | 25.3 |
| Llama 3 scale | 16,384 | 3.1 hours (observed) | 236 | 1.8 hours (Meta's projection) | 405 |
A month is 730 hours. The H100 and A100 rates are applied to Blackwell unadjusted, for lack of public data. At 32 to 128 nodes that's between one interruption a week and one a day: manageable with automation, expensive without it.
Why per-node uptime overstates cluster availability
If each node is independently up a fraction a of the time, all N are up together a to the power N of the time. Spare nodes change the requirement from "all N" to "any N of N plus k".
| Per-node availability | Nodes the job needs | No spare | 1 spare | 2 spares | 4 spares |
|---|---|---|---|---|---|
| 99.9% | 64 | 93.8% | 99.8% | 100.0% | 100.0% |
| 99.9% | 128 | 88.0% | 99.2% | 100.0% | 100.0% |
| 99.5% | 64 | 72.6% | 95.8% | 99.5% | 100.0% |
| 99.3% (Delta H100) | 64 | 63.8% | 92.4% | 98.9% | 100.0% |
| 99.3% (Delta H100) | 128 | 40.7% | 77.2% | 93.6% | 99.8% |
Failures aren't fully independent, and a switch failure takes out several nodes at once, so treat this as the best case. A per-node percentage tells you little about a synchronous job. What decides cluster availability is how many healthy nodes stand by and how fast one replaces a failed node.
Checkpoints: what each interruption costs
Every interruption costs detection, restart, and the work since the last checkpoint, on average half the interval. Saving more often cuts rework but adds blocking time. The Young-Daly rule balances the two: the best interval is the square root of 2 times the blocking save time times the MTBI.
Checkpoints are large. Mixed-precision training with Adam keeps BF16 weights, FP32 master weights and two FP32 optimizer moments, about 14 bytes per parameter (our arithmetic; a 16-byte figure adds gradients, which aren't saved), so a 70B dense model's checkpoint is about 1 TB. NVIDIA's DGX B300 reference architecture recommends 40 GB/s of aggregate write per 72-node scalable unit at the standard level and notes that a synchronous checkpoint stops training while it writes (NVIDIA (opens in a new tab)). At that rate 1 TB takes about 25 seconds if storage is the only limit, and a large MoE takes minutes.
Asynchronous checkpointing copies the state to host memory and writes it in the background. ByteDance measured 0.01 to 0.04 seconds of blocking per save that way, against 6.8 to 13 seconds for a synchronous Megatron save (ByteDance (opens in a new tab)). Meta's research-cluster paper assumes 5 minutes per checkpoint and 5 to 20 minutes to restart (Meta FAIR (opens in a new tab)).
Training goodput for a 32 to 128-node cluster
Three cases, our estimates. A and B assume a healthy spare node replaces a failed one at restart, so the job always resumes at full size. C has no spare: when a node has to be pulled for repair, the job waits for it.
| Assumption | A: async checkpoints, hot spare | B: sync checkpoints, hot spare | C: sync checkpoints, no spare |
|---|---|---|---|
| Interruption rate | Llama 3 (Meta research rate in brackets) | Same | Llama 3 |
| Blocking time per checkpoint | 30 seconds | 5 minutes | 5 minutes |
| Detection plus restart | 5 + 15 minutes | 10 + 30 minutes | 10 + 30 minutes |
| Checkpoint interval | Young-Daly optimum | Young-Daly optimum | Young-Daly optimum |
| Interruptions that need a node pulled | Covered by the spare | Covered by the spare | 40% |
| Wait for the pulled node | None | None | 8 hours (part on site) or 24 hours |
| Cluster | A | B | C, 8-hour repair | C, 24-hour repair |
|---|---|---|---|---|
| 32 nodes, 256 GPUs | 98.9% (98.5%) | 96.8% (95.6%) | 95.1% | 91.9% |
| 64 nodes, 512 GPUs | 98.4% (97.7%) | 95.2% (93.5%) | 92.0% | 85.5% |
| 128 nodes, 1,024 GPUs | 97.5% (96.4%) | 92.8% (90.1%) | 86.4% | 73.5% |
Case B uses the checkpoint and restart times Meta's research-cluster paper assumes. Case A is close to the 97% ByteDance and Google report after years of tooling work. Case C with a multi-day RMA is why nobody runs a large job that way: teams hold spares or re-shard onto fewer nodes.
These figures leave out planned maintenance. Meta reports at least one interruption a day from firmware and kernel upgrades at 16,384 GPUs, so ask for rolling maintenance that drains a few nodes at a time.
Our B300 and GB300 NVL72 ownership models take 1% to 3% of GPU-hours off for failures, counting the days a node is out and a restart per event. A single synchronous job also loses the rework since its last checkpoint, which is why cases B and C fall further.
Each point of goodput on 512 GPUs is about 3,700 GPU-hours a month: US$3,700 for every US$1 of rate per GPU-hour, or about US$18,000 at US$4.84, the all-in cost per productive GPU-hour our model gives for owning B300s. The gap between cases A and B on 64 nodes is roughly US$58,000 a month at that rate, and it's recovered with checkpoint and restart tooling, not hardware.
Spares, hot standby and replacement time
A hot spare is a node that's powered, imaged, burned in and ready to join the job. ByteDance sizes its standby pool at the 99th percentile of simultaneous failures and brings standbys straight into the job instead of requeuing it, which it reports cut recovery time almost elevenfold (ByteDance (opens in a new tab)). The same sizing on Llama 3 rates, assuming 40% of interruptions pull a node (our estimate):
| Cluster | Nodes pulled per month | Spares at P99, repair in 8 hours | Repair in 24 hours | Repair in 72 hours |
|---|---|---|---|---|
| 32 nodes | 1.5 | 1 | 1 | 1 |
| 64 nodes | 3.0 | 1 | 1 | 2 |
| 128 nodes | 5.9 | 1 | 2 | 3 |
The lease checklist takes the conservative view that every hardware interruption takes a node out. That roughly doubles the pulls, to about 3 a month at 32 nodes, and adds one spare to each row of the 72-hour column and to 64 nodes at 24 hours.
SLAs often blur two numbers. Time to restore is how long until a healthy node is back in your job: minutes if a hot spare is already running, hours if one has to be brought up. Time to repair is how long the broken node takes to come back: hours with parts on site, days with an RMA. Ask for both.
Then ask whose capacity the spare is. If it's one of your reserved nodes, a 64-node reservation trains on 63, about 1.6% of capacity, though the spare can run evaluations between failures. If the provider holds spares outside your reservation, you train on all 64.
NVL72 racks change the unit. NVIDIA's partition guide says a failed compute tray marks its four GPUs as having no NVLink while the partition stays healthy, so work can continue on the remaining GPUs. A failed NVLink switch tray marks every GPU connected to it the same way, and the partition can't run NVLink workloads until it's fixed (NVIDIA (opens in a new tab)). One option is to size jobs to 16 of the 18 compute trays, leaving two in-rack spares. Ask which NVLink software release the racks run, how it handles a failed switch tray, and how fast switch trays are replaced, since that sets the worst-case outage.
How to write a GPU cluster SLA
A general-purpose cloud SLA counts whether an instance responds. A GPU cluster SLA has to count whether the hardware is fast enough to train or serve on.
| Clause | Training cluster | Inference deployment |
|---|---|---|
| Unit measured | Healthy nodes in your reserved set, per node and as N of the reserved total | The replica set behind each endpoint, and the capacity it serves |
| What healthy means | All GPUs visible, diagnostics pass, NCCL bandwidth and fabric links within an agreed margin of the acceptance baseline | Serves the agreed load within the agreed latency |
| Degraded | A slow node or flapping link counts as down | Capacity below the agreed replica count counts as degraded |
| Time to respond | Minutes, around the clock | Minutes, around the clock |
| Time to restore | A healthy node back in the job, from a spare | Replica back in service; with N+1 sizing, users shouldn't notice |
| Time to repair | Failed node fixed or replaced, separately stated | Same |
| Credit basis | Node-hours below the commitment, at the GPU-hour rate | Hours below the availability or capacity commitment |
| Repeat failures | A node that fails twice is replaced, not repaired again | Same |
| Reporting | Incident report per event; monthly availability with the raw health data | Same, plus latency percentiles |
For inference, design does most of the work. Size replicas for the busiest hour plus one spare node per model, and a node failure costs headroom, not availability.
Exclusions to read carefully
- Scheduled maintenance: notice, hours a month, rolling or cluster-wide
- Your software, images and drivers, and changes you request
- Failures outside the provider's boundary, such as your own links into the cluster
- Force majeure, and whether utility power events count as one
- Credits as the sole remedy, and the cap on them
Questions to ask a GPU provider
- What failure rate do you see per node per year on this platform, and can we see the data behind it?
- Is availability measured per node or per cluster, what counts as healthy, and does a slow node count as down?
- What are the time to respond, time to restore and time to repair, and are they in the contract?
- How many spare nodes are on site for our cluster, and are they inside or outside our reservation?
- Which parts are held on site: GPUs or whole nodes, NICs, optics, power supplies, switch trays?
- How are switch and fabric failures handled, since they affect many nodes at once?
- What does burn-in cover before acceptance, and can we watch and re-run the tests?
- How many nodes can planned maintenance drain at once?
- What happens to a node that keeps failing, and what monitoring data do we get?
The GPU lease agreement checklist covers the rest of the contract.
What Lightyard's SLA covers
Service levels are written into each contract, so they're set in the order form you sign rather than published here. Every reservation, bare metal or managed, includes monitoring, replacement of failed parts and running the cluster to an SLA. Nodes are single-tenant; on bare metal you hold root on every node, and firmware and the BMC stay with us. See how delivery works, contract terms, or talk to us about the service levels your workload needs.
Questions this note answers
One that measures what your workload needs. For training, the number of healthy nodes in your reserved set, tested against the acceptance baseline, and a restore time for swapping in a spare. For inference, the availability and capacity of the replicas behind each endpoint.
The best public figure is Meta's Llama 3 run: 419 unexpected interruptions in 54 days on 16,384 H100s, about 78% of them hardware. That's about 1.1 hardware interruptions per 8-GPU node a year. Nobody has published comparable data for B200 or B300 yet.
The share of reserved GPU time that ends up as training progress you keep. It leaves out time lost to failures, restarts, checkpoint writes and work redone since the last checkpoint. Large published runs report 85% to 97%.
On Llama 3 failure rates and our assumptions, one hot spare covers a 32 or 64-node cluster when a failed node is back within a day. A 128-node cluster, or repairs that take days, needs two or three. What matters as much is whether the spares come out of your reservation or the provider's pool.
The Young-Daly rule sets the interval at the square root of twice the checkpoint write time times the mean time between interruptions. With a 5-minute blocking save on 64 nodes that's about every 4 hours. With an asynchronous save that blocks for 30 seconds, it's about every 75 minutes.
Sources
- Meta: The Llama 3 Herd of Models, section 3.3.4 reliability (2024) (opens in a new tab)
- Meta FAIR: Revisiting Reliability in Large-Scale ML Research Clusters (2024) (opens in a new tab)
- UIUC: Story of Two GPUs, resilience of H100 and A100 on NCSA Delta (SC '25) (opens in a new tab)
- ByteDance: Robust LLM Training Infrastructure at ByteDance (SOSP '25) (opens in a new tab)
- Google DeepMind: Gemini technical report, training infrastructure (opens in a new tab)
- Google: Machine Learning Fleet Efficiency with ML Productivity Goodput (2025) (opens in a new tab)
- NVIDIA DGX SuperPOD reference architecture, DGX B300: storage architecture (opens in a new tab)
- NVIDIA Blackwell architecture: RAS Engine (opens in a new tab)
- NVIDIA GB200 NVL Partition User Guide v1.2: partition fault handling (opens in a new tab)