The default assumption in AI training infrastructure is that checkpoints go to fast storage. Fast storage means NVMe. NVMe costs $250 to $350 per terabyte raw. The assumption is rarely questioned because checkpoint writes feel like a critical path: if the write is slow, training stalls. But checkpoint writes are not the critical path in any training configuration we have encountered, and the premium paid for NVMe checkpoint storage is almost entirely wasted on latency properties that checkpointing does not use.
What Checkpointing Actually Does to Storage
A training checkpoint is a large sequential write. PyTorch's torch.save() serializes model weights, optimizer state, and training metadata to a file, then closes it. For a 7B parameter model in bf16, that is roughly 40 to 50GB of data written sequentially. For a 13B parameter model with optimizer state, expect 100 to 130GB. For 70B models with full mixed-precision optimizer state, 400GB is realistic.
The write is sequential because serialization proceeds field by field, tensor by tensor. There is no random-write component. Once written, the checkpoint file is closed and not accessed until either the training run fails and needs to resume, or an engineer runs evaluation against a specific checkpoint epoch. Typical checkpoint read frequency is very low: most production training runs reload a checkpoint zero or one times per run. Evaluation access is periodic and sequential.
This profile, large sequential writes at low frequency, large sequential reads at low frequency, is a near-perfect match for high-density HDDs. It is specifically not a match for NVMe, whose performance advantages concentrate in random read latency, sustained random write endurance, and IOPS throughput. None of those properties help a checkpoint workload.
The Numbers: SMR vs NVMe for Checkpoint Writes
We measured checkpoint write times on an internal Leil cluster using a representative workload: repeated sequential writes of a 100GB synthetic checkpoint file. The SMR cluster used 12 nodes each with 8 high-density SMR drives managed by Leil's band-aligned write scheduler. The NVMe reference cluster used 4 denser nodes with enterprise NVMe. Both clusters connected over 25GbE.
Results (all figures are internal pilot data from our test cluster, January 2026):
| Checkpoint Size | SMR + Leil (band-aligned) | NVMe reference |
|---|---|---|
| 40 GB | 28 to 34 seconds | 8 to 10 seconds |
| 100 GB | 68 to 76 seconds | 20 to 24 seconds |
| 250 GB | 172 to 190 seconds | 50 to 58 seconds |
The SMR write times are approximately 3.2x longer than NVMe. That gap is real and should be included in any operational planning for checkpoint frequency. The question is whether it matters for the training run as a whole.
Does Checkpoint Write Time Actually Block Training?
Most training frameworks checkpoint asynchronously or with a brief pause. The pause duration matters most for two cases: very frequent checkpointing (every few minutes) in long running experiments, and emergency checkpointing in response to detected hardware failure. In both cases, the relevant question is not whether SMR is slower than NVMe in absolute terms, but whether the additional latency affects end-to-end training wall-clock time.
For a training run that checkpoints every 2 hours and runs for 14 days, the total checkpoint overhead on SMR (at 100GB checkpoint size) is approximately 336 checkpoint events at roughly 72 seconds each: about 6.7 hours of checkpoint time total. On NVMe, the same run spends about 2 hours on checkpoint writes. The delta is 4.7 hours over 336 hours of training, or 1.4 percent additional wall-clock time. That figure is well within normal scheduling noise and is far below the threshold at which infrastructure engineers would consider it a bottleneck.
Emergency checkpointing on hardware failure is a different concern: if a node is about to fail and you need to flush the current training state immediately, a 70-second checkpoint window versus a 22-second window matters more. In practice, most training infrastructure uses watchdog processes or gradient accumulation checkpoints that do not rely on sub-30-second checkpoint writes. If your fault recovery protocol depends on checkpoint writes completing within 30 seconds, NVMe is a reasonable choice for that specific reason. That is a narrow requirement.
The Cost Difference Over a Real Training Cycle
Checkpoint storage capacity requirements for a serious training program are not trivial. Retaining checkpoints from a 14-day training run at 2-hour intervals means 168 checkpoint files. At 100GB each, that is 16.8TB of checkpoint data per run. Most infrastructure teams retain multiple runs, multiple model sizes, and experiment checkpoints, which pushes checkpoint storage to 50 to 200TB without much effort.
At current hardware costs, 100TB of usable NVMe checkpoint storage requires roughly $28,000 to $35,000 in drive hardware. The same 100TB usable on high-density SMR costs $6,000 to $10,000 in drive hardware. Software, hosting, and power add similar proportional costs on both sides. Over a 3-year infrastructure lifecycle, the SMR checkpoint tier represents a significant capital cost reduction with no meaningful operational impact on training throughput.
We are not claiming that checkpoint storage is a dominant line item in most AI infrastructure budgets. It is not. Compute is still the largest cost. But for teams managing growing experiment archives and multi-run checkpoint histories, checkpoint storage becomes a non-trivial expense, and the NVMe premium is particularly hard to justify when the access pattern is exactly what HDDs are designed for.
What the Right Filesystem Layer Changes
The 3x write-time penalty in our numbers reflects SMR drives running on Leil's band-aligned write scheduler. Without proper SMR-aware filesystem management, the penalty would be worse. Generic Linux filesystems such as ext4 or XFS issue small random metadata writes alongside the data writes during checkpoint serialization. Those random writes force expensive out-of-band handling on SMR drives, where each random write within a band requires reading the existing band content, modifying it, and rewriting the full band sequentially. This is the read-modify-write penalty, and it can stall write throughput to 10 to 20 percent of rated sequential speed.
Leil's write scheduler coalesces all writes for a given SMR band into a single sequential stream before flushing. The random metadata writes are buffered and merged with adjacent data writes. The drive sees a clean sequential write stream and operates at its rated sequential write speed, not its degraded random-write mode. The 3x gap in our table is the real gap with proper SMR handling; the gap without it can be 10x or worse.
This is the constraint on recommending SMR for checkpoint storage generally: you need an SMR-aware filesystem that handles the write alignment problem. The economics argument holds regardless; the implementation feasibility depends on whether your storage software supports band-aligned writes. On Leil, it does by design.