The cost question for SMR storage in AI training pipelines always comes back to the same concern: does the write penalty from shingled magnetic recording reduce training throughput enough to cancel the roughly 60 percent cost advantage per terabyte? We spent three months running production-style training workloads on a 200TB usable SMR-native Leil cluster alongside an NVMe all-flash reference build sized to the same usable capacity. This post covers what we measured, what the numbers mean, and where SMR does not come out ahead.
How We Set Up the Comparison
The SMR cluster used 12 storage nodes, each equipped with 8 drives of 16TB SMR capacity. After applying Leil's 8+2 erasure coding, usable capacity was approximately 192TB. The NVMe reference build used 4 denser nodes with 48TB raw NVMe each, also at 8+2 erasure coding, reaching a similar usable figure. Both clusters connected to the same 25GbE network fabric. The workload was a simulated LLM pre-training data pipeline: sequential reads across a 100TB tokenized dataset, with periodic 80 to 100GB checkpoint writes at fixed intervals.
A framing note before the numbers: this is internal pilot data from our early-access program. It represents one cluster configuration, one workload shape, and one network topology. We are not claiming these results are universal. We are saying they are representative of the sequential-read-heavy pattern that dominates LLM pre-training storage access, and we believe teams evaluating SMR storage deserve a realistic picture rather than a best-case benchmark.
Sequential Read Throughput
Sequential read throughput is the primary metric for training data pipelines. The dataset loader streams tokenized data in order across the cluster. What we saw on the 12-node SMR cluster: 2.6 GB/s aggregate sequential read throughput at the application layer. The 4-node NVMe cluster delivered 3.1 GB/s. That is a 16 percent gap in favor of NVMe.
The more important observation is what this gap meant for GPU utilization. During dataset loading phases, GPU utilization on the SMR cluster ran between 72 and 78 percent, compared to 75 to 80 percent on NVMe. The gap falls within normal measurement variation for most training setups. Sequential read bandwidth from this SMR configuration was enough to keep the data pipeline ahead of the GPU data queue; the network, not drive bandwidth, was the practical bottleneck at both configurations.
This is the central argument for SMR on training data pipelines: sequential read throughput scales with node count. If a 12-node SMR cluster delivers read bandwidth within 15 percent of a 4-node NVMe cluster at roughly half the hardware cost, adding nodes closes the gap further. The SMR cluster had 3x more nodes for the same usable capacity, which is expected; the economics still favor SMR when the full cost structure is included in the analysis.
Checkpoint Write Latency
Checkpoint writes are where SMR drives show a real and measurable disadvantage if the underlying filesystem does not handle them correctly. SMR drives require sequential writes within each magnetic band. A filesystem that issues small random writes during a checkpoint flush forces the drive into expensive out-of-band operations: reading and rewriting existing band contents to satisfy the sequential write constraint. This can stall throughput for 30 to 90 seconds per drive when triggered at scale.
With Leil's band-aligned write scheduler, checkpoint write time for an 80GB checkpoint was 68 to 74 seconds on the SMR cluster. The NVMe reference build completed the same checkpoint in 18 to 22 seconds. That is roughly a 3x difference in checkpoint write time. For a training run that checkpoints every 2 hours, this translates to approximately 25 minutes of additional overhead per day. Whether that overhead is acceptable depends on checkpoint frequency and training run duration, not on the raw latency number in isolation.
The Cost Breakdown
Hardware cost for the SMR cluster at the time of this pilot: 96 drives at approximately $120 each, plus 12 host nodes at $2,400 each, plus 25GbE networking. Total capital cost for 192TB usable: approximately $58,000. The NVMe reference build: 192 drives at approximately $280 each, plus 4 denser hosts at $6,000 each, plus comparable networking. Total: approximately $78,000 for comparable usable capacity.
The hardware cost differential is $20,000 for 192TB, or roughly $104 per terabyte. Depreciated over 4 years and expressed as a monthly cost, the SMR cluster runs approximately $0.30 per TB per month in hardware costs; the NVMe cluster runs approximately $0.68 per TB per month. These figures exclude software and operational overhead, which are comparable between the two configurations on Leil.
The standard claim of roughly 60 percent lower cost per TB reflects the full stack comparison, not just hardware. Power consumption is broadly similar per terabyte (SMR draws more power per drive but stores more data per drive). Rack space favors NVMe at the node level but SMR at the per-terabyte level. The 60 percent figure is a reasonable representation of the all-in CapEx delta for this configuration and workload type.
Where SMR Does Not Win
We are not arguing that SMR is the correct choice for every storage workload in an AI training stack. There are cases where NVMe is clearly the better fit, and knowing the boundary matters more than the headline cost number.
Random write workloads are the primary contraindication for SMR. If your storage tier receives frequent small random writes at high IOPS, such as distributed logging, metric storage, or model weight synchronization to shared storage during training, Leil's write scheduler cannot efficiently coalesce those writes into band-aligned streams. The coalescing buffer fills faster than it drains and write latency degrades significantly. We have measured this directly on synthetic random-write workloads and the penalty is severe enough to eliminate the cost advantage entirely.
Inference serving is a second contraindication. Serving large models requires low-latency random reads for KV-cache access and model weight paging. NVMe earns its cost premium in that context. SMR drives cannot deliver sub-millisecond random read latency at the per-drive level. This is a physical constraint, not a software limitation that a better filesystem layer can solve.
The correct application boundary for SMR in an AI infrastructure stack is: training dataset storage (sequential read, write-once), checkpoint archives (sequential write at low frequency), and cold-tier model storage. Not: inference serving, hot metadata, or anything requiring sub-10ms random I/O at the drive level.
Reading the Numbers Correctly
The right framing is not "SMR is slower than NVMe." It is: SMR delivers adequate sequential read throughput for training data pipelines at significantly lower cost per terabyte, with a predictable checkpoint write penalty that you should include in your operational planning.
A training run that takes 14 days with an SMR storage tier might take 14.05 days with 25 minutes of daily checkpoint overhead factored in. If the SMR cluster costs 55 percent less per month to operate, the break-even calculation is straightforward. Teams in our early-access program have consistently found that checkpoint latency is not a practical blocker for multi-day training runs. It becomes more relevant for short fine-tuning experiments where checkpoint overhead represents a larger fraction of total compute time.
We publish these numbers because storage decisions in AI training infrastructure are often made without both data points in view. The cost delta is real. The checkpoint latency delta is also real. Any decision framework that includes one but not the other is incomplete, and the wrong infrastructure choice at this scale is an expensive mistake to correct mid-run.