When people talk about the cost of training a large language model, the conversation goes straight to GPU hours. H100 on-demand pricing, spot instance availability, infiniband interconnect overhead. Storage gets a line in the budget, if it gets a line at all. In our experience working with early-access partners, storage is almost always undercosted at the planning stage and over-invoiced by the time the run finishes.
This post is a line-item breakdown of storage costs for a plausible mid-size pre-training run: a 30-billion parameter model, 1.5 trillion tokens of training data, running for roughly three weeks on a 64-GPU cluster. We are not attaching a named company to this scenario. These are illustrative figures built from realistic hardware pricing, typical erasure coding overheads, and the operational patterns we see across the teams we work with.
The Three Storage Budget Lines Nobody Unifies
Most ML infrastructure budgets split storage into three distinct line items that live in different spreadsheets: training dataset storage, checkpoint storage, and artifact storage (logs, eval outputs, model weights after training). The problem is these are billed differently, planned separately, and often provisioned with different systems. You end up with NVMe for checkpoints, object storage for training data, and a network-attached volume for artifacts. Each is sized for peak load with safety margins. None is sized with any awareness of the others.
Total storage spend across a mid-size run often falls in the $8,000 to $18,000 range for the duration of the training run alone. That is before replication, before cold archival, and before the artifact sprawl that accumulates when you run hyperparameter sweeps against the same base dataset.
Training Data: The Sequential Read Budget
For a 1.5 trillion token pre-training corpus with a typical 2-byte token encoding, you are working with roughly 3TB of tokenized training data. In practice, you usually hold 2 to 4 copies: a working set on fast storage for the active run, a replica for redundancy, and often a cold copy on cheaper storage kept around for re-runs or ablations.
The hardware cost for 3TB of NVMe-based training data storage on a dedicated cluster is roughly $900 to $1,500 per month depending on drive density and raid configuration. This feels cheap. The problem is that teams rarely hold just 3TB. Pre-training corpora for general-purpose models are assembled from multiple data sources with different quality tiers, and you tend to keep earlier versions around. Active training data storage for a production-grade pre-training effort is commonly 15TB to 50TB across versions.
At those scales, the $/TB/month difference between NVMe and SMR-class HDDs becomes meaningful. A 50TB NVMe training data pool at $35 to $50/TB/month all-in runs $1,750 to $2,500 per month. The same 50TB on a purpose-built SMR filesystem like Leil runs closer to $8 to $12/TB/month total cost (hardware amortization plus software), landing at $400 to $600 per month. For sequential read workloads, the training pipeline cannot tell the difference because read throughput is network-limited long before it is drive-limited.
We are not saying NVMe training data storage is wrong. We are saying the latency premium of NVMe is not buying anything for sequential reads, and the capacity density penalty compounds over the multi-week duration of a pre-training run.
Checkpoints: Where the Write Costs Hide
For a 30B parameter model in bf16, a single checkpoint is approximately 60GB. Training teams checkpointing every 500 steps at a typical 2,048 tokens per step and batch parallelism across 64 GPUs will hit checkpoint cadences on the order of every 30 to 90 minutes of wall-clock time, depending on gradient accumulation settings.
Over a three-week run, you generate 300 to 700 checkpoints. You almost never need all of them. Most teams keep the last five plus checkpoints at logarithmic intervals (1k, 2k, 5k, 10k steps), which means you are holding 15 to 25 checkpoints at any time, or roughly 1TB to 1.5TB of live checkpoint storage.
Where the cost surprises people: the write pattern. Checkpoints are large, sequential writes arriving in bursts. NVMe is fast but checkpoints do not need fast random writes. They need sustained sequential write bandwidth. An SMR-native filesystem handles this well because band-aligned sequential writes are exactly what SMR drives are optimized for. The cost difference on the checkpoint tier specifically is often 60 to 70% compared to an all-flash checkpoint store at equivalent capacity.
The bigger checkpoint cost issue is often the S3-compatible object store that teams default to for checkpoint durability. Object store checkpoint writes add 10 to 30 seconds of overhead per checkpoint due to multipart upload mechanics and the lack of POSIX rename atomicity. Over hundreds of checkpoints, this adds up to hours of GPU idle time per run. The storage cost of object store checkpoints is lower per TB, but the compute cost of the latency overhead often exceeds the storage savings.
Artifact Sprawl: The Unbounded Third Line
Training logs, evaluation outputs, RLHF annotation datasets, experiment tracking metadata: none of these are managed with any discipline at the early stages of a project. They grow as a side effect of iteration. A 10-run hyperparameter sweep against the same base dataset produces 10 times the checkpoint count, 10 times the TensorBoard log volume, and 10 times the eval output.
In our early-access experience, artifact storage is typically 2 to 4 times the size of the active training data at any point during active development. For a 15TB training corpus, expect 30 to 60TB of artifacts. This is almost always stored on whatever was convenient when someone needed space, which means it is usually the most expensive storage tier.
The practical fix here is not a storage technology choice. It is lifecycle policy: automated deletion of eval outputs beyond a configurable retention window, deduplication of repeated dataset versions, and explicit archival tiering for checkpoints beyond a defined age. Most teams implement this only after they get surprised by a storage invoice.
Software and Operational Overhead
The hardware line in the storage budget is what gets modeled. The software and operational lines are where the surprises live.
Storage software licensing for commercial distributed filesystem products typically runs $10 to $30 per TB per month, which on a 100TB operational footprint is a significant cost center. Open-source alternatives (Lustre, GPFS in community form, Ceph) carry lower direct cost but substantial operational engineering overhead: tuning, monitoring, failure response, and the implicit cost of the person-hours spent on filesystem operations rather than model development.
Operational overhead includes the time cost of dealing with drive failures, rebuilds, monitoring alert tuning, and the quarterly audits of what is actually stored. On a 100TB cluster with spinning HDDs, drive failure rates run around 1 to 2% per year, meaning one to two drives fail per year on a 100-drive cluster. Rebuilds on large HDDs take 12 to 20 hours for sequential verification passes, during which redundancy is reduced. This is not a crisis, but it is an operational rhythm that requires staff time.
Putting the Budget Together
For our illustrative 30B parameter, 1.5T token pre-training run over three weeks, a realistic storage budget looks like this:
- Training data (50TB active, NVMe): $2,000 to $2,500 for the three-week period
- Checkpoint storage (1.5TB live, NVMe or fast HDD): $200 to $400
- Artifact and log storage (30TB, object store or NAS): $300 to $600
- Replication overhead (1.5x to 2x total): adds 50 to 100% to hardware line
- Software licensing or operational engineering: $500 to $2,000 depending on stack
Total storage spend: $6,000 to $12,000 for a three-week run. On a compute bill of $40,000 to $80,000 for the same run (64 H100s at blended spot/on-demand pricing), storage is 10 to 20% of total spend. Not the majority, but not trivial, and entirely optimizable once you are explicit about what each tier actually needs from the storage layer it is using.
The recurring pattern we see is that training data storage is overprovisioned on flash (sequential reads do not need IOPS), checkpoints are underprovisioned on durability (async object store writes lose minutes per run), and artifacts are never proactively managed. Fixing the first two with purpose-built sequential storage, and the third with a lifecycle policy, typically brings total storage spend to 6 to 10% of the compute line without any throughput regression.
Storage is not the dominant cost in AI training. But it is one of the few costs where the spend-to-value ratio is genuinely poor on the default stack, and where the engineering investment to fix it is bounded and knowable.