Where HDD-native storage makes a real difference.
Three workload patterns where SMR drives at 60% lower cost match or beat NVMe-backed alternatives.
Trillion-token corpora without trillion-dollar storage bills.
Pre-training datasets are read sequentially, many times over multiple epochs. They grow in strict append-only fashion as new crawl data is added. Both of these patterns play directly to SMR drives: sequential reads run at full rated bandwidth, and sequential appends hit the band-aligned write path without triggering read-modify-write cycles.
A 200TB cleaned corpus stored on Leil costs roughly the same per month as 80TB on a managed NVMe storage service. The filesystem interface is identical: mount and run your DataLoader.
- WebText, RedPajama, CommonCrawl-scale datasets
- Append-only corpus updates compatible
- Multi-reader access for parallel GPU nodes
- No tokenization-output format change required
Checkpoint every 1,000 steps without a write-cliff.
Checkpoints are large sequential writes: the entire model state dict, optimizer states, and gradient accumulators written to disk in one batch. This is the ideal access pattern for SMR drives when the writes are routed through the band-aligned path.
Leil coalesces checkpoint writes at the write-scheduler layer. The GPU cluster issues concurrent writes from multiple nodes; the scheduler fans them into sequential band streams per drive. A 100GB checkpoint writes at full sequential bandwidth, completing in roughly 72 seconds on a 12-node cluster.
- torch.save(), zarr, and safetensors compatible
- Atomic rename for crash-safe checkpoint commits
- Concurrent writes from multi-node training run
- Retention policy: keep last N, delete older
One mount point. All tiers of your data pipeline.
Most AI training pipelines separate hot data (frequently accessed training examples), warm data (recent checkpoints and versioned datasets), and cold data (archived runs, historical snapshots). Managing this across object storage buckets and NVMe pools requires custom code, S3 SDKs, and cache invalidation logic.
With Leil, all three tiers live under one POSIX mount. Your pipeline reads from one path regardless of whether the data is a hot example being streamed during training or a checkpoint from six months ago. No SDK glue, no cache layer code, no format conversion between tiers.
- Single mount point replaces hot/warm/cold split
- Versioned datasets with standard directory structure
- No SDK or format change between access patterns
- Capacity scales to 5PB on a single cluster
Which of these workloads is yours?
Most early-access partners start with a single cluster and a specific dataset. We help size it from your current storage spend.