At petabyte scale, storage architecture decisions are no longer reversible. The choices you make about erasure coding overhead, metadata replication topology, and failure domain granularity determine your operational costs and resilience profile for years. Getting them wrong is expensive. Getting them wrong on SMR drives specifically is expensive in ways that differ from the equivalent mistakes on a flash or CMR HDD cluster.
We built Leil as an SMR-native distributed filesystem, which means we have had to reason through every one of these decisions with SMR's specific write geometry in mind. This post describes the architecture choices we made and why, written for engineers who are either evaluating purpose-built SMR storage or designing their own distributed storage layer for sequential-workload AI infrastructure.
Why Petabyte Scale Changes the Calculus
At 100TB, you can survive with aggressive replication ratios and loose failure domain design. Drive failures are infrequent enough that you can rebuild before the next one arrives. Metadata is small enough to fit entirely in RAM on a single coordinator node. Network topology is simple enough that any node can reach any other node with predictable latency.
At 1PB, none of these assumptions hold cleanly. A 2% annual drive failure rate on a 1,000-drive cluster means roughly 20 drive failures per year, or one every 2.5 weeks. Rebuild times for 20TB SMR drives using sequential passes run 16 to 24 hours. At that failure rate and rebuild duration, you will have periods where a second drive fails before the first rebuild completes. Your erasure coding scheme must tolerate that scenario.
Metadata also grows. File count on a petabyte-class AI training cluster can reach tens of millions of inodes when you account for checkpoint files, sharded dataset files, eval outputs, and versioned artifacts. That metadata must be replicated and must survive coordinator node failures. Keeping it in RAM on a single node is no longer viable.
Erasure Coding: Choosing the Right (k, m) Parameters
Erasure coding with Reed-Solomon allows a (k, m) configuration where k is the number of data shards and m is the number of parity shards. The cluster can sustain m simultaneous drive failures without data loss. The storage overhead is k/(k+m). Leil defaults to (8, 2): 10 drives store what would otherwise take 12.5 drives with 2-copy replication, while tolerating 2 simultaneous drive failures.
The choice of k relative to your node count matters more than the absolute values. An 8+2 configuration on a 10-node cluster means each shard lives on a different node; you are tolerating 2 full node failures, not just 2 drive failures. On a larger cluster where multiple drives can fail on the same node, your failure domain design determines whether the 2-parity budget is consumed by node-level or drive-level failures.
For SMR specifically, the stripe width must align to band boundaries. Leil's write scheduler ensures that erasure-coded stripe writes land at SMR band boundaries across all nodes involved in the stripe. This is what makes SMR-native erasure coding possible: generic distributed filesystems that ignore band alignment cause constant out-of-band write handling on the drives, degrading effective write throughput by 60 to 80%. Our scheduler eliminates this by treating the band size as a first-class constraint in the write path.
A common alternative is (4, 2), which gives higher overhead (66% efficiency vs 80% for 8+2) but faster rebuilds because each rebuild processes smaller stripe width. For AI training workloads where datasets are written once and read sequentially, we lean toward the larger k (better capacity utilization) unless the cluster is running intensive rewrite workloads that trigger frequent band garbage collection.
Metadata Replication: Separating the Hot Path from Durability
Leil separates metadata from data storage and replicates metadata independently using a 3-node Raft consensus group. This is not a novel design (GlusterFS, WekaIO, and Lustre all separate metadata to varying degrees), but the key decision for an SMR-native system is ensuring that metadata writes never land on SMR drives in the write path.
SMR drives handle small random writes poorly. Metadata updates are inherently small and random: directory entries, inode modification timestamps, block allocation maps. Sending these to SMR drives would force constant out-of-band handling and destroy write performance. Leil routes all metadata to dedicated CMR (conventional magnetic recording) or SSD nodes in the cluster configuration. This adds a small hardware cost but eliminates the single largest performance pathology for general-purpose filesystems running on SMR hardware.
The Raft group provides strong consistency for metadata without requiring synchronous writes to all nodes. The leader handles all writes; followers replicate asynchronously with a bounded lag. On a well-tuned 10GbE cluster, replication lag is sub-millisecond under normal load. Metadata reads can be served from any follower, distributing read load across the 3-node group.
Failure Domain Design
Failure domains define which set of drives Leil considers as a single failure unit when placing erasure-coded stripes. The options are: drive-level (each drive is an independent failure domain), node-level (all drives on a node fail together), or rack-level (all nodes in a rack share a power and switch failure fate).
For most AI training clusters, node-level failure domains are the right default. In practice, drive failures are more common than node failures, but the most damaging failure modes involve power supply or network switch failures that take an entire node offline. Designing stripes to span failure domains at the node level means a single node failure consumes only one parity shard, not multiple.
Rack-level failure domains are appropriate for large multi-rack clusters where top-of-rack switch failures are a real concern. The tradeoff is that rack-level domains require stripes to span racks, which means cross-rack network traffic for every write. For a training cluster where all writes to the storage layer are large sequential writes anyway, this is usually acceptable. For small random workloads it would be prohibitive, but those are not the target scenario.
What we advise against for SMR-native clusters specifically: drive-level failure domains with small node counts. With 4 or 6 drives per node and a (8, 2) erasure code, drive-level failure domains spread stripes across drives within the same node. A single node failure then takes out 4 to 6 failure domains simultaneously, consuming all or most of the parity budget. This is a configuration error that looks fine in theory and fails badly in practice.
Handling Rebuilds at Petabyte Scale
Rebuild after a drive failure is the most operationally intensive event in a distributed storage cluster. For SMR drives specifically, rebuilds require reading all surviving shards and writing the reconstructed data to a replacement drive. The replacement drive must receive its data in sequential, band-aligned writes to perform well.
Leil's rebuild path honors the band-alignment constraint on the destination drive. The rebuilder reads surviving shards, reconstructs missing data in memory, and writes to the replacement drive in full-band sequential passes. This is slower per-unit-of-data than unaligned writes would be in terms of latency, but it produces a replacement drive that performs at full rated throughput from the moment rebuild completes, rather than one that is fragmented with out-of-band regions that continue to degrade performance for months.
Rebuild throughput on a 20TB SMR drive using Leil's band-aligned rebuilder is approximately 80 to 110 MB/s, giving rebuild completion times of 50 to 70 hours at full bandwidth utilization. During this window, the cluster operates at reduced redundancy. Monitoring and alerting on rebuild progress is essential at petabyte scale: a second drive failure during a rebuild window, while survivable with 2-parity erasure coding, should trigger immediate human review of the cluster health state.
What This Means for AI Training Workloads
The architecture decisions above are not hypothetical exercises. They determine concrete operational properties: how long you are exposed after a drive failure, what fraction of your cluster capacity is usable versus overhead, and how much rebuild traffic competes with your active training reads.
For sequential AI training reads, rebuild I/O is the main interference source during degraded-mode operation. Leil rate-limits rebuild I/O to a configurable fraction of total cluster bandwidth (default: 30%) so that training throughput degrades gracefully during rebuilds rather than experiencing spikes. This is a policy decision that reflects the AI training use case: a training run that takes 3 hours longer than scheduled is better than one that encounters read errors or bandwidth starvation mid-run.
Petascale SMR storage is not appropriate for every workload. If your primary access pattern involves frequent small random writes across a large namespace, CMR or flash is the right choice and no amount of write scheduling fixes the fundamental mismatch. But for AI training data at scale where the dominant operations are sequential reads and large periodic writes, an SMR-native architecture with correct failure domain design delivers the capacity density and cost profile that flash cannot match.