A week-long training run that fails in the final 12 hours and must restart from the beginning is an expensive mistake. A training run that checkpoints every 200 steps adds 8 to 15% overhead to every GPU-hour and creates terabytes of checkpoint artifacts to manage. These are two failure modes of checkpoint strategy, and most teams default closer to the latter: over-frequent checkpoints on expensive storage, managed with no lifecycle policy, creating a growing burden on both storage costs and training throughput.
Checkpointing strategy has two distinct dimensions that are often collapsed into a single decision. The first is checkpoint frequency: how often you save, which governs maximum recovery loss and training overhead. The second is checkpoint storage architecture: where you save, which governs checkpoint write latency, storage cost, and the operational complexity of managing checkpoint artifacts. Both matter and they interact.
Checkpoint Frequency: The Recovery Window Tradeoff
The argument for frequent checkpoints is recovery time: if your cluster fails, you lose at most the training progress since the last checkpoint. The argument against is overhead: every checkpoint is a large synchronous write that blocks GPU forward passes while checkpoint data serializes from VRAM to storage.
For a 30B parameter model in bf16, a single checkpoint is approximately 60GB of model state. On a 64-GPU cluster with 10GbE storage network, writing 60GB takes 45 to 75 seconds depending on filesystem write throughput and compression. If you checkpoint every 500 steps and a step takes 2.5 seconds of wall clock time, checkpointing adds roughly 6 to 10% overhead to training time. At every 100 steps, that overhead exceeds 25%.
The practical default we see working well for week-long runs: checkpoint every 800 to 1,200 steps, with asynchronous checkpointing to minimize blocking time. This gives a recovery window of roughly 30 to 60 minutes of lost training at typical step times, with overhead below 5%.
The more important consideration for multi-week runs is the checkpoint retention policy. You do not need 500 checkpoints from a completed run. You need the last 3 to 5 for immediate recovery, plus checkpoints at logarithmic intervals (1k, 2k, 5k, 10k steps) for training trajectory analysis and resume points. Automated deletion of intermediate checkpoints is the single highest-impact operational change most teams can make to their checkpoint strategy. We have seen teams running 3-week runs accumulate 2TB of checkpoints when the required durable storage for the same retention policy is under 200GB.
Synchronous vs Asynchronous Checkpoint Writes
Synchronous checkpointing blocks the forward pass until the full checkpoint write completes. For a 60GB checkpoint on a system that writes at 1.2 GB/s to disk, that is 50 seconds of GPU stall per checkpoint. On a 64-GPU cluster where idle GPU-seconds are expensive, this is the dominant cost of a naive checkpoint implementation.
Asynchronous checkpointing copies model state to CPU RAM (which is fast, typically 5 to 15 seconds for a 60GB model across a high-bandwidth CPU interconnect), then writes from CPU RAM to disk in the background while training continues. The GPU resumes before the disk write completes. The tradeoff: if the cluster fails while the async write is in progress, you may lose that checkpoint. In practice, the window of exposure is the disk write time (50 to 100 seconds for large models), which is small relative to the checkpointing interval.
PyTorch's torch.save() is synchronous by default. Asynchronous behavior requires either a separate background writer thread or a framework like Fabric or Lightning that abstracts this. The HuggingFace Trainer supports async checkpoint writes via the save_on_each_node parameter and non-blocking save paths in recent versions. For JAX, orbax's AsyncCheckpointer handles this cleanly.
We are not saying synchronous checkpointing is always wrong. For short runs or small models where checkpoint overhead is under 2%, the simplicity of synchronous writes is worth it. The complexity of async writers introduces edge cases in failure scenarios that need careful testing. The point is: on week-long runs with 30B+ parameter models, synchronous checkpointing to slow storage is a non-trivial compute cost and worth engineering around.
Checkpoint Storage Architecture: The Four Options
Where you write checkpoints determines write latency, storage cost, and the durability profile during the checkpoint window.
Local NVMe on training nodes. Fastest writes (saturates NVMe at 4 to 6 GB/s), zero network overhead. The downside is durability: if you lose the node, you lose the checkpoint. For multi-node training runs, local storage is only viable if checkpoint files are immediately replicated across nodes via a distributed write, which typically requires a custom wrapper around torch.save(). Most teams use this for fast ephemeral checkpoints that are promoted to shared storage every N checkpoints.
NFS-mounted shared storage. The universal fallback. Every cluster has NFS. The problem is NFS protocol overhead on writes: each write system call carries TCP round-trip latency, and the NFS locking protocol is not designed for the large sequential writes of checkpoint operations. On a 10GbE NFS mount, checkpoint write throughput for a 60GB checkpoint is commonly 500 to 800 MB/s, with wide variance. Also, NFS provides no atomicity guarantee on writes, so a crashed write leaves a partially written checkpoint file that looks valid but isn't.
S3-compatible object storage. The default for cloud training environments. Object stores are durable and cheap per TB, but the multipart upload API and the absence of POSIX rename atomicity create two problems. Multipart upload of a 60GB file requires assembling, uploading, and completing multiple 100MB to 1GB parts, which adds 20 to 40 seconds of overhead relative to a direct sequential write of the same data. And because object stores lack rename atomicity, checkpoint code that writes to a temp file and renames to final is broken silently on S3-compatible backends. torch.save() to S3 via s3fs or boto3 does not give you the same durability semantics as writing to a real filesystem, and several checkpoint corruption bugs in production ML systems trace back to this.
Distributed POSIX filesystem (e.g. Leil, Lustre, WekaIO). Gives you POSIX semantics on a shared namespace across your cluster: rename atomicity works, fsync is respected, file permissions are enforced. Sequential write throughput scales with the number of storage nodes. Checkpoint code written for local disk works without modification. The cost is infrastructure: you need dedicated storage nodes and the operational overhead of managing the filesystem. For training clusters running week-long runs where checkpoint integrity matters and write latency affects training throughput, this is the architecture that avoids the failure modes of NFS and object storage.
What Write Latency Actually Costs
Consider a 30B parameter run with a 3-week training duration, checkpointing every 1,000 steps at a 2.5-second step time. That is one checkpoint every 42 minutes, or roughly 1,000 checkpoints over 3 weeks. If synchronous checkpoint writes take 75 seconds on a slow NFS mount instead of 20 seconds on a POSIX distributed filesystem with fast SMR-native write paths, the difference per checkpoint is 55 seconds. Over 1,000 checkpoints, that is 55,000 seconds, or about 15 hours of additional GPU idle time.
At $4/GPU-hour for spot H100s on a 64-GPU cluster, 15 hours of idle time costs roughly $3,800 in wasted compute. The storage cost difference between a purpose-built sequential filesystem and a slower NFS mount over the same 3 weeks is typically $500 to $1,500. The compute savings exceed the storage infrastructure cost by a factor of 2 to 7x.
This calculation is sensitive to model size, step time, checkpoint frequency, and the specific storage systems being compared. But it illustrates why checkpoint storage architecture deserves more engineering attention than it typically receives. The storage bill is the visible cost. The GPU idle time is the invisible one.
Checkpoint Compression and its Tradeoffs
Model weights in bf16 or fp16 compress poorly. The numerical values in a trained model are not random, but they are high-entropy enough that lz4 compression typically achieves 5 to 15% size reduction at best. For a 60GB checkpoint, you save 3 to 9GB. The CPU compute cost of compression/decompression at a typical rate of 2 to 5 GB/s takes 15 to 30 seconds.
In most cases, checkpoint compression is not worth it on the checkpoint write path because the compression time exceeds the storage time saved. The exception is long-term archival: checkpoints stored for more than 2 weeks with low access probability benefit from compression because the space cost is ongoing and the decompression cost is paid at most once per checkpoint file.
For active training runs, skip compression on the write path. Apply it as a post-processing step on checkpoint files being promoted to long-term archival storage, where a background job can handle it without affecting training throughput.
Testing Your Checkpoint Recovery Path
The most important operational discipline for long training runs is testing the recovery path before you need it. Checkpoint file corruption is rare but not unheard of, and the worst time to discover your recovery procedure is broken is at hour 140 of a 168-hour run.
Before starting a week-long run, validate that you can load the most recent checkpoint and resume training with identical model state. For PyTorch runs, this means loading the checkpoint dict, instantiating the model, loading state_dict, and running 10 forward/backward passes to verify gradient flow is consistent with a non-resumed run. This takes 15 minutes and prevents the scenario where your checkpoints write successfully for 5 days and turn out to be unresumable due to a serialization incompatibility introduced by a package update.
Checkpoint strategy is an engineering discipline, not a configuration parameter. The default settings of your training framework were set to be reasonable for the median use case. Week-long runs at petabyte scale are not the median use case.