Eight models, six distributed modes, three solo baselines, 63 runs.
Three mismatched GPUs across two hosts, faster than the fast one alone.
Hardware: RTX 5060 Ti (sm_120, 16 GB) on the controller host,
plus two GTX 1060 (sm_61, 6 GB) inside a VM on the same box — a real
multi-host cluster coordinated over TCP, with the slowest card behind a
PCIe x1 riser. Up to ~12x delivered-throughput gap between fastest and
slowest rank. No pre-built libtorch covers both architectures —
compiled from source with fdl libtorch build.
200 epochs, every DDP mode beats the published reference.
Published: 91.25% accuracy
(He et al. 2015, Table 6).
This is the FlowBuilder Graph version — the same ResNet-20 architecture
expressed as a computation graph. One Trainer::run() call turns it
into a 3-rank multi-host training run. The manual-Module ResNet-20
confirms parity below.
| Mode | Eval | vs Paper | Wall time | Speedup |
|---|---|---|---|---|
| solo-0 (5060 Ti) | 91.46% | +0.21pp | 696s | 1.0x |
| cpu-async-diloco | 92.29% | +1.04pp | 500s | 1.39x |
| cpu-async | 91.56% | +0.31pp | 498s | 1.40x |
| cpu-cadence | 91.79% | +0.54pp | 503s | 1.38x |
| nccl-cadence | 91.78% | +0.53pp | 512s | 1.36x |
| nccl-sync | 91.60% | +0.35pp | 1137s | 0.61x |
| cpu-sync | 91.59% | +0.34pp | 1607s | 0.43x |
Sweep of 2026-07-29. Eval = held-out test accuracy (10K samples).
Speedup relative to solo-0 (which pays per-epoch eval). Every mode
beats the paper, and the async family now leads on wall time:
the averaged-model writeback streams through reused staging instead of
materializing model-sized copies. The 1060 solo baselines are deferred
this pass (hours each behind the x1 riser; the earlier sweep measured
~0.2x). The *-sync modes reduce once per slow-rank step,
so they pay the rendezvous thousands of times — correctness
baselines, not speed contenders.
The most interesting row above is not the wall time.
Solo-0 drove its training loss all the way down to 0.013, and
lands at 91.46% eval. cpu-async-diloco stops at 0.026,
twice the training loss, and lands at 92.29%: it memorized measurably
less and generalized measurably more, +0.83pp with 28% less wall time.
The mechanism: each replica only sees its own data partition, so
whatever one replica memorizes, the others did not. The
consensus average attenuates replica-private memorization every round
while shared structure survives, and DiLoCo's outer Nesterov momentum
smooths the consensus trajectory on top. The distributed run behaves
like a solo run with built-in regularization, no early-stopping
oracle required, and faster.
ResNet-20 without the Graph builder. Same architecture,
same convergence — confirming both DDP entry points work.
| Mode | Eval | vs Paper | Wall time |
|---|---|---|---|
| solo-0 (5060 Ti) | 91.41% | +0.16pp | 685s |
| nccl-cadence | 92.08% | +0.83pp | 510s |
Trainer::builder() path, no Graph required. The eager and
graph engines land within 1.6% on wall time (solo-0: 685s eager vs 696s
graph; nccl-cadence: 510s vs 512s) and within single-seed run noise on
eval, so the full mode suite runs on the graph flavor and the eager
model keeps these two parity cells.
Every mode runs the same ElChe-scheduled engine with work-weighted averaging.
The choice is where each mode sits on the throughput / freshness curve.
Equal data split; the reduce fires as soon as every alive rank has made at least one step. Degenerates to classic DDP on homogeneous rigs; on mixed hardware the fast GPU runs several steps per reduce inside its equal share, then idles. Correctness baseline.
ResNet-20 Graph: 91.60%, 1137s (NCCL) / 91.59%, 1607s (CPU)
Data dispatched proportionally to measured throughput; the reduce fires once every rank completes its planned window. The slow card anchors the interval, the fast card contributes proportionally more.
ResNet-20 Graph: 91.78%, 512s (NCCL) / 91.79%, 503s (CPU)
Cadence plus bounded overshoot and barrier-free application, with EASGD elastic blending on by default: the averaged model streams back through reused staging while ranks keep training, taking the reduce barrier off the wall clock. Fastest wall time on this rig; add the DiLoCo outer optimizer for the best eval of every mode tested.
ResNet-20 Graph: 91.56%, 498s / 92.29%, 500s with DiLoCo
Small models with 5–50 epochs. DDP converges correctly across the board,
though short runs penalize El Che calibration time.
| Model | Published | solo-0 | Best DDP | Mode | Fastest DDP | Mode |
|---|---|---|---|---|---|---|
| LeNet-5 (MNIST) | ~99% | 98.70% | 98.94% | nccl-sync | 5.3s | cpu-cadence |
| MLP (MNIST) | ~97% | 96.28% | 97.42% | cpu-async-diloco | 4.4s | cpu-async |
| Logistic (MNIST) | ~92% | 92.43% | 92.30% | cpu-async-diloco | 4.4s | cpu-async |
| Conv AE (MNIST) | — | 0.0006 | 0.0007 | nccl-sync | — | — |
| GPT-nano (Shakespeare) | ~1.5–2.0 | 1.7543 | 1.7773 | cpu-async-diloco | 48.6s | cpu-async-diloco |
| Char-RNN (Shakespeare) | ~1.5 | 1.7457 | 1.5604 | nccl-sync | 29.4s | nccl-cadence |
Eval is test-set accuracy (classification) or validation loss (language models, lower is better). Short runs are correctness confirmation, not peak performance — gradient averaging slows early convergence and El Che needs calibration time. The 200-epoch flagship above is where DDP demonstrates its advantage.
Reproducible. Ships with fdl ddp-bench.
Nine modes: three solo baselines (one per GPU) and six cluster modes — nccl/cpu × sync/cadence, cpu-async, and cpu-async + DiLoCo outer optimizer. Every cluster cell runs as a real 3-rank multi-host cluster (process-per-rank, TCP control plane, one rank host being a VM).
Eval protocol: Solo modes evaluate against the held-out test set every epoch. Cluster modes run a final evaluation after training, loading the consensus parameters into a fresh model. Both paths use the same eval function on the same test data.
GPU utilization: every physical device is covered,
host-qualified. The controller box is sampled densely at 100ms; every
rank additionally reports its own GPU samples over the metrics wire at
reduce-window cadence, so remote devices carry an honest sparse duty
cycle (marked ~ in the report). Records were collected
live for the whole of every run and archived as each cell's
self-contained portal at teardown; no live server ran and per-window
report emission was off, so published wall times carry zero monitoring
overhead.
Reproducibility: Seed 42, deterministic data loading, identical hyperparameters across all modes. Every run produces a timeline (JSON/CSV/HTML) with GPU traces, sync events, and idle analysis.
fdl @cluster ddp-bench --model resnet-graph --mode cpu-cadence --epochs 200 fdl ddp-bench --report --charts resnet-graph # report + SVG charts
The full report — per-epoch trajectories, per-rank schedules, sync cadence charts, idle analysis — lives in docs/ddp-benchmark.md.