Heterogeneous & Multi-Host DDP

When the rigs get interesting - mixed GPU generations, mixed hosts, mixed libtorch variants - the same Trainer::builder shape from Multi-GPU Training keeps working. This tutorial covers the knobs that earn their keep on real heterogeneous deployments.

Prerequisites: Multi-GPU Training. The universal Trainer::builder shape is assumed.

Time: ~25 minutes.

Canonical reference: DDP Reference.

Runnable example: auto_promote is the binary the cluster path promotes; nothing in it changes between single-host and multi-host.

ElChe - the cadence balancer

Every cadence mode (*Cadence, *Async) routes through ElChe. NcclSync and CpuSync are the tightest-cadence case: an equal data split with the reduce firing as soon as every alive rank has made at least one step since the last one — classic per-batch DDP on a homogeneous rig, but NOT per-batch lockstep on a heterogeneous one (the fast GPU runs several work-weighted steps per reduce; see What “sync” means). ElChe’s job is to keep AllReduce overhead bounded while respecting weight-space divergence.

Phase machine

ElChe progresses through four phases as calibrations accumulate:

The phase gate prevents the multiplicative overhead-tune scale from compounding sparse early-reading noise (the historical “anchor jumps from 10 to 22 on the first measurement” bug).

Weighted gradient averaging

When ranks process different batch counts in a window, each replica’s gradient contributes proportionally to its batch count:

weight[rank]  = count[rank] / sum(counts)
grad_avg      = sum(weight[rank] * grad[rank])

This produces the mathematically correct mean gradient regardless of per-device batch counts. No accuracy degradation from uneven splits.

Anchor auto-tune

After each averaging cycle (gated to Phase::Stable+):

overhead = sync_ms / max(compute_ms across ranks)

The proposal is proposed, not committed - the convergence guard’s verdict decides whether it lands (see below). The anchor is then clamped to [min_anchor, max_anchor].

Convergence guards - the safety belt

overhead_target alone would happily grow the anchor on any stable model, eventually starving the gradients. The convergence guard monitors weight-space divergence between replicas and pulls back.

Guard authority over overhead_target

ElChe’s report_timing proposes an overhead-tune anchor change but doesn’t apply it. The guard’s verdict commits:

So SuppressGrowth doesn’t just not relax up - it actively vetoes the overhead-tune growth before it lands. The convergence guard is authoritative over overhead_target by construction. See DDP Reference: Guard authority over overhead_target.

LR-aware meta-controller

Above ElChe sits the LR-aware meta-controller (on by default - .meta_controller(true)). It watches LR trajectory + anchor trend + guard verdicts in a rolling window and reactively nudges the anchor down on sharp LR drops or sustained divergence patterns. Reports is_settled() once the metric stops moving - useful as an early-stop signal.

Opt out for instrumentation:

let elche = ElCheConfig::nccl_cadence()
    .meta_controller(false);     // unconditioned trajectory

EASGD elastic averaging (CpuAsync)

On the CpuAsync path, EASGD-style elastic blending smooths divergence in long async runs:

local_t1   = (1 - α) * local_t0  +  α * center_t0
center_t1  = (1 - α) * center_t0 +  α * mean(local_t0)

Each rank’s local params blend toward the center; the center blends toward the mean of locals. α controls the blend rate (0 < α ≤ 1.0, typical 0.4-0.8). Honored on CpuAsync only; ignored elsewhere.

.elche(ElCheConfig::cpu_async().easgd_alpha(0.6))

A/B testing modes - the recipe

Five modes via ElCheMode. Each switch is one line:

let base = || Trainer::builder(model_factory.clone(), optim_factory.clone(), train_step)
    .dataset(dataset.clone())
    .batch_size(64)
    .num_epochs(5)              // just enough to see the trend
    .max_grad_norm(5.0);

let a = base().elche(ElCheConfig::cpu_async()).run()?.join()?;     // best-in-class candidate
let b = base().elche(ElCheConfig::nccl_cadence()).run()?.join()?;  // default; recommended NCCL pick
let c = base().elche(ElCheConfig::nccl_sync()).run()?.join()?;     // tightest-cadence baseline

Suggested order (refined from the ddp-bench published numbers):

Compare on: loss at epoch N, wall time per epoch, and loss per wall-second - the last is usually the decider. A slightly higher loss in half the time often wins.

CpuSync and CpuCadence exist for A/B against the NCCL variants when peer-access is unavailable; they’re rarely faster or more accurate than NCCL for typical workloads. CpuAsync is where the CPU backend shines.

The ddp-bench suite drives every mode through the same harness with the same train_step closure. See ddp-bench for the canonical worked example and the published convergence numbers.

NcclAsync used to exist as a sixth mode (NCCL + per-rank cross-epoch dispatch). It was dropped - measured benefit over NcclCadence was within noise on every tested rig, and the in-place AllReduce writeback raced with autograd on heterogeneous Pascal+Blackwell setups. CpuAsync is the real async mode: averaging is decoupled from the GPU pipeline through a separate channel.

Heterogeneous-rig real-world example

A two-host cluster with mixed-generation GPUs:

• node-a (Blackwell): 2× RTX 5060 Ti (sm_120, 16 GB each), libtorch precompiled/cu128
• node-b (Pascal VM via virtiofs): 2× GTX 1060 (sm_61, 6 GB each), libtorch builds/sm61-sm120 (source-built, multi-arch)

NCCL handshake across mixed major.minor versions sometimes fails (node-a may ship NCCL 2.27, node-b may have 2.26 from libtorch). The fix: build a matching libnccl on the easier side and LD_PRELOAD it into libtorch:

fdl nccl build              # auto-detects target version + local archs

Then wire it into the worker’s env: block in fdl.cluster.yml:

workers:
  - host: node-b
    local_devices: all
    nccl_socket_ifname: enp1s0
    path: /srv/flodl
    arch: builds/sm61-sm120
    env:
      LD_PRELOAD: /srv/flodl/libtorch/nccl/builds/v2.27.5-sm61/lib/libnccl.so.2

fdl probe flags the version skew before launch:

fdl @cluster probe           # SSHes each worker; aggregates GPU + libtorch + NCCL inventory

Cluster topology - fdl.cluster.yml

The structured schema with controller: and workers[]: blocks. See DDP Reference: Multi-host clusters for the full field listing.

Key conventions:

• One process per rank; each worker owns one rank per visible CUDA device.
• Global ranks are assigned sequentially by worker order: worker 0 owns ranks [0..N0), worker 1 owns [N0..N0+N1), etc.
• local_devices: all probes the host at dispatch time via SSH + nvidia-smi. Explicit lists carry their own count.
• nccl_socket_ifname: is required on every worker when the cluster spans multiple hosts.
• arch: selects the libtorch variant per host - heterogeneous rigs run different variants without changing the convention path (<path>/libtorch/<arch>/).
• docker: (optional) names the compose service for training on this host. Mixed deployments (controller in Docker, worker bare-metal) are common.

Launch via the env overlay:

fdl @cluster train           # = fdl --env cluster train ; SSHes each worker
fdl @cluster probe           # readiness gate before launch

Programmatic clusters - ClusterBuilder

For tests, embedded launchers, or any binary that wants to launch without a yml on disk:

use flodl::ClusterBuilder;

let cluster = ClusterBuilder::new()
    .controller("192.168.122.1")
        .port(1337)
        .path("/opt/flodl")
    .done()
    .host("node-a")
        .ranks([0, 1])
        .devices([0, 1])
        .nccl_socket_ifname("enp1s0")
        .path("/opt/flodl")
        .arch("precompiled/cu128")
    .done()
    .host("node-b")
        .ranks([2, 3])
        .all_devices()
        .nccl_socket_ifname("enp1s0")
        .path("/srv/flodl")
        .arch("builds/sm61-sm120")
        .ssh_port(2222)
        .ssh_identity_file("/keys/cluster")
    .done()
    .build()?;

let cfg = TrainerConfig::new(dataset)
    .batch_size(64)
    .num_epochs(50)
    .elche(ElCheConfig::cpu_async().easgd_alpha(0.6))
    .cluster(cluster);

Trainer::run(model_factory, optim_factory, train_step, cfg)?.join()?;

ClusterBuilder::all_local_gpus() is the single-host shortcut - synthesizes the same topology auto-promote uses on a multi-GPU host.

Elastic membership

Ranks can die without aborting the run — the survivors carry on with the dead rank’s work redistributed (membership only shrinks; a dead or new rank cannot join a formed world, scale-up is not yet implemented):

• Heartbeat miss → coordinator transitions the rank to Dead and renormalizes partition_ratios across survivors.
• Lone NCCL survivor short-circuits and exits (no dead-quorum AllReduce wait).
• max_failure threshold triggers a clean ShutdownWithSave - coordinator drives a final checkpoint through whichever survivor has the freshest state.
• NCCL rendezvous-timeout retry rebuilds the comm on the largest contiguous survivor subset.

Tune via:

TrainerConfig::new(dataset)
    .batch_size(64)
    .num_epochs(100)
    .save_path("ckpts/run")
    .checkpoint_every(5);
// max_failure comes from cluster-coordinator config (cluster.yml or
// programmatic ClusterCoordinatorConfig).

The save bundle is <stem>.fdl + <stem>.meta.json (carries ElCheState - phase, calibration trajectory, ring buffer, guard history). Resume:

TrainerConfig::new(dataset)
    .batch_size(64)
    .num_epochs(100)
    .save_path("ckpts/run43")
    .resume_from("ckpts/run42")     // loads ckpts/run42.{fdl,meta.json}
    .checkpoint_every(5);

A resumed run inherits ElChe’s calibration trajectory - no warmup re-run.

EpochCallbackPolicy::Fastest - free-compute callbacks

By default, per-epoch callbacks (checkpoint_fn, epoch_fn, eval_fn) fire on the fastest rank (lowest smoothed_ms_per_batch). The reasoning: on heterogeneous rigs the fastest rank has the most idle time at the sync barrier, so eval / save runs as free compute. Sticky within a run; re-resolves only on rank death.

Pin to a specific global rank with EpochCallbackPolicy::Rank(n) when the research convention demands it. n is the global rank (0..world_size cluster-wide), assigned sequentially by worker order in the cluster topology.

TrainerConfig::new(dataset)
    .epoch_callback_policy(EpochCallbackPolicy::Rank(0));

Quick reference

ElCheConfig presets

Knob cheat-sheet

Cluster launch

fdl probe                  # single-host readiness
fdl @cluster probe          # multi-host readiness (SSHes each worker)
fdl @cluster <cmd>          # fan out
FDL_ENV=cluster fdl <cmd>  # equivalent
fdl nccl build             # build libnccl for LD_PRELOAD bridge (heterogeneous NCCL versions)