DDP Troubleshooting

Start with fdl probe

fdl probe (single host) or fdl @cluster probe (cluster) is the first stop for any “it should work, why doesn’t it” question. It surfaces:

• Missing libtorch variant / wrong arch for the local GPUs.
• Missing or mismatched libnccl across hosts.
• Missing nccl_socket_ifname on multi-host workers.
• Stale legacy schema keys in fdl.cluster.yml.
• Shared-data path resolution failures.
• Dashboard port already in use.

NCCL init failure

ncclCommInitAll failed typically means NCCL can’t establish peer-to-peer between devices.

nvidia-smi topo -m            # check device connectivity
fdl probe                     # check NCCL availability + libtorch wiring

Falls back to shared memory transport if peer-to-peer is unavailable. Or switch to a Cpu* mode in ElCheConfig to bypass NCCL entirely.

NCCL version skew across hosts

If one host has libtorch shipping NCCL 2.27 and another has 2.26, the handshake fails. Build a matching libnccl on the easier side and LD_PRELOAD it via the worker’s env: block:

fdl nccl build              # auto-detects target version + archs

Parameter count mismatch

GpuWorker rank N: model has M params but config has K. The model factory produced a model with a different parameter count than the initial model used to extract starting parameters. Make sure model_factory(dev) produces an identical architecture for every device.

CUDA context corruption

CUBLAS_STATUS_EXECUTION_FAILED or SIGABRT after NCCL init usually means ncclCommInitRank was called from multiple threads on heterogeneous GPUs. The framework uses the init-on-main + split() pattern everywhere, but if you’re driving NcclComms manually, make sure you follow the same pattern.

Also covered by the “no CUDA before Trainer::run” invariant - any CUDA tensor created in main() before the launcher trampoline poisons spawned children’s contexts.

OOM on smaller GPU

Any anchor-based mode (NcclCadence, CpuAsync, CpuCadence) routes through ElChe, which assigns proportionally fewer batches to the slower/smaller GPU. The DataLoader’s per-device backend selection also helps: the large GPU goes resident while the small GPU streams.

.elche(ElCheConfig::nccl_cadence().max_anchor(50))   // or any anchor-based preset

CPU averaging timeout

The CPU averaging path now waits indefinitely for survivors and lets the elastic-membership machinery handle the dead-rank decision. If you need a hard time bound (e.g. CI gating), max_failure + ShutdownWithSave is the right knob - it triggers a clean checkpoint exit rather than hanging.

Cluster progressive hangs

If fdl @cluster runs hang several epochs in, the cause is usually:

1. Stale child processes from a previous aborted run holding GPU memory or rendezvous ports. fdl @cluster cleans these up pre-spawn, but a kill -9 on the launcher bypasses cleanup.
2. Shared-mount staleness when the project mount is NFS or virtiofs and the controller and a worker see different file states. fdl probe flags mount-state divergence.