Multi-Host Cluster Guide

A cluster spans hosts via fdl.cluster.yml (deployment) or ClusterBuilder (programmatic). The orchestrator host fdl-cli runs on is the controller and is never a NCCL rank itself; every rank-carrying host lives under workers.

fdl.cluster.yml schema

cluster:
  controller:
    host: 192.168.122.1           # controller bind address
    port: 1337                    # the single controller port (default 1337)
    path: /opt/flodl              # controller's view of the shared project root
    # docker: cuda                # optional pre-flight build service
    # arch: precompiled/cu128     # optional libtorch variant for pre-flight build
    # join:                       # membership-window overrides (see below)
    #   min_rank_start: 2
    #   join_timeout: 300
    #   target_ranks: 4
    #   max_join_timeout: 600
    #   open_admission: false

  workers:
    - host: node-a                # worker identifier; default SSH target
      local_devices: [0]          # 1 device -> 1 rank
      nccl_socket_ifname: virbr0
      path: /opt/flodl
      arch: precompiled/cu128     # libtorch variant under <path>/libtorch/
      docker: cuda                # optional: training runs in this compose service

    - host: node-b
      ssh:                        # optional SSH sub-block
        target: node-b
        port: 2222
        user: ubuntu
        identity_file: /home/me/.ssh/cluster_key
        options:
          - ProxyJump=bastion
          - StrictHostKeyChecking=no
      # tunnel: true              # route training traffic through the SSH
      #                           # session (CPU ElChe modes only; see below)
      local_devices: all          # probed at dispatch via SSH+nvidia-smi
      nccl_socket_ifname: enp1s0
      path: /srv/flodl
      arch: builds/sm61-sm120     # different variant per worker is fine
      # data_path: /flodl/data    # dataset source root on THIS host.
      #                           # DECLARING IT CHANGES A RUN: the value
      #                           # reaches every rank here and supplies the
      #                           # training binary's data dir. Unset ships
      #                           # nothing (the binary keeps its default);
      #                           # the convention default /flodl/data then
      #                           # governs only the pre-flight checks.
      # env:                      # per-worker env for this host's rank
      #   NCCL_SOCKET_NTHREADS: "2"   # children (reserved keys refused:
      #                           # the visibility masks, FLODL_INTERNAL_*)

  # Cluster-scope env vars (apply to every rank child)
  # env:
  #   NCCL_DEBUG: INFO

Conventions:

• One process per rank; each worker owns one rank per visible CUDA device. Global ranks are assigned sequentially by worker order: worker 0 owns [0..N0), worker 1 owns [N0..N0+N1), etc.
• local_devices: all probes the host at dispatch time via SSH + nvidia-smi. Explicit lists carry their own count.
• nccl_socket_ifname: is required on every worker when the cluster spans multiple hosts.
• path: is the project checkout dir on this host (heterogeneous mounts are fine - /opt/flodl on one host, /srv/flodl on another).
• arch: is the libtorch variant subpath under <path>/libtorch/ on this host. For heterogeneous rigs, each worker can select a different variant (e.g. one host on precompiled/cu128, another on builds/sm61-sm120); the convention path stays stable while the variant differs per host.
• docker: (optional) names the compose service for training on this host. Per-host: mixed deployments (controller in Docker, worker bare-metal) are common.
• tunnel: (optional) routes this worker’s training traffic through its fan-out SSH session instead of a direct TCP connection - see below.
• gpu_ram_share: (optional, APU hosts only) the fraction of this host’s physical RAM its integrated GPU’s aperture claims, e.g. 0.5 (the library’s gpu_ram_share knob; discrete-GPU hosts ignore it, and above 1.0 is legal where the platform under-states the aperture). Host-hardware truth like data_path:: it rides the envelope and fills the training binary’s config when that left the knob unset - an explicit with_gpu_ram_share in code still wins. A cluster-scope gpu_ram_share: (sibling of the cluster env: block) sets a fleet default for the identical-APU-farm case; a host entry overrides it, and a walk-in’s own join.gpu_ram_share: overrides both, because each later road knows the box better.

Dial-in membership: the join window

Workers join a run; the controller admits them. At launch the controller opens a join window on its port; every worker - fan-out- managed and self-deployed alike - dials in with a hello (host name, GPU inventory, libtorch variant, dataset signature) and is assigned its global ranks in admission order (contiguous by construction). When the window closes, the world freezes: world_size is whatever actually joined, and all coordination infrastructure (ElChe schedule, heartbeats, rendezvous) is sized to that world.

One thing admission refuses outright: mixing GPU vendors under an NCCL data plane. NCCL and RCCL export the same symbols and the same 128-byte unique-id format, so nothing structural rejects a mixed cohort - it hangs at formation, after the window deadline was spent. The hello’s libtorch label already names the vendor, so a walk-in whose vendor disagrees with the cohort’s is rejected at the door, with both fixes named. The CPU ElChe modes (cpu_sync / cpu_cadence / cpu_async) genuinely work cross-vendor - each box compiles its own binary against its own libtorch, and the averaging plane is vendor-blind - so under those modes mixing is allowed, and it is one of the reasons the CPU plane exists.

The dataset signature is inert today, on every path. The hello carries a 32-byte dataset_sig and both verifiers are built and working (the membership ledger seeds from the first member and refuses a mismatch by name; the NCCL rendezvous does the same independently). Nothing computes one. The trainer passes all-zeros, and fdl join sends none, which is the same all-zeros by convention. Every member therefore presents the same value, they agree trivially, and no comparison happens: a cohort can read as coherent on dataset while nothing was compared.

What does bind a cohort: the vendor label above, run identity, NCCL version, and the model signature. The model signature is the one that catches the realistic version of this failure, since a stale tree or a wrong bin: usually changes the model too.

Treat “every box sees the same data” as your own precondition, not something admission enforces. The one place you can supply a real signature today is Cluster::rendezvous(sig), if you drive ranks manually via Ddp::wrap rather than through Trainer.

fdl @cluster <cmd> fan-out is sugar over this protocol: it starts one worker agent per host over SSH, and those agents dial back in like any worker would. The defaults make fan-out behave exactly like a fixed topology - quorum and early-close target both default to the configured capacity, so the window closes the instant every configured rank is in (zero added latency) and the run cannot start below full strength.

The big picture, end to end

One run, every moving part - a discovery controller holding a manual window, one worker walking in through a guardrailed sshd tunnel (fdl join --ssh), the operator firing the start. Fan-out agents and direct-dial walk-ins enter the exact same sequence at the “dial + hello” step; only the road differs:

%%{init: {"theme":"base","themeVariables":{"actorBkg":"#e8eaf6","actorBorder":"#5c6bc0","actorTextColor":"#1a237e","signalColor":"#5c6bc0","signalTextColor":"#263238","noteBkgColor":"#eceff1","noteBorderColor":"#90a4ae","noteTextColor":"#37474f","labelBoxBkg":"#eceff1","labelBoxBorderColor":"#90a4ae","labelTextColor":"#37474f","activationBkgColor":"#e8f5e9","activationBorderColor":"#66bb6a"}}}%%
sequenceDiagram
    autonumber
    participant OP as Operator
    participant CTRL as Controller<br/>(mux :1337)
    participant SSHD as join sshd<br/>(guardrailed)
    participant FJ as fdl join<br/>(worker box)
    participant AG as Agent<br/>(training binary)
    participant RR as Relay + Ranks

    OP->>CTRL: fdl @cluster-x train …
    activate CTRL
    Note over CTRL: join window opens<br/>(status: waiting)
    FJ->>FJ: prepare: GPU gate, source root,<br/>writable stage dirs, model-sig probe
    FJ->>SSHD: ssh -L ⟶ forward to 127.0.0.1:1337
    FJ->>AG: exec binary in agent role<br/>(spec in env, never argv)
    AG->>CTRL: dial + hello (HMAC: token / salt)
    CTRL-->>AG: Accept — ranks assigned in admission order
    Note over CTRL: quorum met<br/>(status: staging — held)
    OP->>CTRL: fdl status  ⟶  "roster startable"
    OP->>CTRL: fdl start (POST /start, authenticated)
    CTRL-->>AG: WorldFormed — envelope + relay spec
    Note over AG: stamps in what only THIS box knows:<br/>the join-verified address, its source root
    AG->>RR: spawn relay + one rank per GPU
    RR->>CTRL: relay dials in — training runs
    RR-->>AG: children exit
    AG-->>CTRL: RankExited reports, then link EOF
    Note over CTRL: every host link closed<br/>= run complete
    deactivate CTRL

Three properties worth reading off the picture:

• The join connection never closes. The same TCP stream carries the hello, the admission reply, the formed-world artifacts, per-rank exit reports upstream, and abort orders downstream - and its EOF is the host’s exit event. A walk-in host is nobody’s child process, so that link IS how the controller supervises it: the run is over when every host link has closed, and a failing run sends Abort down the links instead of leaving agents to guess.
• The agent trusts its own road, not the controller’s map. The formed-world artifacts name a controller address, but the controller authors it from its view of the topology - wrong on a host whose tunnel sits on an ephemeral local port, wrong behind NAT. The agent rewrites that address with the one its own join just provably used, so the relay dials a road that is known to work.
• Ranks never learn any of this exists. Everything downstream of WorldFormed is byte-identical to the fixed-topology era; the training code cannot tell a walk-in world from an enumerated one.

The window itself is a small state machine. auto runs entirely on the clock; manual inserts an operator-held staging phase between quorum and formation; hybrid is both (the clock still closes, the operator may fire earlier):

stateDiagram-v2
    [*] --> waiting: window opens —<br/>joins accumulate
    waiting --> forming: auto/hybrid —<br/>target or expiry, quorum met
    waiting --> staging: manual/hybrid —<br/>quorum met
    staging --> forming: fdl start
    staging --> forming: hybrid —<br/>target or expiry
    waiting --> failed: hard cap,<br/>quorum never met
    staging --> failed: hard cap, start<br/>never fired (manual)
    forming --> training: artifacts shipped,<br/>relays dialed in
    training --> done: every host<br/>link closed
    training --> failed: coordinator lost /<br/>cohort below tolerance
    note right of staging
        the hold: walk-ins are still
        admitted, and window expiry
        holds too (manual) — only
        fdl start or the hard cap
        ends it
    end note

    classDef good fill:#e8f5e9,stroke:#66bb6a,color:#1b5e20
    classDef cost fill:#faf0e6,stroke:#c9924f,color:#8a5320
    class done good
    class failed cost

Every state here is what fdl status prints as the lifecycle phase, live from before the window opens to the terminal verdict.

Override via controller.join: to allow degraded starts or to hold the window open for extra dial-in workers:

Admission is authenticated by the join frames’ HMAC key: fan-out agents receive the per-run session salt through their SSH session, so a peer without it cannot join. A loopback bind (every remote worker tunneled) is open by construction - the only path to the port is through sshd, so reachability itself is the authentication, and the salt is handed out in the accept reply. open_admission: true extends that hand-out to a network bind: any peer that can reach the port can then join (and therefore influence) the run, which is why flodl warns loudly - sound only on a fully trusted segment.

A self-deployed worker needs nothing but the controller address: fdl join (below) starts one on any GPU host. Under the hood it is a process started with FLODL_INTERNAL_AGENT_JSON set to the hex-encoded spec {"host": "...", "controller_host": "...", "controller_port": 1337} (see AgentSpec in the API docs) - it resolves its own GPUs, joins, receives the formed-world artifacts, and spawns its relay and rank children; the training code is byte-identical to the fan-out path. Pair it with target_ranks above the configured capacity (or a bare-bones one-host config) so the window waits for it.

discovery: true takes that shape to its limit: no roster at all. The controller opens the window from its bind address plus the join credential (token, or sshd reachability under tunnel_only), holds it for join_timeout seconds, and the world is whoever walked in - the cloud shape, where worker addresses do not exist before the VMs boot. Fan-out and discovery compose: an enumerated rig fans out as usual while cloud legs self-register into the same window.

The staging hold and fdl start

With start: manual (or hybrid) the window becomes an operator surface: once quorum is met the run shows as staging in fdl status - the roster is startable but held - and fdl start fires the topology freeze. Manual is the scavenged-credit shape (“launch instances until the money runs out, start when the roster looks full enough”): the window never closes on its own, only the hard cap bounds the hold, so pair it with a generous max_join_timeout. Firing follows the same trust model as joining: on the controller host (or through the sshd tunnel) no credential is needed; from anywhere else pass --token with the run’s join.token. Refusals name their reason - auto mode, quorum not met, window already closed - and arming below quorum is refused rather than queued, so the operator always knows what they started.

Walking in: fdl join

The worker-side command for all of the above - it dials a window, offers the box’s GPUs, and runs your training binary in agent role:

# Direct dial (LAN / trusted segment), authenticated by the run token:
fdl join 10.0.0.1:1337 --token <hex> --bin target/release/train -- --model resnet

# Through a guardrailed sshd on the controller box (reachability =
# authentication; the controller binds loopback under `tunnel_only`):
fdl join --ssh [email protected] --bin target/release/train

--ssh [user@]host[:port] brings up a local ssh -L forward of the controller port (fresh per attempt, ExitOnForwardFailure, never a password prompt) and dials through it; the positional address is then the controller as seen FROM the ssh host, defaulting to 127.0.0.1:1337 - the sshd-on-the-controller-box convention. The forward’s local port is ephemeral, which is safe because of the address rewrite in the walkthrough above: the agent stamps the formed-world artifacts with the address its own join used, so the relay dials through the same forward. Arguments after -- go to the binary verbatim and must match the run: rank children re-enter the binary with them. --devices 0,1 scopes the offer (default: every GPU on the box), --host names the worker in the roster, and when fdl runs inside a project the active libtorch’s lib/ rides LD_LIBRARY_PATH automatically.

Every flag defaults from a top-level join: block in fdl.yml, so a golden image bakes the whole recipe and boots into bare fdl join:

join:
  ssh:                        # full ssh shape: target / port / user /
    target: ctrl.example.com  #   identity_file / options
    user: flodl-join
    identity_file: /etc/flodl/join_key
  bin: target/release/train
  args: ["--model", "resnet"]
  persist: true               # re-dial with backoff when the agent
                              #   exits — the systemd loop

With persist the command never gives up: no window open yet, run finished, controller rebooted - the agent exits, fdl join backs off (5s doubling to 60s) and dials again, so a fleet of workers can sit ready before the operator ever launches, walk into the staging hold, and be re-armed for the next run the moment one ends.

Preparing the box, before the dial

Admission starts a window deadline, so a box acquires what it needs first - and re-acquires it on every attempt, which turns persist into a provisioning loop: new source, next re-dial, no reprovisioning.

join:
  data_path: /flodl/data      # the LOCAL root this box's ranks read
  data_source:                # optional: mount it when it is not there
    sshfs://[email protected]:/flodl/data

data_path plays the same role for a walk-in that cluster.workers[].data_path plays for a fan-out host - and travels a different road to get there, because the controller never configured this host and has nothing to say about where its data lives. The box resolves the path itself, and its agent writes it into the host block of the formed-world envelope on arrival: the same rewrite that stamps the join-verified controller address, for the same reason. Downstream, a rank reads it through the same LocalCluster::data_path() a fan-out rank uses, so the training binary needs no data flag either way.

The mount goes up read-only, which is not a precaution but the enforcement of an invariant: a rank reads the source root and never writes it (anything missing is acquired into ~/.flodl/data, the node-local cache). A root your provisioning already mounts needs no scheme at all - name its path in data_path and leave data_source unset; declare neither and nothing is checked or shipped.

The mount authenticates with the ssh: block’s own key, so a guardrailed join sshd has to permit sftp for it to come up at all - see the recipe below, which is where that costs a decision.

join.gpu_ram_share: is the same class of fact for an APU box: the fraction of its RAM the integrated GPU claims, which only this box knows. It rides the same envelope rewrite as data_path, overriding the cluster-scope default the controller may have declared for the whole farm (cluster.gpu_ram_share: in the controller’s overlay - the one-line answer for a fleet of identical APU boxes). Discrete-GPU boxes need neither.

Compiling on the node

bin: names a binary to run as given. Its alternative builds one here, which is how the ABI stops being a matter of manifest discipline: the box links against the exact libtorch it holds.

join:
  libtorch: auto                # acquire into ~/.flodl/libtorch/ and
                                #   activate; `auto` reads THIS box's
                                #   devices, so one image serves both vendors
  source:
    from: rsync://[email protected]:/home/op/my-train
    cwd: ddp-bench              # project dir inside the tree; governs
                                #   the build AND the run
    build: cargo build --release --features "$FDL_GPU_FEATURE" --bin ddp-bench
    bin: target/release/ddp-bench

Publishing a run: the controller is the authority

fdl publish resolves a source spec into a served directory, builds it once as a gate, and writes a run manifest at the tree’s root:

fdl publish file:///home/op/my-train --bin target/release/my-train \
            -- --model resnet --epochs 20

A worker pointed at that tree then needs nothing but the pointer, because cwd / build / bin / args come from the manifest:

join:
  source:
    from: rsync://[email protected]:/home/op/.flodl/run/tree   # plain key
    # from: rsync://[email protected]:/tree               # rrsync'd key

The two spellings are not interchangeable: behind a guardrailed key’s command="rrsync -ro <served>" the path is re-rooted under the served directory, so the worker asks for /tree — the guardrail recipe below has the pairing, and fdl publish prints both.

So chaining runs on a standing fleet is one command: publish again, and every box picks the new run up on its next dial with nothing to edit anywhere. args is why this is a correctness matter rather than an ergonomic one — they must match the run, since rank children re-enter the binary with them, so a fleet carrying its own copy trains the next run with the previous one’s hyperparameters. The manifest’s PRESENCE is the commit point: publish clears it before touching the tree and writes it only after the build passes, so a box dialing mid-publish finds no manifest and waits rather than training something unvalidated. That wait is a transient failure, and a failed gate publishes nothing at all. And every publish mints a fresh run identity (run: in the manifest) that rides each worker’s join hello: the window refuses a cohort whose members hold different ids, so two boxes that fetched across a publish boundary can never form one world — the stale side just picks the new run up on its next dial.

The gate is one build, not N: every worker still compiles its own, since a controller producing binaries for each worker variant is the build matrix this design deleted. It needs no GPU libtorch either, so a coordinator-only box pays rustup plus fdl libtorch download --cpu.

Three transports, and the scheme names the tool rather than a wire protocol, the same convention data_source uses: file:///abs/path for a directory already on the box, rsync://[user@]host[:port]:/abs/path for a working tree over ssh, git+https://host/owner/repo#<ref> (or git+ssh://) for a pinned checkout. rsync is the one that carries UNCOMMITTED work, which is what a training crate living in no repo at all needs; git+ insists on a ref because a default branch floats, and a floating ref is not a pin. Over ssh the credentials come from the ssh: block, so that key’s forced command has to permit rsync (`rrsync -ro