Cluster and Hardware Commands

fdl probe

Cluster readiness audit. Single-host (default) probes the local box; in cluster context (fdl @cluster probe or FDL_ENV=cluster fdl probe) SSHes every host in fdl.cluster.yml and aggregates the report. Use it as a CI gate before launching multi-hour training runs.

fdl probe                       # local host: GPU + libtorch + NCCL + shared-data
fdl @cluster probe               # multi-host: SSHes each worker, aggregates
fdl probe --json                # machine-readable for CI gating
fdl @cluster probe --json        # cluster JSON aggregate
fdl probe --skip-mount          # skip shared-data-mount check on single-host setups
fdl probe --data-path /flodl/data        # override the shared-data path
fdl probe --libtorch-path /opt/libtorch  # override the libtorch directory
fdl probe --docker cuda         # NCCL is provided by a Docker image (compose service)

Exit code: 0 when every checked component is green, 1 when any issue was surfaced. The green path is silent enough to use as a post-deploy smoke test.

What it checks:

Output splits results into warnings (informational; do not block) and errors (block dispatch). fdl probe is a manual readiness gate — run it yourself before a cluster run; it is not invoked implicitly by fan-out. (The automatic pre-flight step fdl @cluster <cmd> performs is the per-host binary build, skippable with --no-prebuild.)

fdl status

Live status of a running cluster: lifecycle phase (waiting / staging / forming / training / done / failed), who has joined with what hardware, the join-window countdowns while it is still open, and — on start: manual / hybrid runs — the start switch’s state (a staging roster renders “roster startable, fire with fdl start“). The controller serves the state as state.json over plain HTTP on its training port, so no extra port or config is involved — and curl works where fdl isn’t installed.

fdl @cluster status              # controller from the overlay's cluster.yml
fdl status --addr host[:port]    # explicit controller (default port 1337)
fdl @cluster status --json       # raw state.json for scripts
curl http://<controller>:1337/state.json   # same truth, no fdl

Address resolution: --addr wins; otherwise the active env’s cluster.controller (with a loopback retry, so all-tunneled runs are found when running on the controller box); otherwise the convention default 127.0.0.1:1337 (single-host auto-promoted runs), noted on stderr.

Exit code: 0 when the state was fetched and printed, 1 when no endpoint answered. The endpoint lives exactly as long as the launcher process — connection-refused after a run ends is the expected “no run listening” signal, not a fault.

fdl start

Fire the operator start switch of a staging cluster run. A join window opened with controller.join.start: manual (or hybrid) holds the roster once quorum is met instead of forming on the clock — the scavenged-credit shape: launch instances until the money runs out, watch fdl status, start when the roster looks full enough.

fdl @cluster start               # controller from the overlay's cluster.yml
fdl start --addr host[:port]     # explicit controller
fdl start --token <hex>          # non-loopback fire: the run's join token

Trust mirrors join admission: fired from the controller box (or through the sshd tunnel) the loopback peer address is the credential and no token is needed; from anywhere else --token must match the run’s controller.join.token. Address resolution is the same as fdl status.

Refusals name their reason and are never queued — auto mode (no switch to fire), quorum not met (with counts), window already closed, bad token. Exit code: 0 when the start was armed (the world forms at the next poll — watch fdl status), 1 otherwise.

fdl publish

Put a training run where a fleet can pull it. The controller side of compiling on the node: it resolves a source spec into a served directory, builds it once as a gate, and writes the run manifest workers read.

# The operator's own crate, wherever it lives, with the run's arguments:
fdl publish file:///home/op/my-train --bin target/release/my-train \
            -- --model resnet --epochs 20

# A pinned checkout, fetched by the controller itself (a remote
# controller has no copy of your tree):
fdl publish git+https://github.com/me/train#v3 --bin target/release/train

# This repo's own vehicle: a workspace-excluded crate, so `--cwd`:
fdl publish file:///home/op/rdl --cwd ddp-bench \
            --build 'cargo build --release --features "$FDL_GPU_FEATURE" --bin ddp-bench' \
            --bin target/release/ddp-bench -- --model olmo-graph --epochs 1

The tree lands in <served>/tree (default ~/.flodl/run/tree), which is what a worker’s --source points at, and the manifest sits at its root so one fetch carries both.

Flags: --bin (required — the artifact relative to the project dir; a workspace member’s build lands in the WORKSPACE target/, so no rule fdl invented would be right for everyone), --cwd (project dir inside the tree), --build (a shell recipe, default cargo build --release; it receives LIBTORCH_PATH, FDL_GPU_FEATURE and LD_LIBRARY_PATH, with the system ROCm runtime — $ROCM_PATH / $HIP_PATH / $HSA_PATH, else /opt/rocm — resolved ahead of libtorch’s bundled copy), --to <dir> (the served directory, default ~/.flodl/run; the guardrail key’s rrsync -ro scopes to exactly this, so pick it deliberately), --identity <key> (for an rsync:// source the controller itself pulls over ssh), --no-build (skip the gate — recorded in the manifest, loudly), --gate <variant> (extra check-builds, below) and --json (the report as JSON on stdout, notes on stderr — the machine twin of the human report, same data). One deliberate asymmetry with fdl join: publish owns the whole command so its flags are bare (--cwd), while join prefixes its source flags (--source-cwd) because join also carries data, tunnel and libtorch surfaces.

A standing publish: block in fdl.yml (or the active env overlay) carries all of the above, so re-publishing a run is one bare fdl publish. Flags win field by field, and a -- tail replaces the block’s args: outright — even an empty tail, because the args belong to the run and “explicitly none” must be sayable. --no-build has no block field on purpose: a standing config that skips the gate would ship every future typo to the fleet.

publish:
  source: file:///home/op/rdl
  cwd: ddp-bench
  build: cargo build --release --features "$FDL_GPU_FEATURE" --bin ddp-bench
  bin: target/release/ddp-bench
  args: [--model, olmo-graph, --epochs, "1"]

The manifest is .fdl-run.yml, at the tree root:

field meaning
cwd project directory inside the tree (default: its root); governs the build AND the run
build build recipe, a shell line (default cargo build --release)
bin built artifact, relative to cwd — what workers run
args the binary’s own arguments (everything after --)
origin the source spec the controller resolved, for provenance
rustc rustc -V on the controller — advisory; a worker reports a mismatch, never enforces it
published_epoch unix seconds at publish, so a box can say how old its run is
run this publish’s identity nonce; it rides each worker’s join hello, and the window refuses a cohort whose members hold different ids (two boxes that fetched across a publish boundary would train two different runs as one world)
built false when --no-build skipped the gate; workers say so out loud

Do not hand-edit it: the next publish overwrites it, and its presence is what tells a worker the run is ready.

Chaining runs on a standing fleet is then one command. Publish again and every box picks the new run up on its next dial, with nothing to edit on any worker. That is the point of the manifest: a worker’s own config keeps only what is stable for that box (its credentials, its libtorch policy, where to pull from), while cwd / build / bin / args belong to the run and come from the controller. args is the sharp case rather than a convenience: they must match the run, because rank children re-enter the binary with them, so a fleet carrying its own copy would train the next run with the previous one’s hyperparameters.

The build is validation, not an artifact. One build gates the publish; every worker still compiles its own, because a controller producing binaries for N worker variants is the build matrix this design deleted. It also needs no GPU libtorch — compiling without a GPU feature against the cheap CPU variant catches user-code errors just as well — so the cost of having it on by default is rustup plus fdl libtorch download --cpu. What it buys is that a tree which cannot compile never reaches the fleet, where N boxes would each discover it separately in logs nobody is watching. It proves the tree for the controller’s variant only: a break that exists solely under --features rocm passes a CUDA gate and lands on a worker. --no-build skips it and the manifest records that nothing has compiled this tree.

--gate <variant> closes that per-vendor hole from the controller: each one runs the same recipe as an extra check-build against a named libtorch variant (--gate precompiled/rocm70 on a CUDA controller, and vice versa), under its own CARGO_TARGET_DIR so every variant’s incremental cache stays warm. No GPU is needed — linking is the proof — but a flodl-linking crate does need that vendor’s dev headers on the controller (libtorch bundles runtime libraries, not headers); a gate on a box without them fails loudly with the exact package line to install. A failed check-build publishes nothing, exactly like the primary gate.

The manifest’s presence is the commit point. fdl publish removes it before it touches the tree and writes it only once the build has passed, so a box that dials mid-publish, or after a publish whose build failed, finds no manifest and waits for the next dial instead of training something unvalidated. A failed gate publishes nothing, and the fleet keeps running whatever it had.

The served directory is what a source key must be scoped to: command="rrsync -ro <served>". rrsync re-roots every requested path under that directory, so a worker behind it points its source.from at rsync://<host>:/tree, not the absolute path (which double-roots and fails). fdl publish prints both spellings, each labelled with the key it pairs with (see the guardrail recipe).

Exit code: 0 when the run is published, 1 otherwise.

fdl join

Join a cluster run’s window as a self-deployed worker: the worker-side walk-in for discovery windows (controller.join.discovery: true), where the window alone defines the world and worker addresses need not exist in any roster. It dials the controller, offers the box’s GPUs, and runs your training binary in agent role — the binary joins, then spawns and supervises this host’s relay and rank children itself; training code downstream is byte-identical to the fan-out path.

# Direct dial (trusted segment), authenticated by the run token:
fdl join 10.0.0.1:1337 --token <hex> --bin target/release/train -- --model resnet

# Through a guardrailed sshd on the controller box (the controller
# binds loopback under `tunnel_only`; reachability = authentication):
fdl join --ssh [email protected] --identity ~/.ssh/join_key \
         --bin target/release/train -- --model resnet

Every flag defaults from a top-level join: block in fdl.yml (see fdl.yml.example), so a golden image boots into bare fdl join.

Preparation, before the dial

Admission starts a window deadline, so everything a box needs is acquired before it dials, and re-acquired on every attempt — which is what makes --persist a provisioning loop: a box picks up a changed source on its next re-dial, with no reprovisioning.

A source your provisioning already mounts needs no scheme at all: name its path in --data-path. Nothing is checked and nothing is shipped when neither field is set.

Exit codes:

code meaning --persist
0 the agent’s own: this host’s ranks all finished cleanly re-dials
1 transient failure — the controller unreachable or the agent exiting 1, a mount or fetch attempt failed, the source did not compile re-dials
2 permanent failure — no GPU, a spec that does not parse, a missing binary or toolchain, unwritable stage stops

A compile error being transient is deliberate rather than lenient. The source is remote, so the fix is a push away, and the systemd recipe below pairs code 2 with poweroff: a box that stopped permanently over a typo would take the fleet with it. One exception: a compile failure on a box whose vendor toolkit headers are demonstrably missing is permanent — re-dialing cannot install a package, and the error names the apt line.

One-shot mode (no --persist) passes the agent’s own exit code through verbatim, so a training binary that exits 2 for its own reasons is indistinguishable from fdl’s permanent class. The systemd recipe below is therefore written for --persist, where agent exits re-dial inside fdl and only classed preparation failures ever reach systemd — pair the recipe with persist: true (or treat 2 as reserved in your training binary).

fdl never powers a box off itself; 2 is how the thing that owns the instance hears about it:

# /etc/systemd/system/flodl-join.service  (fdl join --persist ...)
Restart=always
RestartPreventExitStatus=2   # stop hot-looping a misprovisioned box
FailureAction=poweroff       # ... and self-deprovision it

Full protocol walkthrough, trust model, and the join-sshd guardrail recipe: DDP reference.

fdl join-config

The once-per-farm wizard: everything the guardrail recipe asks an operator to assemble by hand, produced in one pass on the controller.

fdl join-config b300                       # interactive: prompts have defaults
fdl join-config b300 --controller [email protected]:2222 \
                     --install-key --cloud-init --yes    # scripted
fdl @b300 join-config --regen              # new farm instantiation: rotate credentials

A farm is an env overlay: the wizard scaffolds fdl.<label>.yml (discovery window, start: manual, the admission token stamped) and fdl @<label> <cmd> targets it afterwards with the machinery that already exists: deep-merge onto the base fdl.yml, inherit-from: for sharing a base between farms, fdl config show provenance. On an existing overlay the wizard only ever touches the token: line (byte-preserving), and a user-authored overlay without one is never edited; the snippet is printed instead.

One invocation produces, under ./.fdl/<label>/ (which gitignores itself, since it holds keys):

The wizard also offers to install its authorized_keys line into the invoking user’s own ~/.ssh/authorized_keys, because the composed guardrail line is the artifact most likely to be mangled by hand. Consent is explicit only: the prompt, or --install-key (--yes deliberately does not count; it accepts ordinary defaults, and a security-relevant mutation is not one). Only the wizard’s own line is ever touched (identity is the public key material; foreign lines are preserved byte for byte), permissions are reported with the exact chmod and fixed on confirm, the rewrite is atomic, and /etc/ssh is never edited: the dedicated-user hardening stays in the notes. After installing, the wizard checks something is listening where workers will dial and says so if not (on macOS: Remote Login).

The wizard guides the setup, not just the artifacts. Before writing anything it reports what THIS box is still missing for the chosen door: an ssh daemon, something listening on the door port, rrsync for door b or an sftp server for door a, the served directory, plus the two traps that produce no useful error on their own (Debian hands the ssh listener to ssh.socket, and while it holds it the Port directive is ignored outright; SELinux refuses a non-standard ssh port with a message that never mentions SELinux). Each gap is reported with the fix in this platform’s own spelling — apt, dnf or brew; ssh.service or sshd.service; ufw or firewalld; semanage where it applies.

It then writes a ready-to-install sshd_config.d drop-in into the farm directory and prints the setup as numbered steps whose commands read those files rather than repeating their contents, so what you paste is a copy of something you can review first. The last of the controller-side steps is a door self-test specific to the door: every door must refuse a shell and permit the forward, and only b must additionally list the served tree, only a must open sftp.

The drop-in scopes its guardrail with Match LocalPort, not Match User. Binding it to the exposed port confines every key that arrives there, including ones added later, while leaving port 22 and your own logins untouched — which also means the wizard can install its line into your own ~/.ssh/authorized_keys unaided, where a dedicated no-shell account would have needed root. ForceCommand appears only for the tunnel-only door: doors a and b carry their command in the key line, and a daemon-level forced command would override it, leaving the tunnel working while the mount or the source pull failed.

--authorized-keys <path> names a different door for the setups the default cannot reach at all: an sshd in a container, whose key file is a bind mount (writable only from the host, while the in-container run sees a read-only filesystem), or a host configured with AuthorizedKeysFile /etc/ssh/authorized_keys.d/%u. Every other rule holds — the same explicit consent, only the wizard’s own line, the same atomic rewrite — except that the named file’s parent directory is left exactly as found, since it belongs to a layout the operator already owns rather than to a ~/.ssh the wizard may have to create. A path under /etc/ssh is still refused by name.

The scaffolded overlay also carries a commands: entry, because a join window only opens for a command running in launcher mode and cluster: true is what puts it there. Without one, fdl @<label> <cmd> resolves the base command and trains locally: no window, no walk-ins, and no complaint, since training on that box is a legitimate thing to do. The wizard names the entry after the training crate when it can read one, and comments a placeholder when it would be guessing. Running a non-cluster command under a farm overlay warns for the same reason.

Credentials are reused on re-runs, and so is the farm’s shape: a re-run without --door or --controller recovers both from the farm’s own worker yml rather than falling back to flag defaults, so reprinting the authorized_keys line cannot silently re-render the farm around it. Naming either flag is how you intend a change. --regen (or the prompt) rotates key and token together for a new farm instantiation, after which workers holding the old ones stop being admitted. --json emits the machine twin; secrets appear as file paths, never payloads.

Exit code: 0 with the report, 1 on any refusal (contradictory flags, a decision needed without a tty, an unfixable permission).

Platform notes: the token comes from OS entropy (cross-platform; the wizard refuses rather than degrade if the system call fails) and the key from ssh-keygen, which Linux and macOS ship natively (on macOS, Remote Login is the sshd) and Windows carries with its OpenSSH feature — though for cluster work on Windows the supported path stays WSL2, where the permission checks also actually mean something. Running the wizard inside a container works for generation — the farm lands in the mounted project — but the authorized_keys install step must run where the workers’ sshd actually lives, which is not the container’s ~/.ssh.

fdl nccl

Build NVIDIA’s libnccl from source. Required for heterogeneous-rig clusters when the bundled NCCL versions across hosts don’t match (NCCL refuses handshake across major.minor skew). The build runs in a dedicated Docker context (Dockerfile.nccl.source) so no host-level NCCL/CUDA toolchain is required.

fdl nccl build                              # auto-detect target tag + local GPU archs
fdl nccl build --tag v2.27.5                # explicit NCCL git tag
fdl nccl build --archs "6.1;12.0"           # explicit archs (heterogeneous rig)
fdl nccl build --jobs 8                     # parallel compilation jobs (default 6)
fdl nccl build --dry-run                    # print build plan, do nothing

Auto-detection:

Output path: libtorch/nccl/builds/v<version>-<archs>/lib/libnccl.so.2.

Wire it into a worker via the env: LD_PRELOAD: block in fdl.cluster.yml:

workers:
  - host: node-b
    arch: builds/sm61-sm120
    env:
      LD_PRELOAD: /srv/flodl/libtorch/nccl/builds/v2.27.5-sm61/lib/libnccl.so.2

Build time: 5-15 minutes depending on CPU cores and arch count.

fdl --gpus

Scope GPU visibility for a single command. Two forms:

fdl --gpus all <cmd>            # use every visible CUDA device
fdl --gpus 0,1 <cmd>            # explicit physical indices

Behavior depends on the command kind:

Cluster context interaction: on fdl @cluster <cmd>, --gpus overrides per-worker local_devices: for the local controller host; remote workers continue to use their cluster.yml-declared devices. Loud-errors on duplicate, missing value, or invalid spec.

fdl --gpus 0 test                # CPU-style: scope tests to GPU 0
fdl --gpus 0,1 ddp-bench --mode nccl-cadence   # synthesize 2-rank single-host cluster
fdl --gpus all @cluster ddp-bench --mode nccl-cadence   # override local host devices in cluster mode

fdl @cluster <cmd> - multi-host fan-out

fdl @cluster <cmd> selects the cluster env overlay via the @ sigil. When fdl.cluster.yml exists alongside fdl.yml, it deep-merges as an overlay and triggers SSH fan-out for any command marked cluster: true.

Three equivalent forms:

fdl @cluster <cmd>           # sigil form
fdl --env cluster <cmd>      # explicit flag
FDL_ENV=cluster fdl <cmd>    # env-var form

Pre-flight per-host build: before fan-out, fdl @cluster <cmd> auto-builds the target binary locally for every remote host. Per-host CARGO_TARGET_DIR=target/cluster/<host>/<arch>/, libtorch resolved from each host’s arch: declaration, CUDA feature derived from the host’s GPU arch metadata. (Keying the target dir on arch as well as host means changing a host’s arch: rebuilds cleanly instead of reusing a binary linked against the old libtorch.) Builds run in parallel per host; first failure aborts fan-out. Remote dispatch invokes the prebuilt binary directly - no cargo, no rustc on remote.

Pass --no-prebuild to skip the pre-flight phase (when binaries are known fresh, or when iterating on a build-only issue).

Heterogeneous-rig flow (a Blackwell host + a Pascal VM, say):

  1. fdl @cluster probe - confirm GPU + libtorch + NCCL match per host.
  2. fdl nccl build on the host with the older NCCL - produces a matching libnccl.so.2 to wire via env: LD_PRELOAD:.
  3. fdl @cluster <cmd> - fan out.

See DDP Reference: Multi-host clusters for the fdl.cluster.yml schema and conventions.

← Previous Project and libtorch Commands