Multi-GPU Execution

The CodefyUI server is one process. A run executes in that process by default, or in a subprocess of a worker agent when you target one, and it uses the devices of its target. On a machine with several GPUs that gives you three things: several runs at once, one per card (or more, if you allow it), a single run that sees several cards, and a TrainingLoop that trains one model across them. This page is the how-to; Execution Architecture explains the design, and Run Queue is the reference for queue order and options.

Run or queue

	Run (the canvas)	Queued run
Started from	The toolbar's Run	Queue on... next to Run, cdui run, POST /api/runs, a sweep or study
Executes in	Always the server process	The server process, or a worker when its target says so
Waits for a free card	No: starts at once (two at most, one per tab)	Yes: joins the queue of its device
Devices	One: the graph's device, else Settings	One device, or a device set on a worker
Keeps weights and cache between clicks, captures outputs for the Inspector	Yes	No (outputs only with record_outputs; a worker keeps them and the Inspector reads them from it)

Use Run while you build and debug a graph. Queue a run for anything long, anything you want on a particular card, and anything that needs more than one card.

Queue on...

The caret next to Run in the toolbar opens a menu whose Queue on... item submits the active tab's graph as a queued run (the same POST /api/runs that cdui run uses):

• Target: This server, every worker the server knows with its status and devices, Any available once a worker is connected, and A one-shot worker once the server knows any worker (the one-shot pool, see One-shot workers). A run for a worker that is offline or disconnected waits in the queue until it connects; a draining or disabled one is listed but cannot be chosen. An instance of a shared credential that left is not listed. With no worker at all, the dialog says so with a link that opens the Workers panel.
• Devices: one device for This server; for a worker, a set of one kind, kept in the order the worker lists its devices, with each CUDA card's in-run name shown as you tick it (cuda:2 -> cuda:0 in this run); for Any available, a kind (CPU, CUDA GPU, Apple MPS). Unless the tab names a device, a worker and Any available start on the first accelerator offered, whatever Settings picks for this server; the New sweep dialog does the same.
• Name, Random seed, Deterministic algorithms and Record node outputs start from the tab's settings. On a worker the recorded outputs stay on the worker, and the dialog says so; Record node outputs is off and disabled for a one-shot worker or the one-shot pool, whose agent exits when its run ends.

Queue submits the run, closes the dialog and opens it in the Runs tab. A refusal from the server is shown in the dialog.

Start a worker on this machine

A worker agent executes each run it is given in a subprocess of its own, so a crash or an out-of-memory kill ends that run and leaves the server and the other runs alone. This section starts one on the same machine as the server; for a GPU server elsewhere, see Remote Workers.

cdui start                 # the server, if it is not running yet
cdui agent start           # a worker named after this machine's hostname
cdui agent stop            # when you are done

cdui agent start needs no flags next to cdui start: it finds the server's address and session token on its own and reconnects when the server restarts. --name gpu-box sets the worker name, --devices cuda:0,cuda:1 limits the cards it offers (the CPU is always offered), and --foreground keeps it attached for systemd. All flags: CLI Commands, Agent commands.

Within a few seconds the worker appears in the Workers panel and in GET /api/workers.

Choose devices

A queued run names where it executes in the Queue on... dialog, with --target and --devices on cdui run, or with options.target in POST /api/runs:

# One card on this server, as before
cdui run train.json --device cuda:1

# One card on the worker gpu-box
cdui run train.json --target gpu-box --device cuda:1

# Two cards on the worker: the run sees them as cuda:0 and cuda:1
cdui run train.json --target gpu-box --devices cuda:2,cuda:3

# Whichever has a free CUDA card first: a connected worker, else this server
cdui run train.json --target any --device cuda

{"graph": {...}, "options": {"target": {"worker": "gpu-box", "devices": ["cuda:2", "cuda:3"]}}}

• local (the default) is this server. A run here executes on one device: with a device set it holds a slot on every device of the set, so nothing else is placed on them, and executes on the first. Every device of the set must be one this server has, or the submit fails with 400 (this server does not offer cuda:0 (it offers cpu, mps:0)). A run without a set keeps the fallback it always had: --device cuda on a machine without CUDA runs on the CPU. That fallback, and a node whose device names a device the machine lacks, is recorded in the run's event log as a run_warning of kind device_fallback, which the Runs panel shows.
• A worker name sends the run to that worker. The worker must have connected since the server started and must offer the devices, or the submit fails with 400. A worker that is away now is not a refusal: the run waits for it to reconnect.
• any takes the first connected, non-draining worker with a free slot, in name order, and this server last. An any run with a CUDA set of more than one card, or of one card other than cuda:0, runs on a worker only, because only a worker renumbers the set from cuda:0 for the run (see below); the submit fails with 400 when no worker the server has seen offers the whole set.

In-run device names. A run on a worker sees only its own cards, renumbered from 0 in the order you listed them. With --devices cuda:2,cuda:0, the run's cuda:0 is the machine's cuda:2 and its cuda:1 is the machine's cuda:0. A node whose device parameter is auto follows the run's device (the first card); to put a node on another card of the set, set its device to that card's in-run name, here cuda:1. The node's device list in the config panel offers these in-run names under Inside a run's device set (cuda:0 up to the most cards any known worker has), so a control plane without a GPU can pin a node to a card of a worker run's set; it never lists a worker's own card numbers, which a run with a set does not see. A device the graph holds that this server lacks is shown as stored, with a note that a run here falls back. A node naming a card its run does not have falls back too, and the device_fallback warning names the run's cards and which of the worker's cards they are (Node train (TrainingLoop) is set to device cuda:1; this run has 1 card (cuda:0 = the worker's cuda:1), so when it executes, it runs on cuda:0.). The run records the mapping in execution_facts as visible_devices and device_aliases. See Device Backends.

Sweeps and studies. POST /api/sweeps and POST /api/studies pass options.target to every child. With "target": {"worker": "any"} and no devices, each child takes the next free slot of the sweep's device kind, so a four-variant sweep on two agents with one card each runs two variants at once; seeded children on workers run side by side, each in a process of its own. A target a single run would be refused for refuses the whole sweep or study with 400 before anything is queued. Each variant and attempt reports worker_id, devices and queue_key. The New sweep dialog has the same Target and Devices picker as Queue on...: This server, a connected worker with a device set, or Any available with a device kind. Left at This server and the tab's device, it sends the sweep as before, with no target. The sweep view shows each variant's Worker and Devices. Studies take target through the API. See Run Queue, Sweeps.

Train one model on several cards

A TrainingLoop trains in one process. Set its strategy (under Advanced) to ddp and queue the graph on a worker with a device set of two or more CUDA cards, and the loop trains one model across those cards with PyTorch's DistributedDataParallel:

cdui run train.json --target gpu-box --devices cuda:0,cuda:1

The run subprocess starts one process per card, a rank: rank k trains on the run's cuda:k. The ranks find each other through a file in the run's directory on the worker and communicate with NCCL (with gloo when the PyTorch build has no NCCL) over the loopback interface only: the worker opens no port that another machine could reach (GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME are set to lo for the ranks unless the agent's environment sets them, and OMP_NUM_THREADS to 1). Nothing else changes about the run: it is one run in the Runs panel, with one event log, one set of metrics and one result.

• Data. Each rank reads its own part of every DataLoader the loop is given (dataloader and val_dataloader): one Nth of the dataset, through a DistributedSampler, at the DataLoader's batch_size. batch_size is therefore per card, and one optimizer step averages the gradients of N batches: two cards at batch_size 32 train like one card at 64. The loop's Config panel shows effective_batch_size. With shuffle on, the order changes every epoch and is the same on every rank, and on a seeded run it is the same on every run. When the dataset size is not a multiple of N, the last part repeats a few samples, so that every rank runs the same number of batches; in validation those repeats are left out of val_loss, val_accuracy and val_loss_step, so every validation sample counts once. The dataset must be map-style (indexable) and the DataLoader must use its own sampler; an iterable dataset or a custom sampler fails the run with a message.
• Randomness. On a seeded run, rank k seeds its random number generators with the run's seed plus k, so dropout masks and augmentations differ between the ranks, as they do between the batches of one process; rank 0 is seeded exactly as a single-process run is. The shuffle order is the one exception: it is rank 0's, shared with every rank.
• Weights. Every rank starts from rank 0's weights, and the gradients are averaged across the ranks in every backward pass, so all ranks hold the same weights after every step. accumulate_steps, grad_clip_norm, precision, max_steps and the LR scheduler work as on one card.
• What is reported. train_loss is the mean of every rank's batch losses; val_loss and val_accuracy are computed over every rank's part of the validation set; train_loss_batch and val_loss_step are averaged over the ranks too. The live progress bar shows rank 0's latest batch. The Config panel and the metrics output add strategy: ddp, world_size and backend. Peak memory lists every card.
• Rank 0 is the run. Rank 0 executes the whole graph, and only its events, metrics and files reach the server: the checkpoints (checkpoint_every, checkpoint_every_steps and the interrupt checkpoint) and the TensorBoard files are written once, by rank 0, and load on one card as well. The other ranks execute only the loop and the nodes that feed it, and they start once rank 0 reaches the loop, so a dataset that is downloaded on first use is downloaded once. Those nodes still run on every rank: a dataset that is built or preprocessed in memory is built N times, on N times the host RAM, and a rank must reach the loop within 30 minutes of rank 0 (the time a rank waits in the loop's first collective), or the run fails with a collective timeout. The nodes after the loop (an evaluation, a ModelSaver) run on rank 0 alone, with the trained model. Each rank writes its own output log on the worker: the run's log is rank 0's, and rank k's is output-rank<k>.log beside it.
• Stop. Stop reaches every rank. The ranks agree after every batch on whether to stop, so they stop at the same batch, and rank 0 writes the interrupt checkpoint.
• Failures. A rank that fails or exits ends the run as failed with that rank's error, such as DDP rank 1 of 2 failed: ..., and the other ranks are stopped. The run is reported only after every rank has exited, so its cards are free by then.
• Memory budgets. On a card shared by memory, each rank is capped to its own card's budget, and to nothing on the other budgeted cards: a node that pins another card of the set by its device param fails at its first allocation there, instead of using memory the budget does not cover on a card another rank is capped to. Peak memory reports, for each card, the most any rank used of it.

Where ddp cannot apply, the loop trains in one process on its own device, exactly as with single, and says so in the Runs panel (a warning of kind ddp_not_available) and in the node's log: Run on the canvas, a run on this server (with a device set too), a worker run on one card or on the CPU, a TrainingLoop inside a block or a preset, and an exported Python script (Export as Python warns about such a loop). Two ddp loops in one graph must run one after the other, one loop's model into the next; in a run that trains with ddp, two on separate branches fail the run before it starts.

Slots and limits

A slot is one device on one machine. Each slot admits one run at a time on an accelerator and two on the CPU, and everything else waits in submit order. A device-set run needs a free slot on every device of its set and takes them all at once; while it waits at the front of its queue it reserves them, so a stream of single-card runs cannot starve it.

CODEFYUI_RUN_QUEUE_MAX_CONCURRENT changes one slot's limit. A bare device is this server's; worker/device=N is a worker's:

CODEFYUI_RUN_QUEUE_MAX_CONCURRENT="cuda:0=2,gpu-box/cuda:1=3" cdui start

A card that several offerers share counts as one card: a worker on the same machine and this server (matched by the card's UUID), or two workers that offer the same card (two agents on one GPU server, such as an SSH worker and an enrolled agent). The runs through every offerer count against it, under the smallest limit among the offerers that are online (a worker on the same machine without a <worker>/<device> override of its own uses this server's limit for the card, so cuda:0=2 raises it), so a run on the server and runs on those workers never take one card past its limit. A card shared by memory keeps one ledger, and admits by memory only while every online worker offering it is set to share it. The Workers panel says "Same physical card as ..." on such a card and names the offerer whose limit applies; an override that the smaller limit lowers is logged as a warning when a worker connects. Raising a limit lets runs share a card's memory; nothing checks that they fit. To share a worker's card by memory instead, see Sharing a card. Details: Run Queue, Configuration.

Sharing a card

A worker's card admits one run at a time by default (Exclusive). To let several runs share it, set the card to Shared by memory in the Workers panel. The choice is stored per worker and card, and survives a reconnect and a restart of the server. Only CUDA and MPS cards can be shared, and only on a worker: a run on this server shares one process with every other run here, so nothing could hold it to a budget.

A run sent to a shared card declares a memory budget, in GiB per card of its device set: target.memory_gib, or the Memory budget field of Queue on.... The field is filled in with the CUDA peak of the last succeeded run of the same graph on the same number of cards plus 20%, when there is one (a per-card peak of a run on two cards says nothing about the same graph on one; MPS reports no peak, only what the driver holds at the end, so an MPS run declares its budget). That measured figure is also the default the server applies to a run that declares none. "The same graph" is the graph as it executes: node types, the params that shape its memory or pick its data (a Dataset node's name among them), and the wiring, topology included. Node ids and positions, display labels, seeds, how long it runs (epochs, max_steps, lr) and the loader's worker processes do not count, so a rerun with another seed or more epochs finds its peak. The params of a node type this server does not know (a plugin installed only on a worker) count too: they are left out of the stored graph, but not out of the key. A shared MPS card reports no memory total, so it cannot admit a run by budget: the dialog says so in place of the field, and the server refuses the run until the card is set back to Exclusive. A CUDA card that has not reported its memory yet keeps the field, with a note that the budget is judged once its worker's next heartbeat arrives.

Without such a run, the field is filled with a static estimate, tagged Estimate in the dialog and never presented as a peak: every parameter of the graph's models at 4 bytes, plus, for a graph that trains, what each trainable one holds for training (its gradient and optimizer state: 12 more bytes with Adam, 8 with SGD and momentum, 4 with plain SGD; RMSprop with momentum or centered one more state each) and, under bf16/fp16 autocast, a 2-byte copy of every weight; plus the activations of one training step at the graph's batch size or of an evaluation pass at its own batch, whichever is larger; plus 20% on all of that for the allocator's reserve; plus 128 MiB for the CUDA libraries' workspaces. The dialog lists each of those terms, so they add up to the figure. The server counts each model by building it on PyTorch's meta device (shapes only, no memory, in a thread of its own with a time limit), and the activations by one forward pass on a meta input whose shape comes from the model's first layer and the graph's data nodes (image_size or seq_len set explicitly on a node that provides data, or a known dataset; a default value never decides the size). A build that overruns its time limit is abandoned, and the dialog offers no estimate; the estimate is not built at all when a measured peak exists. On the built-in training examples the estimate came out 1.2 to 1.8 times the measured peak for runs above 1 GiB, and at most 0.25 GiB for the small ones, whose peak is mostly fixed overhead. An estimate that had to skip or guess part of the graph is shown as a Rough estimate with the reason, and the field stays empty: a model node it cannot count (such as one from a plugin), no input shape for a model's activations, an input size the graph does not give (a recurrent net or an adaptive-pooling CNN runs at any size, so a default size would be a guess), or no batch size.

The server never applies a static estimate on its own. A measured suggestion left as it is is not sent: the server applies the same default, and the run's Budget column names the run it came from. An estimate left as it is is sent as the run's declared budget, marked as an estimate (target.memory_source: "estimate"), and the Budget column marks it est.. An estimate submitted after the graph already has a larger measured peak (from the dialog, the API or the CLI) gets that peak plus 20% instead. When a run of the same graph succeeds with a larger peak while such a run is still waiting, the waiting run's budget is raised the same way, and its reason and Budget cell in the Runs panel say so; a raised budget that no card the run may use can hold fails the run with that reason rather than leaving it waiting forever. A budget you typed is never changed. A run that runs out of memory under its budget says where the budget came from (typed, measured, or an estimate) and asks for a larger target.memory_gib. With no measured peak and no budget declared, the server refuses the run and asks for one. A sweep or a study is checked whole before any of its runs is written, each run with its own target.wait: when one of its runs would be refused (no budget, one larger than every card it may use, or one that does not fit now with wait: false), the sweep or study is refused; a sweep of a graph that has no measured peak yet declares its budget.

{"options": {"target": {"worker": "gpu-box", "devices": ["cuda:0"], "memory_gib": 6}}}

The server keeps a ledger per card: its total and free memory from the worker's heartbeat (free counts every process on the card, other users' jobs included), the budgets of runs assigned but not started (reserved), and the budgets of runs that started (running). A run on a CUDA card also holds its CUDA context (cuBLAS handles, loaded kernels), which the cap below does not cover, so the server counts each one as its budget plus CODEFYUI_EXECUTION_CUDA_CONTEXT_MIB (768 MiB by default; about 650 MiB was measured on a CUDA 12 card). A run fits when that sum is at most min(free - reserved, total - reserved - running), and a refusal names both: needs 6.0 GiB (+0.8 GiB CUDA context), .... The worker caps each run's allocator to its budget (torch.cuda.set_per_process_memory_fraction) before the run starts, so a run that outgrows its budget runs out of memory alone and the other runs on the card go on.

When a budget does not fit:

• a run for a named worker is refused at once, with the reason: needs 12.0 GiB, cuda:1 on gpu-box has 6.3 GiB free (runs a1b2c3d4, c3d4e5f6). With target.wait: true (Wait in the queue when it does not fit now in the dialog) it waits instead;
• a run for any waits until a card has room, unless its budget is larger than every candidate card (refused), or target.wait is false (refused instead of waiting). A run for the one-shot pool (target.pool) is judged the same way, on the one-shot workers' cards;
• a run waiting for memory says so under its worker in the Runs panel.

A run without a budget never goes to a shared card, and a run on a shared card that holds no budget (a canvas run on a same-host worker's card, or one that started before the card was shared) keeps it to itself until it ends. The Runs panel's Budget column shows each run's budget, and the Workers panel draws each shared card's memory: in use, reserved, and what a new budget may take (its CUDA context already taken off). A run that waited for a card while it was Exclusive gets its graph's default budget when the card is set to Shared, or later, when a run of the same graph on the same number of cards succeeds; until then it waits, and the Runs panel says so (cancel it and queue it again with a budget to start it sooner). A run admitted by its budget is never started on an agent that cannot cap it: if the worker reconnected with an older agent in between, the run fails with the reason. A run the server adopts after a restart (Remote Workers) counts its budget on its card again from the start, as running.

The Workers panel

Open it with Workers in the Runs panel header, or from Settings → Execution → Workers → Manage. It refreshes every 5 seconds while open and shows, for each worker:

• its status (Online, Degraded when a heartbeat is late or its node types differ from the server's, with the reason beside it, Draining, Disconnected, and Disabled for an SSH worker you disabled), its version and whether it matches the server's, its environment digest, whether its node types match the server's (and which differ), the reason its last connection was refused, mount labels (see Remote Workers, Storage) and the age of its last heartbeat;
• each device with its free and total memory (GiB) from the latest heartbeat, marked "when last seen" while the worker is away, and its slots taken / limit, and for a CUDA or MPS card how it admits runs (Exclusive or Shared by memory, Sharing a card) with a shared card's memory bar;
• the runs it is executing, each a link to the run in the Runs panel.

Agents the server refused (another version, for example) and that have no card are listed under Refused connections with the reason (Remote Workers, Keeping workers in sync).

Drain stops new runs from being placed on a worker while its running runs finish; Undrain reverses it. A queued run whose worker is draining or away waits, and the Runs panel shows why on a line under the run, for an any run too when the only workers that offer its device are draining. Add worker generates an enrollment command for another machine, or shows the same-host command above with its options; on a server that listens on 127.0.0.1 only, which no other machine can reach, it opens on Over SSH. Re-key gives an enrolled worker a new credential (a same-host agent on the session token and an SSH worker have none to replace, so they do not offer it), and Remove revokes it, disconnects the worker and fails the runs queued for it; for an SSH worker it also ends the ssh session and deletes the entry and what it keeps on the remote host (Remote Workers). Removing a worker, deleting an SSH worker and disabling an SSH worker that has runs on it each take a second press, and the card says what that press does first. A disabled SSH worker reads Disabled here and in Queue on..., where it cannot be chosen.

What the canvas does

Run on the canvas always executes in the server process, on the graph's device (else the one in Settings), and never goes to a worker. It skips the queue, so it can share a card with a queued run even when that card's limit is 1. It keeps the tab's cached outputs and model weights between clicks, which a worker run cannot do, and records outputs for the Inspector (a worker run does so only with record_outputs). To send the tab's graph to a worker, use Queue on....

Peak memory in the Runs panel

When a run finishes, its detail header shows Peak memory for CUDA (the allocator's high-water mark, every card on hover) or Memory held at end for MPS (what the driver still holds, cache included; MPS has no peak counter). A run on a worker executes alone in its subprocess, so its figure is exact. Runs that executed in the server process share one CUDA context, so their figure is approximate when another run used the card at the same time. Use the peak of a finished run to decide whether two such runs fit on one card before you raise its limit.

Limits

• Several cards train one model only through ddp, on one worker. A device set makes several cards visible to one run, and nodes can be pinned to them by in-run name; a TrainingLoop with strategy: ddp trains across them (Train one model on several cards). There is no multi-node training, and no sharded (FSDP) or model-parallel strategy. True multi-card NCCL training was not tested on hardware with several GPUs before this release: the strategy was checked with gloo on the CPU and on one CUDA card, and with NCCL on one card.
• Memory admission on workers only. A card on this server is admitted by count per slot: a run is not refused because the card lacks memory, and an over-full card fails with an out-of-memory error. A worker's card can be shared by memory. See Training Memory.
• The ledger counts budgets, not what runs use. A running run is counted at its budget, and free memory comes from the worker's heartbeat (every 10 seconds) for the whole card: memory another user's job takes between two heartbeats is not seen until the next one, and per-run use is not measured.
• Plain HTTP between machines. A remote worker enrolls and connects over whatever the control plane serves; without a TLS proxy or an SSH tunnel its token and run inputs cross the network unencrypted (Remote Workers, Trust).
• Captured outputs stay on the worker (record_outputs, #789): the Inspector reads them only while the worker is connected. No persistent weights: every worker run starts fresh.
• A long disconnect ends the run. A worker that loses its connection keeps its runs going for CODEFYUI_WORKER_RESUME_GRACE_S (5 minutes); past that they end interrupted (Remote Workers, When the connection drops).

See also

• Run Queue — queue order, cdui run, the target option and workers in detail
• Execution Architecture — control plane, executors, worker protocol and invariants
• Device Backends and Training Memory — devices, card addressing and out-of-memory errors