Skip to main content

Device Backends

CodefyUI runs on PyTorch, so it inherits PyTorch's device backends: CPU, NVIDIA CUDA, Apple Silicon (MPS), and AMD ROCm (Linux). For installing the right wheel, see GPU & Device Setup; this page explains how device selection behaves at runtime.

Device selection​

CPU is the default, and nothing switches away from it on your behalf. A run resolves its device in this order, first match wins:

  1. A node's own device parameter, when it is not auto (under Advanced). auto falls through to the graph's device and then Settings, below. Its description reads: "Device for this node. 'auto' follows the run's device. A run sent to a worker with several devices sees them as cuda:0..N-1 in the order they were listed, so a node can be pinned to any of them by that position." A run given several cards on a worker sees them as cuda:0 to cuda:N-1 in the order its target lists them (see cdui agent), so a node set to cuda:1 in a run given cuda:0 and cuda:2 runs on the machine's cuda:2. The value is not rewritten; the run's CUDA_VISIBLE_DEVICES does the renumbering.
  2. The graph's own device, settings.device in the graph file (format below), set from the device select next to Run. It is saved with the graph, so git tracks it and the graph runs on the same device wherever it is opened. The select's first option, Follow Settings (CPU) while Settings is on CPU, leaves the graph without a device of its own.
  3. The Settings device (this browser, remembered between sessions; cpu until you change it). New graphs and graphs with no assigned device use it; Settings also shows the best device this server can see as a hint. A Settings device this server does not list is marked in both selects; see When a device is unavailable.
  4. cpu, the value the server assumes when a request names no device at all.

Every dropdown (Settings, the graph control and the node parameter) lists the same devices, the ones PyTorch can actually see (device_utils.describe_accelerator()), including cuda:N per card on a multi-GPU box. A requested device is checked against what's available and falls back to CPU with a warning if it isn't present (except an out-of-range cuda:N, which goes to the current CUDA device; see below). Only an explicit auto (--device auto on cdui run or an exported script, or "device": "auto" on the run API) resolves to the best accelerator present. Changing a graph's device invalidates the interactive cache, so every node runs again.

The graph's settings object​

The graph's own device and run seed are stored in an optional top-level object of the graph JSON, as settings.device and settings.seed: "settings": {"device": "cuda:1", "seed": 0}.

  • Values: cpu, auto, cuda, cuda:N, mps or mps:N, case-insensitive. An empty string means no assignment. A Mac has one MPS device, so mps:N runs on mps.
  • Written only when set. Save writes settings only when the graph has a device or a seed, with only the keys that are set (a settings.seed of 0 counts), so a graph with neither stays byte-identical to what it was. In a project directory it goes in the git-tracked graphs/<name>.graph.json, not in the layout file.
  • Any other value is refused: 422 from POST /api/graph/save, /api/graph/validate and /api/graph/export; 400 from POST /api/runs and POST /api/sweeps (so cdui run reports a failed submit); an execution_error frame when the canvas runs the graph; and an invalid_settings finding from cdui project validate. The offline runner (backend/run_graph.py), POST /api/graph/run/{name} and POST /api/apps/{slug}/invoke read settings.device only when the call names no device (--device, or the body's device); then they ignore an invalid value, log a warning, and run on the CPU.
  • Seed: settings.seed is a whole number from 0 to 4294967295; any other value gets 422 from save, validate and export. It is the tab's Random seed (Settings → Training Behavior): Save writes it, Open and Import set the tab's seed from it, and Export Python bakes it in. A file without settings.seed, such as any graph saved before 2.8.9, leaves the tab's seed as it is. Headless runs read it the way they read settings.device: backend/run_graph.py, cdui run, POST /api/runs, POST /api/graph/run/{name} and POST /api/apps/{slug}/invoke use the graph's seed when the call names none. An explicit --seed N or "seed": N wins, and --no-seed or "seed": null runs a seeded graph unseeded. A sweep's variants are the exception: the sweep owns their seeds (sweep_spec.seed with seed_variants), so they ignore settings.seed. POST /api/runs and POST /api/sweeps refuse an invalid settings.seed with 400; the offline runner and the two function routes ignore it with a warning and run unseeded.

cdui run without --device prints the file's device as <device> (graph), and without --seed it prints the file's seed as <seed> (graph); run_graph.py logs it the same way.

Device alignment is guaranteed by the engine​

You do not have to reason about where a tensor happens to live. Before a node runs, graph_engine.invoke_node moves every tensor in its inputs to the device that node runs on — its own device parameter when it declares one and it is not auto, otherwise the run's (resolved as above). Because every path into a node goes through that one function, the guarantee covers builtin nodes, plugin nodes and your own custom nodes alike.

This matters because a device mismatch cannot happen on a CPU-only machine, so it is invisible during most development and shows up only on someone else's GPU box. Two shipped graphs died that way — Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same — before alignment moved into the engine.

What alignment deliberately does not touch:

  • Modules. nn.Module.to() is in-place, so relocating a model handed from one node to another would flip weights out from under the node that owns it. A node that wants a model on its own device says so with an explicit to_device.
  • Datasets, DataLoaders, environments and other non-tensor values. They pass through untouched, so a dataset stays lazy and TrainingLoop keeps streaming batches to the GPU one at a time instead of resident VRAM.
  • Nodes that declare align_inputs = False. A node whose work is host-side by nature — it hands its input straight to numpy, sklearn, matplotlib or PIL — opts out, because Tensor.numpy() raises on anything but the CPU. TrainTestSplit is the builtin example. Write align_inputs = False on your own node if it does the same; see Custom nodes.
  • Tensors a node creates inside its own execute. Alignment sees a node's inputs, not torch.zeros(...) called two lines into its body. If your node builds a tensor to combine with one it was handed, build it with device=<the input>.device.

:::note The exported script follows the same rule python graph.py with no --device runs on the device the graph was saved with (settings.device), else on the CPU, the answer the canvas gives for that graph while Settings is on cpu. Pass --device auto to take the best accelerator on the machine you are running on, or --device cuda:1 to pin one. :::

Addressing one card out of several​

On a machine with more than one CUDA device, every dropdown also lists the cards individually — cuda:0, cuda:1, and so on — alongside the bare cuda, which means "whichever card torch is currently pointed at". A single-GPU machine shows only cuda, because there the two name the same hardware.

An index that does not exist on this machine falls back to the current CUDA device, not to the CPU: a graph pinned to cuda:2 on a workstation should still train on the GPU when it is opened on a laptop. Each card also gets its own run queue. See Training Memory for the full picture, including what several GPUs change.

:::note Device sets A queued run sent to a worker agent can name several cards of one kind, target.devices (--devices cuda:2,cuda:0 on cdui run). The run's subprocess gets them as CUDA_VISIBLE_DEVICES and sees them as cuda:0 to cuda:N-1 in the order listed, so a node's device set to cuda:1 runs on the second card of the set. The node's value is never rewritten, and the rules above apply unchanged inside the run. A run in the server process executes on the first device of its set. A TrainingLoop with strategy: ddp in a worker run trains on every card of a set of two or more, one process per card (Multi-GPU Execution). An exported script takes one --device; to give it a subset of cards, set CUDA_VISIBLE_DEVICES before you start it. See Multi-GPU Execution. :::

The float64 + MPS constraint​

MPS is float32-native and rejects float64 tensors. CodefyUI normalizes this in device_utils.to_device, but if you write a custom node that creates tensors directly, keep them in float32 on Apple GPUs to avoid runtime errors.

CodefyUI also sets PYTORCH_ENABLE_MPS_FALLBACK=1 before importing torch. When MPS does not support an operation, PyTorch runs it on the CPU instead. This is slower but prevents an error for the unsupported operation. Export the variable as 0 before cdui start to make unsupported operations raise an error.

Performance on Apple Silicon​

Numbers below are from an M3 MacBook Air (24 GB, torch 2.11), one epoch of three shipped examples, with cool-down gaps between runs. With the first two items below applied: CNN-MNIST 11.2 s → 4.1 s, ResNet-CIFAR10 12.1 s → 5.1 s, GPT-Mini 19.9 s → 13.1 s on mps; 13.7 s → 11.5 s, 27.4 s → 24.4 s, 22.4 s → 20.8 s on cpu.

  • The training loop does not read the loss back per batch. loss.item() is a host/device synchronisation. On MPS it drains the Metal command queue, so the CPU cannot prepare the next batch while the GPU runs the current one. TrainingLoop, EvaluateModel and DiffusionTrainingLoop accumulate the loss on the device and read it back once per epoch, at each batch_metrics point, and for each progress frame (at most two per second). This change alone: CNN-MNIST 11.2 s → 5.5 s, ResNet-CIFAR10 12.1 s → 5.6 s, GPT-Mini 19.9 s → 15.8 s per epoch on mps.
  • The default ToTensor + Normalize pipeline runs per batch. For MNIST, FashionMNIST, CIFAR10 and CIFAR100 with no transform wired, Dataset applies both steps to the whole batch. The output is bit-identical to the per-sample path; host time per epoch drops from 1.5 s to 0.13 s for MNIST. A wired transform chain uses torchvision's per-sample path.
  • Keep num_workers at 0 for in-memory datasets. macOS starts DataLoader workers with spawn. For MNIST-sized data the start-up and per-batch IPC cost more than they save: CNN-MNIST took 11 s with 0 workers, 25 s with 2, 16 s with 4. Workers help for datasets that decode image files (ImageFolderDataset).

When benchmarking: the first MPS run in a process spends 0.2–0.6 s initialising Metal and 0.1–0.2 s per new kernel shape. A later process on the same Mac is faster because macOS caches compiled shaders. A fanless Mac throttles after a few minutes of sustained GPU load; leave cool-down gaps between runs you compare.

Mixed precision is not used on MPS. bf16 and fp16 autocast run on torch 2.11 but measured 1.8–3.3× slower than fp32 with no memory saving. See Training Memory.

ROCm presents as CUDA​

On AMD + Linux with a ROCm build of PyTorch, torch.cuda.is_available() returns True because ROCm exposes a CUDA-compatible interface. Settings and the graph's device select label the device AMD ROCm; the value stored in the graph and used by the run is cuda (cuda:N per card), and a node's device parameter lists it as cuda. That is expected.

Intel GPUs (XPU)​

cdui install --gpu xpu installs PyTorch's Intel GPU build, and torch.xpu works in that venv. The run's device controls do not offer xpu yet: the device options are cpu, auto, cuda (or cuda:N) and mps, auto resolves to CUDA, then MPS, then CPU, a run whose device is xpu is refused as an invalid device, and a node parameter set to xpu falls back to the CPU with a warning in the log. Runs on a machine with the xpu build therefore execute on the CPU.

Experimental: native MLX (spike)​

There is a proof-of-concept that ports a small MLP's forward inference from PyTorch to Apple's MLX framework, producing numerically identical results (max abs difference ~1.9e-7). Key points:

  • Apple acceleration in the real graph engine is PyTorch MPS, which is wired up and verified end-to-end. MLX is not a shipped execution backend.

  • MLX is a distinct array framework, not a PyTorch backend — there is no torch.device("mlx") — so it can't be a value in any of the device dropdowns (they drive torch).

  • The spike is inference-only and float32, runnable ad-hoc:

    uv pip install mlx        # Apple Silicon only
    python scripts/mlx_spike.py

  • mlx is not a committed dependency; the main app never imports it. Surface it only via device_utils.mlx_available() (detection) and the spike script.

Recommendation: keep MPS as the Apple default for all execution (training + inference); treat MLX as an optional inference accelerator to revisit only if there's a measured win for inference-heavy teaching demos.