Training Memory

Most training runs do not stop because the maths was wrong. They stop because the card ran out of room. This page covers the four things CodefyUI gives you to push that ceiling up, and the one thing it deliberately will not do for you.

Everything here is off by default. A graph saved before any of it existed trains exactly as it did before.

Mixed precision

TrainingLoop has a precision parameter under Advanced:

Value	What it does	When to use it
`fp32`	Nothing. Full 32-bit throughout.	The default. Always correct, always the most memory.
`bf16`	Runs the forward pass and the loss under `autocast(torch.bfloat16)`. No gradient scaler.	The one to reach for. Ampere and newer — RTX 30xx, 40xx, 50xx, A100, H100.
`fp16`	Same, in `float16`, plus a `GradScaler`.	Cards without bfloat16 — Volta and Turing, i.e. GTX 16xx and RTX 20xx.

Activations, not weights, are what fills a card on a deep model, and autocast halves them. Parameters stay in float32 either way, so a model whose memory is mostly weights sees a smaller win than one whose memory is mostly feature maps.

Why bf16 and not fp16 on a modern card. bfloat16 keeps float32's exponent range and spends its bits on mantissa instead. Small gradients therefore do not underflow, there is nothing for a loss scaler to do, and there are no skipped steps. float16 has five exponent bits, so it needs a scaler that multiplies the loss before backward() and divides the factor back out before the step — and backs off, skipping that step entirely, whenever a gradient overflows.

What each device can honour. The choice is resolved against the device before the run starts, and a device that cannot honour it falls back to fp32 with a warning rather than failing:

CUDA without bfloat16 support asks for bf16 and gets fp32.
MPS gets fp32 for anything but fp32. Both 16-bit modes run on torch 2.11 but are slower: on an M3, ResNet-18 on CIFAR measured 17 ms/step in fp32, 30 ms in bf16 and 37 ms in fp16, with no change in allocator memory. MPS kernels are float32-native; half precision adds a conversion around each op.
CPU honours all three. Neither 16-bit mode is fast on a CPU — this is about being able to run the lesson on the machine in front of you.

The node's config frame and its metrics output both report precision (what ran) and, when they differ, precision_requested (what you asked for). Validation runs under the same autocast as training, so the two loss curves stay comparable.

Resuming an fp16 run

The loss scale is part of the training state. CheckpointSaver and CheckpointLoader carry it in a grad_scaler_state port, and TrainingLoop has one on each side:

TrainingLoop.grad_scaler_state  →  CheckpointSaver.grad_scaler_state
CheckpointLoader.grad_scaler_state  →  TrainingLoop.grad_scaler_state

Leave both unconnected for fp32 and bf16 — there is no scale to carry. Losing the state is survivable rather than fatal: a fresh scaler re-finds its level within a few hundred steps, and the steps in between are taken at the wrong scale.

Gradient accumulation

accumulate_steps (also under Advanced) runs N batches, divides each one's loss by N, and steps the optimizer once.

The gradient that reaches the optimizer is then the gradient of one batch N times larger — exactly, not approximately, when the loss is a mean and every micro-batch is full. So:

batch_size 8 with accumulate_steps 4 is batch_size 32, at a quarter of the activation memory.

That is a tested claim, not a description: the test suite computes the full-batch gradient by hand and requires the accumulated run to land on the same weights.

Three interactions worth knowing:

Gradient clipping happens at step time, over the whole accumulated gradient. Clipping each micro-batch instead would let N clipped gradients sum to N times the threshold.
max_steps counts optimizer steps, not batches, so it means the same amount of learning at any accumulate_steps. The metrics output reports both: total_steps (optimizer steps) and total_batches (forward passes).
The reported loss is undivided. The /N is a detail of how the gradient is assembled and never reaches the chart, so the same run at accumulate_steps 1 and 4 draws the same curve.

An epoch whose batch count is not a multiple of accumulate_steps ends with a short window, and that window is still stepped — those batches already paid for their forward and backward passes. It is divided by N like every other, so a short window takes a proportionally smaller step, which is the honest treatment of a smaller sample.

An accumulation window never spans an epoch boundary, and a Stop discards the pending window rather than taking one more step on the way out.

Picking a specific GPU

On a machine with more than one CUDA device, every device dropdown — the selector in Settings, the graph's own device control, and each node's device parameter — lists the cards individually:

CPU
NVIDIA CUDA · 20.1 / 24.0 GiB free                     (whichever card torch is currently pointed at)
NVIDIA CUDA #0 · cuda:0 · RTX 4090 · 20.1 / 24.0 GiB free
NVIDIA CUDA #1 · cuda:1 · RTX 3060 · 11.4 / 12.0 GiB free

In the Settings and graph toolbar selects, each accelerator ends with its free and total memory as the server read it when the page loaded (NVML for NVIDIA cards, PyTorch otherwise); it is a reading for orientation, nothing refuses a run because of it, and it is left out when the server cannot read the figure.

A single-GPU machine shows only NVIDIA CUDA, because there cuda and cuda:0 are the same piece of hardware and offering both would be a choice with nothing behind it.

Each card gets its own run queue, so a six-hour job on cuda:0 never delays a run submitted to cuda:1. See Run Queue.

An index that does not exist — cuda:3 on a two-card box, or a graph saved on a workstation and opened on a laptop — falls back to the current CUDA device, with a warning naming the count. Not to the CPU: you asked to train on a GPU, and answering that with a silent forty-minute CPU run is the worse surprise.

A device string that is not addressable at all — cuda:, cuda:abc (an unexpanded ${CUDA_INDEX}, say), cuda:0:1, cuda: 0 — lands in the same place, for the same reason. The cuda prefix is still an unambiguous request for a GPU; only the index is wrong. Before this, those strings reached PyTorch verbatim and came back as RuntimeError: Invalid device string, which named neither the graph nor the parameter they came from.

Several GPUs

A run that executes in the server process uses one card. To train one model on several cards, set TrainingLoop's strategy (under Advanced) to ddp and queue the graph on a worker with a device set of two or more CUDA cards: the loop then trains with PyTorch's DistributedDataParallel, one process per card. The how-to, what each process does, and where ddp falls back to one process are in Multi-GPU Execution, Train one model on several cards.

What it does to memory:

Every card holds the whole model, its gradients and its optimizer state, plus the activations of one batch at the DataLoader's batch_size, which is per card. DDP adds cards; it does not split a model that does not fit on one. A model too large for one card needs the levers above (bf16, a smaller batch with accumulate_steps), and there is no sharded (FSDP) or model-parallel strategy.
The effective batch grows with the cards. Each optimizer step averages the gradients of every card's batch, so two cards at batch_size 32 step like one card at 64, and accumulate_steps multiplies on top. The config frame reports effective_batch_size = batch_size × accumulate_steps × cards.
A memory budget applies per card. On a card shared by memory, each process is capped to its own card's budget, and to nothing on the other budgeted cards of the set, so a node pinned to another card fails at its first allocation there instead of taking memory outside the budget. Peak memory reports the most any process used of each card.

There is no multi-node training: the cards are those of one worker.

When the card runs out anyway

An out-of-memory error on CUDA, on MPS or on the CPU is reported as a NodeOOMError: which node, which device, what the allocator was holding at the time, and what to change. On CUDA it reads roughly like this:

Node TrainingLoop (n7) ran out of memory on cuda:0.

What to try, cheapest first:
  - reduce batch_size on the DataLoader node
  - set TrainingLoop's precision to bf16 (roughly halves activation memory on Ampere and newer)
  - raise TrainingLoop's accumulate_steps and lower batch_size by the same factor, which keeps the effective batch identical
  - make the model smaller, or shorten the sequence / shrink the image

CUDA memory on cuda:0: 14.82 GiB held by live tensors, 15.44 GiB reserved by
the caching allocator, 15.61 GiB peak this process, 15.99 GiB on the card.

Original error: CUDA out of memory. Tried to allocate 2.00 GiB ...

On MPS the memory line gives what live tensors hold, what the allocator has reserved, and the recommended working-set ceiling for the process (unified memory, shared with everything else on the Mac) instead of the card's size. A CPU MemoryError gets the same message without a memory line.

Alongside the message, two things happen so the next run starts from a clean card: whatever that node had cached is dropped, and on CUDA and MPS the caching allocator's free blocks are handed back.

The run is not retried, and the batch size is not reduced for you. Re-running the same allocation gets the same answer. Halving the batch behind your back would change the numbers your run produces without telling you, so the same graph would mean two different things depending on how much VRAM happened to be free. You get the message and you make the change.

The server's own memory

Everything in this section is what the server is holding in memory because you ran something. Your saved graphs, your project files and your model_weights.pt are on disk and are not counted here — restarting the server empties these stores and loses none of that.

Three in-memory stores hold tensors between runs, and all three are bounded by bytes as well as by count:

Store	What it holds	Budget	Setting
Execution cache	Node outputs, for "edit one node, only that subtree re-runs"	1 GB per open editor connection	`CODEFYUI_EXECUTION_CACHE_MAX_MB`
Run output store	Captured port values for the Teaching Inspector	2 GB	`CODEFYUI_RUN_OUTPUT_STORE_MAX_MB`
Node state store	Persistent `nn.Module` weights per node	1 GB	`CODEFYUI_NODE_STATE_STORE_MAX_MB`

Set any of them to 0 to disable the byte budget and leave the older count limit in charge.

Counts alone were never a memory limit: 256 cached node outputs is either 40 MB of scalars or 200 GB of feature maps, and a count cannot tell the two apart. Eviction is least-recently-used, and sizes are measured over tensor storage — recursively through lists, tuples and dicts, counting a tensor and a view of it once rather than twice.

GET /api/health reports what each store currently holds against its budget:

{
  "status": "ok",
  "caches": {
    "execution_cache": {"instances": 2, "entries": 41, "bytes": 918212608, "max_bytes_each": 1073741824},
    "run_output_store": {"runs": 6, "max_runs": 20, "bytes": 244318208, "max_bytes": 2147483648},
    "node_state_store": {"modules": 12, "max_modules": 200, "bytes": 51380224, "max_bytes": 1073741824}
  }
}

Two caveats on the numbers, since a memory report that quietly rounds is worse than none. Module sizes are measured when the module is built, so the gradients a module grows while training are not re-counted. And CPU and CUDA bytes are added together — the budget compares one number, and this is that number.

Mixed precision​

Resuming an fp16 run​

Gradient accumulation​

Picking a specific GPU​

Several GPUs​

When the card runs out anyway​

The server's own memory​