Running Graphs

When you click Run, the frontend sends the graph to the backend over a WebSocket (ws://host/ws/execution) and the backend streams results back as each node completes.

Real-time execution

• The backend validates the graph (DAG check, type safety, at least one Start node), topologically sorts it (Kahn's algorithm, with cycle detection), and runs independent nodes in parallel.
• Each node reports status as it goes: running → completed (or error), with a small output summary embedded inline for quick viewing.
• The Execution Log tab lists each node's outcome and the text nodes write; see The results panel.

A node without a trigger can still run

Removing a node's trigger edge does not, by itself, take it out of the run. If a data edge still connects its output into something that does run — whether that's a required input or an optional one makes no difference — the node runs too, with or without a trigger of its own. A Dataset or the first node of a transform chain typically has no trigger at all and is expected to run this way; the same rule applies to anything else you have wired in.

The practical effect: disconnecting the trigger edge alone no longer parks a branch. If you want a node to stay wired for later but not run right now, disconnect its data edge(s) instead — that is what actually removes it from the run. Bypass (right-click a node, or Ctrl/Cmd+B) is a one-click alternative for a node partway through a chain, skipping it while passing its input straight through to whatever it fed — but it only works when the node has an input of the same type as its output to forward, so it is refused on a source node with no inputs at all (CSVReader, ImageReader, Dataset, and the rest of the file-reading nodes). For one of those, disconnecting the edge is the only way to park it.

One consequence worth knowing: a reader node (CSVReader, ImageReader, and the like) left wired to an input — even an optional one — but never triggered now executes where it used to be silently skipped. If the file it points at has since been deleted or moved, a graph that previously ran without error can start failing with FileNotFoundError on that node.

Training loops and loss charts

The TrainingLoop node emits progress events during training. The Training tab of the results panel follows them live: the current Epoch, the latest and Best training loss, a progress bar, a Loss Curve of the training loss per epoch, the loop's Config, and a per-epoch table with each epoch's change and duration. When a val_dataloader is wired, the curve also draws the validation loss, labelled train_loss and val_loss, and the table gains a Val loss column.

Every value the loop records is also stored with the run as a named series. A run's detail in the Runs panel charts all of them and downloads them as CSV, and a sweep ranks its variants by one of them (objective.metric):

Series	Recorded	Step axis
train_loss, lr	Every epoch. lr is the learning rate the epoch started with, also when scheduler_step is optimizer_step.	Epoch number; a run resumed through start_epoch continues the numbering.
val_loss	Every epoch, when val_dataloader is wired.	Epoch
val_accuracy	Every epoch, when val_dataloader is wired and loss_fn is CrossEntropyLoss or NLLLoss.	Epoch
patience_counter, best_epoch	Every epoch, when early stopping is on.	Epoch
train_loss_batch	Every log_interval-th batch, with batch_metrics on.	Batch, counted across epochs
grad_norm, plus grad_norm_clipped when grad_clip_norm is set	Every log_interval-th optimizer step, with log_grad_norm on. The norm is measured before clipping.	Optimizer step
update_ratio	Every log_interval-th optimizer step, with log_update_ratio on: ‖lr × grad‖ / ‖weights‖.	Optimizer step
val_loss_step	Every val_every_steps optimizer steps, when val_dataloader is wired.	Optimizer step

The config panel shows log_interval (Advanced) only while batch_metrics is on, but the grad_norm and update_ratio rows above follow its value either way.

EvaluateModel adds one eval_accuracy point when it runs. The classical and RL evaluation nodes record theirs the same way, so a sweep can rank those graphs too: Accuracy records accuracy; RegressionMetrics records mse, mae and r2; PolicyRollout records success_rate and mean_return (only mean_return for an env that names no episode outcome, such as a Gymnasium env, which has no goal); RLTrainingLoop records episode_return and mean_return (the mean of the last 20 episodes) as it trains, and solved_at_episode once it reaches target_return; and BradleyTerryTrain records train_loss, train_accuracy and holdout_accuracy once per epoch. When two nodes in one graph record the same name, a sweep objective on that name is ambiguous; name the node with objective.node_id.

The loop's other options:

• Early stopping. early_stopping_patience, when above 0, stops training after that many epochs without improvement in monitor: val_loss, lower is better (the training loss when no val_dataloader is wired), or val_accuracy, higher is better (when it is not recorded, the loop warns and watches val_loss). With early stopping on, the model output holds the best epoch's weights rather than the last epoch's, except after Stop.
• Periodic checkpoints. checkpoint_every saves a checkpoint every N epochs, and checkpoint_every_steps every N optimizer steps (0, the default, turns either off), to models/periodic/ in the data folder. The files are listed with the run's artifacts in the Runs panel, and retention deletes them with their run. Each one is roughly the model plus its optimizer state, often several times the model's size, and nothing limits how many a run writes while it is still going. An exported script and the CLI Graph Runner write none. Resume from a checkpoint_every file the same way as from any checkpoint: Stopping and resuming. A checkpoint_every_steps file stores its optimizer-step count where the epoch belongs, so a resume through start_epoch reads that count as completed epochs and, when it is at or past epochs, trains nothing.
• Step-based schedules. scheduler_step (Advanced) sets when a wired LRScheduler advances: after every epoch (the default) or after every optimizer_step (ReduceLROnPlateau stays per epoch). It also sets the unit of the scheduler's lengths (step_size, T_max, total_steps). LRScheduler's warmup_cosine, warmup_linear and constant_with_warmup raise the rate from near zero over warmup_steps (default 100), then decay it with a cosine, decay it linearly or hold it, over total_steps in all; use them with optimizer_step, since per epoch 100 warmup steps are 100 epochs. max_steps (Advanced), when above 0, ends training after that many optimizer steps, whatever epochs says. ReduceLROnPlateau follows the validation loss (the training loss without a val_dataloader) at its default mode = min; at mode = max it follows val_accuracy instead, and the run stops before the first batch if there is no val_dataloader or the loss is not a classification loss. See Gotchas worth knowing for matching a schedule's length to the run.

The optimizer output is the optimizer that actually trained: the one wired in, or a rebuilt one when that one's parameters did not match the model. Wire it, not the Optimizer node, into CheckpointSaver.optimizer, and wire the epoch output (complete epochs, early stopping and max_steps included) into CheckpointSaver.epoch so the checkpoint resumes where training stopped. The memory options precision and accumulate_steps are covered in Training Memory, tensorboard in TensorBoard, and strategy (Advanced: single, the default, or ddp, which trains one model across the GPUs of a worker run with two or more cards) in Multi-GPU Execution.

Partial re-execution (dirty tracking)

CodefyUI tracks which nodes are dirty. When you change a node's parameters or inputs, only that node and its downstream dependencies are marked for re-execution. Unchanged nodes return cached outputs (shown as cached), so iterating on a single hyperparameter re-runs just the affected part of the graph — a big speedup during development.

Deterministic nodes are cached automatically; non-deterministic ones (training loops, random ops, or any node with cacheable = False) always re-run.

Content-aware caching for file-reading nodes

A cache entry is keyed by a hash of the node's type, its parameters, its upstream nodes' cache keys, the run device and, on a seeded run, the seed the node runs under. Anything a node reads from outside the graph is invisible to params alone — a path parameter records where to read, never what is there. A node that reads external state therefore also folds a content fingerprint into its key: the resolved file's size and modification time, plus (for files up to 8 MB) a content hash, so a same-size edit landing inside one filesystem timestamp tick still changes the key. CSVReader, FileReader, ImageReader, ImageBatchReader, Dataset, ImageFolderDataset and DocumentLoader all do this — editing the file (or, for Dataset/ImageFolderDataset, one of the dataset's own files) and clicking Run again gives you the new content; leaving it untouched gets you the cached result instead of a re-read. DocumentLoader fingerprints every file under its directory, subfolders included even with recursive off, or the single uploaded file.

Dataset fingerprints only the directory its own dataset lives in — MNIST/ for MNIST, cifar-10-batches-py/ for CIFAR-10, and so on — not the whole of data_dir. That matters because in a project directory every dataset shares one assets/data/, alongside assets/models/: before this scoping, saving a model or downloading a second dataset invalidated the first one and made it re-read on the next run.

GraphInput with type=image uses the same mechanism on a canvas run: the API path already has the caller's value in params, but a canvas run instead loads the default path from disk, and the fingerprint is what makes editing that image between canvas runs pick up the new pixels.

What is never cached

Some nodes still opt out of caching entirely with cacheable = False, for four different reasons:

External state a fingerprint can't describe. HuggingFaceDataset and KaggleDataset hit the network and, for Kaggle, KAGGLE_* environment credentials — a fingerprint of the local cache directory cannot tell "the remote revision changed" or "the credentials changed" from "nothing changed", so both re-execute on every run. LLMChat reaches a remote model API for the same reason.

The node's purpose is a side effect. ImageWriter, ModelSaver, and CheckpointSaver exist to write a file; a cache hit would return the recorded {"path": ...} without touching disk, which is wrong when the node's whole point is the write (deleting the output and re-running must recreate it). These re-execute every run regardless of what feeds them.

TrainingLoop is in this group too, for the same reason in a different medium: what it produces is not the model handle it returns but the change to the weights that handle points at, plus a metric series per epoch. Neither survives a cache hit — so it re-executes every run, regardless of whether everything feeding it was unchanged. EvaluateModel is here for the narrower half of that: the accuracy is in its outputs, but the point it writes to the accuracy chart is not, so a hit left the chart empty on a run reported as successful. DatasetBatch is here too: it re-reads its samples every run, because on a seeded run reading a batch draws from the dataset's augmentation, and every later reader of the dataset must augment as on a fresh run.

The node hands back a live, mutable object. A model, an optimizer and a learning-rate scheduler are references to things training changes in place. The cache key describes how such an object was built — architecture, hyperparameters, upstream keys — and stops describing it the moment the first batch runs, so replaying the recorded handle hands the next run the object the previous run already trained. ModelLoader, CheckpointLoader, Inference, Optimizer, LRScheduler, DQN, PPO and RewardModel therefore all re-execute every run.

ModelLoader and CheckpointLoader sit in that group despite having a perfectly good content fingerprint, because the fingerprint describes what they read and not what they write: load_state_dict writes into the model you wired in, and a cache hit skips exactly that. It costs less than it sounds — opting out propagates downstream (see below), and whenever their model input is connected it comes from a weight-owning node, which is already non-cacheable, so on the usual wiring they were never being served from cache to begin with.

The node might have a side effect and only its author knows. PythonScript runs code you type on the canvas. code is a cache-keyed parameter, so an edited script re-runs — but an unedited script over unchanged inputs is exactly where a hit happens, and a script can mutate an input tensor or model in place, change process-global torch/numpy state, or use whatever an ANY-typed input port handed it. None of that is visible to the node's type or to any check on its source, so it opts out unconditionally. See the PythonScript node.

The same opt-out covers nodes whose output escapes the cache key for other reasons — GaussianNoise, DDPMSampler, BackwardOnce, DiffusionTrainingLoop, and every node that owns weights (SequentialModel, DiffusionUNet, and every layer node: Linear, Conv2d, LSTM and the rest), whose parameters drift as training proceeds. Map always runs as well: its body is a saved preset that can be edited between runs under the same name, and the body's nodes may log metrics or write files, which a cache hit would skip.

For nodes that own weights, Settings → Training Behavior → Persist weights between runs controls subsequent runs independently of output caching. It is off by default, so every run initialises a new model, the way an exported Python script always does. Turned on, a run keeps the weights it trained and the next run continues from them, and the Execution Log says so after each successful run. Reset all weights now discards stored weights, so the next run initialises a new model. SequentialModel reports which action it took in the Execution Log on every run.

Opting out propagates downstream: every node fed by one of these re-executes too, because a cache key records only the keys of upstream nodes, not their actual outputs. A cached downstream node would otherwise hand back a stale result computed from data that has since changed.

Two Settings switches bypass the cache wholesale. With Verbose internals or Capture gradients on, nothing is served from cache and every node re-executes: a cached output carries no __steps__ trace and no gradient-tracked tensors, so a hit could not produce what was asked for.

What a preset or subgraph box reports

A preset node and a subgraph instance are single boxes on the canvas standing for several nodes underneath, so they report one status for the lot. Which one depends on what those inner nodes actually did:

The box shows	It means
Completed	At least one node inside really executed. A box that was half cache hits still counts as work done.
Cached	Every node inside was a cache hit. Nothing ran — the outputs you see are the ones the previous run produced.
Skipped	Every node inside was passed over because something feeding the box failed.
Error / Interrupted	A node inside failed, or stopped early. Either settles the whole box immediately.

The Cached case is the one worth knowing about: a preset holding a TrainingLoop that reports Cached did not train this run. Before this was distinguished the box said Completed either way, so "did my change actually re-run anything?" had no answer at the preset level. If you want it to genuinely re-run, change something it depends on, or clear the cache.

Text that a node inside the box writes, such as a Print line or a TrainingLoop note, appears in the Execution Log as soon as that node finishes, tagged with the box. Once the box shows Error or Interrupted, later lines from inside it are not shown.

Reproducible runs (seed)

By default a run draws its randomness from whatever entropy PyTorch picks, so two runs of the same graph give slightly different weights, a different shuffle order, and therefore a different loss curve. Set a Random seed in Settings → Training Behavior to make a run repeatable. The seed is saved in the graph file as settings.seed and comes back on Open and Import; a new tab starts with the seed of the tab you were on, and Export Python bakes the seed in as the script's default --seed. Headless runs (cdui run, run_graph.py, the run API and graph-as-a-function) use it too when they name no seed of their own; --no-seed runs the graph unseeded.

With no seed set, the edu pack's FFNLayer and TrainAndEvaluate are the exception: they start from the weights seed 0 gives, the same on every run and in an exported script. With a seed set:

• Every node is seeded from a value derived from (seed, node id), so what a node draws depends on the seed and on its own identity — not on how much randomness the rest of the graph consumed first, and not on the order the engine happened to schedule things in.
• DataLoader gets its own generator, so the epoch shuffle order is fixed too, and each worker process gets its own independent stream.
• The run executes one node at a time, and it does not overlap another run. Seeding writes process-global RNG state, so a second node — or a second run — drawing from it at the same moment moves the numbers. A seeded run therefore waits for the runs already in flight, then runs alone, and anything submitted behind it waits for it. Unseeded runs are unaffected: they still run in parallel with each other, and their nodes still run in parallel.
• Results are cached per seed: changing the seed, or opening a graph saved with a different one, runs every node again instead of serving what the old seed produced.

Two runs of the same graph with the same seed produce bitwise-identical loss curves on CPU. Different seeds produce genuinely different ones.

"Another run" means every graph this server is executing, including a headless POST /api/graph/run/{name} call: it waits while a seeded run is going, and a seeded run waits for it. Headless calls that are not seeded still overlap each other freely. A run executed by a worker agent has a process of its own, so it is alone in that process and neither waits for this server's runs nor holds them up.

The cost is worth stating plainly: a seeded run is slower (about 3-4x on a graph of independent branches, near zero on the usual mostly-linear teaching graph), and it can wait behind a long job even when started from the canvas. Reproducibility is opt-in, and a run that asked for it would rather be late than wrong.

Deterministic algorithms is the other half. It asks PyTorch for kernels that combine those draws the same way every time (torch.use_deterministic_algorithms(True, warn_only=True)). It is warn_only on purpose: an operation with no deterministic implementation prints a warning rather than killing the run, so you get "everything reproducible was made reproducible" instead of a stack trace from inside cuDNN.

The seed is stored with the run and shown in the Runs panel, so an interesting result can always be traced back to the settings that produced it.

From the CLI:

cdui run graph.json --seed 1234 --deterministic

note

A seed fixes the software's randomness. Exact bitwise agreement is promised on CPU; across different GPUs, driver versions or PyTorch builds, floating-point reduction order can still differ.

Stopping

Click Stop to cancel an in-flight run. Stop is the only thing that cancels a run.

Closing the browser tab, navigating away, or losing the connection no longer stops anything: the run lives on the server, not in the socket. When you come back, the tab reconnects to the run it was watching, replays everything you missed into the results panel, and carries on following it live — so a long training job survives a reload, a laptop lid, or a flaky Wi-Fi hop. Watch in the Runs panel attaches the tab to any other run the server owns the same way.

Cancelling is cooperative rather than immediate, because there is no safe way to interrupt arbitrary node code partway through. The long-running nodes check for it every batch, step or item — a training loop stops within one batch and writes an interrupt checkpoint on its way out — and every other node runs to the end of its current call, after which the run stops at the next node boundary. Either way it is recorded as cancelled.

Beyond the browser

• cdui run submits a saved graph to the running server and follows it from the terminal; the run survives the terminal.
• The CLI Graph Runner executes a graph in-process with no server at all.
• Export as Python (Tabs & Persistence) writes a standalone program that delegates to the same node implementations. Secret parameters are blanked in the export; --seed defaults to the seed set on the canvas when the file was exported (--no-seed for fresh entropy); --timeout is soft — the node already running finishes and no further node starts. File parameters (the upload fields of CSVReader, ImageReader, DocumentLoader, TextCorpusDataset, ModelLoader and CheckpointLoader) are written into the script as they stand, so an absolute path such as C:\Users\..., \\server\share\... or /home/... ties the script to the computer it was exported on. The script still downloads, and a warning then names up to three such nodes, each with the blocks and presets it sits in and its path, and counts the rest. A bare file name has no such tie: Relative file paths says where it is looked up. A TrainingLoop with strategy: ddp is exported as it stands too, and the script trains it in one process; the same warning names such loops (Multi-GPU Execution).
• TensorBoard — TrainingLoop.tensorboard writes event files under the run's artifact directory.