ComfyUI: images, video and 3D on your own cards

Level IntermediateEstimated time 1 hRead first Docker with access to your GPU, Several models, two graphics cards: a grid instead of a juggling actA word you don't know? Glossary

ComfyUI is an open-source editor where you wire generative models as nodes: text encoder, diffusion model, decoder, output. It draws images, makes video clips, turns a picture into a 3D object. Ours makes the mascot's renders and the shots of an automatic video pipeline, on the same cards that serve language models at night.

What you get. Your own image, video and 3D studio, on the cards you already have, ordered by your agents: a five-second video shot in about two minutes, no subscription, no upload.

1. Install it as a service, listening on the machine only

git clone https://github.com/comfyanonymous/ComfyUI.git ~/comfyui
cd ~/comfyui && python -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python main.py --listen 127.0.0.1 --port 8188

Then make it a user service, so it restarts by itself. Keep --listen 127.0.0.1: ComfyUI has no login, and its custom nodes run arbitrary code. Reach it from your laptop through an SSH tunnel, never by opening the port.

ssh -L 8188:127.0.0.1:8188 server    # then open http://localhost:8188

2. One instance per card

With two cards, run two instances, each pinned to its card and writing to its own folder:

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 \
  .venv/bin/python main.py --listen 127.0.0.1 --port 8189 --output-directory ~/comfyui-b/output

Without CUDA_DEVICE_ORDER=PCI_BUS_ID, "card 1" for CUDA may not be "card 1" for nvidia-smi. Set it everywhere, or you will load a model on the card you meant to keep free.

3. Which model for what

Need	Model family	Note
Photo-like images, fast	SDXL (Juggernaut, RealVis), Turbo variants	fits in 8 GB
Images that follow the prompt closely	Flux dev, quantized (GGUF Q8)	16 GB with an fp8 text encoder
Video from text or an image	Wan 2.2 14B (two experts), LTX-Video 2B	a 4-step LoRA divides the time
An object in 3D from a picture	Trellis 2	shape and texture decoders

Measured on one of our cards: a 5-second Wan 2.2 shot (81 frames at 16 fps) takes about 2 minutes in fast mode and about 13 minutes at 20 steps. Plan a video in shots, not in minutes of film.

4. Drive it from scripts and agents

Save a workflow in API format; a script posts it to /prompt with the prompt and seed changed, then polls the history for the output files. That is how an agent orders a render: it fills in a few fields, the workflow stays fixed and tested.

Traps

ComfyUI and a language model cannot share a card comfortably. Decide who owns which card when (see the GPU grid), and stop ComfyUI's instance rather than let both fight for memory.

Install custom nodes as you would install any program from the internet: few, from known authors, read before running. They execute with your rights.

Sources

ComfyUI, official repository

Found this useful? Tell us what to measure next.