Video and audio in the graph

ComfyUIComfyUIThe dominant node-based interface for running diffusion models locally. Started by 'comfyanonymous' in January 2023, now stewarded by Comfy Org. stopped being an image tool a while ago. The current video models are supported natively, meaning you update the app and they work, no third-party nodeNodeA single block in a visual pipeline like ComfyUI that does one operation and wires to others to form a workflow. required. Alibaba's Wan (versions 2.1 and 2.2, including a five-billion-parameterParameterOne individual weight. Model size is quoted in parameters, in the millions or billions. More parameters can mean more capability and more cost. hybrid that does both text-to-videoText-to-videoGenerating video from text alone, with no starting image. and image-to-videoImage-to-videoStarting from a still image and generating a video that animates it according to a prompt., and a smaller variant that runs in about eight gigabytes of VRAMVRAM (Video RAM)The memory on a GPU. It sets a hard ceiling on the model size and image resolution you can run locally.), Tencent's Hunyuan Video, Genmo's Mochi, and Lightricks' LTX line are all first-party. LTX-2 is the notable one: a nineteen-billion-parameter audio-video model that generates motion, dialogue, sound effects, and music together in a single pass, which is the native-audio shift the foundations described, now arriving inside a node graphNode graphA visual pipeline where each box performs one operation and wires to others, the interface ComfyUI made standard..

Video changes two things about the graph. The VAEVAE (Variational Autoencoder)A neural network that learns to compress images into a small numerical representation (a latent) and decompress them back. Used in latent diffusion models to make the generation process tractable. is three-dimensional, compressing across time as well as space, and the empty-latent node carries a frame-count length parameter that image latents do not; set that length to one on a video model and it produces a still image. Audio is its own small world now too. Music and sound models such as ACE-Step and Stable Audio run as native nodes, and Wan ships a sound-to-video model that animates from an audio track through a dedicated audio encoder. The one thing to know about the old workhorse, AnimateDiffAnimateDiffAn early, widely used community extension that adds motion to image models to produce short video, notable as a custom node rather than a natively supported model., is that it is a community custom nodeCustom nodeA community-written extension that adds new operations to ComfyUI. Because it is arbitrary Python running on your machine, custom nodes are powerful and a genuine security consideration., not a native model, which is exactly the distinction the next section is about.