Generative AI: The 2026 Field Guide

Everything we have discussed so far applies to images. Video is much harder, and the reasons it is harder are worth understanding because they explain why video models lag behind image models in quality, why they cost so much more to train and run, and why some labs have made the architectural choices they have.

Why video is hard

A video is not just a sequence of independent images. If you take a state-of-the-art image model and use it to generate a hundred frames one at a time, you get visible flicker. Textures shimmer. Objects subtly reshape between frames. Lighting drifts. A character's eye color might change halfway through the clipCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward.. The model has no memory of what the previous frame looked like, so it has no way to ensure consistency.

To generate video, you need a model that understands time as well as space. It needs to know that a tree in frame fifty should be the same tree in frame fifty-one, just possibly with leaves rustling slightly. It needs to know that physics applies, water flows downward, balls roll, faces do not abruptly change shape. It needs to handle motion in a way that looks natural rather than like a series of cross-faded images. None of these properties emerge from simply running an image model many times.

The technical name for the property you want is 'temporal coherenceTemporal coherenceThe property of video where elements remain consistent across frames. The central challenge of video generation.,' and getting it right is the central challenge of video generation. There are roughly three ways to attack the problem, and modern video models fall into combinations of all three.

Approach one: bolt temporal layers onto an image model

The simplest approach is to take a pretrained image model and add new layers that specifically look across time. You leave the existing image processing alone, and you interleave attention layers that, at each spatial position, attend to the same position in nearby frames. This is much cheaper than training a video model from scratch, because you inherit all the visual knowledge of the image model and only have to learn the temporal dynamics.

Early Runway models worked this way. So did the first version of Stability AI's video model, called Stable Video Diffusion. The results were okay, coherent enough to be recognizable as video, but not great. The motion was often stiff, characters were unstable, complex scenes broke down. Bolting time onto a model that was not designed for it was always going to have a ceiling.

Approach two: full spatiotemporal attention

The second approach is to design the model from scratch with the assumption that it is processing video. The latents are now three-dimensional volumes, width, height, and time, and the transformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks.'s attention layers can look across all three dimensions at once. Every patch in every frame can in principle attend to every patch in every other frame. This is enormously expensive (the compute scales quadratically with the number of patches, and the number of patches scales linearly with the number of frames), but it produces dramatically better results.

Fig.diagram

■Full spatiotemporal attention. Inside a video transformer every patch attends across its own frame in space and across every other frame in time, so a tree in frame fifty stays the same tree in frame one. This is what Sora, Veo, and Kling all do.

This is what Sora does. It is what Veo does. It is what Kling does. It is what every major frontier video model from 2024 onward does, with various tricks to make the quadratic cost manageable. The reason these models can produce coherent, physically plausible motion is that the transformer is genuinely reasoning about time as a first-class dimension, not treating it as an afterthought.

Approach three: 3D VAEs and the compression trick

Even with full spatiotemporal attentionSpatiotemporal attentionAttention that operates across both spatial dimensions (within frames) and temporal dimensions (across frames). Essential for high-quality video generation., video latents are huge. A five-second clipCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward. at 720p is something like twenty thousand patches per second times five seconds, a hundred thousand patches, all attending to each other. That is a vast amount of compute. To make this tractable, modern video models use a 3D VAE3D VAEA VAE that compresses both spatial and temporal dimensions of video, used to make video diffusion tractable. that compresses across both space and time.

Fig.diagram

■The 3D VAE compresses video across both space and time. The quality of your VAE puts a hard ceiling on the quality of your final video outputs.

Recall that an image VAEVAE (Variational Autoencoder)A neural network that learns to compress images into a small numerical representation (a latent) and decompress them back. Used in latent diffusion models to make the generation process tractable. compresses spatial dimensions by a factor of eight in each direction, so a 1024x1024 image becomes a 128x128 latent. A 3D video VAE additionally compresses across time. A common ratio is four times temporally and eight times spatially, so a hundred-frame 720p video becomes a twenty-five-frame 90x160 latent. That is a forty-eight-fold reduction in the volume of data the transformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks. has to process. Without this trick, modern video generation would not be economically viable at all.

ANALOGY

Imagine you are filming a slow-motion shot of a flower opening. You have a thousand frames of high-resolution footage. Most of those frames are nearly identical to the frames next to them, the petal moves a fraction of a millimeter between frame 412 and frame 413. A naive video model would process every frame as if it were brand new information. A 3D VAE notices that consecutive frames are highly redundant in time the same way that adjacent pixels are highly redundant in space, and it compresses both dimensions at once. Instead of a thousand high-res frames, you get something like two hundred fifty low-res keyframe-summaries, each one capturing the meaningful change since the last summary. The transformer never sees the original thousand frames, it sees the compressed two hundred fifty, does its expensive math there, and then at the very end the VAE decoder reconstructs the full thousand-frame video. This is the same compress-edit-decompress pattern as image latent diffusion, just with one more dimension. It is also the dirtiest secret of modern video generation, the quality of the VAE puts a hard ceiling on the quality of the final video, no matter how good the diffusion model on top of it is.

Different labs have developed their own custom 3D VAEs and treat them as a competitive advantage. Kuaishou's Kling has a proprietary 3D VAE that they describe as performing 'synchronous spatiotemporal compression.' Alibaba's Wan 2.2 has a 3D VAE that achieves a 4x16x16 compression ratio, the most aggressive in any open-source model. ByteDance's Seedance uses what they call a 'time-causal VAE', meaning the encoding of any given frame can only depend on previous frames, never future ones, which is useful for streaming applications.

Audio: the newest frontier

Until the second half of 2025, every major video generation model produced silent video. You could add audio in post-production, but the audio was not generated jointly with the visuals, so things like lip sync and sound effects synchronized to actions were difficult. In the second half of 2025, two models broke this barrier: Google's Veo 3 and OpenAI's Sora 2. Both generate audio and video together, in the same diffusion process, by treating audio as another modality in the latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run.. The model's training data includes videos with their sound tracks, and the model learns the joint distribution.

The result is that Veo 3 and Sora 2 produce videos where the audio is naturally tied to what is happening visually. Footsteps match the rhythm of walking. Dialogue lip-syncs to the character's mouth. The crash of a wave is timed to the visual moment of impact. This is a meaningful upgrade for narrative content, and it is one of the reasons these two models stand out in the current generation. Most other video models still treat audio as a separate, post-hoc step, Krea, for example, has a feature where after you generate a silent video, a separate model attempts to add matching audio. This is a clever workaround, but it is not the same thing as joint generation.