For most of this field's short history, generative AI meant pixels. Sound was the afterthought, added later if at all. That is over. In filmmaking especially, audio is arriving as the thing that was missing: a silent clip is a demo, but a clip with the right voice, a score that fits, and footsteps that land on the right frame is a scene. This part treats audio as a first-class citizen next to image and video, and it splits the field into the three tracks that actually behave differently. Voice is the human-performance layer, the one that matters most for narrative and the one changing fastest. Sound is the effects and ambience that make a space feel real. And music is the hardest of the three. We start with how audio generation works, then profile the companies the same way we did for the image and video labs.
The good news is that audio generation runs on the same ideas as image generation, adapted to a signal that moves in time. If you followed the VAE and the diffusion loop in the foundations, most of this will feel familiar. The pieces just have different names, and one of them, timing, is a great deal less forgiving than it is in a still image.
The codec is the audio VAE
Raw audio is a firehose of numbers. A CD samples sound about 44,100 times a second, so even a few seconds is hundreds of thousands of values, and a model can no more read that directly than it can read a photo pixel by pixel. The fix is a neural audio codec: a learned compressor that squeezes a waveform down into a short sequence of discrete symbols, called tokens, and can expand those symbols back into sound that still resembles the original. Turn a continuous stream of water into a manageable string of numbered beads, and now a model can predict the next bead the way a language model predicts the next word. This is exactly the job the VAE does for image models, and it carries the same consequence: your audio quality is capped by how good the codec is, just as image quality is capped by the VAE.
The trick that makes these codecs work is residual vector quantization. Rather than mapping each slice of sound to one entry in one codebook, the model quantizes, measures the leftover error, quantizes that residual with a second codebook, and repeats across several codebooks, coarse to fine. That yields high fidelity at a low bitrate and lets you trade quality for size by using fewer codebooks. Google's SoundStream introduced the approach in 2021, Meta's EnCodec refined it and became the tokenizer under a wave of speech and music models, the Descript Audio Codec pushed fidelity further, and Kyutai's streaming Mimi codec cut latency to the point where live conversation is possible.
Two ways to make a voice, and how cloning works
Old text-to-speech was either concatenative, gluing together tiny recorded snippets of a voice actor, which showed its seams, or parametric, describing the voice with a few statistical knobs, which sounded smooth but robotic. Neural TTS threw both out and learns the mapping from text to sound directly from data. Two modern recipes now dominate, and they mirror the split you already know from images. The first treats speech like a language: convert the target voice and the text into audio tokens, then predict those tokens one after another, exactly the way a chatbot predicts words. This is the VALL-E and AudioLM lineage, and it gets voice cloning almost for free, because a few seconds of reference audio simply sits in the model's context window. The second treats speech like an image being sharpened out of noise: start from static and, guided by the text, denoise it into a clean waveform in a few steps. This is the flow-matching lineage, Voicebox and F5-TTS, and it tends to be faster and less prone to the rambling and repetition that the token approach can fall into.
Voice cloning is the capability that changed everything, and it is simpler than it sounds. Older systems trained a separate network to summarize a voice into a fixed fingerprint. Modern zero-shot systems skip that: you hand the model three to ten seconds of someone speaking, it reads that reference in context, and it then says anything in that voice. That is why a clone can be built from almost nothing, and it is also, unavoidably, why convincing voice deepfakes are now trivial. The same mechanism that lets a filmmaker give a character a consistent voice lets a bad actor impersonate a real person, and watermarking and detection remain an unsolved arms race rather than a fix.
Why music is harder than speech
Music is speech's harder cousin, for reasons worth naming because they explain why the music companies lag the voice ones on control. Speech has one voice and a clear target: say these words clearly. Music stacks a melody, chords, drums, bass, and often a singer all at once, and every layer has to stay in tune, in time, and in a consistent key and style for minutes rather than seconds. Our ears are also ruthless about music in a way they are not about speech: a single off-key note or a drummer drifting out of time is instantly obvious, while slightly odd speech still passes. Add higher sample rates, stereo, the joint problem of singing intelligible lyrics over accurate instrumentation, and the fact that musical quality is aesthetic and hard to measure, and you have a much harder problem.
The two approaches are the same two families again. Token models like Google's MusicLM and Meta's MusicGen compress audio into tokens and predict them in order, which naturally produces vocals and instruments together because the model just learns whatever sound was in the training audio. That entanglement is also why clean stem separation and precise editing are hard for these systems. Diffusion models like Stable Audio sculpt the whole clip out of noise at once, which is faster and strong on texture, stereo, and length. The commercial leaders, Suno and Udio, do not disclose their exact recipes, but they clearly output full mixes with sung vocals from a prompt, and the honest read is that the internals beyond that are trade secrets.
Sound effects, and audio that is born inside video
Sound effects generation is the audio version of text-to-image: describe a dog barking, rain on a roof, a spaceship whoosh, and the model produces it, using the same toolkit of tokens or diffusion. The catch is timing. Words are a loose way to describe sound, and models are weak at ordering, so asking for a dog barking and then a bird chirping can get you the two overlapping or in the wrong order, because the model captures which sounds are present better than exactly when they happen.
The bigger shift is native audio in video. Until 2025, AI video was silent film: you generated the picture, then bolted sound on afterward and tried to line it up. Native audio means the model produces picture and sound together in one pass, so footsteps land when the foot hits the floor, the dialogue matches the lips, and the room tone matches the room. This is hard for a blunt reason: sound and vision have to agree at millisecond precision, and lip-sync is the cruelest case because humans are expert lip-readers who notice the tiniest mismatch. Google's Veo 3 was the first major model to ship it, Sora 2 followed, and by 2026 the Chinese labs had joined, which is why so much of the standalone sound-effects and voiceover market is now under pressure for short-form work, a dynamic we return to later in this part.