How audio generation fits, in brief

Everything so far has been about pixels. The third medium this book treats as first-class, audio, runs on the same core ideas with three twists worth holding in mind before the models arrive later on. The mechanics get a full chapter of their own in the audio part, this is only the one-page orientation.

First, audio has its own compressor. Where an image model works in the compressed latent spaceLatent spaceThe compressed numeric representation an image is turned into before diffusion. Working here instead of on raw pixels is what makes modern image models fast enough to run. of a variational autoencoder, an audio model works on tokens produced by a neural codec, which turns a waveform into a short stack of discrete codes and back. The codec is the audio equivalent of the VAEVAE (Variational Autoencoder)A neural network that learns to compress images into a small numerical representation (a latent) and decompress them back. Used in latent diffusion models to make the generation process tractable., and just as image quality is capped by the VAE, audio quality is capped by the codec.

Second, there are two main ways to make a voice. One predicts audio tokens one after another, the way a language model predicts words, the lineage that runs from VALL-EVALL-EA Microsoft speech model that treats text-to-speech as predicting audio codec tokens with a language model, cloning a voice from a roughly three second sample. Its lineage defines the token-prediction approach to speech.. The other sharpens a whole clipCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward. out of noise using the same flow-matching idea behind modern image models, the lineage of VoiceboxVoiceboxMeta's text-to-speech model built on flow matching, which sharpens a whole clip out of noise rather than predicting tokens one by one. and F5-TTS. Music is harder than speech, because it has to stay coherent across minutes and carry structure, melody, and lyrics at the same time.

Third, and most consequential for the industry, sound is increasingly generated jointly with video. The video models now produce synchronized dialogue, effects, and ambience in the same pass as the picture, which reshapes where the standalone audio companies can win. The full treatment, including the figures for the codec and the two speech families, is in the audio part.