The audio frontier: speech, music, and sound catch up

For most of the current wave, audio has trailed image and video. That gap is closing, and the interesting work now clusters around a handful of threads. The through-line is that audio, like text and pixels, is increasingly modeled as a stream of learned tokens, which means progress downstream is capped by how good those tokens are.

The most visible shift is from batch text-to-speechText-to-speech (TTS)Generating spoken audio from written text. Modern neural TTS learns the mapping from text to sound directly from data rather than gluing together recorded snippets., type a sentence, wait, receive a file, to live conversation. Older systems took a full turn of speech, processed it, and replied. Full-duplexFull-duplexA voice system that listens and speaks at the same time, handling interruptions and back-channel sounds the way people actually talk, rather than taking strict turns. models instead listen and speak at the same time, the way people actually talk, handling interruptions and back-channel sounds like mm-hmm. Kyutai's Moshi (arXiv 2410.00037, September 2024) is a representative open example, reporting a theoretical latencyLatencyThe delay between asking for a generation and getting the first result. Distinct from throughput. around 160 milliseconds and leaning on a streaming codec called MimiMimiKyutai's streaming neural audio codec, designed to generate sound incrementally with low enough latency for live conversation. so speech is generated incrementally rather than all at once. Those latency figures are the authors' own.

In parallel, expressive text-to-speech matured into reliable zero-shot voice cloningZero-shot voice cloningReproducing a specific voice from just a few seconds of reference audio with no per-speaker training, because the model reads the sample in context and then speaks new lines in that voice., imitating a speaker from a few seconds of reference audio. The lineage is well marked. VALL-EVALL-EA Microsoft speech model that treats text-to-speech as predicting audio codec tokens with a language model, cloning a voice from a roughly three second sample. Its lineage defines the token-prediction approach to speech. (arXiv 2301.02111, January 2023) framed speech as codec tokens predicted by a language model and cloned a voice from a roughly three-second sample. Meta's VoiceboxVoiceboxMeta's text-to-speech model built on flow matching, which sharpens a whole clip out of noise rather than predicting tokens one by one. (arXiv 2306.15687, June 2023) used flow matching, learning to turn noise into speech along a smooth path, and VALL-E 2 (arXiv 2406.05370) pushed the codec-language-model approach to human-parity claims by its authors. F5-TTS (arXiv 2410.06885, October 2024) is a widely used open successor that combines flow matching with a Diffusion TransformerTransformerThe neural network architecture built on attention that underpins modern language and, increasingly, image and video models. It scales well and handles long-range relationships better than convolutional networks.. The capability is real and increasingly commoditized, which is exactly why watermarkingWatermarkingHiding a detectable signal inside generated media so its origin can later be identified. For audio and AI media it remains an arms race rather than a guarantee. matters.

Music generation is tackling two harder problems, staying coherent over minutes and being editable. YuE (arXiv 2503.08638, March 2025) is an open foundation model for long-form lyrics-to-song generation, with authors reporting up to five minutes while keeping structure and lyrical alignment. MusicGenMusicGenMeta's open music generation model, which compresses audio into tokens and predicts them in order, producing instruments and vocals together.-Stem (arXiv 2501.01757, January 2025) targets control rather than length, generating and editing separate stemsStemsThe separated individual tracks of a piece of music, such as vocals, drums, and bass, which can be edited or remixed on their own. so a section or single instrument can change without regenerating the whole track. A newer direction makes video and sound together: rather than dubbing a silent clipCLIPA text encoder developed by OpenAI in 2021 that learns to align text and images in a shared embedding space. Foundation of most text-to-image models from 2022 onward. afterward, video-to-audioVideo-to-audioGenerating a soundtrack timed to an existing video clip, so effects land on the exact frame of the on-screen action. Also called video-to-sound. models like MMAudio (arXiv 2412.15322, December 2024) generate sound timed to the picture, where frame-accurate timing, not just plausible sound, is the open challenge.

Underneath all of it sits the codec, the component that compresses audio into the tokens generative models predict. SoundStreamSoundStreamGoogle's 2021 neural audio codec, the model that introduced residual vector quantization for audio. (arXiv 2107.03312, 2021) and EnCodecEnCodecMeta's neural audio codec, a refinement of residual vector quantization that became the tokenizer under a wave of speech and music models. (arXiv 2210.13438, 2022) established residual vector quantizationResidual vector quantization (RVQ)The technique behind modern audio codecs, which quantizes a slice of sound, measures the leftover error, quantizes that residual again with another codebook, and repeats across several codebooks, giving high fidelity at a low bitrate., stacking several rounds of quantizationQuantizationShrinking a model by storing its weights at lower numeric precision, for example 8 or 4 bits instead of 16, so it fits in less memory and runs faster, usually with a small quality cost. to keep fidelity high at low bitrate. Successors optimize for generation rather than pure compression: Mimi folds in semantic information for streaming, and SpectroStream (arXiv 2508.05207, August 2025) extends the line to 48 kHz stereo music. Because everything downstream inherits the codec's ceiling, codec quality quietly gates the whole field.