The audio model landscape, in practice

Audio splits into two families that behave very differently in production, speech and music, with sound design sitting alongside both. Start with voice. The axis that matters most day to day is not raw naturalness, which most leading models now handle well, but the latencyLatencyThe delay between asking for a generation and getting the first result. Distinct from throughput. profile. A real-time voice agent that talks back over a phone line needs its first audio in well under a quarter second, while an audiobook can take all the time it wants for a better read. Cartesia built its Sonic model explicitly for the live case, reporting streaming latency around 90 milliseconds. ElevenLabs splits its own lineup along the same seam, with a Flash tier aimed at roughly 75 milliseconds for conversational use and the more expressive Eleven v3 trading speed for range and aimed at narration rather than live dialogue. Treat those numbers as vendor figures, but the shape is real: you choose a voice model by how fast it has to answer.

The other voice differentiators are cloning fidelity, control, and language coverage. Zero-shot cloning can imitate a voice from a short sample, but the convincing, consistent results still come from longer enrollmentEnrollmentThe longer reference recording, often many minutes, used to build a high-fidelity voice clone that holds up under close listening., and ElevenLabs professional cloning asks for thirty minutes of audio at minimum and prefers a few hours. Control is where the field is moving. Hume's Octave is pitched as a speech model that interprets meaning and takes acting instructions, so you can direct a line to be whispered or weary rather than just typed. PlayAI conditions on conversational context so multi-turn speech carries rhythm across a scene. Language coverage ranges widely, with the top models citing dozens of languages each.

Music models differ along song structure, vocal quality, length, and editability. The practical questions are whether a tool produces a coherent arrangement, verse into chorus into bridge, rather than a looping texture, whether it sings intelligible lyrics, how long a track it can hold together, and whether it exposes stemsStemsThe separated individual tracks of a piece of music, such as vocals, drums, and bass, which can be edited or remixed on their own. for real editing in a digital audio workstation. Suno and Udio lead on full vocal songs and stem export. Google's Lyria line, delivered through Google's own surfaces and watermarked by default, is built for ecosystem reach. Stability's Stable Audio targets brand and enterprise sound. Meta's MusicGen is the open-weight option you can run and fine-tune yourself.