Key takeaways
- Modern text-to-speech (TTS) produces voices close to human quality, a transformation driven by neural networks over the past decade.
- DeepMind’s WaveNet (2016) was a watershed — the first neural TTS to match human mean opinion scores on naturalness.
- Voice cloning — synthesizing a specific speaker from seconds of audio — is now commercially accessible, raising serious security and ethical concerns.
- Commercial tools span low-cost high-quality (Google Cloud TTS, Azure Speech) to premium realistic voices (ElevenLabs, PlayHT, Lovo, Resemble).
- Real-time low-latency TTS is enabling live applications — voice AI agents, live translation, accessibility tools.
How TTS works
Classical TTS (through the 2000s) used concatenative approaches — recorded snippets stitched together — or parametric synthesis that modelled speech acoustics explicitly. The results were usable but distinctly robotic.
Modern TTS is neural. The core pipeline has three stages. First, a text-analysis frontend converts text into a phonetic and prosodic representation — which words, which phonemes, where to emphasize, how to break sentences, how to pronounce tricky cases. Second, an acoustic model generates a spectrogram — a time-frequency representation of the intended audio. Third, a vocoder converts the spectrogram into the actual waveform. WaveNet, WaveRNN, HiFi-GAN, BigVGAN, and newer codec-based vocoders have progressively improved quality and efficiency. See our deep learning primer for the underlying techniques.
The major quality leaps
WaveNet (2016)
DeepMind’s WaveNet was the first neural vocoder to produce speech indistinguishable (by naive listeners) from real humans. Originally slow to run, later efficient versions made it deployable. Google Cloud TTS and Google Assistant adopted WaveNet variants widely.
Tacotron and FastSpeech (2017-2019)
Sequence-to-sequence acoustic models that simplified the TTS pipeline. Tacotron 2 + WaveNet produced the first widely-available high-quality neural TTS. FastSpeech brought non-autoregressive inference for real-time use.
Large-scale voice models (2023-present)
Microsoft’s VALL-E, Meta’s Voicebox, ElevenLabs’ models, and several open-source projects treated speech synthesis as a large-model learning problem. Trained on tens of thousands of hours of multi-speaker audio, these models can generate any voice from a short reference clip — a technique called zero-shot voice cloning.
Voice cloning
Given 3-30 seconds of a person speaking, modern voice-cloning systems produce new speech in that person’s voice. Quality ranges from “obvious but recognizable” (short reference, basic models) to “indistinguishable from the original” (longer reference, top commercial models).
Legitimate uses include accessibility (cloning a patient’s voice for speech devices before disease progression), entertainment and voice-over (resurrecting an actor’s voice for a production with appropriate consent), and personalization (cloning your own voice for a personal AI assistant).
Harmful uses include financial fraud (impersonating a CFO on a phone call), romance scams using a loved one’s voice, political disinformation (fake audio of a politician), and non-consensual voice use. The Hong Kong Arup case in 2024 — a finance employee transferred $25 million after a deepfake video and audio call — made the threat concrete. See our ai safety coverage for the broader picture.
Applications
Accessibility
Screen readers for blind and low-vision users. Communication devices for people with ALS, cerebral palsy, or laryngectomies. Real-time captioning with natural-sounding read-back.
Media and entertainment
Audiobook production at scale — publishers are exploring AI-narrated titles, particularly for backlist and less-commercial works. Podcast production. Game narration and NPC voices. Language dubbing for film and television (HeyGen, Papercup, ElevenLabs partnerships with major studios).
Corporate and education
E-learning narration. Training video voiceovers. Corporate communications in multiple languages. Customer-service voice agents (see our customer-service coverage).
Personal AI
AI assistants with natural voices. Real-time voice conversation with LLM-based assistants (OpenAI’s Advanced Voice Mode, Google Gemini Live, ElevenLabs Conversational). Latency has dropped to real-time in 2024-2025, enabling natural dialogue.
Translation and dubbing
Dubbing video from one language to another while preserving the original speaker’s voice. Real-time voice translation (Apple AirPods real-time translation, Samsung Galaxy AI, Microsoft Teams). Quality is usable for many settings and rapidly improving.
Commercial offerings
- Enterprise/general: Google Cloud TTS, Azure Speech, Amazon Polly. Large voice libraries, reliable infrastructure, enterprise contracts.
- Premium realistic: ElevenLabs, PlayHT, Resemble AI, Lovo. Higher-quality voices, explicit voice-cloning support, creator-friendly pricing.
- Open source: Coqui TTS, Piper, MetaVoice, Tortoise TTS, XTTS. Free, self-hostable, quality approaching commercial for English voices.
- Specialized: Speechify (consumer reading app), Descript (editing with AI voices), HeyGen (avatar + voice video generation).
Security and consent frameworks
Major platforms have built consent and detection mechanisms. ElevenLabs requires voice-cloning training to include a recorded consent statement. Microsoft publishes detection APIs for its VALL-E and other synthesized voices. Amazon’s Alexa and Apple’s Siri avoid deploying voice cloning broadly despite having the capability.
Detection of AI-generated speech is an active research area. The best detectors identify most samples correctly but can be fooled by high-quality targeted attacks. The US Federal Communications Commission ruled in 2024 that AI-generated voices in robocalls are illegal under existing rules. EU AI Act deepfake transparency rules apply to manipulated voice content. For the underlying neural network technology, see our neural networks primer.
What’s next
End-to-end speech models that bypass separate text and audio representations (SeamlessM4T, Spirit-LM) promise richer prosody and better multi-lingual capability. On-device TTS is improving fast — iPhones and high-end Android devices now produce quality previously requiring cloud infrastructure. Multimodal models that integrate speech naturally with language understanding will continue the integration of voice into general-purpose AI assistants.
Frequently asked questions
Can I tell AI voices from real ones?
Increasingly hard. Top commercial voice-generation systems produce speech that casual listeners cannot distinguish from real humans, especially on short samples. Long samples, emotional complexity, and specific prosodic patterns still sometimes reveal synthesis — but the gap has narrowed dramatically over the past two years. Audio deepfake detection tools exist but are not reliable enough to be sole evidence.
Is cloning a voice legal?
Depends on jurisdiction, consent, and use. Cloning your own voice for personal use is generally fine. Cloning a living person’s voice without their consent for commercial use is restricted — the SAG-AFTRA AI provisions in US entertainment union contracts, the Right of Publicity in many US states, Europe’s GDPR treatment of voice as personal data, and state laws like Tennessee’s ELVIS Act (2024) all create legal risks. Major voice platforms require consent documentation for cloning.
Can AI voice replace voice actors?
Partially, and the industry is adjusting. Large productions still hire human voice actors for leads, with AI augmentation for secondary roles, language adaptations, and temp tracks. The 2024-2025 SAG-AFTRA video-game strike negotiated specific protections and compensation for AI use of actor voices. Commercial voice work is seeing genuine displacement — explainer videos, corporate training, low-budget e-learning are increasingly AI-voiced. Character work, brand voice-over, and high-production audio drama remain primarily human for now.
