Audio AI covers converting between speech and text (in both directions) and, increasingly, models that can reason about audio content directly — the foundation of voice assistants, transcription tools, and voice-driven applications.
The Core Conversions
| Direction | Technology | Deep Dive |
|---|---|---|
| Speech → Text | Speech-to-text (STT) / transcription | Speech-to-Text |
| Text → Speech | Text-to-speech (TTS) / voice synthesis | Text-to-Speech |
A Typical Voice Assistant Pipeline
User speaks
↓ Speech-to-Text
Transcribed text
↓ LLM (text-only reasoning, prompting, tool calling, etc.)
Text response
↓ Text-to-Speech
Spoken response back to userThis is the "multimodal system, not single model" pattern discussed in the Multimodal AI overview — most production voice assistants today chain specialized STT, text-LLM, and TTS components rather than using one unified audio-native model, though this is an actively evolving area.
Practical Considerations Specific to Audio
- Latency compounds across the pipeline — STT, LLM generation, and TTS each add their own latency; a voice application's total response time is the sum of all three
- Transcription errors propagate — an STT mistake becomes the LLM's actual input, potentially causing a wrong or irrelevant response to what the user actually said
- Audio quality affects everything downstream — background noise, accents, and audio quality affect STT accuracy, which affects everything after it
Practical Use Case
A customer support voice line using this pipeline needs to account for cumulative latency (users notice multi-second pauses) and needs a strategy for handling STT misrecognition gracefully (e.g. confirming ambiguous requests) rather than confidently acting on a likely-mistranscribed input.
Common Mistakes
- Not accounting for compounding latency across a multi-stage voice pipeline when designing for real-time interaction
- Assuming STT transcription is always accurate, without any strategy for handling likely misrecognitions
Interview Relevance
"Design a voice assistant's high-level architecture." — the STT → LLM → TTS pipeline, with attention to compounding latency and transcription error propagation, is the expected shape of a strong answer.
Practice Question
A voice assistant occasionally responds to the wrong request. List one likely cause specific to the audio pipeline (not the LLM's reasoning itself).
