Apple's New On-Device AI Voice Tech Makes Siri Sound More Human, Without Sending Your Audio to the Cloud
A research paper from Apple's machine learning team reveals the audio engine behind Siri's more expressive new voices. All the processing happens on your iPhone, using a chip you already have.
Key points
- Apple's AFM 3 Core Advanced model, the company's most capable on-device AI, now powers Siri's new Expressive Voices feature.
- All audio is generated and processed directly on the device, meaning no voice data is sent to Apple's servers.
- The system runs on the Apple Matrix Coprocessor, a specialised processing unit built into Apple's own chips.
- A new audio architecture converts compressed audio tokens into full, natural-sounding speech in real time.
- Apple ML Research published the underlying research, detailing how tight memory limits on a phone were overcome.
When Siri speaks to you, it now sounds noticeably more natural. That is not a coincidence. Apple has quietly shipped a sophisticated new audio generation system, and a paper published by Apple ML Research explains exactly how it works.
The feature is called Siri Expressive Voices. It uses a process called audio synthesis, which is software generating spoken words from scratch rather than stitching together pre-recorded clips. Think of it like a very advanced text-to-speech engine, except instead of sounding robotic, the result is rich and configurable in real time.
How does it actually work?
The system converts text into speech through two stages. First, a large language model called AFM 3 Core Advanced, a type of AI that understands and generates language, produces compact packets of audio information called semantic tokens. These are not sound yet. They are more like a compressed shorthand for what the audio should be.
A second component, called the detokenizer, then unpacks those tokens into actual sound waves you can hear. The paper describes this second stage in detail, because it is the genuinely hard part.
The difficulty is memory. Phones have strict limits on how much memory software can use at any moment. To squeeze high-quality audio generation into those limits, Apple's engineers built a three-part design that converts the simple tokens into a richer audio format called RVQ, which stands for residual vector quantization. Think of RVQ as a layered set of audio instructions that progressively sharpen the sound quality, similar to how a JPEG image loads blurry and then sharpens.
All of this runs on the Apple Matrix Coprocessor, or AMX, a dedicated calculation unit embedded in Apple Silicon chips like the A-series and M-series. The AMX handles the intense number-crunching without draining the main processor or the battery.
Why does it matter that everything stays on the device?
Privacy is the short answer. Because audio synthesis happens entirely on-device, your voice and what Siri says back to you never travel to Apple's servers. There is nothing to intercept, store, or subpoena. For users who have worried about voice assistants listening in, this architecture is a meaningful design choice, not just a marketing claim.
Performance is the other benefit. Processing locally means no waiting for a round trip to a distant server. The voice response starts almost immediately.
| Component | Role | Runs on |
|---|---|---|
| AFM 3 Core Advanced | Generates semantic audio tokens from text | On-device |
| Detokenizer | Converts tokens into audio waveforms | Apple AMX chip |
| RVQ Representation | Sharpens and layers audio detail | On-device |
What does this mean for ordinary users?
If your device supports the feature, Siri simply sounds better. You do not need to change a setting or opt in to anything. The improvement arrives through a software update.
The broader signal is that Apple is pushing serious AI capability into the chip rather than relying on cloud servers. That approach prioritises privacy and speed together, and this audio engine is one of the clearest examples yet of what that looks like in practice.
