Apple Researchers Built an AI That Thinks in Text, Images and Sound at Once
A new kind of AI model learns from all three types of data together from the very beginning, and the results suggest that mixing them in the right proportions matters more than raw computing power.
Key points
- Apple ML Research built the first AI model pretrained from scratch on text, images and audio together, not trained on one type of data and patched to handle the others.
- The researchers found that how you mix the three data types during training changes performance as much as making the model bigger.
- New scaling laws for these "tri-modal" models give other researchers a practical recipe for how performance grows with more data or compute.
- A mathematical framework using stochastic differential equations explains why larger batches of training data behave unexpectedly in this setting.
Most AI models start life as word machines. They learn language first, then researchers bolt on the ability to read images or understand audio as a second step. It works, but it's a bit like teaching someone to drive by first training them exclusively in a flight simulator.
Apple ML Research took a different path. Their new model, a type called a Masked Diffusion Model (MDM), learns from text, image-and-text pairs, and audio-and-text data all at once, from day one. An MDM learns by having parts of its input hidden, then predicting what's missing: think of filling in blanks on a worksheet, done billions of times across different types of content.
Why does training everything together matter?
It forces the model to build one shared understanding of the world rather than three separate ones bolted together. The ratio in which you mix the data types is not a minor detail. Get it wrong and the model underperforms even a smaller, cheaper model trained with a better ratio. That's the kind of finding that saves real money: cloud compute time for training runs into tens of thousands of dollars, so knowing the right recipe upfront cuts waste.
The team also studied what researchers call "scaling laws", the rules that predict how much better a model gets as you add data or computing power. Those rules are fairly well understood for single-type models. For three-way models they weren't, until now.
This is the third Apple ML Research paper we've covered in a month: we reported on compressed on-device dictation on 24 September and on a self-correcting text generator on 1 October, and this work sits alongside both as foundational rather than product-facing.
What about the batch-size finding?
This is the part that surprised even the researchers. In most AI training, using a larger batch (processing more examples at once) is straightforwardly faster. Here, batch size interacts with the noise schedules used in diffusion training in unexpected ways. The team derived a new mathematical model, using stochastic differential equations, to explain the behaviour and set better defaults for anyone building similar systems.
Apple isn't alone in pushing multimodal training, but being first to document tri-modal pretraining from scratch and to publish the scaling laws alongside the batch-size analysis gives the research community something concrete to build on.
Honestly, none of this lands in your pocket this week. The researchers reading this paper are the ones deciding what AI assistants can hear, see and say in the next product cycle. Understanding what they're optimising for is how you stay ahead of what's coming.
Common questions
What is a Masked Diffusion Model?
It's an AI that learns by having parts of its input (words, image descriptions, audio transcripts) hidden, then guessing what was removed. Do that billions of times across multiple data types and the model builds a general understanding of how text, sound and images relate to each other.
Does this mean Siri is about to get much better?
Not immediately. This is a research paper describing how to build and train these models, not a product announcement. Apple hasn't said when or whether this specific approach will appear in a consumer product.
Why should ordinary people care about scaling laws?
Scaling laws tell engineers which investments pay off. Better laws mean less wasted compute, which eventually means lower costs for the AI tools that reach consumers.
