How to Find Out Why Your AI Model Is Slow on AWS Trainium
A profiling walkthrough from Hugging Face tests three versions of the same AI operation and shows exactly where computation stalls on Amazon's purpose-built chip.
Key points
- A single unfused operation called
aten::mulconsumed 49.5% of Neuron device time in a basic AI normalisation test on AWS Trainium. - The walkthrough, published by Hugging Face, uses three versions of the same calculation to show how code structure directly changes hardware performance.
- AWS Trainium is a chip designed specifically for the heavy maths involved in training and running AI models, unlike general-purpose processors.
- The total device time for the naive (unoptimised) implementation of one small AI operation was 2.355 milliseconds, spread across four arithmetic steps.
Most engineers who build AI systems know when something runs slowly. Fewer know how to prove which part is slow, or why. That gap matters enormously when AI inference, the work a model does every time it answers a question, has to run thousands of times a second at real-world scale.
A new technical walkthrough published by Hugging Face tackles that problem directly. It covers AWS Trainium, Amazon's chip built from the ground up for AI workloads, and a software layer called TorchNeuron. TorchNeuron is an open-source backend for PyTorch, the most widely used framework for building AI models, letting developers run standard PyTorch code on Trainium hardware with minimal rewrites. We covered PyTorch performance on Apple hardware in yesterday's story about PyTorch 2.15; the Trainium walkthrough is the cloud-side counterpart.
What does profiling actually show?
Profiling means recording exactly what a chip does and when, so you can read the timeline afterwards. The walkthrough profiles a small but common AI operation called RMSNorm, which rescales numbers inside a neural network to keep calculations stable.
Three versions of the same RMSNorm calculation were tested back to back:
| Implementation | Description |
|---|---|
| Naive PyTorch | Five separate operations, each dispatched individually to the chip |
Built-in torch.rms_norm() |
One fused call the chip handles as a single instruction |
Naive + torch.compile |
Same five operations, with the compiler trying to combine them automatically |
The naive version is most revealing. A profiling table called key_averages() showed that three calls to a multiply operation (aten::mul) alone ate 49.5% of the chip's time. Adding a tiny constant called epsilon, a number that stops a division-by-zero error, triggered an unexpected data copy from main memory to the chip, adding further cost. Total for the unoptimised path: 2.355 milliseconds for one pass of one small operation.
That sounds tiny. At thousands of requests per second, it compounds fast.
What tool makes the data readable?
A browser-based viewer called Neuron Explorer lets developers see each operation as a horizontal bar on a timeline. Time flows left to right; each instruction moves from framework dispatch through runtime scheduling and down to hardware execution on the chip's processing lanes, called NeuronCores.
Separate operations leave visible gaps. A fused operation fills those gaps and runs faster, and the timeline makes that difference obvious in a way that a wall of log text simply doesn't.
This kind of chip-level visibility is increasingly valuable. Our earlier story on Apple running a nearly-trillion-parameter model across four Mac Studios showed how even consumer hardware can handle serious AI workloads when inference is set up carefully. The engineering lesson there and here is the same: visible bottlenecks get fixed; invisible ones don't.
Writing AI operations in the most readable way is often the slowest way, and this walkthrough proves it with numbers. Reading a profiler trace is now a basic production skill, not something you can leave to the chip architects. What to watch next is whether the torch.compile path closes the gap with the fused built-in, which Part 1 leaves open for follow-up.
