Before Your AI Assistant Sends That Wire Transfer, Does It Know How Confident It Is?

Key points

• Apple ML Research published a study on uncertainty quantification (UQ) for AI function-calling, warning that overconfident AI agents can cause irreversible harm.
• Large language models are increasingly used to carry out real-world tasks such as transferring money or deleting files on a user's behalf.
• The research argues that measuring an AI model's own confidence before it acts could prevent costly, hard-to-reverse mistakes.
• Current function-calling systems often execute actions without any internal check on whether the AI is actually sure it is doing the right thing.

Imagine asking a voice assistant to pay your electricity bill. It hears your instruction, decides which account to use and sends the payment. Done. Except it misread you. The money's gone.

That's the problem researchers at Apple ML Research set out to tackle. Their work focuses on what happens when a large language model, the technology behind chatbots like ChatGPT and Claude, is given the ability to call functions, meaning it can reach out and actually do things in the world rather than just answer questions in a chat window.

Function-calling is already everywhere. When an AI books a restaurant or fires off an email on your behalf, it's using this capability. The convenience is real. So is the risk.

The researchers point to a gap most users would never think to ask about: does the AI know how confident it is before it acts? Most systems don't check. The model makes a decision and the action runs, regardless of whether the AI was barely guessing or completely certain.

Uncertainty quantification, or UQ, is the technical term for measuring that confidence level. Think of it like a surgeon's checklist: before you cut, you confirm you have the right patient and the right procedure. The idea here is to give AI agents a similar pause, a built-in moment where the system asks itself whether it's truly sure enough to proceed.

This matters most for irreversible actions. Deleting a file, sending a payment: these are the moves where a wrong call is expensive or impossible to undo. A confident, wrong AI agent is a much bigger problem than a hesitant one that asks for confirmation.

Apple ML Research argues that UQ methods should be built into the function-calling pipeline itself, not bolted on afterward. This is the second function-calling story we've covered since July, and the virtual-user testing framework Apple published on 14 July points in the same direction: the lab is systematically working to catch AI mistakes before real users bear the cost.

Should ordinary users be worried right now?

Not immediately, but you should pay attention to which tasks you hand to AI agents. The risk grows as AI tools gain broader access to your accounts and services.

• If an app offers AI automation for financial actions, check whether it asks you to confirm before completing a transaction.
• Be cautious about granting AI tools wide permissions. Narrow access, limited to the one task you need, reduces the damage any mistake can do.
• If an AI assistant acts in a way that surprises you, report it through the app's feedback channel. These reports genuinely help researchers find failure cases.

The researchers aren't saying AI agents are too dangerous to use. They're saying the tools need a confidence check before they act. That's a reasonable thing to ask for, and the fact that Apple's lab is publishing this work suggests the industry knows it hasn't solved the problem yet.