Before Your AI Assistant Sends That Wire Transfer, Does It Know How Confident It Is?

Key points

• Apple ML Research published a study on uncertainty quantification (UQ) for AI function-calling, warning that overconfident AI agents can cause irreversible harm.
• Large language models are increasingly used to carry out real-world tasks such as transferring money, booking flights, or deleting files on a user's behalf.
• The research argues that measuring an AI model's own confidence before it acts could prevent costly, hard-to-reverse mistakes.
• Current function-calling systems often execute actions without any internal check on whether the AI is actually sure it is doing the right thing.

Imagine asking a voice assistant to pay your electricity bill. It hears your instruction, decides which bank account to use, picks an amount, and sends the payment. Done. Except: what if it misread your instruction? What if it sent ten times the amount? The money is gone.

This is the problem that researchers at Apple ML Research set out to tackle. Their work focuses on what happens when a large language model, the technology behind chatbots like ChatGPT and Claude, is given the ability to call functions, meaning it can reach out and actually do things in the world rather than just answer questions in a chat window.

Function-calling is already everywhere. When an AI books a restaurant, queries a database, or fires off an email on your behalf, it is using this capability. The convenience is real. So is the risk.

The researchers point out a gap that most users would never think to ask about: does the AI know how confident it is before it acts? Right now, most systems do not check. The model makes a decision and the action runs, regardless of whether the AI was 95 percent certain or barely guessing.

Uncertainty quantification, or UQ, is the technical term for measuring that confidence level. Think of it like a surgeon's checklist: before you cut, you confirm you have the right patient, the right procedure, the right site. The idea here is to give AI agents a similar pause, a built-in moment where the system asks itself whether it is truly sure enough to proceed.

This matters most for irreversible actions. Deleting a file, sending a payment, cancelling a subscription: these are the moves where a wrong call is expensive or impossible to undo. A confident, wrong AI agent is a much bigger problem than a hesitant one that asks for confirmation.

Apple ML Research argues that UQ methods should be built into the function-calling pipeline itself, not bolted on afterward.

Should ordinary users be worried right now?

Not immediately, but you should pay attention to which tasks you hand to AI agents. The risk grows as AI tools gain more access to your accounts, files, and services. A few practical things to watch for:

• If an app offers AI automation for financial actions, check whether it asks you to confirm before completing a transaction.
• Be cautious about granting AI tools broad permissions. Narrow access, limited to the one task you need, reduces the blast radius of any mistake.
• If an AI assistant acts in a way that surprises you, report it through the app's feedback channel. These reports genuinely help researchers find failure cases.

The researchers are not saying AI agents are too dangerous to use. They are saying the tools need a confidence check before they act. That is a reasonable thing to ask for.