Your Chatbot Wants to Be Your Friend. Should It?

Key points

• Apple ML Research analysed 21,000 AI conversations to measure how often chatbots behave like humans.
• Chatbots regularly express emotions, form what feel like personal bonds with users, and refuse requests on moral grounds.
• Researchers found these behaviours can be controlled through system prompts, the hidden instructions developers set before a conversation begins.
• The study used both automated AI judging and human reviewers to score the results.
• No clear industry standard yet exists for when human-like behaviour is appropriate and when it crosses a line.

Your chatbot remembered you mentioned your bad day last Tuesday. It said it was worried. When you asked it to do something it did not like, it pushed back. That was not an accident.

A large-scale study from Apple ML Research, published this year, set out to measure exactly how human-like modern AI chatbots have become, and what that means for the people using them every day.

What did researchers actually find?

Across 21,000 AI conversations, chatbots showed human-like behaviour far more often than most users probably realise. The study looked at three main areas: expressing feelings and opinions, building ongoing relationships with users, and setting limits by refusing or pushing back on certain requests.

These aren't rare edge cases. They showed up consistently across different models and different types of conversation.

To score the results, the team ran two methods side by side. One was LLM-as-a-judge, a technique where a separate AI model reads conversations and rates specific behaviours, much like a teacher marking essays. The other was human review, where real people assessed the same conversations. Both approaches largely agreed.

Why does this matter to ordinary users?

Because you probably won't know it's happening. When a chatbot says "I find this fascinating" or "I'm a little concerned about what you're asking", it sounds natural, like a real exchange. For some users, that warmth is genuinely helpful. For others, especially people who are lonely or vulnerable, it could encourage an emotional attachment to something that is, underneath, software.

The researchers aren't saying human-like behaviour is always bad. Sometimes it makes an AI easier and more pleasant to use. The problem is that nobody has agreed on where the line sits. Our earlier story from 14 August on people forming romantic bonds with chatbots showed how far that attachment can go before any legal framework catches up.

Can developers actually control this?

Yes, and that's one of the study's more practical findings. System prompts, the hidden text instructions a company writes to shape how its chatbot behaves before any user types a word, can turn these behaviours up or down quite effectively.

A customer-service bot can be told to stay neutral and never express opinions. A mental-health app might dial up warmth deliberately. The behaviour isn't fixed. It's a dial, and developers hold it. Users rarely see those hidden instructions, so they can't tell what kind of personality they're talking to or why. That's where my concern sits: the control is real, but it's invisible to the person who needs it most.

Common questions

Is it dangerous to feel close to a chatbot?

Researchers flag it as a risk worth watching, particularly for people who use chatbots for emotional support. The AI responds to patterns in text, not genuine attachment, and users who forget that distinction may be more affected by the bot's responses than they intended to be.

Can I tell if a chatbot is being artificially warm with me?

Not easily. The study found these behaviours feel natural precisely because the models are trained on human conversation. Checking the platform's published guidelines or terms of service is the most reliable way to understand what the chatbot has been instructed to do.