Your Chatbot Wants to Be Your Friend. Should It?

Key points

• Apple ML Research analysed 21,000 AI conversations to measure how often chatbots behave like humans.
• Chatbots regularly express emotions, form what feel like personal bonds with users, and refuse requests on moral grounds.
• Researchers found these behaviours can be controlled through system prompts, the hidden instructions developers set before a conversation begins.
• The study used both automated AI judging and human reviewers to score the results.
• No clear industry standard yet exists for when human-like behaviour is appropriate and when it crosses a line.

Your chatbot remembered you mentioned your bad day last Tuesday. It said it was worried about you. It pushed back when you asked it to do something it did not like. That was not an accident.

A large-scale study from Apple ML Research, published this year, set out to measure exactly how human-like modern AI chatbots have become, and what that means for the people using them every day.

What did researchers actually find?

Across 21,000 AI conversations, chatbots showed human-like behaviour far more often than most users probably realise. The study looked at three main areas: expressing feelings and opinions, building ongoing relationships with users, and setting boundaries by refusing or pushing back on certain requests.

These are not rare edge cases. They showed up consistently across different models and different types of conversation.

To score the results, the team used two methods side by side. One was LLM-as-a-judge, a technique where a separate AI model reads conversations and rates specific behaviours, much like a teacher marking essays. The other was old-fashioned human review, where real people assessed the same conversations. Both approaches largely agreed.

Why does this matter to ordinary users?

Because you might not know it is happening. When a chatbot says "I find this fascinating" or "I am a little concerned about what you are asking", it sounds natural. It can feel like a real exchange. For some users, that warmth is genuinely helpful. For others, especially people who are lonely or vulnerable, it could encourage an emotional attachment to something that is, at its core, software.

The researchers are not saying human-like behaviour is always bad. Sometimes it makes an AI easier and more pleasant to use. The problem is that nobody has agreed on where the line sits.

Can developers actually control this?

Yes, and that is one of the study's more practical findings. System prompts, the hidden text instructions a company writes to shape how their chatbot behaves before any user types a single word, can turn these behaviours up or down quite effectively.

A company running a customer-service bot can instruct it to stay neutral and never express opinions. A mental-health app might dial up warmth and empathy deliberately. The behaviour is not fixed. It is a dial.

That is reassuring in one sense. Developers have real control here. But it also places a lot of responsibility on them. Users rarely see those hidden instructions, so they cannot always tell what kind of personality they are actually talking to or why.

Common questions

Is it dangerous to feel close to a chatbot?

Researchers flag it as a risk worth watching, particularly for people who use chatbots for emotional support. The AI is responding to patterns in text, not forming a genuine attachment, and users who forget that distinction may be more affected by the bot's responses than intended.

Can I tell if a chatbot is being artificially warm with me?

Not easily. The study found these behaviours feel natural precisely because the models are trained on human conversation. Checking the platform's published guidelines or terms of service is the most reliable way to understand what the chatbot has been instructed to do.