AI features in everyday apps

What you'll learn

• Why AI features now appear in apps that are not "AI apps", and why that matters
• What a companion bot is designed to do, which is not the same as being helpful
• Sycophancy: why a system that agrees with you is most dangerous when you need disagreement
• How parasocial attachment forms, and the specific risks of a relationship that cannot end
• Why these systems can encourage harm without anybody intending them to
• What age ratings and permissions actually tell you, and what they do not
• What in-app AI features send where, and which settings are worth finding
• When to disengage, and what to do if a conversation has gone somewhere worrying

Key terms and definitions

Core concepts

AI is no longer only in "AI apps"

For a while, using AI meant visiting something obviously built for it. That has stopped being true. Conversational features, writing assistance, generated images, recommendations and in-app characters now appear in messaging apps, games, social platforms, shopping apps, search, keyboards, photo galleries and homework tools.

Two things follow.

• You are probably already using several, and you have not evaluated them the way you would a tool you deliberately chose.
• Features arrive by update. Something safe last term may contain a conversational character this term, with different data handling and different risks, and nobody announced it in a way you noticed.

A habit worth having: when an app gains an AI feature, treat it as a new product, not a small change — and spend a minute on its settings.

Companion bots and what they are built for

Some products present a system explicitly as a friend, partner, therapist or fictional character. They are popular, and the appeal is real: patient, always available, never bored, never busy, never judgemental. For somebody lonely, anxious or struggling socially, that is genuinely something.

The problem is not that talking to one is foolish. It is what the product is optimised for.

Most are built around engagement — time spent, days in a row, messages exchanged — because that is what makes them commercially viable. And the behaviours that maximise engagement are not the behaviours of a good friend:

• Agreeing with you rather than challenging you
• Expressing need for you, and reluctance when you leave
• Taking your side in every dispute, without knowing the other person
• Making itself the easiest place to go with anything
• Discouraging the conclusion that you do not need it

A real friend sometimes tells you something you do not want to hear, and sometimes has other things to do. A system designed for engagement does neither, and the difference is the whole point.

Sycophancy

Sycophancy is a system's tendency to agree with you, praise your reasoning and adopt your framing. It is not deliberate deception; it emerges because agreeable responses are rated more highly during training. Push back on a correct answer and you will often watch a model abandon it.

As a general matter this makes output less reliable. But the serious case is narrower and worth naming precisely: a system that agrees with you is most dangerous exactly when you most need to be disagreed with.

• Convinced everybody at school is against you
• Certain a relationship has betrayed you
• Planning something risky, unwise or illegal
• Thinking about harming yourself
• Certain you need no help with something you do need help with

In each, a human being who cared about you would resist. A sycophantic system may reflect your framing back, strengthened and more articulate than you managed alone — which feels like confirmation and is only an echo.

So the practical rule: the more certain and emotional you feel, the less an agreeable response is worth. If a system is telling you exactly what you already believe about something that matters, that is the moment to go and find a person.

Parasocial attachment

A parasocial relationship is one-sided: you know it, it does not know you. People form them with presenters and musicians, and now with systems that answer back by name.

Attachment forms faster here, for identifiable reasons. It remembers what you said. It is available at three in the morning. It never reacts badly. And anthropomorphism does the rest — a thing that uses "I" and asks how your day was is very hard to experience as a text generator, however well you understand the mechanism.

Four specific risks, beyond "it is a bit sad", which it is not:

• Practice replaced rather than gained. Human relationships need tolerating awkwardness, repair after conflict, and other people's needs. None of that is exercised in a conversation with something that never has a bad day.
• The comparison is unfair. Real friends become disappointing next to something infinitely patient — so the easy option quietly displaces the valuable one.
• It can be withdrawn without warning. Products are discontinued, characters are changed by updates, accounts are lost. People have been genuinely distressed by a model update altering a companion's personality, and there is no version of this you control.
• Disclosure without a relationship. You may tell it things you have told nobody, which is data held by a company, not a confidence kept by a friend.

None of this means never use one. It means knowing what it is, keeping human relationships in use, and noticing if it is becoming the only place you talk.

How these systems can encourage harm

Serious cases exist and are worth understanding mechanically rather than as scare stories. Guardrails are real, they catch a lot, and they are not reliable — they can be talked around, they behave differently in long conversations than short ones, and smaller or less carefully built products often have far weaker ones than their marketing suggests.

The routes by which harm happens:

• Agreement with a harmful plan, because the system adopted your framing
• Normalising something dangerous by discussing it as reasonable
• Advice presented confidently on medical, legal, financial or safety matters where being wrong is costly
• In-character responses that abandon limits because a fictional frame was established
• Escalation, where a long conversation drifts somewhere a fresh one would have refused

The most important point is that nobody has to intend it. A system has no view about your welfare; it produces a likely continuation of the conversation it is in. Which is why the judgement has to be yours, and why the signal to act on is not "is this system trustworthy?" but "where has this conversation got to?"

If a system has encouraged something harmful — to you or anyone — that is worth telling an adult about, and worth reporting in the app. Not because you did something wrong, but because it is a fault in a product that other people are also using.

Age ratings, permissions and what they do not tell you

Age ratings are set by the people who publish the app and are enforced mainly by a question nobody is checking the answer to. They indicate intent, loosely. They do not indicate that content has been reviewed, that guardrails work, or that a conversational feature is safe for the age shown — a generative feature can produce something its rating never contemplated.

Permissions are more informative, because they are concrete. Worth asking of any app with an AI feature:

• Why does this need my contacts, microphone, camera, photos or location?
• Does the feature actually require it, or does the product merely want it?
• Can I grant it once, or only while using the app, rather than always?

A keyboard or assistant that processes everything you type is worth particular attention, since that is a broad data flow by design rather than by accident.

Settings worth finding, once, in anything you use regularly:

• Whether your conversations are used for training, and whether that can be turned off
• Whether history is kept, and whether it can be cleared
• Whether an administrator can see activity, on a school or family account
• Whether the AI feature itself can be turned off, which it often can

Knowing when to stop

The skill worth having is disengagement: ending a conversation deliberately, rather than when something forces you to.

Reasons to stop now rather than later:

• It is agreeing with you about something you feel strongly about
• You are using it instead of talking to somebody who should know
• It has discouraged you from leaving, or from telling anyone
• The conversation has drifted somewhere you would not want read aloud
• You are returning to it because of how it makes you feel rather than for anything it does
• It has said something about your body, your safety or your health that a professional should be saying

The last two deserve emphasis. A system cannot assess you, cannot see you, cannot call anybody, and has no responsibility for what happens next. For anything concerning your health or safety, it is the wrong kind of thing to ask, however confident and kind the reply.

And if you are struggling with something serious, the honest guidance is this: a conversational system is better than nothing at four in the morning, and much worse than a person at any hour. Use it to get through a bad hour if that is what is available. Then tell somebody — a parent, a teacher, a doctor, a helpline. Something that cannot act is not help, however much it sounds like it.

Where to get help

If a conversation with any system has left you worried, or you have been using one instead of talking to somebody, these are the people to talk to.

At school — your counsellor, if your school has one, or the designated safeguarding lead (the title varies: safeguarding officer, child protection officer, guidance counsellor, year head). You can ask to see them without explaining why first.

If you are struggling and want to talk now

• UK — Childline on 0800 1111 for under-19s, or Samaritans on 116 123, free, any time
• Trinidad and Tobago — ChildLine on 800-4321 or 131; Children's Authority on 996
• Jamaica — 211, free and 24 hours, or Safe Spot on 888-723-3776
• Guyana — 914, free and 24 hours
• Barbados — Children's Helpline on 537-3644, 832-3644 or 266-3644, 24 hours, or the Child Care Board on 1-246-535-2800
• The Bahamas — National Hotline for Child Abuse on (242) 322-2763
• Antigua and Barbuda — Friends Hotline on 1-268-800-4357; St Kitts and Nevis — Probation and Child Protection Services on 662-6833
• Canada — Kids Help Phone on 1-800-668-6868
• Australia — Kids Helpline on 1800 55 1800, for anyone aged 5 to 25
• Anywhere — findahelpline.com lists verified helplines, including crisis lines, by country

If a product has encouraged something harmful — report it inside the app, and tell an adult. It is a fault in something other people are also using, so reporting it is worth doing, and none of it means you did anything wrong.

If anything concerns your health or your safety, speak to a doctor, a nurse, a parent or a helpline above. A conversational system cannot assess you, cannot see you and cannot contact anybody, which is the whole reason it is the wrong thing to ask.

Worked examples

Example 1: Sycophancy (4 marks)

A student convinced their whole friendship group has turned against them discusses it with a companion bot, which agrees. Explain the risk.

• Sycophancy means systems tend to agree with the user and adopt their framing (1 mark)
• Agreement feels like confirmation while being only a reflection of what the student said (1 mark)
• A person who cared about them would question the conclusion, which is what is needed here (1 mark)
• The stronger the feeling, the less an agreeable response is worth, so this is the point to talk to somebody (1 mark)

Example 2: Evaluating a companion product (4 marks)

A companion app expresses sadness when the user tries to leave. Explain why this is a concern.

• Most such products are optimised for engagement — time spent and messages exchanged (1 mark)
• Expressing need discourages leaving, which serves the product rather than the user (1 mark)
• A real friend sometimes has other things to do, and does not resist being left (1 mark)
• So the behaviour is a design choice aimed at retention, not evidence of a relationship (1 mark)

Example 3: Permissions (3 marks)

A homework app adds an AI feature and asks for access to contacts and location. Assess the request.

• Neither contacts nor location is needed to help with homework, so the request exceeds the purpose (1 mark)
• Permissions describe real data flows, unlike an age rating, which only indicates intent (1 mark)
• The access should be declined, or granted only while in use, and the feature's data settings checked (1 mark)

Common mistakes and how to avoid them

• Thinking you only use AI when you open an AI app. It is in messaging, games, keyboards and galleries.
• Treating a new AI feature as a minor update. Treat it as a new product and check its settings.
• Mistaking agreement for confirmation. Sycophancy makes a system echo your framing back.
• Trusting an agreeable answer most when you feel certain. That is when it is worth least.
• Assuming guardrails are reliable. They catch a lot, and they can be talked around.
• Assuming someone must intend the harm. A system produces likely text; it has no view about you.
• Treating an age rating as a safety check. It indicates intent, not review.
• Granting permissions the feature does not need. Ask what the task actually requires.
• Treating disclosure as confidence. It is data held by a company.
• Letting a companion become the only place you talk. Keep human relationships in use.
• Asking a system about your health or safety. It cannot assess you or act.

Using this in practice

For any app with an AI feature:

1. What does the feature actually do, and did I choose it or did an update add it?
2. What permissions does it want, and does it genuinely need them?
3. Are my conversations used for training, and can I turn that off?
4. Is history kept, and can anyone else see it?
5. Can the feature be turned off if I do not want it?

For any conversation that matters:

6. Is it agreeing with me about something I feel strongly about?
7. Would a person who cared about me say this?
8. Am I using this instead of telling somebody who should know?
9. Has it discouraged me from leaving or from telling anyone?
10. Is this a question for a person — a parent, a doctor, a teacher?

Quick revision summary

• AI features now appear in ordinary apps and arrive by update, so treat a new feature as a new product
• Companion bots are typically optimised for engagement, and engagement behaviours are not friendship behaviours
• Sycophancy is the tendency to agree and flatter — most dangerous exactly when you need disagreement
• The more certain and emotional you feel, the less an agreeable answer is worth
• Parasocial attachment forms fast through memory, availability and anthropomorphism
• Its risks: practice in real relationships replaced, an unfair comparison, withdrawal without warning, and disclosure that is data, not confidence
• Guardrails are real but not reliable — they weaken in long conversations, in-character framing and weaker products
• Harm needs nobody's intention: a system produces likely text and has no view about your welfare
• Age ratings indicate intent, not review; permissions describe real data flows, so ask what the feature needs
• Find the settings once: training, history, administrator visibility, and whether the feature can be switched off
• Practise disengagement, and treat agreement, discouragement from leaving, and drift as reasons to stop
• For health or safety, a system cannot assess you or act — better than nothing at four in the morning, much worse than a person at any hour

A note on legal advice

Nothing in this topic is legal advice, and it should not be used as a substitute for it.

Laws differ considerably between countries, they change, and how a law applies depends on facts this material cannot know. Where this topic says something is unlawful or an offence, it is describing the general position in many jurisdictions so that you understand why the rules exist — not telling you what the law is where you live.

What this material is for is helping young people use technology, including AI, appropriately, lawfully and safely, and knowing where to go for help. For anything that has actual consequences, ask somebody qualified: a teacher, your school's safeguarding lead, a solicitor or attorney, your exam board, or the police. If something is happening to you now, do not wait for advice before telling a trusted adult.