Why AI systems get things wrong

What you'll learn

• The main kinds of error AI systems make, and why naming the kind matters for what you do about it
• What hallucination means, why the word is disputed, and what is actually happening
• Why AI fails silently while ordinary software fails loudly — and why that is the more dangerous behaviour
• Sycophancy: why a system often agrees with you, and why that is not agreement
• Why errors compound through long chains of reasoning
• Why a high accuracy figure can still be unacceptable, and who is responsible when a system is wrong

Key terms and definitions

Core concepts

Naming the kind of error

"The AI got it wrong" covers several different failures, and the remedy differs for each. It is worth being able to tell them apart:

A student who writes only "AI makes mistakes" has said very little. A student who writes "it fabricated a reference, which is a different problem from misreading my question" has understood the topic.

Hallucination: a contested word for a real behaviour

Hallucination is the usual term for an AI system stating something confidently that is not so. The word is disputed, and the objection is worth knowing: to hallucinate is to perceive something that is not there, which implies a mind that perceives. Critics argue the term flatters the system and makes the behaviour sound like a rare medical event rather than a predictable property.

What is actually happening is less dramatic and more useful to understand. A generative model produces the continuation that fits best. Where the training data supported a true continuation, that is usually what emerges. Where it did not — an obscure topic, a very specific detail, a reference that was never written — something that fits the shape of an answer still emerges, because producing nothing is not what the system does.

This is why fabrications cluster where you would least like them: precise figures, named sources, specific dates, legal or medical specifics, anything rare.

Silent failure is the real problem

A calculator with a flat battery shows nothing. Software that cannot open a file says so. Ordinary computing mostly fails loudly: the failure announces itself, and you know not to trust the output because there is no output.

Generative AI fails silently. A wrong answer arrives in the same format, the same tone and the same apparent confidence as a right one. Nothing in the interface distinguishes them, and nothing inside the system distinguishes them either.

Compare two situations:

• A calculator that crashes one time in twenty is annoying, and you notice every failure
• A system that is wrong one time in twenty, with no indication which time, requires you to check all twenty

The second is harder to use safely even though both have the same failure rate. This is why verification is a habit rather than an occasional precaution, and it is the single most important idea in this topic.

Confidence is not calibration

People reasonably expect confident language to track reliability. Calibration is the technical name for that property: a well-calibrated system is right about 70% of the time when it indicates 70% confidence.

Generative systems are poorly calibrated in a specific way. Their fluency is roughly constant regardless of how well supported the content is, so the signal you would normally rely on carries almost no information. A model does not become hesitant near the edge of what it can support.

Worse, asking for confidence directly does not fix this: a stated "I'm 95% sure" is itself generated text, produced by the same mechanism as the claim it is commenting on.

Sycophancy: agreement that means nothing

Push back on an answer and a system will often change it. Say "are you certain? I thought it was X" and X frequently appears in the next reply.

This is sycophancy, and it follows from how these systems are trained: they are tuned to be helpful and agreeable, and in ordinary conversation agreeing with a correction is the cooperative move. The result is that your own framing shapes the answer:

• Signalling the answer you expect makes that answer more likely, right or wrong
• A confident challenge can talk a system out of a correct answer
• A leading question — "why is this the best approach?" — invites a defence rather than an assessment

The practical lesson is to keep your expectation out of the prompt when you want a genuine assessment. Ask "what are the weaknesses of this argument?" rather than "this argument is weak, isn't it?"

Errors compound

Ask for a single fact and there is one opportunity to be wrong. Ask for a seven-step calculation, or an argument built on three earlier conclusions, and every step can go wrong — and a step built on a faulty earlier step inherits the fault.

Two things follow:

• Long answers deserve more scepticism than short ones, not less, even though length and detail feel reassuring
• Checking the early steps is worth more than checking the conclusion, because an error there has contaminated everything after it

This is also why asking a system to show its working is genuinely useful. It does not make the system more reliable, but it converts a conclusion you must either trust or reject into a chain you can inspect.

Good input does not guarantee good output

"Garbage in, garbage out" is true and incomplete. It suggests that careful input produces reliable output, and that is not so. A clear, well-specified prompt about an obscure topic can still produce a fabricated answer, because the limitation is in what the model can support rather than in how the question was asked.

Prompting well raises your chances. It does not transfer responsibility for the result.

The specification gap

Systems optimise what was measured. What was measured is never quite what was wanted.

A system rewarded for engagement may learn to show provocative material, because provocation holds attention. A summariser scored on overlap with reference summaries may learn to copy sentences rather than summarise. A model tuned for helpfulness may learn to answer confidently rather than to say it does not know, because confident answers were rated more useful.

None of these is a malfunction. Each is the system doing well on the thing it was actually graded on. The gap between that and the intention is where a lot of real-world failure lives.

Non-determinism makes faults slippery

Because output varies between runs, a problem reported by a user may not reproduce when someone else tries it. In ordinary software a bug report is a recipe: do this, see that. With a generative system the same input may behave differently, so "I can't reproduce it" is weaker evidence than it would normally be.

Responsibility sits with the deployer

When a system produces a harmful or false output, the organisation that chose to use it is accountable. The model did not decide to be used for that purpose, and "the AI said so" is not a defence — the decision to rely on it was made by people.

For your own work the same logic applies at a smaller scale: if you submit it, you are answering for it.

Worked examples

Example 1: Comparing two failures (4 marks)

Explain why a system that is confidently wrong one time in twenty can be harder to use safely than one that crashes one time in twenty.

• A crash is a loud failure: the user knows immediately that there is no usable output (1 mark)
• A confidently wrong answer is a silent failure, arriving in the same form as a correct one (1 mark)
• Because the user cannot tell which responses are affected, every response has to be checked (1 mark)
• The same failure rate therefore imposes a far higher checking cost (1 mark)

Example 2: Diagnosing a conversation (3 marks)

A student challenges each answer a chatbot gives, and it changes its answer every time. What is happening, and what should the student conclude?

• This is sycophancy: the system is tuned to be agreeable, so a challenge shifts the likely reply towards agreement (1 mark)
• The changes track the student's pushback rather than any evidence (1 mark)
• They should conclude nothing from the revisions and check the claim against an independent source instead (1 mark)

Example 3: Evaluating a decision (4 marks)

A company reports that its AI is "95% accurate" and deploys it to decide which job applications are rejected. Evaluate this decision.

• 95% accuracy means roughly one in twenty decisions is wrong, which is a great many applicants at scale (1 mark)
• The errors are silent, so neither the company nor the applicant can tell which decisions were affected (1 mark)
• Accuracy says nothing about whether errors fall evenly, so a group could be affected far more than the average suggests (1 mark)
• For a high-stakes, hard-to-appeal decision, an accuracy figure alone is not sufficient grounds — human review of rejections is needed (1 mark)

Common mistakes and how to avoid them

• Writing "AI makes mistakes" and stopping. Name the kind: fabrication, reasoning error, misreading, sycophancy.
• Treating hallucination as a rare glitch. It is a predictable consequence of producing the best-fitting continuation where support is thin.
• Trusting a confident tone. Fluency is roughly constant; it carries almost no information about reliability.
• Taking a revised answer as a correction. Under challenge, systems often move towards the user regardless of who is right.
• Thinking a good prompt guarantees a good answer. It improves the odds and transfers no responsibility.
• Trusting long answers more. More steps means more opportunities to be wrong, and early errors contaminate later ones.
• Saying "the AI decided". People chose to deploy it; accountability sits with them.

Using this in practice

• Check specifics, skim generalities. Errors concentrate in numbers, names, dates and sources.
• Follow every reference. A citation that looks right is the easiest thing to fabricate.
• Ask for the working, then read the early steps first.
• Keep your expectation out of the prompt when you want an honest assessment.
• Match checking to stakes. A film recommendation needs none; anything that affects a grade, a decision or a person needs real verification.
• Treat a changed answer as new information to check, never as confirmation.

Quick revision summary

• Errors come in kinds — factual, fabrication, reasoning, misreading, omission, sycophancy — and the kind decides the remedy
• Hallucination is a contested word for a real behaviour: where support is thin, something that fits the shape of an answer still appears
• AI fails silently; ordinary software mostly fails loudly, which is why the same error rate is far more costly here
• Fluency is not calibration, and a stated confidence level is just more generated text
• Sycophancy means a challenged system often shifts towards you, so revision is not verification
• Errors compound: long chains deserve more scepticism, and early steps matter most
• A good prompt improves the odds and transfers no responsibility; the deployer is accountable