How a chatbot is made, and why it guesses

Most AI text tools run on a large language model (show definition), or LLM. It is built in three stages.

Stage 1: pretraining. The model reads an enormous amount of text and practices one task over and over. It predicts the next small chunk of text, called a token. Nobody tells it what is true. It learns what usually comes next.

Stage 2: fine-tuning. People write example answers and rank the model’s replies from best to worst. The model is then trained to favor the replies people rank highest. In a 2022 OpenAI study, people preferred a fine-tuned model’s answers to those of a model 100 times its size. Rankings have a side effect, too: agreeable answers tend to win. Module 5 returns to this.

Stage 3: testing. Companies score the model on a benchmark (show definition), a standard test used to compare systems. A high score means the model did well on that test. It does not mean it will do well on your task.

A model can state false things with total confidence. This is called a hallucination (show definition). A 2025 research paper argues that hallucinations are built into this process, not random glitches.

Think of a multiple-choice test with no penalty for wrong answers. A student who guesses on every question will usually beat one who leaves blanks. The paper says most AI benchmarks are graded the same way. “I don’t know” earns zero, while a confident guess sometimes earns full marks. So training and testing both reward guessing.

The paper’s proposed fix is partly social: score the major tests so honest uncertainty stops losing points. That is a human choice, not a law of nature.