AI for Doctors / Theory / Lesson 6

How ChatGPT, Claude and Gemini actually work

Tokens, next-word prediction, temperature, context windows, and why hallucination is built in.

12 min

A large language model (LLM) is a neural network trained on a huge amount of text to do one thing: predict the next piece of text. It does not look facts up in a database. It has learned which words tend to follow which, so well that the output reads like reasoning.

The pieces of text are called tokens. A token can be a whole word, part of a word or a punctuation mark.

How text gets chopped into tokens (rough illustration)
Pt·c/o·breathlessness·since·3·days,·SpO2·88%·on·room·air.·Started·on·nebulisation·with·salbutamol.
Characters98
Tokens (this sketch)32

Common words are one token; rare drug names and abbreviations get broken into pieces. Real tokenizers differ from model to model, but English text usually runs at roughly 3 to 4 characters per token. Context limits, speed and price are all counted in tokens.

At every step the model scores every token it knows and picks one. Then it adds that token to the text and does it again. A setting called temperature controls how adventurous the pick is.

Guess the next word (toy probabilities)
“Septic shock, so the patient was started on IV ___”
fluids
36.0%
antibiotics
29.4%
noradrenaline
13.2%
ceftriaxone
9.8%
pantoprazole
4.9%
insulin
3.6%
paracetamol
3.0%
chocolate
0.1%

A language model only ever does this: score every possible next word and pick one. Low temperature almost always picks the top word. High temperature flattens the scores, so odd words slip in. Nothing in this step checks whether the word is true.

Hallucination is not a bug that will be patched away

The model produces the most plausible continuation, and a plausible-looking citation, dose or trial name is exactly that. Newer models hallucinate less, especially when they can search or are given the source text, but no model is free of it. For anything you will act on, check against the primary source.

Context window: how much text the model can see at once: your prompt, any pasted documents and its own reply. Beyond it, earlier text is dropped.
Training cut-off: the date its training text ends. It does not know later guidelines unless a search tool feeds them in.
Fine-tuning: extra training on a narrower set of data (say, clinical notes) to adapt a general model.
Reasoning models: models that write out intermediate steps before answering. Better at multi-step problems; still able to be wrong.

Check yourself

1. At its core, an LLM is trained to:
2. Raising the temperature makes output:
3. An LLM cites a trial you cannot find on PubMed. Most likely:

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.