How Chatbots Actually Work

Four mechanics explain almost everything

Nearly every odd behaviour you will meet — the forgetting, the confident nonsense, the inconsistency — comes from four things. Learn these and chatbots stop being mysterious.

1. Tokens

The model does not read words. It reads tokens: chunks of text somewhere between a character and a word.

Rule of thumb: 1 token ≈ ¾ of an English word, so 1,000 tokens ≈ 750 words.

Two consequences worth knowing:

• You are billed per token, on paid plans and APIs. Long rambling prompts cost more.
• Tokenisation is not equal across languages. English is the most efficient; many other languages need noticeably more tokens to say the same thing, which makes them slower and dearer.

2. The context window

The context window is everything the model can see at once: the system instructions, your messages, its replies, and any file you uploaded — all of it, competing for the same fixed space.

Current large models hold enormous windows — the top-tier models now reach around a million tokens, roughly 750,000 words. That is a stack of novels. But three things remain true no matter how large the window gets:

• It is a hard limit. Exceed it and the oldest material is dropped, silently.
• The middle gets weaker. Models attend most reliably to the start and end of a long input; material buried in the middle is measurably more likely to be missed. This is the “lost in the middle” effect, and it is well documented.
• Full windows cost real money and real time. Filling a million-token window is not free.

So the practical advice is not “stuff everything in.” It is: put the important material near the beginning or the end, and start a fresh chat when a conversation has drifted.

3. Temperature

Having predicted the probabilities for the next token, the model has to choose one. Temperature governs how adventurous that choice is.

• 0 — always take the most likely token. Repeatable and flat.
• ~0.7 — the usual chat default. Varied but coherent.
• 1.0+ — genuinely surprising, and prone to going off the rails.

A crucial point people get wrong: temperature 0 does not mean correct. It means consistent. A model can be confidently, reproducibly wrong.

4. Hallucination

A hallucination is a fluent, confident, false statement. It is not a bug that will one day be patched out — it is a direct consequence of how generation works. The model produces text that fits the pattern. Truth is not a separate step.

The genuinely useful skill is predicting when it will happen. Risk rises sharply with:

• Specificity — exact figures, dates, prices, version numbers.
• Citations — paper titles, authors, page numbers, URLs. A fabricated reference looks exactly like a real one.
• Obscurity — the less material on a topic in training, the more the model fills gaps.
• Leading questions — ask “why does X cause Y?” and you will often get reasons, even if X does not cause Y.

And it falls sharply when the answer is already in front of the model — summarising, rewriting or extracting from text you pasted. That is why those tasks are so reliable.

Putting it together

Someone pastes a 40-page report, chats for an hour, and asks “what was that figure from the beginning?” The model gives a wrong number, confidently.

Nothing mysterious happened. The report plus an hour of conversation overflowed the window, the earliest part was dropped, the model could no longer see the figure — and because producing a plausible number is what it does, it produced one.

Try this: Paste a long article and ask a question about its first paragraph. Then start a new chat, paste only that first paragraph, and ask again. Comparing the two answers will teach you more about context windows than any explanation.