How large language models work: tokens, training and prediction

A clear explanation of large language models — tokens, next-word prediction, training, fine-tuning and context windows — and what each means for everyday use.

By the AiTrain editorial team · Reviewed under our editorial standards

Large language models (LLMs) power most of today's AI assistants. Understanding roughly how they work is not just trivia: it explains why they are brilliant at some tasks, unreliable at others, and why the way you phrase a request changes the answer so much.

Key takeaways

• An LLM reads and writes in tokens — chunks of words.
• At its core it repeatedly predicts the next token given everything before it.
• It learns patterns from huge amounts of text during pre-training, then is shaped into a helpful assistant through fine-tuning and human feedback.
• Everything it can "see" in a conversation must fit inside its context window.

Step 1: text becomes tokens

Computers work with numbers, so text is first split into tokens. A token is often a whole common word, part of a longer word, or a punctuation mark. "Training" might be one token; an unusual surname might be three or four. As a rough rule for English, 100 tokens is about 75 words.

Why it matters: limits and prices for AI services are usually counted in tokens, and very long documents may need to be split.

Step 2: predicting the next token

Give the model the text "The capital of France is" and it calculates a probability for every possible next token. "Paris" gets a very high probability, so it is likely to be chosen. The model then adds "Paris" to the text and predicts again, one token at a time, until it decides the answer is complete.

That is the whole trick, repeated extremely fast. The surprising part is how much ability emerges from it. To predict the next word in a legal contract, a recipe or a Python file well, the model has to absorb a great deal about grammar, facts, reasoning patterns and style.

A pencil-and-paper version

Try this with a colleague. Write: "Thank you for your email. Unfortunately, we are unable to…" Ask them to guess the next three words. Most people write something like "accept your request" or "process your order". They are not looking anything up — they are drawing on patterns from every polite rejection they have ever read. An LLM does the same thing, with patterns from far more text.

Step 3: pre-training

During pre-training, the model reads a vast collection of text and practises predicting the next token billions of times. Each time it is wrong, its internal settings (called parameters or weights) are nudged slightly to make the right answer more likely next time. The method behind those nudges is called gradient descent; the Build with AI Line has visual videos that show it beautifully.

After pre-training, the model is a powerful text predictor but not yet a helpful assistant. Ask it a question and it might continue with three more questions, because that is a common pattern on the internet.

Step 4: fine-tuning into an assistant

Developers then train the model further on examples of good conversations, and use human ratings to reward answers that are helpful, honest and harmless. This stage teaches the model to follow instructions, answer in a useful format, and decline clearly harmful requests.

Step 5: the context window

When you chat, the model does not remember you the way a person does. Each time it answers, it is given the conversation so far — your messages, its replies, and any documents you pasted — up to a maximum size called the context window. Anything outside that window is invisible to it.

Practical consequences:

• In a very long chat, early instructions can lose influence. Restate important constraints.
• Paste the actual source material when you want answers based on it.
• Start a fresh conversation for an unrelated task, so old context does not leak in.

Why answers vary

Most assistants add a little randomness when choosing tokens, so the same prompt can produce different wording each time. That is useful for brainstorming and less useful when you need consistency. Clear formats and examples reduce variation a lot — see few-shot prompting.

Why it can be confidently wrong

The model is optimised to produce plausible text. Most of the time plausible and true overlap. When they don't — an obscure fact, a recent event, a specific statistic — it can produce something that reads perfectly and is simply invented. Tools that connect the model to search or to your own documents (see retrieval-augmented generation) reduce this, but checking still matters.

Summary table