How large language models work comes down to one repeated step: predict the next token, add it to the text, and predict again. Chat assistants and coding tools are both built around that loop. This guide works through the loop with real numbers, then covers the three stages a model goes through (pretraining, fine-tuning and inference) and why setting temperature to zero doesn't stop an LLM from making things up. You only need basic Python to follow the code.
What is a large language model?
A large language model (LLM) is a neural network trained to model patterns in language. The most common kind, a causal language model, has one central task: given the tokens so far, predict the next one.
"Large" refers to the number of learned parameters, often billions, and to the amount of training data. Size alone doesn't make an answer correct, though. A model can write fluent, confident text with no reliable source behind it, and most of the engineering around LLMs exists to deal with that.
You'll see two terms constantly. Parameters are the learned numbers inside the model; training sets them, and they stay fixed while you use it. Context is the information in the current request: your prompt, the conversation so far and any documents you supplied. When an answer is wrong, start by asking whether the knowledge was missing from the parameters or the context was missing or wrong.
Tokens: what the model reads
LLMs read tokens, not words. A token can be a whole word, part of a word, a punctuation mark or a sequence of bytes. A tokenizer splits text into tokens and maps each one to an ID from a fixed vocabulary.
For example, a common word like "the" is usually a single token, while a rarer word like "unbelievable" might be split into pieces such as "un", "believ" and "able". The exact split depends on the tokenizer, so two models can count the same sentence differently.
Tokens are also the unit you pay for. API pricing and context windows count tokens, not words or characters. Text in languages that are less represented in a tokenizer's training data often needs more tokens for the same meaning, and product codes, IDs and long numbers can break into many small pieces.
How large language models work: the generation loop
Every causal LLM runs this loop when it writes:
- Read all the tokens so far.
- Produce a score for every token in the vocabulary.
- Turn the scores into probabilities.
- Pick one token according to a decoding rule.
- Append it and repeat until a stop condition, such as an end token or a length limit.
Suppose the prompt is "The capital of France is". The model scores every token it knows, and " Paris" gets a much higher score than " Berlin" or " banana". The decoder picks " Paris", appends it, and the next step predicts what comes after, perhaps a full stop.
The model isn't looking the fact up in a database; it predicts a likely continuation. When the training data strongly supports the right answer, prediction and fact line up. When it doesn't, the model still predicts something.
From logits to probabilities
The raw scores are called logits, and they can be any real number, positive or negative. A function called softmax turns them into probabilities that are positive and add up to 1.
Take three candidate tokens with logits 2.0, 1.0 and 0.1:
import math
logits = [2.0, 1.0, 0.1]
exps = [math.exp(x) for x in logits]
probs = [e / sum(exps) for e in exps]
print([round(p, 2) for p in probs]) # [0.66, 0.24, 0.1]
Softmax exponentiates each score, so a bigger logit gets a disproportionately bigger share. The top token gets 66% of the probability even though its logit is only one point ahead.
Decoding is the rule that picks the next token from these probabilities. Greedy decoding always takes the highest. Sampling draws at random according to the probabilities, which gives more varied text.
Temperature: what it changes and what it doesn't
Temperature rescales the logits before softmax by dividing each one by the temperature T. With the same logits as above:
| Temperature | Scaled logits | Probabilities |
|---|---|---|
| 0.5 | 4.0, 2.0, 0.2 | 0.86, 0.12, 0.02 |
| 1.0 | 2.0, 1.0, 0.1 | 0.66, 0.24, 0.10 |
| 2.0 | 1.0, 0.5, 0.05 | 0.50, 0.30, 0.19 |
Lower temperature sharpens the distribution toward the top token, and higher temperature flattens it so less likely tokens get picked more often. As temperature approaches 0, sampling becomes greedy decoding.
Many people expect a low temperature to make answers more accurate. It makes them more predictable. If the model's most likely continuation is wrong, temperature 0 gives you that wrong answer on every run.
For instance, ask a general-purpose model about your company's internal leave policy. It has never seen that policy, so its most likely answer is a plausible guess, and at temperature 0 you get the same guess every time. That repeatability can make the guess look more trustworthy than it is. Numerical and serving differences can also make outputs vary slightly even at temperature 0.
The three stages: pretraining, fine-tuning and inference
People often mix these stages up, in interviews and in design discussions, so it's worth keeping them apart.
Pretraining
Pretraining builds broad capability from very large datasets by making the correct next token more likely. Formally, training minimizes the negative log probability of the correct next token, averaged over the training tokens.
A worked example: if the model gave the true next token a probability of 0.25, the loss at that position is −ln(0.25) ≈ 1.39. If it gave it 0.9, the loss falls to about 0.11. Over billions of positions, training adjusts the parameters to push those probabilities up.
Fine-tuning and preference training
Supervised fine-tuning updates the weights to teach a narrower behaviour or task, such as following instructions or answering in a particular format. Preference training then nudges the model toward responses that people, or a reward model, preferred.
For example, a team might fine-tune on thousands of support question-and-answer pairs so the model learns the tone and structure of good answers. Preference data might then show raters two answers to the same question and record which one helped more.
Inference
Inference is what happens when you use the model: the trained weights stay fixed, and the model applies them to your input. It does not normally change the weights.
So chatting with a model all day teaches it nothing permanent. The conversation shapes behaviour inside that context only, and a new session starts from the same weights.
Few-shot examples are not fine-tuning
Putting examples in a prompt, known as few-shot prompting, is inference. The examples change the model's behaviour for this request only.
Let's say you add three labelled support tickets to a prompt before asking the model to classify a fourth. Accuracy may improve right away, and if you remove the examples the improvement disappears, because nothing in the parameters changed. Fine-tuning becomes worth considering when you need the behaviour across thousands of requests without paying for long prompts, ideally after you have measured a real gap.
Why LLMs hallucinate
A hallucination is output that sounds plausible but isn't supported by facts or by the provided sources. It follows from how the model works: it generates likely text, and likely text can be false.
The usual causes are missing knowledge (the fact is too recent, too niche or private), missing or outdated context, and the model producing an answer-shaped response when the honest reply is "I don't know". Neither parameter count nor fluent wording guarantees correctness, and a larger model can't make up for being given the wrong company's policy.
How to build around an LLM
A model can sound right and be wrong, so reliable applications split the work. In an employee policy assistant, for example, the LLM writes the answer, a retriever supplies the current policy passages as context, and application code checks that the user is allowed to see those passages.
A few habits do most of the work:
- Measure on your task. Build a small set of real test cases, including ambiguous and unanswerable ones, and score answers against it.
- Supply evidence for facts that matter by retrieving and citing them, instead of relying on the model's memory.
- Keep permissions and business rules in your application code, not in the prompt.
- Decide what happens when evidence is missing. The right answer is often "I can't find that".
When an answer is wrong, check those parts in order: was knowledge missing from the model, was the supplied context wrong or incomplete, or did a tool or check fail?
How to choose a model for a task
Leaderboards and parameter counts tell you little about your own task. A better process:
- Write representative evaluation cases with expected results.
- Set requirements for quality, latency, privacy and cost.
- Run two or three candidate models, or decoding settings, on the same cases.
- Compare results, including cost per correct answer rather than cost per request.
For example, design 20 classification or question-answering cases, some of them ambiguous or unanswerable, and compare two models without changing the test cases. Before you start, write down how you'll score correctness, appropriate refusals, p95 latency and cost.
Frequently asked questions
What is the difference between pretraining, fine-tuning and inference?
Pretraining learns broad patterns from large datasets. Fine-tuning updates the weights for a narrower behaviour or task. Inference applies fixed weights to a new input. Supplying examples in a prompt is inference, not fine-tuning.
Does setting temperature to zero eliminate hallucinations?
No. It makes selection close to greedy decoding, but the highest-probability completion can still be wrong. Numerical and serving differences can also affect repeatability.
Do LLMs learn from my conversations?
Not during inference. The weights stay fixed while you chat. A provider may use conversations to train future models, depending on its policies, but your current session doesn't update the model you are talking to.
Is a bigger model always better?
Not for a specific task. A smaller model with the right context and good evaluation often beats a larger one used without them, so choose by measuring on your own cases.
Key takeaways
- An LLM predicts the next token, appends it and repeats. It does not look up facts.
- Softmax turns logits into probabilities, decoding picks the token, and temperature reshapes the distribution.
- Temperature 0 means predictable, not correct.
- Pretraining, fine-tuning and inference are different stages, and prompt examples are inference.
- Reliable systems pair the model with retrieved evidence, permission checks in code and measured evaluation.
The free Hugging Face LLM course is a practical next step. The next post in this series covers how the transformer architecture works, the network design behind almost every modern LLM.