How Large Language Models Work

Every answer from ChatGPT, Claude, or Gemini comes from one mechanism repeated billions of times: predicting the next token. A large language model is a probability machine. It reads a sequence of text and computes which fragment is most likely to come next. That single trick, scaled up with enough data and compute, produces essays, working code, and translations. It also explains why these systems sometimes invent facts with total confidence.

A large language model (LLM) is a neural network trained on massive text collections to predict the next token in a sequence, which is what lets it generate, summarize, and translate language.

Where did language models come from?

Before deep learning, computers processed language with hand-coded rules and statistical models such as n-grams. Those systems counted word sequences but could not generalize: they had no notion of meaning.

The turning point was word embeddings, which represent words as dense vectors in continuous space. Distance between vectors started to encode similarity in meaning, and machines could finally learn semantics from data. Four milestones built the path to modern LLMs:

• word2vec (2013). Tomas Mikolov and colleagues at Google showed that unsupervised learning could place words in a vector space where arithmetic captures meaning: king minus man plus woman lands near queen. The paper is Efficient Estimation of Word Representations.
• Transformers (2017). The paper Attention Is All You Need by Vaswani et al. introduced self-attention, letting a model weigh every word in a sequence at once. That enabled parallel training and made long-range dependencies tractable.
• ELMo (2018). Embeddings from Language Models produced contextual vectors: the word "bank" gets different representations in "river bank" and "savings bank", a leap toward handling words with multiple meanings.
• BERT (2018). Google's bidirectional transformer masked random words and trained the model to fill them in, reading context from both directions. Question answering and sentiment analysis improved sharply.

What is a transformer and why did it change everything?

The transformer architecture is the foundation of nearly every modern LLM, including GPT, Claude, LLaMA, Gemini, and Mistral.

Earlier recurrent networks (RNNs) read text one token at a time, which made training slow and long sentences hard to model. Transformers process whole sequences in parallel through self-attention: each token decides how much weight to give every other token in the sequence.

The main components:

• Tokenization. Input text is split into subword units, often with Byte Pair Encoding (BPE).
• Embedding layer. Each token becomes a dense vector that preserves semantic relationships.
• Self-attention. Every token gets three vectors: Query (Q), Key (K), and Value (V). Attention scores come from the dot product of Q and K, normalized with softmax, and they weight the V vectors so the model focuses on the relevant parts of the input.
• Feedforward layers. After attention, each representation passes through a dense network for further transformation.
• Positional encoding. Since nothing is processed in order, position information is injected explicitly.
• Residual connections and layer normalization. These keep gradients flowing and training stable in very deep networks.

The video below is a strong visual companion to the original paper. It shows the architecture in a way that works for technical and non-technical viewers alike.

How does a model generate text, step by step?

Language models do not understand text the way humans do. They compute probability distributions over a vocabulary and pick the next token. The full path from input to output:

1. Tokenization. "The dog barks" becomes ["The", " dog", " barks"], then a list of token IDs.
2. Embedding. The IDs pass through an embedding layer and become dense vectors.
3. Transformer blocks. The vectors flow through stacked self-attention and feedforward layers.
4. Logits and softmax. The output layer produces raw scores for every token in the vocabulary, which softmax converts into probabilities.
5. Decoding. Greedy decoding picks the single most likely token. Top-k sampling draws from the k most likely. Nucleus sampling draws from the smallest set whose cumulative probability passes a threshold, typically 0.9.

How are LLMs trained?

Training means exposing the model to enormous text datasets and adjusting billions of parameters until its predictions improve. Pretraining a frontier model can run for weeks on clusters with thousands of GPUs and cost tens of millions of dollars.

Pretraining

The objective is simple: predict the next token (GPT-style) or fill in masked tokens (BERT-style) across books, websites, code, and academic text. Scale varies enormously. GPT-3 was trained on roughly 300 billion tokens in 2020; Meta's LLaMA 3 used about 15 trillion in 2024, including code and multilingual data, with open weights. Gemini 1.5 combines a Mixture-of-Experts architecture with multimodal input. Anthropic trains Claude with Constitutional AI, where the model critiques its own outputs against a set of written principles.

Fine-tuning

Pretraining produces a text predictor, not an assistant. Fine-tuning aligns the model with human expectations: following instructions, refusing harmful requests, staying on task.

Instruction fine-tuning pairs prompts with high-quality responses until the model generalizes to instructions it has never seen. Reinforcement Learning from Human Feedback (RLHF) goes further:

1. The model generates several answers to the same prompt.
2. Human annotators rank the answers from best to worst.
3. The rankings train a separate reward model that scores outputs.
4. Reinforcement learning, commonly PPO, updates the base model to maximize that reward.

RLHF powered InstructGPT, one of the first aligned models. Anthropic extended the idea with Constitutional AI, which replaces part of the human feedback with principle-based self-critique.

Why do LLMs sound smart without understanding?

An LLM writes essays, explains jokes, and debugs code, yet it holds no beliefs and has no intent. It is a probabilistic engine trained to continue text in plausible ways.

The fluency comes from patterns memorized across huge datasets, attention that retrieves the relevant context at each step, and sensitivity to phrasing, which is why "let's think step by step" changes results.

The risk has the same structural origin. In the 2021 paper known as Stochastic Parrots, Bender and colleagues warned that ever-larger models produce authoritative-sounding text without factual grounding or accountability. Hallucination is not a bug awaiting a patch; it is the default behavior of a system that optimizes plausibility.

Where are LLMs heading?

Three research directions define the next generation:

• Mixture-of-Experts (MoE). Only a small subset of parameters activates for each input, so capacity scales without proportional compute. Introduced by Shazeer et al., 2017.
• Chain-of-thought reasoning. Prompting a model to reason step by step measurably improves math and logic performance, as shown by Wei et al., 2022.
• Retrieval-Augmented Generation (RAG). The model fetches external documents before answering, grounding output in current sources. The Cohere RAG guide is a practical introduction, and our own article on RAG covers the current research.

Models are also becoming multimodal, working across text, images, and audio; holding far longer contexts; and acting as agents that call tools and APIs.

Why does this matter for your visibility?

LLMs now answer questions people once typed into Google. When a model summarizes your market, its training data and its retrieved sources decide whether you appear. Content that defines terms clearly and makes verifiable claims is easier for a probability machine to quote.

That shift from ranked links to generated answers is the subject of our article on generative engines. Whether the answers actually mention you is measurable: RankWit monitors your brand across ChatGPT, Perplexity, Gemini, Claude and Grok.

FAQ

What is a token?

A token is the unit of text an LLM actually processes, usually a word fragment. "Understanding" might split into "under", "stand", "ing". Models read and produce tokens rather than words, and every context window limit or API price you see is counted in tokens. In English, a token averages around four characters.

Do LLMs understand what they write?

Not in the human sense. An LLM has no beliefs, goals, or model of truth. It computes which token is most probable next, given everything before it. The behavior often looks like understanding because language patterns encode a lot of real-world structure, but the mechanism is statistical prediction, which is why confident errors happen.

What is the difference between pretraining and fine-tuning?

Pretraining teaches the model language itself: next-token prediction over huge general datasets, at enormous cost. Fine-tuning is a shorter, cheaper second phase that shapes behavior, using instruction examples or human feedback (RLHF) to make the model helpful, safe, and able to follow directions. Pretraining builds capability; fine-tuning builds an assistant.

Why do LLMs hallucinate?

Because they optimize plausibility rather than truth. When the training data contains no reliable answer, the model still produces the most statistically likely continuation, which can be a fabricated citation or a wrong date delivered fluently. Retrieval-augmented generation reduces the problem by grounding answers in fetched documents, but no current system eliminates it.