How Large Language Models Work

The Origins of Language Models

‍

Before deep learning, computers processed language using hand-coded rules and statistical models like n-grams.

These early systems could not capture the nuances of human language and failed to generalize.

The turning point came with word embeddings, a way to represent words as dense vectors in continuous space.

This marked the beginning of teaching machines the meaning of words through patterns in data.

‍

Key Historical Milestones

‍

Below are the major milestones that paved the way for the LLMs we use today.

• word2vec (2013)
  Developed by Tomas Mikolov and colleagues at Google, word2vec was a groundbreaking model that used unsupervised learning to represent words as vectors in a continuous space. It captured semantic similarity. For istance, the vector for king minus man plus woman is close to queen. This simple yet powerful idea introduced the concept of distributed word representations.
  🔗 Efficient Estimation of Word Representations
• Transformers (2017)
  Although BERT and many others use the Transformer architecture, its origin lies in the 2017 paper Attention Is All You Need by Vaswani et al.
  This paper introduced the self-attention mechanism, allowing models to weigh the importance of all words in a sequence at once.
  This not only enabled parallel processing (faster training) but also made it easier to model long-range dependencies, such as connecting a noun in the first sentence to a pronoun in the fourth.
  🔗 Attention Is All You Need
• ELMo (2018)
  ELMo, or Embeddings from Language Models, took things further by generating contextualized embeddings. Unlike word2vec, ELMo produced different vector representations for the same word depending on its sentence context, "bank" in “river bank” vs. “savings bank.”
  This was a key leap toward understanding polysemy (words with multiple meanings).
  🔗 Deep Contextualized Word Representations
• BERT (2018)
  Google’s BERT (Bidirectional Encoder Representations from Transformers) introduced a bidirectional pretraining method, enabling the model to understand context from both the left and right of a word. Instead of predicting the next word, BERT masked random words in a sentence and trained the model to fill them in, which significantly improved performance on tasks like question answering and sentiment analysis.
  🔗 BERT: Pre-training of Deep Bidirectional Transformers

‍

‍

What Is a Transformer and Why It Changed Everything

‍

The Transformer architecture, introduced in the 2017 paper “Attention Is All You Need,” is the foundation of nearly all modern large language models, including GPT, Claude, LLaMA, Gemini, and Mistral.

Unlike recurrent neural networks (RNNs), Transformers process entire sequences simultaneously using self-attention, which lets each token decide how much attention to give to others.

In other words, it replaced sequential processing with self-attention, enabling models to process entire sequences in parallel and model long-range dependencies more effectively than Recurrent Neural Networks (RNNs).

But what exactly makes a Transformer work?

Let’s break down its core components, each playing a crucial role in how these models encode, attend to, and transform text.

• Tokenization
  The input text is split into subword units (tokens), often using methods like Byte Pair Encoding (BPE).
  
• Embedding Layer
  Each token is converted into a dense vector using a learned embedding matrix.
  This gives the model a way to work with numerical input that preserves semantic relationships.
• Self-Attention Mechanism
  At the heart of the Transformer is self-attention, which allows the model to weigh the importance of each token in relation to others in the sequence.
  • Each token is assigned three vectors: Query (Q), Key (K), and Value (V)
  • Attention scores are computed as the dot product of Q and K, scaled and normalized using softmax.
  • These scores are used to weight the V vectors, allowing the model to dynamically focus on relevant parts of the input.
• Feedforward Network
  After self-attention, each token’s representation is passed through a fully connected (dense) neural network to enable more complex transformations.
• Positional Encoding
  Since Transformers do not process input sequentially, they use positional encodings to inject information about token order into the model.
• Residual Connections & Layer Normalization
  These architectural features help stabilize training and speed up convergence by ensuring better gradient flow through deep networks.

This video serves as an excellent companion to the "Attention Is All You Need" paper by Vaswani et al. (2017).

It visualizes the architecture and inner workings of the Transformer in a way that's accessible to both technical and non-technical audiences, highlighting key ideas.

‍

‍

From Tokens to Language

‍

Language models don’t understand text like humans.

They predict the most likely next token based on everything seen before, using probability distributions over vocabulary.

‍

Step-by-Step Flow

‍

1. Tokenization:
   Input like “The dog barks” becomes [“The”, “ dog”, “ barks”], then mapped to token IDs.
2. Embedding:
   These IDs are passed through an embedding layer to produce dense vectors.
3. Transformer Blocks:
   These vectors are processed through multiple self-attention and feedforward layers.
4. Logits & Softmax:
   The output is a vector of logits (raw scores) that are converted into probabilities using softmax.
5. Decoding Strategies:
   • Greedy decoding: Choose the highest probability token.
   • Top-k sampling: Sample from the top k most likely tokens.
   • Nucleus sampling: Sample from the smallest set of tokens whose cumulative probability exceeds a threshold (usually 0.9).

‍

‍

‍

‍

Training LLMs: The Brains Behind the Words

‍

Training a large language model involves exposing it to vast text datasets and teaching it to predict tokens.

This process can take weeks on supercomputers with thousands of GPUs.

Training a large language model involves exposing it to massive text datasets and teaching it to predict tokens, a process that can take weeks on AI supercomputers with over 10,000 GPUs, consuming hundreds of zettaFLOPs of compute and costing tens of millions of dollars.

A zettaFLOP — short for zetta floating-point operations per second — is a unit of computational power equal to 10²¹ operations per second (that’s 1 sextillion, or a 1 followed by 21 zeros). While zettaFLOP-scale performance remains largely theoretical for sustained tasks, it’s a useful way to express the total cumulative compute required to train today’s most advanced AI models.

‍

Pretraining

‍

• Objective: Learn statistical patterns in language by predicting next (GPT-style) or masked (BERT-style) tokens.
• Data: Books, websites, code, social media, academic texts.
• Models:
  • GPT-3 (OpenAI, 2020)
    Trained on 300B tokens using causal language modeling
  • Claude (Anthropic)
    Uses Constitutional AI: a reinforcement learning approach where models self-critique based on a set of principles
  • LLaMA 3 (Meta AI, 2024)
    Trained on 15T tokens, including code and multilingual data; open-weight models available
  • Gemini 1.5 (Google DeepMind, 2024)
    Uses a Mixture-of-Experts (MoE) architecture and supports multimodal inputs (text, images, audio)
  • PaLM 2 (Google, 2023)
    Trained on multilingual corpora, code, and scientific data; improved reasoning and translation capabilities
  • Grok (xAI, 2023–2024)
    Trained on real-time X (Twitter) data, with access to proprietary user-generated content
  • Command R+ (Cohere)
    Retrieval-augmented generation (RAG) optimized for long-context enterprise tasks
  • Mistral 7B / Mixtral (Mistral AI, 2023)
    Highly efficient dense and sparse MoE models — open weights and strong performance at small scale
  • Phi-2 (Microsoft Research, 2023)
    A small (1.3B parameter) model trained with textbook-style data, optimized for reasoning efficiency
  • GatorTron (UF Health + NVIDIA, 2022)
    Trained on clinical and biomedical records for medical NLP applications
  • WuDao 2.0 (Beijing Academy of AI)
    One of the largest multilingual/multimodal models — trained on 1.75T parameters and diverse corpora including Chinese and English

‍

Fine-Tuning

After pretraining, large language models can be fine-tuned to perform better on specific tasks or align more closely with human expectations. This step is optional but widely used to make models more useful in real-world applications.

Fine-tuning helps the model:

• Follow human instructions more accurately
• Be more helpful, honest, and harmless
• Align with specific goals (e.g., customer support, legal advice, education)

Instruction Fine-Tuning

This is the most common approach. The model is trained on examples where inputs are paired with high-quality desired responses. Over time, it learns to generalize and follow similar instructions even if they weren’t part of training.

Reinforcement Learning from Human Feedback (RLHF)

One of the most powerful fine-tuning techniques. It improves alignment through feedback from human evaluators:

1. Generate outputs: The model produces multiple answers to a prompt.
2. Human ranking: Annotators rank the outputs from best to worst.
3. Train a reward model: The rankings are used to train a separate model that scores outputs.
4. Fine-tune the main model: Using reinforcement learning (commonly PPO – Proximal Policy Optimization), the base model is updated to maximize this reward signal.

This method was used in InstructGPT, one of the earliest aligned models (Ouyang et al., 2022), and later extended by Anthropic with Constitutional AI, which teaches models to critique and revise their own responses based on predefined ethical guidelines (Bai et al., 2022).

‍

Why LLMs Sound Smart (But Aren’t)

‍

Despite writing essays, explaining jokes, or writing code, LLMs don’t actually “understand” anything.

They don’t form beliefs or possess intent.

They are probabilistic engines trained to continue text sequences in plausible ways.

They simulate intelligence by:

• Memorizing patterns in huge datasets
• Using attention to retrieve relevant context
• Reacting to prompt phrasing (e.g., "Let's think step-by-step")

‍

“Training ever-larger language models without addressing underlying limitations risks creating systems that sound authoritative but lack accountability or factual grounding.” 📚 Stochastic Parrots: Bender et al., 2021

‍

‍

‍

The Future of LLMs: Agents, Memory, and Reasoning

‍

Next-generation language models are evolving rapidly — not just in scale, but in capability.

These models are becoming:

• Multimodal: Processing and generating across text, images, and audio
• Long-context aware: Maintaining memory over hours or even weeks (e.g., Claude 3.5, Gemini 1.5)
• Agentic: Taking actions via tools, APIs, and dynamic environments

‍

Want to Understand What’s Next for LLMs?

‍

As language models evolve into more capable and autonomous systems, several core paradigms are shaping their future.

3 foundational ideas:

🧠 Mixture-of-Experts (MoE)
Tip: Read Shazeer et al., 2017
MoE models improve efficiency by activating only a small subset of their parameters for each input — making it possible to scale up without proportionally increasing compute.

🧩 Chain-of-Thought Reasoning
Tip: Read Wei et al., 2022
This prompting strategy encourages models to think step by step, significantly improving performance on complex reasoning and math tasks.

🔍 Retrieval-Augmented Generation (RAG)
Tip: Explore the Cohere RAG Guide
RAG combines language models with external knowledge sources, allowing them to pull in relevant information from databases or documents before generating responses.

‍

These techniques are the building blocks of next-gen AI systems.

Start with these papers to see where the future is heading.

‍

Understanding LLMs Is the New Digital Literacy

‍

Large Language Models represent a seismic shift in human-computer interaction.

They are probabilistic engines of knowledge synthesis.

If search engines were about keywords, LLMs are about context, clarity, and citations.

‍

‍