ABCs of the Math Behind Large Language Models

You don't need a PhD to understand how ChatGPT or Claude works under the hood. The mathematics behind Large Language Models (LLMs) boils down to a handful of ideas — and once you see them clearly, the "magic" of AI becomes wonderfully logical. This article walks through each concept in plain language, with analogies and examples throughout.

Step 1 — Numbers Are the Only Language Computers Speak

Before anything else, understand this: computers only work with numbers. They can't read the word "cat" — but they can work with the number 0.82, or a list of numbers like [0.82, 0.14, 0.56].

So the first challenge of AI is: how do we convert human language into numbers in a meaningful way?

Vectors: Lists of Numbers with Direction

A vector is just an ordered list of numbers. Think of it like coordinates on a map — but instead of 2 coordinates (x, y), we might use 768 or even 4096 numbers to describe a word.

Words with similar meanings end up with similar vectors. This is the core insight of word embeddings.

Matrices: Tables of Numbers

A matrix is a grid (table) of numbers. If a vector is a single row, a matrix is many rows stacked together. The entire vocabulary of an LLM — say, 50,000 words — is stored as a matrix where each row is that word's vector.

Matrix multiplication (multiplying two grids of numbers together) is the most important operation in all of deep learning. Modern GPUs are designed to do this billions of times per second.

Step 2 — Turning Words Into Meaningful Numbers (Embeddings)

An embedding is a learned vector representation of a word (or token). "Learned" means the model figured out good number values through training — nobody hand-coded them.

The famous example from Word2Vec captures the idea beautifully:

This works because the model learned that the difference between King and Man encodes the concept of "royalty," and adding that concept to Woman produces Queen. The math is capturing real semantic meaning.

Tokens, Not Just Words

Modern LLMs don't just embed whole words. They break text into tokens — which might be whole words, parts of words, or punctuation. The word "unbelievable" might become ["un", "believ", "able"]. This lets the model handle any word, even ones it's never seen before.

Step 3 — Attention: How the Model Focuses

This is the core innovation of the Transformer. Before attention, models read text left-to-right like a person with a very short memory. Attention lets the model look at all words simultaneously and decide which ones are relevant to each other.

The Spotlight Analogy

Imagine reading the sentence: "The trophy didn't fit in the suitcase because it was too big."

What does "it" refer to? You instantly know it's "the trophy" — because you can look back and connect the words. This is exactly what attention does mathematically.

Queries, Keys, and Values (Q, K, V)

Attention uses three matrices — Q (Query), K (Key), and V (Value). The analogy is a library search:

Mathematically, the attention formula is:

This one equation is the heart of every LLM in existence today.

Multi-Head Attention

Real transformers run this attention mechanism in parallel multiple times (8, 12, or 16 "heads"). Each head learns to pay attention to different aspects — one might focus on grammar, another on meaning, another on long-range references. Their outputs are then combined.

Step 4 — The Full Transformer Architecture

A Transformer stacks many layers, each containing attention + a feed-forward network. Here's what one layer does:

Positional Encoding — Teaching Order

Since attention looks at all words simultaneously, the model needs to be told the position of each word. Positional encodings are mathematical patterns (sine and cosine waves at different frequencies) added to each word's embedding. This way "dog bites man" and "man bites dog" produce different patterns even though they use the same words.

Softmax — Turning Scores Into Probabilities

The final layer of an LLM outputs one number per word in the vocabulary (50,000+ numbers). Softmax converts these into probabilities that sum to 1.0 — and the highest probability word is selected as the next token.

Step 5 — Training: How Models Learn

Training is how a model adjusts its millions (or billions) of internal numbers to get better at predicting text. The process:

Loss Functions

The loss function measures how wrong the model's prediction is. For LLMs, this is typically cross-entropy loss — it's large when the model is very wrong and near zero when the model is confident and correct.

Gradient Descent & Learning Rate

Imagine the loss as a hilly landscape. Gradient descent is like rolling a ball downhill — the model keeps adjusting its weights in whatever direction reduces the loss. The learning rate controls how big each step is. Too large → overshooting. Too small → takes forever.

Putting It All Together

An LLM is, at its core, a very deep stack of matrix multiplications guided by attention. There is no "understanding" in the human sense — but through training on hundreds of billions of words, the patterns of human language are encoded into billions of numbers so precisely that the output is often indistinguishable from human writing.