How Does AI Work: Transformers, Neural Networks & Next-Token Prediction Explained

AI works by prediction. When you ask a chatbot a question, it does not look up an answer, understand your meaning the way a person does, or reason from first principles. It turns your words into numbers, weighs those numbers against each other to work out which parts of your message matter most, and then guesses the most likely next piece of text, one piece at a time, until it has produced a full reply. Everything that looks like thinking, creativity, knowledge or conversation in modern AI comes out of that single mechanism, repeated at enormous speed and scale. This article explains the whole pipeline step by step, in plain language first and then in real technical detail, so that by the end you can explain not just what AI does, but why it is fluent, why it is sometimes wrong, and why it sounds so confident either way.

One clarification before we start, because the word AI covers a lot of territory. There are older kinds of AI, such as systems that sort email into spam and inbox, or that recognise a face in a photo. This article is about the kind of AI most people mean when they ask how AI works today: large language models, the systems behind ChatGPT, Claude, Gemini and their rivals. They are all built on the same architecture, called the Transformer, and they all work the way this article describes. Where another kind of AI works differently, the article says so explicitly.

The simple explanation

Think of the autocomplete on your phone, grown up beyond recognition. When you type a message, your phone suggests the next word based on what you usually write. A large language model does the same thing, except that instead of learning from your messages alone, it has learned from an enormous slice of human writing — books, websites, code, articles — and instead of guessing from the last word or two, it considers your entire message at once, weighing every part of it. That is the whole trick. Fluency comes from having seen so much real writing that its guesses sound human. Creativity comes from the fact that it does not always pick the single most likely next piece; it samples from the plausible ones, so the same prompt can produce different answers. And mistakes come from the same source: the model has no fact-checker inside it, only a sense of what sounds likely, so when the likely-sounding continuation is false, it will still produce it, confidently.

To turn that intuition into a real understanding, you need five stages. Every time a language model produces a reply, your text passes through the same five stages, in order: ingestion, tokenization, embeddings, self-attention, and softmax. The rest of this article takes each stage in turn, then traces one sentence through all five, then explains how the system learned to do any of this in the first place.

Stage 1: Ingestion — gathering the raw material

Before a model can learn anything, it needs text to learn from, and the scale of that text is the first thing that separates modern AI from everything before it. Pretraining corpora are built from books, websites, code repositories, academic papers and encyclopaedias, running to hundreds of billions or trillions of pieces of text. GPT-3, published by OpenAI in 2020, trained on roughly 300 billion tokens, and later models have trained on far more. The point of this scale is not memorisation, although models do memorise some passages. The point is coverage: with enough varied writing, the patterns of grammar, facts, reasoning styles and genres recur often enough for the model to absorb them as statistical tendencies rather than as stored quotes.

Ingestion is also where the model’s limits are set. A model cannot know what was never in its training data, or what happened after its training data was collected, unless it is given a way to look things up at reply time, such as web search or a document you paste in. That is why models have a knowledge cutoff date, and why a model asked about last week’s news will either say it does not know or invent a plausible-sounding answer. The data it ingested is its entire world, and everything downstream works only with what ingestion provided.

At reply time, ingestion means something smaller and more immediate: your prompt itself is the input being ingested, along with any conversation history and any documents attached to it. All of that together is the text the next four stages will process. The model’s working memory for this input is called its context window, the maximum amount of text it can consider at once. Early chatbots managed a few thousand tokens, roughly a long email thread. Modern systems claim windows in the millions of tokens, whole codebases or whole books, though in practice attention dilutes as the window grows, and the marketing claim and everyday usefulness are not always the same thing.

Stage 2: Tokenization — chopping text into pieces

Models cannot read. They compute, and computation needs numbers, so the first thing that happens to your text is that it is chopped into tokens. A token is a chunk of text, often a whole common word, often a piece of a word, sometimes a single character or punctuation mark. The sentence “The cat sat on the mat” might become tokens like “The”, ” cat”, ” sat”, ” on”, ” the”, ” mat”. An unusual word like “tokenization” might split into “token”, “ization”, or smaller pieces. In English, tokens average roughly three to four characters each, and a model’s vocabulary, the full list of tokens it knows, runs into the tens of thousands or beyond a hundred thousand entries.

Tokenization matters more than it looks, because the model never sees your words, only your tokens. Two prompts that mean the same thing to you can tokenize differently, and the model may respond differently to each. Typos, unusual spellings and rare words tokenize into stranger pieces, which is one reason models handle familiar phrasing better than odd phrasing. It also explains a quirk people notice: models are unreliable at counting letters in a word or doing exact character arithmetic, because they never see individual letters unless the tokenizer happens to split that way. They see tokens, and a token is already a bundle of characters whose internal structure is invisible to them.

Each token is then mapped to a number, its ID in the vocabulary. From here on, the model works entirely with numbers. Nothing downstream ever sees the word “cat”. It sees token number whatever-it-is, and everything the model knows about cats must be carried by what happens to that number in the next stages.

Stage 3: Embeddings — turning tokens into meaning as numbers

A token ID on its own carries no meaning. The number 4821 does not know it represents “cat”. The embedding stage fixes that by converting each token ID into a long list of numbers, a vector, that places the token in a kind of meaning-space. Tokens that appear in similar contexts end up with similar vectors. “Cat” and “dog” land near each other, because both appear in sentences about pets, vets and feeding. “Cat” and “democracy” land far apart. The vector does not contain a definition of a cat. It contains a position, and that position encodes, implicitly, everything the training data revealed about how “cat” is used.

These vectors are not hand-designed. They are learned during training, starting from random values and gradually adjusted as the model practices next-token prediction. Early in training, “cat” and “dog” are no closer than “cat” and “democracy”. By the end, the geometry of the space reflects the structure of language itself: directions in the space correspond to relationships, so the difference between “king” and “queen” points roughly the same way as the difference between “man” and “woman”, and the difference between a country and its capital is a consistent direction too. This is why people say the model “understands” words. It does not, in the human sense. But its number-space captures relationships between words well enough to behave as if it does, for a surprising range of tasks.

One more ingredient is added at this stage, because embeddings on their own have no sense of order. A list of token vectors is a bag, not a sentence: “dog bites man” and “man bites dog” would look identical. The Transformer solves this with positional encodings, extra numbers added to each embedding that record where the token sits in the sequence. The original 2017 design used sine and cosine waves of different frequencies for this, so that each position has a unique signature and the model can learn relative distances between positions. Order is therefore injected, not built in, which is a subtle but important fact about the architecture: the model does not read left to right. It receives the whole sequence at once, plus position markers, and works out relationships from there.

Stage 4: Self-attention — how each word weighs every other word

Self-attention is the mechanism that made the Transformer famous, and it is the heart of how AI works. For each token in your input, the model asks a question: which other tokens in this sequence matter most for understanding this one? It answers by computing a relevance score between every pair of tokens, then rebuilding each token’s representation as a weighted mix of all the others, weighted by those scores. In the sentence “The cat sat on the mat because it was tired”, understanding “it” means attending strongly to “cat”, not to “mat”. The model learns to make exactly that kind of judgement, for every token, in every sentence, without being told any grammar rules.

Mechanically, each token’s embedding is projected into three vectors: a query, a key and a value. Think of the query as the question “what am I looking for?”, the key as the label “what do I offer?”, and the value as the content itself. The attention score between two tokens is the dot product of the first token’s query and the second token’s key: a high score means high relevance. Those scores are scaled, passed through a softmax function so they sum to one, and used as weights to blend the value vectors together. The formula, from the 2017 paper, is Attention(Q, K, V) = softmax(QK^T / √d_k)V, where the scaling by the square root of the key dimension keeps the scores from growing so large that learning stalls.

Two refinements make this powerful rather than merely clever. The first is multi-head attention: instead of computing one set of relevance scores, the model computes many in parallel, called heads, each learning to track a different kind of relationship. One head might track subject-verb agreement, another might track pronoun references, another might track which words belong to the same phrase. The original paper used eight heads per layer. The second refinement is stacking: the output of one attention layer feeds into the next, and the original Transformer stacked six layers in its encoder and six in its decoder, each layer also containing a feed-forward network that processes each token’s representation independently. By the top of the stack, each token’s representation has absorbed context from the whole sequence, refined through many rounds of attention. That deep, context-soaked representation is what the final stage uses to predict.

Why did this beat everything before it? Older models read text word by word, in sequence, carrying a summary forward in a hidden state. That worked, but it forgot the start of long sentences, and it could not be parallelised: word ten could not be processed until word nine was done. Self-attention removes the sequence bottleneck. Every token attends to every other token at once, so the whole input can be processed in parallel, which is what allowed training to scale to the enormous datasets and model sizes of today. The 2017 paper’s title was a claim about exactly this: attention is all you need, no recurrence, no convolution, just attention plus feed-forward layers.

Stage 5: Softmax — turning understanding into a prediction

After the stack of attention layers, the model holds a rich representation of your entire input. The final stage converts that representation into a prediction: a probability for every token in its vocabulary, answering the question “given everything that came before, how likely is each possible next token?” The softmax function is what turns raw scores into probabilities that sum to one. If your prompt ends “The cat sat on the”, the model might assign high probability to ” mat”, moderate probability to ” floor” or ” couch”, and near-zero probability to ” democracy” or ” quantum”. It then picks one token, not necessarily the most likely one, and appends it to the sequence.

That picking step is where the model’s personality comes from. A setting called temperature controls how much randomness enters the choice. At low temperature, the model almost always picks the top-ranked token, producing predictable, consistent, sometimes repetitive text. At higher temperature, it samples further down the probability list, producing more varied and creative text, and also more errors, because the less likely continuations include the false ones. This is a genuine trade-off, not a bug: the same mechanism that makes a model creative makes it unreliable, and the same mechanism that makes it reliable makes it dull. Products tune temperature behind the scenes, which is why the same underlying model can feel precise in one app and playful in another.

Then the loop repeats. The chosen token is appended to the input, the whole sequence runs through the stages again, and the next token is predicted. One token at a time, hundreds or thousands of times, a paragraph appears. That is why generation speed is measured in tokens per second, and why a long answer takes visibly longer than a short one: the model is doing the full computation once per token. It is also why a model can paint itself into a corner. Once it has committed to a wrong token, everything after it is conditioned on that mistake, and the model will often continue fluently down the wrong path rather than backtrack, because backtracking is not part of the mechanism. There is no draft, no edit pass, no moment where the model reviews its answer. There is only the next token, again and again.

Worked example: one sentence through all five stages

Take the prompt “The cat sat on the” and watch what happens when the model completes it. At ingestion, the prompt plus any conversation history enters as raw text. At tokenization, it is chopped into tokens, roughly “The”, ” cat”, ” sat”, ” on”, ” the”. At embeddings, each token becomes a vector positioned in meaning-space, with positional encodings marking first, second, third, fourth, fifth. At self-attention, each token weighs the others: “sat” attends strongly to “cat”, because cats are things that sit; “the” before the blank attends to the whole phrase, because it signals a noun is coming. The stack refines these representations through its layers until the final representation encodes, in numbers, something like “a sentence about a cat sitting, awaiting the thing it sat on”.

At softmax, that representation becomes a probability distribution over the vocabulary. The numbers below are illustrative, not measured from any particular model, but their shape is realistic:

At low temperature the model picks ” mat” and the sentence completes as expected. At higher temperature it might pick ” couch”, giving a slightly more original sentence that is still correct. It will essentially never pick ” democracy”, because the probability is vanishingly small, but note what the mechanism does not do: it does not check whether cats actually sit on mats, or whether this particular cat exists. It picks by likelihood. Now extend the loop. Suppose the model has written “The cat sat on the mat. It was”. The next-token distribution now depends on the whole sequence, and ” tired” and ” happy” will both rank highly, because both complete cat sentences plausibly. The model picks one, and the story continues from there. Every paragraph you have ever read from a chatbot was built exactly this way, one sampled token at a time, with no plan beyond the next distribution.

How AI learns: training in two phases

Nothing above explains where the model’s abilities come from. The attention layers, the embeddings, the probability judgements are all learned, not programmed, and learning happens in two very different phases. Understanding the phases matters, because they explain both the model’s strengths and its characteristic failures.

Phase one is pretraining. The model starts with its billions of internal numbers, its parameters or weights, set to random values, and is given one relentless task: predict the next token. It reads a passage with the next token hidden, guesses, and is corrected against the real token. Each correction adjusts the weights a hair in the direction that would have produced a better guess, through a process called backpropagation. Repeat that across trillions of tokens and the model gradually absorbs grammar, facts, writing styles and reasoning patterns, all as a side effect of getting better at prediction. This phase is enormously expensive in computing power, and it produces what is called a base model: fluent, knowledgeable, but not yet an assistant. Ask a base model a question and it may complete it as if it were continuing a document, because completion is all it was trained to do.

Phase two is post-training, which turns the base model into the chatbot you actually use. First comes supervised fine-tuning: the model is trained on curated examples of good assistant behaviour, questions paired with good answers, so it learns the format of helpful responses. Then comes alignment, often through reinforcement learning from human feedback, where reviewers rank different responses and the model is nudged toward the answers people prefer: helpful, honest, safe. This phase uses far less data than pretraining, but it changes behaviour enormously. It is also where many of the model’s apparent manners come from. Politeness, refusals, structured answers with headings: those are post-training, layered on top of the raw prediction engine, not properties of the engine itself.

Training and inference are different jobs, and confusing them causes confusion about AI generally. Training is the slow, expensive process of adjusting weights, done once by the model’s maker. Inference is what happens when you use the model: your prompt runs through the finished, frozen network, and no learning happens. A chatbot does not learn from your conversation, does not remember you between sessions unless the product stores conversation history and feeds it back in as input, and does not update its weights because you corrected it. When a product seems to remember you, that is conversation history being re-ingested as context, not learning. The weights change only when someone trains a new version.

Scale is the last piece of the learning story, and it is the least intuitive. Nothing in the five stages says the system must be huge. Small models use the same stages. But across the field, researchers found that as models, datasets and computing grew, capabilities improved smoothly and sometimes jumped: grammar, then facts, then multi-step reasoning, then the ability to follow complex instructions. No one programmed those jumps. They emerged from more prediction practice at larger scale. That is why the frontier of AI is measured in parameters, tokens and computing power, and why the same architecture that translated sentences in 2017 now writes essays, code and poetry. The mechanism did not change. The scale did.

Discriminative vs generative AI: the distinction that defines the era

Most AI before the current wave was discriminative: it learned boundaries between categories. Given an email, is it spam or not? Given a photo, is it a cat or a dog? Given a transaction, is it fraud or legitimate? Discriminative models learn to draw lines. They are enormously useful, and they still run much of the world’s software, from fraud detection to medical screening. But they cannot produce anything new. They can only judge what you show them.

Generative AI, the subject of this article, learns the distribution itself rather than the boundary. Instead of learning what separates cats from dogs, it learns what cat-like text, or cat-like images, look like in general, well enough to produce new examples. A language model is generative because it produces new text token by token. An image model is generative because it produces new pixels. The table below captures the difference, because confusing the two is the source of many wrong expectations about AI.

The practical consequence is that generative AI’s failures look different from older AI’s failures. A spam filter that errs puts a good email in spam, and you notice. A language model that errs writes you a fluent paragraph with a false citation, a made-up statistic or a non-existent book, and you may not notice, because the failure mode is fluency itself. That is the single most important thing to carry from this article into everyday use: generative AI fails by sounding right.

What AI does not do

AI does not think. Thinking, in the human sense, involves beliefs, intentions, understanding and a model of the world that exists independently of language. A language model has none of these. It has weights shaped by text, and a mechanism for producing likely continuations. When it “reasons” through a maths problem step by step, it is producing the tokens that, in its training data, typically follow the setup of such a problem, and the steps often lead to the right answer because correct reasoning patterns are common in the data. When the pattern is unusual, the same mechanism produces confident nonsense. The reasoning is in the text patterns, not in a mind behind them.

AI does not know things. It has no beliefs it can check, no access to the world except through its training data and whatever you provide as input. When it states a fact, it is reproducing a pattern that was frequent in training, not reporting something it verified. That is why it can be wrong about simple, checkable facts while being right about obscure ones: frequency in training data, not truth, is what it tracks. And it does not feel anything. It has no experience of the words it produces, no awareness of itself as an entity, no continuity between conversations beyond the text fed back in as context. Claims that a model is sentient mistake fluent self-description for inner life. A model will describe itself as conscious if that is the likely continuation of the conversation, just as it will describe itself as a toaster if you set the conversation up that way.

None of this makes AI useless. A system that predicts the likely continuation of text, trained on humanity’s written output, is an extraordinary tool for drafting, summarising, translating, brainstorming and coding, precisely because so much human work consists of producing likely continuations of existing patterns. But the tool’s limits follow directly from its mechanism. For anything where correctness matters, dates, numbers, money, health, law, the output must be checked against a source outside the model. The model cannot check itself, because checking would require a different mechanism from the one it is built on.

What this one idea explains

Once you hold the next-token mechanism in mind, the daily headlines about AI become much easier to parse. Models are fluent because fluency is what they were trained to optimise. They are creative because sampling introduces variation into the same mechanism. They hallucinate because nothing in the mechanism checks facts. They are biased because their training data is biased, and likelihood tracks frequency. They improve with scale because more prediction practice sharpens the probability judgements. They struggle with exact counting, spelling and arithmetic because they see tokens, not characters or numbers. And they sound confident when wrong because confidence is not a separate signal in the system; it is the shape of the probability distribution, and a sharply peaked distribution around a false token looks exactly like a sharply peaked distribution around a true one.

It also explains why the same technology produces such different products. A chatbot, a coding assistant, a summariser and a search feature can all be the same prediction engine with different post-training, different temperature settings, different system instructions and different tools attached, such as web search or a code interpreter. The engine does not change. The scaffolding around it does. When you compare AI products, you are usually comparing scaffolding and post-training, not different kinds of intelligence.

Glossary

Token: a chunk of text, often a word or part of a word, that is the model’s unit of input and output. Models see tokens, never raw words or letters.

Embedding: a list of numbers representing a token’s position in meaning-space. Similar tokens have similar embeddings.

Weight / parameter: an internal number the model adjusts during training. Billions of weights together encode everything the model has learned.

Self-attention: the mechanism by which each token weighs every other token in the sequence to decide what matters for understanding it.

Softmax: a function that turns raw scores into probabilities summing to one, used both inside attention and at the final prediction step.

Temperature: a setting controlling randomness in token sampling. Low temperature means predictable; high temperature means varied and less reliable.

Pretraining: the first, largest training phase, where the model learns by predicting the next token across a vast corpus.

Fine-tuning / alignment: post-training phases that teach the base model to behave as a helpful, safe assistant, using curated examples and human feedback.

Inference: running a finished model on a new input to produce output. No learning happens during inference.

Context window: the maximum amount of text, in tokens, the model can consider at once, including your prompt and conversation history.

Hallucination: fluent, plausible output that is false, produced because the mechanism optimises likelihood, not truth.

Frequently asked questions

How does AI work, in simple terms? It learns patterns from an enormous amount of human writing, then uses those patterns to guess the most likely next piece of text, one piece at a time. Fluency, creativity and mistakes all come from that single guessing mechanism.

How does AI learn from data? By practicing next-token prediction. During pretraining it reads text with the next token hidden, guesses, and adjusts its billions of internal weights slightly toward the correct answer each time. Repeated trillions of times, those adjustments encode grammar, facts and reasoning patterns as statistical tendencies.

How does AI work step by step? In five stages: ingestion gathers the text; tokenization chops it into tokens; embeddings turn tokens into number-vectors carrying meaning; self-attention lets each token weigh every other token; softmax turns the result into probabilities for the next token, which is sampled and appended, and the loop repeats.

Does AI think or understand what it says? No. It has no beliefs, intentions or inner experience. It produces the continuations that are statistically likely given its training. When those continuations look like reasoning, that is because correct reasoning patterns are common in the training data, not because a mind is at work.

Why does AI make things up? Because nothing in its mechanism checks facts. Its only objective is plausible continuation. When the truthful continuation is rare in training data and a false one is common, or when it has no relevant data at all, it produces the most likely text anyway, fluently and confidently.

What is the difference between AI learning and AI remembering? Learning happens during training, when weights change. Using the model, called inference, does not change weights. If a chatbot seems to remember your conversation, that is because the conversation history is fed back in as input text, not because the model learned anything from you.

Will AI keep working this way? The Transformer architecture has dominated since 2017, and every major language model today is built on it. Future systems may add new mechanisms around it, such as tools, memory stores and verification steps, but the next-token prediction engine at the centre is the foundation everything else is built on.

What to read next

This pillar is the foundation of a five-article Silo 1 series on how AI works. Next in the series: how generative AI works, covering language models, diffusion models and multimodal systems; how neural networks learn, going deeper into weights, layers and backpropagation; how AI training data is collected and why it matters; and how transformers compare with the architectures they replaced. Those spokes will link back here, and this pillar will link out to them as they publish.

Related reading on this site: Google Backs One Robots.txt Switch for All AI Training, on how the data AI learns from is controlled, and Claude Learns to Flag Its Own Hidden Harm, on what a model can and cannot assess about its own behaviour.

Sources and further reading

The architecture in this article comes from the paper that introduced it: Vaswani et al., “Attention Is All You Need” (2017), arXiv:1706.03762, which proposed the Transformer as an architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely, and reported 28.4 BLEU on English-to-German and 41.8 on English-to-French translation. The full paper text is available at Attention Is All You Need. The training account, pretraining and post-training phases, tokenization, temperature sampling and the hallucination mechanism are standard, textbook-level descriptions of large language models, consistent across current technical explainers; where this article gives a specific historical figure, such as GPT-3’s roughly 300 billion training tokens and 175 billion parameters, it is attributed to OpenAI’s 2020 publication. The probability figures in the worked example are illustrative, not measured from any particular model, and are labelled as such.