BERT, Llama, Whisper and RAG Explained: The Four Ideas Behind Most Modern AI

The four ideas in one paragraph: Almost everything you call “AI” today is built from four ideas, and you can understand all four without a computer science degree. BERT, from Google in 2018, taught machines to read a sentence in both directions at once, so they could truly understand what a piece of text means. Llama, from Meta starting in 2023, proved that a powerful language model could be given away for anyone to download, run, and build on — the open-weight movement that now powers half the AI industry. Whisper, from OpenAI in 2022, taught machines to listen, turning speech in nearly a hundred languages into accurate text with no special training for each new accent or room. And RAG — retrieval-augmented generation, from Facebook AI Research in 2020 — gave a model something to look up, so it could answer from your documents instead of guessing from memory. One reads, one is shared, one listens, and one looks things up. Together, they are the working parts inside the chatbots, search tools, meeting transcribers, and coding assistants you use every day.

A product changes every six months; an idea lasts. BERT is seven years old and still inside nearly every search engine and spam filter. Whisper’s checkpoints have not changed since October 2024, yet Whisper transcribes more audio every day than any speech system before it. And RAG, barely five years old, quietly became the standard way to build a business AI that answers from real documents. This guide explains each idea in plain language, with the numbers its own papers and model cards publish.

Checked 7 November 2026. Model names, sizes, and dates on this page are the ones listed in the original research papers and the official model cards on that date, and they change often — Meta’s Llama 4 Behemoth, for example, was previewed in April 2025 and has still not been released, and OpenAI’s hosted whisper-1 model is reported to be retired in February 2027. Where this article could not verify a figure on a primary source in this session, it says so rather than repeating a number as fact.

1. The Short Answer: Four Ideas, One Modern AI

<img src="

” alt=”Four different tools laid out neatly on a workbench — a magnifying glass, a set of building blocks, a microphone, and a small card catalogue — suggesting four ideas that work together” />

If you only read one section, read this one. The four ideas each solve a different problem, and modern AI is what happens when you combine them.

BERT solves understanding. Before BERT, a machine reading a sentence mostly read it left to right, like a person reading with blinkers on — it could only use the words it had already seen. BERT reads the whole sentence at once, in both directions, so the meaning of each word is shaped by the words on both sides of it. That single change made machines dramatically better at the jobs that depend on understanding text: search, question answering, sorting messages, and spotting the sentiment in a review. BERT does not write essays. It understands.

Llama solves access. The most capable language models were locked inside big companies, available only through an API you paid for and could not inspect. Llama broke that pattern: Meta published the model’s weights — the billions of numbers that make it work — so anyone could download them, run the model on their own computers, fine-tune it for their own purpose, and build products on top of it. The weights are free under Meta’s own licence, with conditions. The idea that a frontier-class model can be a shared public resource, not a rented service, is Llama’s real contribution, and it reshaped the whole industry.

Whisper solves listening. Speech recognition used to need a separate model trained for each language, each accent, and each kind of room, and it still fell apart on noisy audio. Whisper was trained on 680,000 hours of audio paired with transcripts from the internet — so much, and so varied, that it learned to transcribe speech it had never been specifically trained on, in language after language, without any extra fine-tuning. One model, many languages, many tasks: transcription, translation into English, and even telling which language is being spoken.

RAG solves knowing. A language model on its own can only answer from what it learned during training, which is frozen in time and sometimes simply wrong. RAG gives the model a library: when you ask a question, the system first searches your documents for the relevant passages, then hands those passages to the model along with your question, so the answer is grounded in real text you can check. It is the difference between asking someone to answer from memory and letting them open the book first.

2. The Transformer: The 2017 Paper Underneath Everything

None of the four ideas makes sense without the Transformer, so this section is the foundation. In 2017, eight researchers at Google published a paper called “Attention Is All You Need”. Its proposal was radical in its simplicity: throw away the step-by-step reading that every previous language model used, and let every word in a sentence look directly at every other word at the same time.

The mechanism that makes this possible is called self-attention. When a model reads the sentence “The bank refused the loan because it was insolvent”, self-attention lets the word “it” look at the other words and work out that it refers to the bank, not the loan — and it does this by weighing how strongly each word relates to each other word, in one parallel pass, rather than by reading left to right and hoping the connection survives. That is the whole trick, and it is why the paper’s title has become a slogan: attention, on its own, is enough.

The Transformer has two halves. The encoder reads an input and builds a rich representation of it — it understands. The decoder generates an output one piece at a time, looking back at what it has already written — it produces. Different models use different halves. BERT uses only the encoder, because BERT’s job is understanding. The GPT family and Llama use the decoder, because their job is generating text. Whisper uses both: an encoder to hear the audio, a decoder to write out the transcript. RAG, as a technique rather than a single model, sits on top of a sequence-to-sequence model that has both halves. Once you know which half a model uses, you can usually guess what it is for.

There is one more reason the Transformer matters: it scales. Because self-attention processes a whole sentence in parallel, it runs efficiently on graphics processors, and it keeps improving as you feed it more data and more parameters. The BERT paper’s authors found the same thing — their largest model beat their smaller one on nearly every task. That observation, that bigger pre-trained models transfer better, is the engine behind everything that followed. For the fuller plain-language version of how a neural network learns, see our guide to how AI works.

3. BERT Explained: Reading in Both Directions

BERT stands for Bidirectional Encoder Representations from Transformers, and the name is a fair summary of the whole idea. It is a Transformer encoder that represents words by looking in both directions. The paper was published by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova at Google AI Language in October 2018, and it changed natural language processing more suddenly than almost any paper before or since.

The problem BERT solved was unidirectionality. The best language models of 2018 — notably OpenAI’s GPT, published earlier that year — read text left to right. Every word could only attend to the words before it. That is fine for generating text, where you cannot peek at the future, but it is a handicap for understanding, where the words after a word often decide what it means. Take the sentence “He went to the bank to deposit his pay cheque.” The word “bank” is a river bank or a financial institution, and you only know which from the words that come after it. A left-to-right model has to guess; BERT sees the whole sentence and does not have to.

BERT comes in two sizes, and the paper reports both. BERT Base has 12 layers, a hidden size of 768, 12 attention heads, and 110 million parameters. BERT Large has 24 layers, a hidden size of 1,024, 16 attention heads, and 340 million parameters. Those numbers look tiny next to a modern model with hundreds of billions of parameters, and that is part of the point: BERT proved that the architecture and the training method mattered more than raw size, and a 340-million-parameter model was enough to set new records on eleven different tasks at once.

The second half of BERT’s idea is what happens after pre-training. Once BERT has learned the structure of language from unlabeled text, you adapt it to a specific job — sentiment analysis, question answering, spam detection — by adding one small output layer and fine-tuning all of its parameters on labelled examples of that job. The paper stresses how cheap this is: every result in the paper could be reproduced with at most one hour of fine-tuning on a single Cloud TPU, and a question-answering model could be trained in about 30 minutes. Pre-training is expensive and done once; fine-tuning is cheap and done per task. That recipe — one big pre-trained model, many small fine-tuned copies — is now the standard pattern for the whole field.

4. How BERT Is Trained: Masked Words and Next Sentences

The cleverest part of BERT is not the architecture but the training trick that makes bidirectional learning possible at all. There is a catch with reading in both directions: if a model can see the word it is supposed to predict, the task is trivial — it just copies the answer. BERT’s authors solved this with a game of fill-in-the-blank, and it is worth understanding because the same idea still shows up in modern training.

The first training task is called the masked language model. During pre-training, 15% of the words in each passage are chosen at random, and the model has to predict them from their context. But the chosen words are not simply hidden. To stop the model from leaning on the mask itself, the authors split the treatment three ways: 80% of the time the chosen word is replaced with a special [MASK] token, 10% of the time it is replaced with a random word, and 10% of the time it is left unchanged. The model must predict the original word in all three cases, which forces it to build a genuine understanding of the surrounding sentence rather than a shortcut. The mask token never appears during fine-tuning, and this 80-10-10 split is the authors’ answer to the mismatch that would otherwise create.

The second training task is next sentence prediction. Many real jobs — answering a question from a passage, deciding whether one sentence follows from another — depend on understanding the relationship between two sentences, and a word-guessing game does not teach that. So half the time, BERT is shown two sentences where the second genuinely follows the first, and half the time the second sentence is a random one from elsewhere in the corpus. The model must say which is which. On this simple test the final model reached 97 to 98% accuracy, and the paper shows the task measurably improved both question answering and natural language inference.

The training data was large for 2018 and modest by today’s standards: the BooksCorpus, about 800 million words of unpublished books, and English Wikipedia, about 2,500 million words of article text, with lists, tables, and headers stripped out. The authors deliberately used document-level text — long continuous passages — rather than shuffled single sentences, because the next-sentence task only works when real sentence pairs exist. Words were split with WordPiece tokenisation into a 30,000-token vocabulary, and each input was built from three embeddings added together: the token itself, which sentence it belongs to, and its position. A special [CLS] token starts every sequence, and its final representation becomes the summary of the whole input used for classification tasks; a [SEP] token separates sentence pairs.

The results were, by the standards of the field, a landslide. BERT Large pushed the GLUE benchmark — a suite of language-understanding tasks — to a score of 80.5%, a 7.7-point improvement over the previous best. On MultiNLI, a test of whether one sentence follows from another, it reached 86.7% accuracy, up 4.6 points. On SQuAD v1.1, a question-answering test built on Wikipedia passages, it reached a test F1 score of 93.2, and on the harder SQuAD v2.0, which includes questions with no answer in the passage, it reached 83.1 — a 5.1-point improvement. Eleven tasks, eleven new state-of-the-art results, from a single pre-trained model fine-tuned eleven times. Within months, BERT was inside Google Search, and the “pre-train, then fine-tune” pattern became the default for the entire field.

5. What BERT Is Actually Good For — and What It Cannot Do

BERT’s strength is understanding text that already exists. Any job that starts with a piece of writing and ends with a judgement about it is a BERT-shaped job: Is this review positive or negative? Is this email spam? Does this passage answer that question? Which named people, places, and organisations appear in this document? Is this support ticket about billing or about a bug? For all of these, you take a pre-trained BERT, add a small classification or span-selection layer, fine-tune it on a few thousand labelled examples, and you have a specialist that is fast, cheap to run, and accurate enough for production.

BERT is also the quiet workhorse of search. A search engine needs to understand a query and a document in the same terms, and BERT-style encoders turn both into vectors — lists of numbers that capture meaning — so that a query about “cheap flights to Lahore” can match a page that never uses the word “cheap”. That embedding idea, which BERT helped popularise, is also the foundation of RAG in section 11: the same encoder family that reads your documents is the one that retrieves them. And because BERT models are small — a base model is 110 million parameters, small enough to run on an ordinary server or even a laptop — they remain the default choice when a task needs understanding at scale and a giant generative model would be slow, expensive, or simply unnecessary.

What BERT cannot do is just as important. BERT does not generate text. It is an encoder, and encoders read; they do not write. If you ask BERT to continue a story, it has no mechanism for producing the next word — that is the decoder’s job, and it belongs to the GPT and Llama family. BERT also does not know anything beyond its training data, and it cannot look anything up. A BERT classifier trained in 2021 will happily classify text about events from 2026, but its judgements reflect the world it was trained on, not the world today. And BERT is not state of the art any more: the GLUE and SQuAD leaderboards it topped in 2018 have long since moved on to larger and more specialised models. What has not moved on is the pattern BERT established. Every modern encoder — RoBERTa, DeBERTa, the embedding models inside today’s search engines and RAG systems — is a descendant of this design.

The practical rule is simple. If the job is “read this and tell me something about it”, reach for a BERT-family encoder first: it is cheaper, faster, and easier to test than a giant generative model. If the job is “write something new”, you need a decoder, and that is where Llama comes in.

6. Llama Explained: Meta’s Open-Weights Family

Llama is Meta’s family of large language models, and its importance is not any single capability but a decision about distribution: Meta gave the models away. The name is a backronym for Large Language Model Meta AI. The first version, stylised LLaMA, was announced on 24 February 2023 as a research release — four sizes, 7, 13, 33, and 65 billion parameters, available only to vetted researchers under a non-commercial licence. Within days the weights leaked, and the leak became the story: thousands of developers who had never touched a frontier model suddenly had one on their own machines, and the open-weight movement went from a niche idea to the centre of the industry in a single week.

Llama 2, released on 18 July 2023 in partnership with Microsoft, made the opening official. Three sizes — 7, 13, and 70 billion parameters — trained on about 2 trillion tokens, with a 4,096-token context window, and this time a licence that permitted commercial use. It also shipped in two flavours that have become standard across the family: a pretrained base model, which predicts the next word and can be adapted to anything, and a chat-tuned version, Llama 2-Chat, fine-tuned with supervised examples and reinforcement learning from human feedback so it behaves like an assistant. That split — base for builders, chat for users — is still how every Llama release is organised.

Llama 3 arrived on 18 April 2024 in two sizes, 8 and 70 billion parameters, trained on more than 15 trillion tokens with an 8,192-token context and a much larger 128,000-token vocabulary. Three months later, on 23 July 2024, Llama 3.1 added a 405-billion-parameter flagship — the first open-weight model in the 400-billion class — and stretched the context window to 128,000 tokens, with official support for eight languages and tool use for the first time. Llama 3.2, on 25 September 2024, went in the opposite direction as well as forward: tiny 1- and 3-billion-parameter text models small enough for a phone, alongside 11- and 90-billion-parameter vision models that could read images. Llama 3.3, in December 2024, refined the 70-billion size. Each generation widened the range of things an open model could do, and each one was downloadable on the day it was announced.

There is a catch, and it matters enough to have its own section later: Llama is open-weight, not open source in the formal sense. The weights are free to download and use, but they arrive under Meta’s own licence, with conditions an Apache or MIT licence would not have. Section 8 takes that apart in detail, because it is the most misunderstood fact about the whole family.

7. Llama 4 in November 2026: Scout, Maverick, and the Missing Behemoth

As of November 2026, the current Llama generation is Llama 4, released on 5 April 2025, and it is a bigger break from the past than any earlier generation. The two models that shipped are Scout and Maverick, and both are built differently from every Llama before them: they are mixture-of-experts models, and they are natively multimodal.

A mixture of experts, usually shortened to MoE, is a way of building a very large model that only uses part of itself for any given word. Instead of one giant network that fires in full for every token, the model has many smaller expert networks, and a router picks a few of them for each token. Llama 4 Scout has 17 billion active parameters out of 109 billion in total, spread across 16 experts; Llama 4 Maverick has the same 17 billion active parameters out of 400 billion in total, across 128 experts. The total size is what the model knows; the active size is what it costs to run. That is the point of the design: Maverick carries roughly the knowledge of a 400-billion-parameter model while doing the computation of a 17-billion one. The trade-off is memory — the full 400 billion parameters still have to sit somewhere, even if only a fraction is used at a time — which is why Maverick ships in both full-precision and FP8 quantised weights, with the FP8 version fitting on a single H100 DGX host.

The second break is multimodality. Earlier Llama models were text-only until Llama 3.2 added vision as a separate capability; Llama 4 is natively multimodal from the start, taking text and images as input and producing text and code as output, with the image and text processing fused early in the architecture rather than bolted on. Scout’s headline feature is its context window: 10 million tokens, by far the longest of any open-weight model, enough to hold entire codebases, long documents, or many images in a single conversation. Maverick’s window is 1 million tokens — still enormous — and it is the stronger of the two on the benchmarks Meta reports: on its own model card, Maverick scores 80.5 on MMLU Pro and 69.8 on GPQA Diamond, against 74.3 and 57.2 for Scout, and 94.4 on DocVQA, the document-understanding test, where both models score identically. Those are Meta’s own reported figures, not independent verification, and they should be read as the vendor’s account of its own models.

Both models were trained on a mix of publicly available and licensed data plus information from Meta’s products and services, including publicly shared posts from Instagram and Facebook and people’s interactions with Meta AI. Scout was pre-trained on about 40 trillion tokens and Maverick on about 22 trillion, with a knowledge cutoff of August 2024 — anything that happened after that date, the models do not know unless you tell them. They officially support twelve languages: Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese.

Then there is the third member of the family that never arrived. Meta previewed a much larger model, Llama 4 Behemoth, alongside Scout and Maverick in April 2025 — roughly 2 trillion total parameters with 288 billion active — describing it as the teacher model from which the smaller two were distilled. Its weights have never been published, and Meta’s Hugging Face organisation lists only Scout and Maverick. As of November 2026 there is no Llama 4.1 and no Llama 5 with public weights; Scout and Maverick remain Meta’s latest open models. Readers should also know that Meta’s developer site, llama.com, no longer leads with Llama at all: it now leads with Muse, a newer closed-weight model family whose flagship, Muse Spark, launched in April 2026. The open-weight line appears, for now, to be paused at the top end — which makes understanding what you can actually download, rather than what was announced, more important than ever. If you are trying to pick between open models for a real project, our guide to choosing an AI model walks through the filters that matter — benchmarks worth trusting, licences, and size trade-offs — from the user’s side.

8. Open Weights, Open Questions: What “Open” Really Means for Llama

The most common misunderstanding about Llama is the word “open”. Llama’s weights are free to download, free to run on your own hardware, free to fine-tune, and free to use commercially — and none of that makes Llama open source. The weights ship under Meta’s own Llama 4 Community License, a custom commercial licence, not under Apache 2.0 or MIT. The distinction is not pedantry; it decides what you are allowed to do, and it has caught real companies by surprise.

The licence’s best-known condition is a scale threshold: if your product reaches 700 million monthly active users, you need a separate licence from Meta. For almost every reader of this article, that threshold is irrelevant — it is a clause aimed at the handful of companies large enough to matter to Meta. The conditions that do affect ordinary users are the everyday ones: you must include a “Built with Llama” attribution, derived models must carry the Llama name, and you must follow Meta’s Acceptable Use Policy, which prohibits certain uses. You also do not get the training data, the training code, or the full recipe — only the finished weights and the model card describing them. That is why the careful term is “open-weight” rather than “open-source”: the model is open to use, not open to inspect in full.

Whether that matters depends on what you want the model for. If you want to run a model on your own servers, keep your data private, fine-tune it on your own documents, or build a product without paying per-token API fees, Llama’s licence gives you everything you need, and the conditions are light. If you want to redistribute the model at the scale of a global platform, audit exactly how it was trained, or guarantee that no future licence change can affect you, the custom licence is a real constraint, and a genuinely open-source model under Apache 2.0 — or a commercial API with a contract — may suit you better. There is no universally right answer, only a trade-off, and the honest version of the trade-off is what this section is for.

The same caution applies to safety. Meta ships Llama with safety fine-tuning and a companion safety model family, Llama Guard, but the responsibility for what a deployed Llama model does sits with the deployer, not with Meta. That is the price of the freedom to run the model yourself — you also own the consequences. For the wider picture of how different governments are treating open models and AI safety, see our guide to AI regulation around the world.

9. Whisper Explained: Teaching a Machine to Listen

Whisper is OpenAI’s speech recognition system, and its story starts with a frustration. By 2022, speech recognition worked well in the lab and badly in the world: models trained on clean, read speech in one language fell apart on accented speech, noisy rooms, overlapping talkers, and languages with little training data, and fixing each failure meant collecting a new labelled dataset and fine-tuning a new model. The Whisper team’s answer was to stop fine-tuning and start scaling — to train one model on so much varied audio, with so many kinds of supervision, that it would simply work wherever it was used.

The paper, “Robust Speech Recognition via Large-Scale Weak Supervision”, was published by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever in December 2022. The training data is the headline: 680,000 hours of audio paired with transcripts collected from the internet. Of that, 117,000 hours cover 96 languages other than English, and 125,000 hours are translation data — audio in one language paired with an English transcript. The transcripts were not gold-standard human annotations; they were the captions and subtitles people had already put on the internet, cleaned up with automated filtering to remove machine-generated text, mismatched languages, and low-quality sources. The bet was that quantity, diversity, and careful filtering would beat a smaller amount of perfect data — and the bet paid off.

Underneath, Whisper is a sequence-to-sequence Transformer: an encoder that listens and a decoder that writes. All audio is re-sampled to 16,000 Hz and turned into an 80-channel log-magnitude mel spectrogram — a picture of the sound’s frequencies over time, computed on 25-millisecond windows with a 10-millisecond stride. The encoder reads that picture; the decoder writes out the transcript. Audio is processed in 30-second segments, and longer recordings are handled by sliding that window along or by splitting the file into chunks and stitching the results together. The model also does more than transcribe: the same network, told which task to perform by special tokens at the start of the sequence, can translate speech into English, identify the language being spoken, and detect whether anyone is speaking at all. One model replaces the whole traditional pipeline of separate components for each of those jobs.

The result the paper is proudest of is not accuracy on a single test but robustness — how little the accuracy drops when the model meets audio it was never trained for. The best zero-shot Whisper model scored 2.7% word error rate on LibriSpeech test-clean, the standard clean-read-speech test, which is roughly the performance of a good supervised model from a few years earlier. But on twelve other speech datasets — noisy, accented, conversational, and far from the lab — it made 55.2% fewer errors on average than a supervised model with the same LibriSpeech score. The paper’s point is that the two models were not equally good: one had memorised a test, and the other had learned to listen. For the company behind Whisper and the story of its other models, see our OpenAI explainer.

10. Whisper’s Sizes and Versions: From Tiny to Turbo

Whisper ships in a family of sizes, and choosing between them is the most practical decision in this whole article. The original paper defined five: tiny at 39 million parameters, base at 74 million, small at 244 million, medium at 769 million, and large at 1,550 million. The smaller four come in both English-only and multilingual versions; the large models are multilingual only. The multilingual checkpoints cover 99 languages. The rule of thumb the paper’s results support is simple and honest: every step up in size buys accuracy, and every step down buys speed. Even the smallest model, tiny, with only 39 million parameters and a 6.7% word error rate on LibriSpeech test-clean, was roughly competitive with the best supervised LibriSpeech model of its day when tested on other datasets — a reminder that a small Whisper is still a Whisper.

Since the paper, the large end of the family has moved twice. Large-v3, released in November 2023, keeps the same architecture as large-v2 with two changes: the spectrogram input uses 128 mel frequency bins instead of 80, and there is a new language token for Cantonese. It was trained on 1 million hours of weakly labelled audio and 4 million hours of pseudo-labelled audio collected using large-v2, trained for 2.0 epochs over that mixture, and its model card reports a 10 to 20% reduction in errors compared with large-v2 across a wide variety of languages. Large-v3 is the model most people mean when they say “Whisper” today, and the one most transcription tools and hosted services run under the hood.

Large-v3-turbo, released on 1 October 2024, is the speed answer. It keeps large-v3’s encoder exactly as it is and cuts the decoder from 32 layers down to 4, then fine-tunes the result for two more epochs. The result has 809 million parameters instead of 1,550 million, and OpenAI’s README reports it running about eight times faster than large-v3 with accuracy close to the full model. There is one real trade-off, and it is easy to miss: turbo was not trained for translation, so if you need speech translated into English rather than transcribed in its original language, you need large-v3, not turbo. For plain transcription, turbo is the sensible default on most hardware; for translation, or for the last few points of accuracy on difficult audio, large-v3 is the one.

One thing Whisper does not do, and buyers should know it before they build on it: it does not tell speakers apart. Diarisation — “who spoke when” — is not a Whisper capability, and any tool that shows you speaker labels is adding a separate diarisation model on top. Whisper can also hallucinate, inventing words that were never spoken, especially in long silences or noisy audio; the model card’s warning on this still applies. And a note on the hosted API, which is a different thing from the open model: OpenAI’s hosted whisper-1 model and its /v1/audio/translations endpoint are reported to be removed on 26 February 2027, in favour of the newer gpt-transcribe and gpt-live-transcribe models — a date this article could not verify on an OpenAI page in this session, so check OpenAI’s deprecations page before building anything long-lived on whisper-1. The open-weight checkpoints are unaffected either way. For the current transcription models, their per-minute prices, and the API behind them, see our OpenAI API explainer, and for a dedicated voice specialist that covers speech generation as well as recognition, see our ElevenLabs explainer.

11. RAG Explained: Giving a Model Something to Look Up

RAG — retrieval-augmented generation — is the youngest of the four ideas and, for most businesses, the most useful. It was introduced in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London, and New York University, and its starting point was a flaw in large language models that everyone in the field recognised but nobody had fixed cleanly: a model stores its knowledge in its parameters, frozen at the moment training ends, and it cannot easily be updated, cannot show its sources, and sometimes invents facts with complete confidence.

The paper’s answer was to give the model two kinds of memory. The parametric memory is the generator itself — a pre-trained sequence-to-sequence model, BART-large with 400 million parameters in the paper’s experiments — which knows how to write fluent language. The non-parametric memory is an external library: a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. When a question arrives, the retriever searches the library for the most relevant passages, and the generator writes its answer with those passages in front of it, rather than from memory alone. Knowledge can then be revised and expanded simply by changing the library — swap the index, and the model’s knowledge updates without any retraining — and the passages the model used can be shown to the reader, which gives the answer provenance: you can see where it came from.

The retriever in the paper is called DPR, Dense Passage Retrieval, and it works by turning text into vectors. There are two encoders, both based on BERT Base: a document encoder that turns each Wikipedia passage into a vector, and a query encoder that turns your question into a vector in the same space. Relevance is then a dot product — passages whose vectors point in the same direction as the question’s vector are the most relevant — and the search itself is a maximum inner product search over the index, fast enough to run over millions of passages. The paper’s index held a December 2018 dump of Wikipedia, split into disjoint 100-word chunks, making 21 million documents in total, searched with the FAISS library’s hierarchical navigable small world approximation. That is the machinery, but the idea is simple: find the right pages, then let the model write from them.

The paper tested two ways of combining retrieval with generation, and the distinction still matters. In RAG-Sequence, the model picks one retrieved document and uses the same document for every word of the answer. In RAG-Token, it can draw on a different document for each word, mixing sources as it writes. Both are trained end to end — the retriever and the generator learn together, with no direct supervision on which document should have been retrieved — and both beat the alternatives of their day: they set the state of the art on three open-domain question-answering tasks, Natural Questions, WebQuestions, and CuratedTrec, outperforming both purely parametric models that answered from memory and task-specific systems that extracted answer spans from retrieved text. For generation tasks, the paper found RAG’s answers more specific, diverse, and factual than a parametric-only baseline of the same size. A chatbot that answers from your documents, with citations, is the direct descendant of this design.

12. How RAG Works in Practice: Chunking, Embeddings, and Retrieval

A modern RAG system follows the paper’s shape with better parts. The pipeline has four steps, and each one is a decision that affects the quality of the final answer. First, chunking: your documents are split into pieces, usually a paragraph or two each, because a retriever works best on passages of a consistent size. Too large, and a chunk contains the answer buried in irrelevant text; too small, and the answer is split across pieces that never get retrieved together. There is no universal right size — it depends on your documents — and most teams test a few options against real questions before settling.

Second, embedding: each chunk is turned into a vector by an embedding model, the descendant of the BERT encoders in the original paper. The embedding captures the chunk’s meaning as a list of numbers, so that two chunks about the same topic sit close together in vector space even if they share no words. Modern embedding models are small, fast, and cheap — the cost of embedding a whole document library is usually measured in dollars, not hundreds — and the choice between them is a trade-off between retrieval quality and cost, much like choosing between BERT Base and BERT Large.

Third, storage and retrieval: the vectors go into a vector database, which is a database built to answer one question very fast — “which vectors are closest to this one?” When a user asks a question, the question is embedded with the same model, the database returns the top few chunks, and those chunks become the context for the answer. The number retrieved, usually called top-k, is typically five or ten, the same range the original paper tested. More chunks give the model more to work with but also more noise and a longer, more expensive prompt; fewer chunks are cheaper but risk missing the passage that matters. Many production systems add a second pass, re-ranking, in which a small model scores the retrieved chunks and keeps only the best before handing them to the generator. Retrieval is also why RAG matters beyond chatbots — our GEO explainer covers what happens when AI answers, rather than links, become the way people find things.

Fourth, generation: the question and the retrieved passages are handed to a language model — a Llama model running on your own servers, a Claude or GPT model through an API, or any generator you choose — with instructions to answer from the passages and say so if the answer is not there. This last instruction matters more than it sounds: a generator told to answer only from the retrieved text is far less likely to invent facts than one left to its own memory, and when it does answer, you can show the reader the exact passages it used. That audit trail is the reason RAG, rather than fine-tuning, became the default for business AI: updating the documents updates the answers, and every answer can be checked.

13. How the Four Ideas Combine in the Apps You Use

So far this guide has treated the four ideas separately. In practice, they rarely appear alone. The products you use every day are combinations, and recognising the parts makes the whole less mysterious — and makes it easier to judge whether a product is doing something real or just putting a chatbot in front of you.

Take a meeting transcriber. Whisper listens to the recording and produces a transcript, including who said what only if a separate diarisation model is added, because Whisper itself does not do that. A BERT-style model then reads the transcript and pulls out the structured parts: action items, decisions, dates, and the names of people and projects. A RAG layer lets you ask questions about the meeting — “what did we agree about the budget?” — by retrieving the relevant passages from the transcript and handing them to a generator, often a Llama model or a closed-weight model, which writes the answer. Three of the four ideas, in one product, each doing the job it is best at.

Take a customer support chatbot. When a customer writes in, a BERT-style classifier routes the message: billing, technical, or refund. A RAG layer retrieves the relevant help articles and policy documents. A generator — Llama running on the company’s own servers, or a closed model such as Claude — writes the reply from those documents, with citations the support agent can check. And if the customer would rather talk than type, Whisper transcribes their voice message first. The chatbot is not one model; it is a pipeline, and the pipeline’s quality depends on the weakest step, which is usually retrieval, not generation.

Take a coding assistant. The model that writes the code is a decoder, a Llama descendant or a closed model, fine-tuned on code. But the assistant also needs to understand your codebase, and that is where the other ideas come back: embeddings index your files so the assistant can retrieve the relevant functions before it writes anything — RAG for code — and a BERT-style model can classify, search, and rank those files. Our GitHub Copilot explainer shows how the same ideas appear inside a programming tool, and our Anthropic explainer covers Claude, the closed-weight model most often used as the generator in these stacks. The lesson is the same in every case: generation gets the attention, but understanding, listening, and retrieval do most of the work.

The pattern to remember is a simple one. Understanding is BERT’s job. Generation is Llama’s job, or the job of any decoder like it. Listening is Whisper’s job. Knowing — answering from real documents, with sources — is RAG’s job. A product that does all four well is rare; a product that does one of them well and is honest about the rest is usually the one worth using. If you want to see these ideas from the user’s side, inside the product most people meet first, start with our guide to using ChatGPT.

14. Frequently Asked Questions

What is BERT in simple terms? BERT is a Google model from 2018 that understands text by reading each sentence in both directions at once. It does not write text; it reads it, and it is used for search, classification, question answering, and anything else that starts with understanding what a piece of writing means. It has 110 million parameters in its base size and 340 million in its large size.

What is Llama in simple terms? Llama is Meta’s family of language models, given away as downloadable weights so anyone can run, fine-tune, and build on them. The current generation is Llama 4, released in April 2025, with two models: Scout, with a 10-million-token context window, and Maverick, the stronger of the two. Llama is open-weight, not open source — the weights are free under Meta’s own licence, with conditions.

What is Whisper in simple terms? Whisper is OpenAI’s speech recognition system, trained on 680,000 hours of audio from the internet. It turns speech into text in 99 languages, translates speech into English, and works without being specially trained for each new accent or room. It comes in six sizes, from tiny at 39 million parameters to large-v3 at 1,550 million, plus a faster turbo version at 809 million.

What is RAG in simple terms? RAG, or retrieval-augmented generation, is a way of building AI that answers from your documents instead of from memory. When you ask a question, the system searches your documents for the relevant passages, then gives those passages to a language model, which writes its answer from them. It was introduced in a 2020 paper by Lewis and colleagues, and it is the standard way to build business AI that needs to be accurate and checkable.

Is BERT still used, or is it outdated? BERT is still used everywhere — in search engines, spam filters, classification systems, and as the encoder inside RAG pipelines. It no longer tops the research leaderboards it won in 2018, but for understanding text at scale, cheaply and reliably, its descendants remain the default choice.

Can I run Llama on my own computer? Yes, depending on the size. The smaller Llama 3 models, at 1, 3, and 8 billion parameters, run on a good laptop or a single consumer graphics card with quantisation. Llama 4 Scout fits on a single H100 server GPU with int4 quantisation, according to Meta’s model card; Maverick needs a data-centre card. The weights are free to download from the Hugging Face Hub, subject to Meta’s licence.

Does RAG stop AI from making things up? No, not completely. RAG makes hallucination much less likely, because the model answers from real passages you can check, but a model can still misread a passage, combine two passages wrongly, or invent an answer when the documents do not contain one. The protection RAG gives you is not certainty but checkability: you can see the sources and catch the mistake.

What is the difference between RAG and fine-tuning? Fine-tuning changes the model itself by training it on your data; RAG leaves the model alone and gives it your documents at question time. Fine-tuning is better when you need the model to learn a style or a skill; RAG is better when you need it to answer from facts that change, because you update the documents instead of retraining the model. Many real systems use both.

That is the map. BERT taught machines to read. Llama gave the models away. Whisper taught them to listen. RAG gave them something to look up. None of the four is the whole of modern AI, but together they are most of it — and once you can see the four parts, the products built on top of them stop looking like magic and start looking like engineering, which is a much more useful way to see them.