Why Does RAG Exist? The Problem It Solves
Large language models are trained on a fixed snapshot of data up to some cutoff date, and they don't know anything about your company's internal documents, your product manuals, or anything published after training. Two common failure modes result:
- The model doesn't know the answer — because the information was never in its training data
- The model "hallucinates" — it confidently generates a plausible-sounding but incorrect answer instead of admitting it doesn't know
RAG addresses both by giving the model the actual relevant text to read before it answers — similar to giving someone an open-book exam instead of asking them to answer purely from memory.
How RAG Works, Step by Step
- Ingestion: Your documents (PDFs, web pages, database records) are broken into smaller chunks of text.
- Embedding: Each chunk is converted into a vector — a list of numbers that captures its meaning — using an embedding model.
- Storage: Those vectors are stored in a vector database, indexed for fast similarity search.
- Retrieval: When a user asks a question, the question is also converted into a vector, and the database returns the chunks whose vectors are most similar (this is cosine similarity in most implementations).
- Augmentation: Those retrieved chunks are inserted into the prompt sent to the language model, alongside the user's question.
- Generation: The model generates its answer using both its general knowledge and the specific retrieved context — and can cite where the answer came from.
RAG vs. Fine-Tuning: What's the Difference?
| RAG | Fine-Tuning | |
|---|---|---|
| What it changes | What context the model sees at answer time | The model's own weights/parameters |
| Good for | Answering from a specific, changeable knowledge base | Teaching a model a new style, format, or behavior |
| How current is the knowledge | As current as your document source — update anytime | Frozen at the time of training; needs retraining to update |
| Cost & complexity | Lower — no model training required | Higher — requires training infrastructure and data |
| Can cite sources | Yes, naturally | No, the knowledge is baked in invisibly |
In practice, most production AI applications use RAG rather than fine-tuning for knowledge questions, because it's cheaper, keeps information current, and lets you show exactly where an answer came from.
What Is a Vector Database?
A vector database is a database optimized to store and search embeddings (vectors) efficiently, rather than exact text matches. Popular options include Pinecone, Weaviate, Chroma, and pgvector (a Postgres extension). What makes them useful for RAG is semantic search — they can find chunks that are conceptually related to a query even if they don't share the exact same words.
Common RAG Applications
- Document Q&A tools ("chat with your PDF")
- Customer support bots that answer from a company's help docs
- Internal knowledge assistants for onboarding and policy questions
- Legal and research tools that cite specific source passages
Limitations of RAG
RAG isn't magic — if the retrieval step returns irrelevant or poor-quality chunks, the generated answer will still be wrong (a "garbage in, garbage out" problem). Getting RAG right in production involves tuning chunk size, retrieval count, and re-ranking — which is exactly the kind of hands-on skill that's hard to learn from a definition alone and needs practice building a real system.