Fine-tuning vs RAG: When to Use Which

The debate around fine-tuning vs RAG is often polluted by vendor bias and misunderstood trade-offs. If you are building LLM applications in production, you cannot afford to get this architectural decision wrong. Choosing the wrong path means wasted GPU cycles, stale data, and hallucination-prone systems.

This post tears down the theoretical noise. We will examine the exact problem space, look at the architecture of both Retrieval-Augmented Generation (RAG) and model fine-tuning, and give you an opinionated framework for when to use which.

The Problem

Large Language Models (LLMs) are frozen in time. A model trained in 2024 knows nothing about the proprietary codebase your team wrote yesterday, nor does it understand the real-time API schema updates pushed an hour ago. When you ask a foundational model to reason over internal documents or private enterprise data, it will either confidently hallucinate or refuse to answer.

You have domain-specific data. You need the LLM to use it. That is the fundamental problem.

Why It's Hard

Bridging the gap between static LLM weights and dynamic, private data introduces severe engineering challenges. You cannot simply dump a 50MB PDF into a prompt context window-even with models supporting 1M+ token windows, long contexts suffer from the "lost in the middle" phenomenon, massive latency, and exorbitant token costs.

If you decide to update the model's weights, you enter the chaotic realm of distributed training, dataset curation, and catastrophic forgetting. Training pipelines break, data formats require aggressive standardization, and evaluating whether a fine-tuned model actually improved requires expensive, manual evaluation frameworks.

Architecture: RAG vs Fine-Tuning

Retrieval-Augmented Generation (RAG)

RAG leaves the LLM weights untouched. Instead, it alters the prompt dynamically at inference time. You convert your proprietary data into high-dimensional vectors, store them in a vector database, and perform semantic similarity searches when a query comes in. The retrieved documents are then injected into the prompt context.

RAG is a data retrieval problem masquerading as an AI problem. The architecture usually looks like this:

1. Data ingestion pipeline (extracting text from PDFs, Confluence, etc.)
2. Chunking strategy (splitting text into 512-1024 token blocks)
3. Embedding model (e.g., text-embedding-3-small)
4. Vector store (e.g., Pinecone, Qdrant, pgvector)
5. Generation model (e.g., GPT-4, Claude 3 Opus)

Fine-Tuning

Fine-tuning permanently alters the neural network's weights. You take a pre-trained model (like Llama-3-8B) and train it further on a curated dataset of prompt-completion pairs. This bakes the knowledge or behavior directly into the model parameters.

Architecture for Fine-Tuning (specifically LoRA/QLoRA):

1. Dataset preparation (JSONL format of instructions and responses)
2. Base model selection
3. Parameter-Efficient Fine-Tuning (PEFT) adapters
4. Distributed training cluster
5. Evaluation loops
6. Inference server with merged adapters

Fine-tuning vs RAG: The Opinionated Breakdown

Use RAG when you need the model to know facts. Use fine-tuning when you need the model to learn a format, tone, or behavior.

Do not fine-tune a model to memorize your company's HR policy. It will forget details, hallucinate numbers, and require complete retraining when the policy changes. Use RAG for this.

Do not use RAG to teach a model how to output highly specific, complex, proprietary JSON schemas or write code in a deeply customized internal DSL. The context window will overflow with few-shot examples, and the latency will kill your application. Fine-tune for this.

Implementation

Building the RAG Pipeline

For RAG, we will use Python 3.11, LlamaIndex 0.10.x, and Qdrant.

# requirements.txt
# llama-index==0.10.15
# llama-index-vector-stores-qdrant==0.1.2
# qdrant-client==1.7.3

import qdrant_client
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.qdrant import QdrantVectorStore

# 1. Initialize Vector Store
client = qdrant_client.QdrantClient(location=":memory:")
vector_store = QdrantVectorStore(client=client, collection_name="internal_docs")
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# 2. Load Documents
documents = SimpleDirectoryReader("./data").load_data()

# 3. Build Index
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
)

# 4. Query
query_engine = index.as_query_engine()
response = query_engine.query("What is the new API rate limit?")
print(response)

Building the Fine-Tuning Pipeline

For fine-tuning, we use Unsloth 2024.4 for 2x faster training and less VRAM usage.

# requirements.txt
# unsloth[cu121-ampere] @ git+https://github.com/unslothai/unsloth.git
# trl==0.8.1

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# 1. Load Base Model and LoRA Adapters
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha = 16,
    lora_dropout = 0, 
    bias = "none",
    use_gradient_checkpointing = "unsloth",
)

# 2. Prepare Dataset
dataset = load_dataset("json", data_files="internal_dsl_dataset.jsonl", split="train")

def format_prompts(examples):
    instructions = examples["instruction"]
    outputs      = examples["output"]
    texts = []
    for instruction, output in zip(instructions, outputs):
        text = f"Instruction: {instruction}\nOutput: {output}<|end_of_text|>"
        texts.append(text)
    return { "text" : texts }

dataset = dataset.map(format_prompts, batched = True)

# 3. Train
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = 2048,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        output_dir = "outputs",
    ),
)

trainer.train()

Pitfalls

RAG Pitfalls

1. Blind Chunking: Cutting text indiscriminately every 512 tokens splits sentences in half and destroys semantic meaning. You must use semantic chunking or recursive character splitting.
2. Ignoring Metadata: Vector similarity search is inherently dumb. If you search for "Q3 revenue", the embedding might retrieve a document from 2021 because it looks mathematically similar. Always filter by metadata (date, author, category) before doing the vector search.
3. Lost in the Middle: Feeding 20 retrieved documents into the context window often results in the LLM ignoring the middle documents. Re-rank your retrieved documents (using tools like Cohere Rerank) to put the most relevant items at the very beginning and very end of the context window.

Fine-Tuning Pitfalls

1. Garbage In, Garbage Out: Fine-tuning amplifies the quality of your dataset. If your training data contains formatting errors, hallucinations, or contradictions, your fine-tuned model will aggressively produce those exact errors.
2. Overfitting: Training for too many epochs on a small dataset will destroy the model's general reasoning capabilities. It will learn to recite the training data verbatim but fail spectacularly on novel inputs.
3. Catastrophic Forgetting: As the model learns your specific task, it will inevitably forget other information. A model fine-tuned heavily on writing SQL queries might suddenly become terrible at writing Python or summarizing text.

Outcome

Stop treating fine-tuning vs RAG as an either/or decision. They solve different problems.

RAG provides external context. Fine-tuning provides internal behavior.

If your application requires reasoning over dynamic, private knowledge bases, build a robust RAG pipeline. If your application requires the model to consistently output complex structures, speak in a hyper-specific brand voice, or operate efficiently on smaller, cheaper open-source models, you must fine-tune.

The most advanced enterprise architectures use both. They fine-tune a small, cheap model (like Llama 3 8B) to perfectly understand their specific JSON output schema and user intents, and then they hook that fine-tuned model up to a massive RAG pipeline to retrieve the actual facts required to populate that schema.

Make the architectural choice based on your data velocity and output requirements, not on the latest Twitter hype. Build aggressively, evaluate relentlessly, and understand the core mechanics of your stack.