# Build a Retrieval‑Augmented Generation (RAG) App with Open‑Source LLMs

> Learn how to create a full‑stack Retrieval‑Augmented Generation app using open‑source LLMs, LangChain, ChromaDB, and FastAPI in a step‑by‑step tutorial.

- URL: https://ferdousazad.com/blog/build-rag-app-open-source-llms
- Author: Ferdous Azad (Founder @ DataXgen AI)
- Published: 2026-11-08
- Updated: 2026-11-08
- Tags: rag, open-source-llm, tutorial, python, langchain, fastapi

## Intro  

In this tutorial we’ll build a simple Retrieval‑Augmented Generation (RAG) web service that answers user questions over a custom document collection using an open‑source LLM (Mistral‑7B‑Instruct). By the end you’ll have a FastAPI endpoint that:

* Retrieves the most relevant chunks with ChromaDB.  
* Sends the retrieved context to the LLM via llama‑cpp.  
* Returns a concise answer to the caller.  

### Prerequisites  

- Basic familiarity with Python 3.9+ and virtual environments.  
- A GPU with at least 8 GB VRAM (or CPU‑only fallback).  
- 10 GB of free disk space for model weights and embeddings.  
- Git installed.  

> [!NOTE]  
> The steps work on Linux, macOS, and Windows Subsystem for Linux (WSL). Adjust paths accordingly.

## Step 1: Create a clean Python environment  

```bash
# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
```

## Step 2: Install core dependencies  

```bash
pip install --upgrade pip
pip install langchain==0.1.5 \
            chromadb==0.4.5 \
            fastapi==0.110.0 \
            uvicorn[standard]==0.27.0 \
            sentence-transformers==2.2.2 \
            huggingface-hub==0.23.0 \
            llama-cpp-python==0.2.68
```

> [!TIP]  
> `llama-cpp-python` provides a thin Python wrapper around the `llama.cpp` inference engine, which works on both CPU and GPU.

## Step 3: Download the open‑source LLM  

We’ll use the 7‑b instruction‑tuned Mistral model hosted on Hugging Face.  

```bash
# Create a folder for model files
mkdir -p models/mistral-7b-instruct
cd models/mistral-7b-instruct

# Use huggingface-cli to download the ggml‑quantized file
pip install huggingface_hub
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="TheBloke/Mistral-7B-Instruct-v0.1-GGML",
    revision="ggmlv3-q4_0",
    local_dir=".",
    allow_patterns=["*.bin"]
)
PY
```

The resulting `ggml-model-q4_0.bin` file is ~4 GB and ready for llama‑cpp.

## Step 4: Prepare the document corpus and create embeddings  

For the demo we’ll index a small set of markdown files located in `data/`.  

```python
# embed_and_store.py
import os
from pathlib import Path

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
import chromadb
from chromadb.utils import embedding_functions

# 1️⃣ Load raw documents
docs_path = Path("data")
raw_texts = []
for file in docs_path.rglob("*.md"):
    raw_texts.append(file.read_text(encoding="utf-8"))

# 2️⃣ Split into chunks (≈ 500 tokens)
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", " ", ""],
)
chunks = splitter.split_text("\n".join(raw_texts))

# 3️⃣ Create embeddings with a sentence‑transformer model
embed_model = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

# 4️⃣ Initialise Chroma (persisted in ./chroma_db)
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(
    name="rag_collection",
    embedding_function=embedding_fn,
)

# 5️⃣ Add chunks to the vector store
ids = [f"id-{i}" for i in range(len(chunks))]
metadatas = [{"source": str(docs_path)} for _ in chunks]
collection.add(
    ids=ids,
    documents=chunks,
    metadatas=metadatas,
)
print(f"Indexed {len(chunks)} chunks.")
```

Run the script:

```bash
python embed_and_store.py
```

## Step 5: Build the LangChain RAG pipeline  

```python
# rag_chain.py
import os
from typing import List

from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import LlamaCpp
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# Initialize the vector store (same collection as Step 4)
embedding = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vector_store = Chroma(
    collection_name="rag_collection",
    embedding_function=embedding,
    persist_directory="./chroma_db",
)

# Load the LLM via llama‑cpp
llm = LlamaCpp(
    model_path="models/mistral-7b-instruct/ggml-model-q4_0.bin",
    n_ctx=2048,
    n_threads=os.cpu_count(),
    temperature=0.2,
    max_tokens=256,
)

# Prompt that forces the model to cite sources
prompt = PromptTemplate(
    input_variables=["context", "question"],
    template=(
        "You are a helpful assistant. Use only the provided context to answer the question.\n"
        "If you cannot answer, say you don't know.\n\n"
        "Context:\n{context}\n\nQuestion: {question}\nAnswer:"
    ),
)

# RetrievalQA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vector_store.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True,
    chain_type_kwargs={"prompt": prompt},
)

def answer(question: str) -> dict:
    """Return answer and source docs."""
    result = qa_chain({"query": question})
    return {
        "answer": result["result"],
        "sources": [doc.metadata for doc in result["source_documents"]],
    }
```

## Step 6: Expose the RAG service with FastAPI  

```python
# main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

from rag_chain import answer

app = FastAPI(title="Open‑Source RAG Service")

class QueryRequest(BaseModel):
    question: str

class QueryResponse(BaseModel):
    answer: str
    sources: list

@app.post("/query", response_model=QueryResponse)
def query(req: QueryRequest):
    if not req.question.strip():
        raise HTTPException(status_code=400, detail="Question cannot be empty")
    result = answer(req.question)
    return QueryResponse(answer=result["answer"], sources=result["sources"])
```

Run the API:

```bash
uvicorn main:app --host 0.0.0.0 --port 8000
```

Visit `http://localhost:8000/docs` for an interactive Swagger UI.  

## Architecture diagram  

```mermaid
flowchart TD
    A["User (HTTP POST)"] --> B["FastAPI endpoint"]
    B --> C["RAG chain (LangChain)"]
    C --> D["Vector store (ChromaDB)"]
    D --> E["Embedding model (MiniLM)"]
    C --> F["LLM (llama‑cpp + Mistral‑7B)"]
    F --> G["Generated answer"]
    G --> B
```

## Common pitfalls  

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `OSError: libcuda.so not found` | Running GPU‑optimized llama‑cpp on a machine without CUDA drivers. | Install the correct NVIDIA driver or add `n_gpu_layers=0` to force CPU mode. |
| Retrieval returns unrelated chunks | Text splitter created overly large chunks, causing loss of context. | Reduce `chunk_size` to 300–400 tokens and increase `chunk_overlap`. |
| API latency > 5 s | Model loaded on CPU with large `n_ctx`. | Use a GPU, or switch to a smaller quantization (q8_0) if GPU memory is limited. |
| `UnicodeDecodeError` while reading markdown | Files contain non‑UTF‑8 encoding. | Open files with `encoding="utf-8", errors="ignore"` or convert to UTF‑8. |

## Next steps  

1. **Scale the vector store** – switch to a distributed solution like Milvus or Pinecone for millions of documents.  
2. **Add streaming** – use `FastAPI` WebSockets and `llama_cpp` streaming callbacks to send partial tokens.  
3. **Secure the endpoint** – integrate OAuth2 or API keys, and rate‑limit with `slowapi`.  
4. **Experiment with prompting** – try the “refine” chain type or a custom RAG prompt that includes citations.  

## FAQ  

**Q: Can I replace Mistral‑7B with another open‑source model?**  
A: Yes. Any GGML‑compatible model (e.g., Llama‑2‑7B, Gemma‑2B) works by changing the `model_path` and, if needed, adjusting `n_ctx` to match the model’s context window.

**Q: Do I need a GPU to run this in production?**  
A: A GPU dramatically reduces latency, but the CPU‑only mode is functional for low‑traffic prototypes. For production, a single A100 or RTX 4090 is a common baseline.

**Q: How do I keep the embeddings up‑to‑date when the source documents change?**  
A: Re‑run `embed_and_store.py` with the `overwrite=True` flag on the Chroma collection, or use Chroma’s incremental `add`/`update` methods to modify only changed chunks.