How do I build a RAG system?

RAG (Retrieval-Augmented Generation) lets an LLM answer questions using your private data -- documents, databases, knowledge bases -- without fine-tuning the model or sending all your data to an API at every call. Here is the technical architecture.

The Four Components of a RAG System

1. Ingestion pipeline -- load source documents (PDFs, web pages, database records, emails), chunk them into 200-500 token segments, generate embeddings for each chunk using an embedding model (OpenAI text-embedding-3-small, Cohere embed-v3, or open-source alternatives), and store embeddings in a vector database (Pinecone, Weaviate, pgvector, Qdrant).
2. Retrieval layer -- given a user query, generate an embedding for the query, perform approximate nearest-neighbor search against the vector database, and return the top-K most semantically similar chunks (typically K=5-20).
3. Reranking -- apply a cross-encoder reranker (Cohere Rerank, BGE-Reranker) to the retrieved chunks to improve precision beyond what vector similarity alone delivers. This step improves answer quality significantly and is frequently skipped in early implementations.
4. Generation -- pass the reranked chunks as context in the LLM prompt alongside the user query. The LLM generates an answer grounded in the retrieved content. Include source citations in the output for auditability.

Key Design Decisions

• Chunk size -- smaller chunks (150-300 tokens) improve retrieval precision; larger chunks (400-600 tokens) preserve more context per retrieved segment. Hybrid approaches split at semantic boundaries.
• Metadata filtering -- store document metadata (date, author, category, access level) alongside embeddings so retrieval can be filtered by structured criteria before semantic search.
• Hybrid search -- combine vector search (semantic) with BM25 keyword search for best results on domain-specific terminology that embeddings may not handle well.

Getting Help Building a RAG System

Code and Trust builds production RAG systems for businesses that need AI to reason over their internal knowledge base, document library, or structured data. Learn about our AI implementation services or contact us to scope your RAG project.