Step-by-Step Tutorial: Build a Simple RAG Agent with LangChain and LlamaIndex

This hands-on tutorial shows you how to build a minimal retrieval-augmented generation (RAG) agent using LangChain and LlamaIndex. By the end you will have a runnable pipeline that ingests documents, builds a vector index, retrieves context for queries, and generates grounded answers. Follow along to implement the example and adapt it to your own data.

What you will build

We will implement a simple RAG agent that:

 
• Loads a small set of text documents.
 
• Creates embeddings and stores them in an index using LlamaIndex.
 
• Uses LangChain to orchestrate retrieval and generation.
 
• Exposes a query function that returns context-aware answers.

Prerequisites

 
• Python 3.9 or later
 
• An API key for your chosen LLM (for example OpenAI) or a local model connector
 
• Basic familiarity with Python and virtual environments

Environment setup

Create a virtual environment and install the required packages.

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install --upgrade pip
pip install langchain llama-index faiss-cpu tiktoken

Note: Replace faiss-cpu with a compatible vector store if you use a cloud vector DB.

Step 1 - Prepare example documents

Create a folder named data/ and add a few plain text files (example1.txt, example2.txt). Keep each file short for testing.

data/example1.txt  # Short product manual
data/example2.txt  # FAQ-style notes

Step 2 - Ingest and build the index with LlamaIndex

Use LlamaIndex to load documents, split text into nodes, create embeddings, and persist a vector index.

from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex
from llama_index import ServiceContext
from langchain.embeddings import OpenAIEmbeddings

# Choose your embedding implementation; many patterns are supported
embeddings = OpenAIEmbeddings()  # or other provider

# Load documents from the data folder
documents = SimpleDirectoryReader('data').load_data()

# Build an index (this uses embeddings internally)
index = GPTVectorStoreIndex.from_documents(documents, service_context=ServiceContext.from_defaults())

# Persist the index to disk
index.save_to_disk('index.json')

Explanation: This creates a vector index where each document chunk is represented as an embedding for nearest-neighbor retrieval.

Step 3 - Create a LangChain-based RAG agent

We will use LangChain to wrap retrieval calls and a language model to generate answers conditioned on retrieved context.

from llama_index import GPTVectorStoreIndex
from langchain.llms import OpenAI

# Load the index
index = GPTVectorStoreIndex.load_from_disk('index.json')

# Initialize your LLM via LangChain
llm = OpenAI(model_name='gpt-4o-mini')  # replace with your available model

# Simple query function
def query_rag(question, k=4):
    # Retrieve top-k relevant nodes
    response = index.as_query_engine(similarity_top_k=k).query(question)
    return response.response

This minimal function retrieves context and asks the LLM to answer using that context. LlamaIndex's query engine handles the retrieval and prompt composition by default.

Step 4 - Run a local test

Start an interactive Python shell or create a small script to test queries.

if __name__ == '__main__':
    while True:
        q = input('Question (or \"exit\"): ')
        if q.lower() in ('exit', 'quit'):
            break
        print('\nAnswer:\n', query_rag(q))

Ask questions related to the sample documents and confirm the responses use the retrieved context.

Tips to improve accuracy and robustness

 
• Chunking: Tune text chunk sizes and overlap when creating nodes to preserve context and reduce hallucination.
 
• Prompt templates: Use explicit system and user instructions to tell the LLM how to use retrieved snippets.
 
• Retrieval augmentation: Increase k for more context or use reranking to surface higher-quality passages.
 
• Safety guardrails: Add a verification step to detect when the model is guessing beyond available context.

Extending this minimal agent

 
• Persist embeddings in a production vector database (e.g., FAISS, Milvus) for larger datasets.
 
• Implement a cached retrieval layer to reduce costs for repeated queries.
 
• Add a feedback loop that logs user satisfaction and fine-tunes prompt engineering or reranker models.
 
• Wrap the agent behind a lightweight API to serve queries from other applications.

Common errors and troubleshooting

 
• Authentication errors: Ensure your LLM and embeddings API keys are set in environment variables before running the script.
 
• Empty results: If retrieval returns no context, verify documents were loaded correctly and embeddings created without errors.
 
• High latency: Use batching for embedding calls and consider a faster vector store for production.

Related RAG Agent Articles

Continue exploring retrieval-augmented generation with these related guides:

• RAG Agents: The Complete Guide to Retrieval-Augmented Generation for Business Automation - Pillar guide covering definitions, architecture, business use cases, and a production implementation checklist.
• How RAG Agents Work: Architecture, Components, and Data Flows - Beginner-friendly breakdown of RAG architecture, retrieval pipeline components, and vector search data flows.
• Selecting the Right Knowledge Base for Your RAG Agent: Vector Stores Compared - Compare Milvus, Pinecone, and Weaviate on latency, cost, scalability, and operations to pick the right vector store.
• Prompt Engineering for RAG Agents: Templates and Strategies to Reduce Hallucinations - Prompt templates and system-message strategies to improve retrieval relevance and reduce hallucinations.
• Cost & Performance Optimization for RAG Agents: Caching, Indexing, and Hybrid Retrieval - Practical techniques to cut cloud spend and latency with caching, smarter indexing, and hybrid retrieval.

Conclusion and next steps

You now have a working RAG agent built with LangChain and LlamaIndex. This tutorial covered ingestion, indexing, retrieval, and a basic generation flow. Next, adapt the sample to your dataset, improve prompts, and experiment with different LLM providers and vector stores to meet your performance and cost goals.

Ready to productionize? Start by migrating your index to a managed vector database, add authentication and rate limiting for the API layer, and instrument monitoring for query quality and system health.

Call to action

Try the example with your own documents. If you need templates for prompt engineering or a starter API wrapper, replicate the code above and iterate - this minimal RAG agent is a solid foundation to build from.