System architecture diagram illustrating a LangChain local RAG pipeline using ChromaDB for vector storage.

Building a Completely Local RAG System and Agents with LangChain

by

Building a Completely Local RAG System and Agents with LangChain

Enterprise demand for Retrieval-Augmented Generation (RAG) and autonomous decision-making agents is higher than ever. However, sending proprietary internal documentation, financial spreadsheets, healthcare records, or legal code bases to third-party cloud APIs poses significant data security risks.

By engineering a LangChain local RAG pipeline alongside air-gapped LangChain local agents, developers can achieve state-of-the-art document intelligence while maintaining complete data privacy.

This comprehensive guide covers building an end-to-end local RAG system using LangChain local embeddings, selecting a LangChain local vector store, and orchestrating autonomous tool-calling agents without making a single outbound HTTP call to external cloud providers.

What is a Completely Local RAG System?

A completely local RAG system is an architecture where every pipeline component operates entirely within your private workstation, local server cluster, or air-gapped internal network.

System architecture diagram illustrating a LangChain local RAG pipeline using ChromaDB for vector storage.
A visual representation of a completely local RAG system, highlighting the integration of LangChain with ChromaDB for private data processing.

Unlike hybrid solutions that combine local LLMs with cloud-hosted embedding services, a truly offline RAG pipeline ensures all three core processing layers remain local:

  1. Local Text Embedding Layer: Converts raw text documents into dense numerical vector representations locally.
  2. Local Vector Database Layer: Indexes, stores, and executes nearest-neighbor vector searches on disk or in-memory.
  3. Local LLM Generation Layer: Processes retrieved context and synthesizes natural language answers inside your local GPU/VRAM hardware.

Before proceeding with complex multi-stage architectures, ensure you have reviewed the core concepts for setting up your initial private LLM.

How to Set Up Local Embeddings in LangChain

To build a zero-cloud RAG application, you must replace API-based embedding wrappers (like OpenAI’s text-embedding-3-small) with locally executed embedding models.

LangChain provides several native integrations for running open-source embedding models locally:

  • HuggingFaceEmbeddings / SentenceTransformers: Runs PyTorch models directly in Python.
  • OllamaEmbeddings: Offloads embedding generation to a local Ollama service using specialized embedding models (such as nomic-embed-text or bge-large).
  • LlamaCppEmbeddings: Uses llama.cpp bindings to extract vector representations from GGUF embedding models. (llama.cpp)

Here is how to set up local embeddings using Hugging Face SentenceTransformers and Ollama in Python:

# local_embeddings_setup.py
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import OllamaEmbeddings

def initialize_huggingface_embeddings():
    """Initializes a HuggingFace BGE local embedding model running on CPU/GPU."""
    print("Loading Hugging Face embedding model locally...")
    model_name = "BAAI/bge-small-en-v1.5" ([BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5))
    model_kwargs = {"device": "cpu"}  # Change to "cuda" if NVIDIA GPU is available
    encode_kwargs = {"normalize_embeddings": True}

    embeddings = HuggingFaceEmbeddings(
        model_name=model_name,
        model_kwargs=model_kwargs,
        encode_kwargs=encode_kwargs
    )
    return embeddings

def initialize_ollama_embeddings():
    """Initializes Ollama Nomic Embeddings local service."""
    print("Loading Ollama Nomic Embeddings...")
    embeddings = OllamaEmbeddings(
        model="nomic-embed-text",
        base_url="http://localhost:11434"
    )
    return embeddings

if __name__ == "__main__":
    hf_emb = initialize_huggingface_embeddings()
    sample_vector = hf_emb.embed_query("LangChain local vector store integration")
    print(f"Generated Vector Dimension: {len(sample_vector)}")

Choosing a LangChain Local Vector Store

Selecting the appropriate LangChain local vector store depends on database persistence requirements, collection scaling targets, indexing speed, and deployment complexity.

ChromaDB vs FAISS for Local Deployments

The two most popular open-source, serverless vector engines for Python developers are ChromaDB and FAISS (Facebook AI Similarity Search).

Feature / Metric ChromaDB (langchain-chroma) FAISS (langchain-community)
Storage Mechanism SQLite + HNSW index on disk In-memory binary file dump (.faiss)
Ease of Use High (automatic metadata handling & persistence) Medium (requires manual index saving/loading)
Metadata Filtering Built-in rich filtering (where={"category": "tech"}) Limited native metadata filtering
Search Speed Excellent for small-to-medium datasets Ultra-fast C++ optimized vector search
Ideal Deployment Production local apps requiring persistence In-memory searches & large-scale static indexes

How to Build a Local RAG System with LangChain

Let’s build a fully functioning, end-to-end local RAG pipeline using ChromaDB, HuggingFaceEmbeddings, and Ollama.

Document Ingestion and Chunking

The first stage of our pipeline loads local PDF or text documents, breaks them down into semantically balanced chunks, converts those chunks into dense vector embeddings, and stores them in ChromaDB on disk.

# ingest_documents.py
import os
from langchain_community.document_loaders import TextLoader, PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma

PERSIST_DIRECTORY = "./local_chroma_db"

def build_local_vector_index(document_path: str):
    print(f"--- Loading Document: {document_path} ---")

    # Select appropriate document loader
    if document_path.endswith(".pdf"):
        loader = PyPDFLoader(document_path)
    else:
        loader = TextLoader(document_path, encoding="utf-8")

    raw_documents = loader.load()

    # Split documents into overlapping chunks
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n\n", "\n", " ", ""]
    )
    doc_chunks = text_splitter.split_documents(raw_documents)
    print(f"Created {len(doc_chunks)} document chunks.")

    # Initialize local embedding engine
    embeddings = HuggingFaceEmbeddings(
        model_name="BAAI/bge-small-en-v1.5",
        model_kwargs={"device": "cpu"}
    )

    # Ingest and persist vectors locally
    vector_db = Chroma.from_documents(
        documents=doc_chunks,
        embedding=embeddings,
        persist_directory=PERSIST_DIRECTORY
    )
    print(f"Successfully saved vector index to {PERSIST_DIRECTORY}")
    return vector_db

if __name__ == "__main__":
    # Ensure sample file exists for demonstration
    sample_file = "sample_data.txt"
    if not os.path.exists(sample_file):
        with open(sample_file, "w") as f:
            f.write("LangChain local RAG systems allow running completely private AI applications offline.\n"
                    "By combining ChromaDB, local embeddings, and Llama 3, sensitive data remains protected.")

    build_local_vector_index(sample_file)

Retrieval and Prompt Generation

Once documents are indexed, we wire the vector store to a local LLM chain using LangChain Expression Language (LCEL).

# query_local_rag.py
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

PERSIST_DIRECTORY = "./local_chroma_db"

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

def execute_rag_pipeline(user_query: str):
    print(f"\n--- Processing Query: '{user_query}' ---")

    # Load persisted ChromaDB instance
    embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
    vector_db = Chroma(
        persist_directory=PERSIST_DIRECTORY,
        embedding_function=embeddings
    )

    # Configure retriever (Similarity search with Top-K = 3)
    retriever = vector_db.as_retriever(search_kwargs={"k": 3})

    # Initialize local LLM engine (e.g., Llama 3)
    # Consider using Llama 3 as the RAG engine for best results
    llm = ChatOllama(model="llama3:8b", temperature=0.1)

    # Construct strict RAG prompt template
    rag_prompt = ChatPromptTemplate.from_template("""
You are a secure, private corporate assistant. Answer the user question based ONLY on the provided local context.
If the answer cannot be deduced from the context, state "I cannot answer based on private local documentation."

Context:
{context}

Question:
{question}

Answer:
""")

    # Assemble LCEL RAG Chain
    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | rag_prompt
        | llm
        | StrOutputParser()
    )

    # Execute chain offline
    answer = rag_chain.invoke(user_query)
    print("\nLocal RAG Answer:\n", answer)

if __name__ == "__main__":
    execute_rag_pipeline("How do local RAG systems protect sensitive enterprise data?")

How to Use LangChain Local Agents Without Cloud APIs

Building LangChain local agents takes local AI beyond fixed query-response chains. Local agents perform autonomous multi-step reasoning, select tools, analyze outputs, and solve problems iteratively—all without calling external APIs.

Workflow diagram illustrating LangChain local agents operating autonomously without reliance on cloud APIs.
A workflow diagram detailing how LangChain local agents perform multi-step reasoning and tool selection in an air-gapped environment, ensuring complete data privacy.

Building a Self-Hosted Reasoning Agent

To run local agents successfully, your local model must excel at structured tool calling and precise JSON generation. Models such as Llama-3-8B-Instruct or Mistral-7B-Instruct-v0.3 are ideally suited for local agent loops.

Below is an implementation of a completely local agent that uses custom Python tools:

# local_agent.py
import math
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_core.tools import tool
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_ollama import ChatOllama

# Define custom local python tools
@tool
def calculate_system_vram_requirement(parameter_count_billions: float, quantization_bits: int) -> str:
    """Calculates estimated VRAM required in GB to run an LLM based on parameters and quantization."""
    bytes_per_param = quantization_bits / 8.0
    base_vram = parameter_count_billions * bytes_per_param
    total_vram_required = base_vram * 1.2  # Add 20% overhead for context KV cache
    return f"Estimated VRAM required: {total_vram_required:.2f} GB"

@tool
def check_local_disk_space(path: str) -> str:
    """Simulates checking available local NVMe disk storage space."""
    return f"Path '{path}' has 142.5 GB available storage."

def run_local_agent_workflow():
    print("--- Initializing Offline Autonomous Agent ---")

    # Initialize tool list
    tools = [calculate_system_vram_requirement, check_local_disk_space]

    # Initialize local LLM with tool calling capabilities
    llm = ChatOllama(
        model="llama3:8b",
        temperature=0.0
    )

    # Define Agent Prompt
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are an autonomous AI infrastructure agent operating in an air-gapped environment. "
                   "You have access to tools. Choose the correct tool to answer user technical requests accurately."),
        MessagesPlaceholder(variable_name="chat_history", optional=True),
        ("human", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ])

    # Create local tool-calling agent
    agent = create_tool_calling_agent(llm, tools, prompt)
    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

    # Execute agent decision-making loop
    result = agent_executor.invoke({
        "input": "How much VRAM do I need to run a 70 billion parameter model at 4-bit quantization? Also check if disk path './models' has room."
    })

    print("\nFinal Agent Response:\n", result["output"])

if __name__ == "__main__":
    run_local_agent_workflow()

Essential Rules for Building Local Agents

  1. Keep Tools Simple & Deterministic: Ensure custom Python tools return clean, stringified outputs that local models can easily parse.
  2. Use Zero Temperature: Set model temperature to 0.0 to minimize hallucinated tool calls or malformed parameter syntax.
  3. Set Step Timeouts: Wrap AgentExecutor with max_iterations=5 to prevent infinite execution loops if a local model fails to parse a tool return value.

To review extended production setup tips, deployment troubleshooting, and performance tuning strategies, check out our guide on local model best practices.

Leave a Comment