Building Production-Ready AI Chatbots with LangChain and RAG

A retrieval-augmented chatbot is roughly a weekend of work to demo and several months of work to trust. The demo answers five questions you chose. Production answers thousands you did not, from users who will not rephrase when the first attempt fails, about documents that change weekly.

The gap between those two states is almost entirely in retrieval quality and in what happens when retrieval comes back empty.

Retrieval Quality Is the Whole Product

The instinct when a RAG system gives bad answers is to blame the model and change the prompt. In practice the failure is upstream almost every time: the right passage was never retrieved, so the model was answering from nothing.

Diagnosing this is straightforward and skipping it wastes weeks. Log the retrieved chunks alongside every answer. When a response is wrong, read what was retrieved. If the correct passage is absent, no amount of prompt engineering will help. If it is present and the answer is still wrong, then you have a generation problem, and those are comparatively rare.

Chunking Is a Real Decision

Fixed-size chunking with a character count is the default in every tutorial and is the wrong approach for most document types, because it cuts through the middle of sentences, tables, and the relationship between a heading and the text under it.

Split on structure first. Markdown headings, document sections, and paragraph boundaries carry meaning that a character offset does not. Then apply a size constraint within those boundaries rather than across them.

Chunk size involves a genuine tradeoff. Small chunks retrieve precisely and lose context — a paragraph that says "this is not supported in the enterprise plan" is useless without knowing what "this" refers to. Large chunks carry context and dilute the embedding, so the passage that matters is averaged together with surrounding material and ranks lower.

The pattern that resolves this is to embed small and retrieve large: index precise chunks for matching, but return the surrounding section to the model. You get the retrieval accuracy of small chunks and the context of large ones.

Prepend document and section titles to each chunk before embedding. A chunk that begins with the product name and the section heading it came from is dramatically easier to retrieve correctly, and this single change often produces a larger improvement than switching embedding models.

Combine Vector Search With Keyword Search

Pure semantic search has a specific and consistent weakness: exact tokens. Error codes, product names, version numbers, API endpoint names, and part numbers are precisely what users search for and precisely what embeddings handle worst, because the embedding captures meaning rather than the literal string.

Hybrid search fixes this. Run vector search and keyword search in parallel and fuse the ranked lists. Reciprocal rank fusion works well and requires no tuning. In most document sets this is the single largest retrieval improvement available, and it is not a difficult change.

Add a reranker on top of the fused results when quality matters more than the additional latency. Retrieve twenty candidates, rerank with a cross-encoder, pass the top four to the model. The reranker sees the query and passage together rather than comparing independent embeddings, which is why it is substantially more accurate.

Rewrite the Query Before Retrieving

Users do not phrase questions the way documents phrase answers. They also ask follow-ups that are meaningless in isolation — a question consisting of "what about the enterprise plan" cannot be embedded usefully, because the actual subject is three messages back.

Two transformations handle most of this.

Condense the conversation into a standalone question before retrieval. This is a small, cheap model call and it is what makes multi-turn conversation work at all. Without it, the second question in every conversation retrieves poorly.

Expand ambiguous queries into several phrasings and retrieve for each, then merge. This costs latency and helps most when your users are non-expert and your documents are written by experts, which is the usual situation for a support chatbot.

Make Refusal a First-Class Path

The most damaging behavior in a production chatbot is a confident answer assembled from irrelevant retrieved text. Users cannot distinguish it from a correct answer, and one instance destroys trust in the entire system.

Handle it structurally. Apply a relevance threshold to retrieval scores and treat a result below it as no result. Instruct the model explicitly that answering from outside the provided context is not permitted and that saying it does not know is an acceptable outcome. Then route the no-result case somewhere useful — a search interface, a support handoff, a contact form — so that not knowing is still a resolved interaction rather than a dead end.

A chatbot that reliably says "I don't have information about that, here is how to reach support" is more valuable than one that is right ninety percent of the time and confidently wrong the rest, because the second one cannot be trusted on any individual answer.

Cite Sources, Always

Every answer should carry links to the documents it drew from. This serves three purposes at once, and the second two are usually underestimated.

For users, citations turn an opaque assertion into something verifiable, which is what makes the answer actionable in a work context.

For you, citations make evaluation possible. A tester can check whether the cited passage actually supports the claim, which is a far more tractable question than judging correctness in the abstract.

And citations constrain the model. Requiring a source for each claim measurably reduces fabrication, because the generation is anchored to specific retrieved text rather than free-running.

Keep the Index Fresh

Documents change. A chatbot answering from a knowledge base six weeks stale is confidently describing a product that no longer exists, and this failure is invisible until a customer acts on it.

Build incremental indexing from the start. Track a content hash per source document, re-embed only what changed, and delete chunks whose source was removed. A full reindex as your only update mechanism means updates happen rarely, which means the index is usually stale.

Store the embedding model version alongside every vector. When you upgrade models, you must re-embed everything — vectors from different models are not comparable — and knowing which vectors came from which model is what makes that migration a background job instead of a rebuild from scratch.

Latency Budget

Users tolerate a chatbot that thinks for a moment. They abandon one that appears frozen.

The pipeline has several sequential stages — query condensation, retrieval, reranking, generation — and they accumulate. Two things keep it acceptable.

Stream the final answer so the first token arrives quickly. This matters more than total time by a wide margin.

Show the intermediate stages rather than hiding them. A brief indication that the system is searching, then that it is reading sources, makes several seconds feel like progress rather than failure. This is presentation rather than engineering, and it changes abandonment rates.

For the pipeline itself, parallelize what does not depend on order, cache condensed queries for repeated questions, and consider skipping reranking on queries where the top vector result is already scoring far above the rest.

Evaluate Continuously

Assemble a set of real questions with known-correct answers and known-correct source documents. Fifty is enough to be useful. Run it on every change to chunking, retrieval, prompts, or models.

Measure retrieval and generation separately, because they fail for different reasons and conflating them makes the results uninterpretable. For retrieval, ask whether the correct document appeared in the top results. For generation, ask whether the answer is supported by what was retrieved.

This separation is what makes debugging tractable. A drop in retrieval recall points at chunking or embeddings. A drop in generation quality with stable retrieval points at the prompt or the model. Without the split, you get a single quality number that moves for unknown reasons.

Where LangChain Earns Its Place

For this class of application, LangChain is a reasonable fit. Multi-step retrieval pipelines with query transformation, hybrid search, reranking, and memory are exactly the composition it was built for, and assembling those pieces yourself is real work.

Two cautions. Keep your prompts in your own code rather than relying on framework defaults, because the defaults are generic and your quality lives in the specifics. And make sure you can observe what each step actually sent and received — a pipeline you cannot inspect is a pipeline you cannot debug, and RAG systems require a lot of debugging.

The Realistic Sequence

Build retrieval first and evaluate it in isolation, before writing any generation prompt. If retrieval is not finding the right passages, nothing downstream will work, and you will waste the time on prompts.

Then add generation with strict grounding and citations. Then add the refusal path. Then optimize latency and cost, in that order, because a fast chatbot that fabricates answers is worse than a slow one that does not.