When RAG is the wrong tool for the AI problem you actually have

Retrieval-augmented generation (RAG) has become the default answer to almost any AI question involving business documents. Someone asks 'can we build a chatbot that knows about our policies,' and RAG gets proposed before anyone has confirmed what 'knows about' actually needs to mean.

RAG is the right tool when three things are true: the underlying knowledge changes often enough that fine-tuning isn't practical, the answer needs to be traceable back to a specific source document, and the volume of source material is too large to fit in a single prompt.

It's often the wrong tool when the real need is simpler. If the business logic is a fixed decision tree, a rules engine will be more predictable and easier to debug than a retrieval pipeline. If the source documents are few and stable, putting them directly in context may outperform a retrieval step that can miss the right chunk.

The cost of defaulting to RAG isn't just engineering time. It's an ongoing system to maintain: an embedding pipeline, a vector store, retrieval tuning, and a new class of failure mode where the model answers confidently from the wrong chunk of text.

Before committing to a RAG architecture, it's worth writing down exactly what 'wrong answer' looks like for the use case, and whether a simpler architecture would make that failure mode rarer or easier to catch.