RAG (2/2) + Flagship 1

User queries are short, ambiguous and phrased differently from documents. Query rewriting uses an LLM to turn a conversational follow-up ('what about for contractors?') into a standalone query, essential in chat.

Multi-query generates several paraphrases, retrieves for each and merges results (RAG-fusion merges them with RRF). HyDE asks the LLM to write a hypothetical answer and embeds that instead of the question, because answers look more like documents than questions do.

Each adds an LLM call (latency and cost) and can drift from the user's intent. Measure the recall gain before keeping it.

Bi-encoders embed query and document separately: fast, but they can't model fine interactions. A cross-encoder reads the query and a candidate together and outputs a relevance score: far more accurate, too slow to run over the whole corpus.

Two-stage pattern: retrieve the top 30–100 with hybrid search, rerank them with a cross-encoder (open models or hosted rerank APIs), keep the top 3–8 for the prompt.

Reranking is often the single largest quality jump in a RAG pipeline, and it lets you send fewer, better chunks, which cuts cost.

Give each chunk an ID in the prompt, and require the answer to cite IDs for each claim. Render citations as links to the source passage so users can verify. Optionally check after generation that cited chunks actually support the sentence.

Instruct the model explicitly: if the documents don't contain the answer, say so. Add a retrieval-score threshold so when nothing relevant is found you abstain before calling the model at all.

Measure both sides: faithfulness (answers supported by context) and abstention quality (does it decline on unanswerable questions without declining answerable ones?).

Retrieval metrics: context precision (are retrieved chunks relevant?) and context recall (was everything needed retrieved?). Generation metrics: faithfulness (are claims supported by the context?) and answer relevancy (does it address the question?). Answer correctness needs reference answers.

Ragas computes these, mostly with an LLM as judge. Spot-check judge scores against your own judgement on 20–30 examples before trusting them.

Build an eval set of 50–200 questions (real ones if possible, including unanswerable ones), record a baseline, and re-run on every change: chunking, embeddings, reranker, prompt. Report the numbers in the README.

Choose RAG when knowledge changes often, answers need citations, the corpus is large or private, or access control matters. Updating is just re-indexing.

Choose fine-tuning for consistent format, tone or style, domain-specific behaviour, distilling a large model's skill into a smaller cheaper one, or shortening long prompts. It is poor at injecting facts reliably and makes them hard to update or cite.

Often both: fine-tune a small model to follow your format and use retrieved context well, with RAG supplying the facts. Start with prompting + RAG, measure, and fine-tune only for a specific measured gap.

Linear RAG is query → retrieve → answer. Agentic RAG puts retrieval behind tools in an agent loop: the model plans, decides whether to search, rewrites queries, retrieves again after reading results, combines sources (vector store, SQL, web) and verifies claims before answering.

Named patterns: Self-RAG (the model critiques its own retrieval and generation), Corrective RAG (a retrieval evaluator triggers re-search or web fallback when results look weak), Adaptive RAG (a classifier picks no retrieval, single-shot or multi-step by question complexity) and multi-hop decomposition (sub-questions answered in sequence).

Costs: more LLM calls, higher latency, harder evaluation. Use it for complex, multi-part or multi-source questions; keep simple RAG for straightforward lookups, ideally with a router choosing between them.

Vector search finds passages similar to the question, but struggles with global questions ('what are the main themes across these 2,000 reports?') and multi-hop ones ('which suppliers of our top customer had incidents?').

GraphRAG uses an LLM to extract entities and relationships into a knowledge graph, clusters it into communities, and pre-writes summaries per community. Queries can then traverse relationships (local search) or combine community summaries (global search).

Trade-offs: expensive indexing (many LLM calls), more moving parts, and extraction errors that propagate. Worth it for relationship-heavy domains and corpus-level questions; overkill for FAQ-style lookups.