RAG Architectures: What Nobody Tells You
The vector search is the easy part. Getting chunking right is where most RAG implementations quietly fail.
Everyone building RAG systems focuses on the retrieval model — the embedding quality, the vector store, the similarity metric. These matter. But in my experience, chunking strategy has more impact on output quality than any retrieval parameter, and it gets a fraction of the attention.
Fixed-size chunking (512 tokens, 20% overlap) is a reasonable baseline and works fine for dense technical documentation. It breaks down on narrative content — legal contracts, meeting transcripts — where meaning spans paragraphs. For those, I use semantic chunking: split on natural boundaries, then merge small adjacent chunks until you hit a token budget.
The other under-discussed lever is query rewriting. A user query like 'what does the contract say about termination' is semantically distant from the clause it's asking about. A simple LLM call to expand the query into 3–5 hypothetical answer fragments before retrieval (HyDE) dramatically improves recall without adding latency you'd notice.
Metadata filtering deserves its own mention. In a multi-tenant knowledge base, namespace-level filters cut retrieval time in half and eliminate cross-tenant bleed. Always store doc_id, section, date, and source as filterable metadata fields — you will need them.
Want this working in your business?
I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.
Or join the monthly newsletter and get my 6 n8n workflow templates free
One email at the start of each month. Nothing more.
Share
Citation
Artenis Alija. "RAG Architectures: What Nobody Tells You About Chunking." 2025. https://artenisalija.com/blog/rag-architectures-chunking-guide/
https://artenisalija.com/blog/rag-architectures-chunking-guide/Related services
More posts
AI Solves a Quantum Computing Conjecture: What It Really Means
What it means when AI helps solve a quantum computing conjecture, why math is a good test for AI, and the human-plus-AI pattern businesses can use today.
Application Development Agency: How to Choose the Right Builder
How to choose an application development agency or builder: five checks, red flags, costs, ownership and how AI changes custom app development for businesses.
Deacom ERP: A Plain Guide for Process Manufacturers
What Deacom ERP is, what process manufacturers need from an ERP, how to evaluate it, and where AI helps with forecasting, scheduling and quality.
Desktop Application Development Services: Desktop vs Web in 2026
When to choose desktop application development vs a web app in 2026: use cases, pros and cons, hybrid setups, and how AI fits into business software.
DOSS ERP: How to Evaluate It and Other Modern ERP Platforms
Considering DOSS ERP? How to evaluate newer, modern ERP platforms: fit, three-year cost, integrations, AI features and data ownership.
ERP Software: How to Choose the Right ERP and Where AI Fits
What ERP software does, the main types, a practical checklist for choosing one, how AI is changing ERPs, and when a custom operations system is the better fit.