artenis.alija
ende
RAG · LLM · Architecture8 min read20 February 2025

RAG Architectures: What Nobody Tells You

The vector search is the easy part. Getting chunking right is where most RAG implementations quietly fail.

Everyone building RAG systems focuses on the retrieval model — the embedding quality, the vector store, the similarity metric. These matter. But in my experience, chunking strategy has more impact on output quality than any retrieval parameter, and it gets a fraction of the attention.

Fixed-size chunking (512 tokens, 20% overlap) is a reasonable baseline and works fine for dense technical documentation. It breaks down on narrative content — legal contracts, meeting transcripts — where meaning spans paragraphs. For those, I use semantic chunking: split on natural boundaries, then merge small adjacent chunks until you hit a token budget.

The other under-discussed lever is query rewriting. A user query like 'what does the contract say about termination' is semantically distant from the clause it's asking about. A simple LLM call to expand the query into 3–5 hypothetical answer fragments before retrieval (HyDE) dramatically improves recall without adding latency you'd notice.

Metadata filtering deserves its own mention. In a multi-tenant knowledge base, namespace-level filters cut retrieval time in half and eliminate cross-tenant bleed. Always store doc_id, section, date, and source as filterable metadata fields — you will need them.

Work with me

Want this working in your business?

I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.

Or join the monthly newsletter and get my 6 n8n workflow templates free

One email at the start of each month. Nothing more.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.

Citation

Artenis Alija. "RAG Architectures: What Nobody Tells You About Chunking." 2025. https://artenisalija.com/blog/rag-architectures-chunking-guide/

https://artenisalija.com/blog/rag-architectures-chunking-guide/
Free monthly newsletter

One email a month with an AI automation you can use

Sent at the start of each month, never more often. Real workflows I build for clients, new free tools first, and the n8n templates behind them. Sign up now and get my 6 n8n workflow templates straight away.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.