artenis.alija
ende
LLM · Ollama · FastAPI7 min read28 March 2025

Building a Self-Hosted LLM Stack That

Running a local language model is easy. Running one reliably under load, with a clean API, proper auth, and logging, is a different problem.

Most self-hosted LLM tutorials stop at 'run Ollama, see it respond.' That's fine for experiments. For production use — a coding assistant used by a 20-person engineering team, for example — you need a layer of infrastructure around the model.

The stack I settled on: Ollama for model serving (Llama 3 70B quantised at Q4_K_M), FastAPI for the gateway, Bearer token auth, Redis for in-flight request dedup, and Prometheus + Grafana for latency dashboards. Containerised with Docker Compose, orchestrated on a single beefy VM to start.

The subtlety is prompt caching. Ollama's KV cache works best when the system prompt is identical across requests — teach your clients to be consistent. A shared system prompt stored in Redis, versioned by hash, reduced average TTFT by 34% in my tests.

Would I run this on Kubernetes? For a team of 5+, yes. The horizontal scaling story for inference is awkward (GPUs are expensive), but you can run the gateway layer on cheap pods and let Ollama autoscale on metal separately. It's a split that's worked well for me.

Work with me

Want this working in your business?

I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.

Or join the monthly newsletter and get my 6 n8n workflow templates free

One email at the start of each month. Nothing more.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.

Citation

Artenis Alija. "Building a Self-Hosted LLM Stack That Actually Scales." 2025. https://artenisalija.com/blog/building-self-hosted-llm-stack/

https://artenisalija.com/blog/building-self-hosted-llm-stack/
Free monthly newsletter

One email a month with an AI automation you can use

Sent at the start of each month, never more often. Real workflows I build for clients, new free tools first, and the n8n templates behind them. Sign up now and get my 6 n8n workflow templates straight away.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.