Building a Self-Hosted LLM Stack That
Running a local language model is easy. Running one reliably under load, with a clean API, proper auth, and logging, is a different problem.
Most self-hosted LLM tutorials stop at 'run Ollama, see it respond.' That's fine for experiments. For production use — a coding assistant used by a 20-person engineering team, for example — you need a layer of infrastructure around the model.
The stack I settled on: Ollama for model serving (Llama 3 70B quantised at Q4_K_M), FastAPI for the gateway, Bearer token auth, Redis for in-flight request dedup, and Prometheus + Grafana for latency dashboards. Containerised with Docker Compose, orchestrated on a single beefy VM to start.
The subtlety is prompt caching. Ollama's KV cache works best when the system prompt is identical across requests — teach your clients to be consistent. A shared system prompt stored in Redis, versioned by hash, reduced average TTFT by 34% in my tests.
Would I run this on Kubernetes? For a team of 5+, yes. The horizontal scaling story for inference is awkward (GPUs are expensive), but you can run the gateway layer on cheap pods and let Ollama autoscale on metal separately. It's a split that's worked well for me.
Want this working in your business?
I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.
Or join the monthly newsletter and get my 6 n8n workflow templates free
One email at the start of each month. Nothing more.
Share
Citation
Artenis Alija. "Building a Self-Hosted LLM Stack That Actually Scales." 2025. https://artenisalija.com/blog/building-self-hosted-llm-stack/
https://artenisalija.com/blog/building-self-hosted-llm-stack/Related services
More posts
AI Solves a Quantum Computing Conjecture: What It Really Means
What it means when AI helps solve a quantum computing conjecture, why math is a good test for AI, and the human-plus-AI pattern businesses can use today.
Application Development Agency: How to Choose the Right Builder
How to choose an application development agency or builder: five checks, red flags, costs, ownership and how AI changes custom app development for businesses.
Deacom ERP: A Plain Guide for Process Manufacturers
What Deacom ERP is, what process manufacturers need from an ERP, how to evaluate it, and where AI helps with forecasting, scheduling and quality.
Desktop Application Development Services: Desktop vs Web in 2026
When to choose desktop application development vs a web app in 2026: use cases, pros and cons, hybrid setups, and how AI fits into business software.
DOSS ERP: How to Evaluate It and Other Modern ERP Platforms
Considering DOSS ERP? How to evaluate newer, modern ERP platforms: fit, three-year cost, integrations, AI features and data ownership.
ERP Software: How to Choose the Right ERP and Where AI Fits
What ERP software does, the main types, a practical checklist for choosing one, how AI is changing ERPs, and when a custom operations system is the better fit.