Kubernetes for ML Pipelines: Six Months on
Running batch ML workloads on EKS taught me things the documentation won't. Node pools, spot interruptions, GPU scheduling — the hard lessons.
Kubernetes is not the obvious choice for ML pipelines — managed solutions like SageMaker or Vertex AI take care of a lot of infrastructure complexity. But when you need full control over your stack, want to avoid vendor lock-in, and have workloads that span training, inference, and data prep, EKS with a well-configured node pool setup is hard to beat.
The critical decision is GPU node pool architecture. We ended up with three pools: a small on-demand pool for interactive inference (always warm, expensive), a spot pool for batch training (90% cheaper, needs interruption handling), and a CPU pool for preprocessing. Spot interruption handling with node affinity rules and graceful checkpointing was the engineering challenge that took longest.
Autoscaling takes time to tune. Cluster Autoscaler works well for CPU workloads; for GPU nodes, the scale-up latency (3–5 minutes for a g4dn instance to join) means you need to keep a warm floor. We used Karpenter in the end — its bin-packing and faster node provisioning made a meaningful difference.
If I were starting over, I'd containerise more aggressively from day one. Debugging Kubernetes issues where half your config lives in the container and half lives in Helm charts is a nightmare. Single-source-of-truth infrastructure, ideally Terraform managing EKS config and Helm charts from the same repo, is the sane path.
Want this working in your business?
I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.
Or join the monthly newsletter and get my 6 n8n workflow templates free
One email at the start of each month. Nothing more.
Share
Citation
Artenis Alija. "Kubernetes for ML Pipelines: Six Months on EKS." 2025. https://artenisalija.com/blog/kubernetes-ml-pipelines/
https://artenisalija.com/blog/kubernetes-ml-pipelines/Related services
More posts
AI Solves a Quantum Computing Conjecture: What It Really Means
What it means when AI helps solve a quantum computing conjecture, why math is a good test for AI, and the human-plus-AI pattern businesses can use today.
Application Development Agency: How to Choose the Right Builder
How to choose an application development agency or builder: five checks, red flags, costs, ownership and how AI changes custom app development for businesses.
Deacom ERP: A Plain Guide for Process Manufacturers
What Deacom ERP is, what process manufacturers need from an ERP, how to evaluate it, and where AI helps with forecasting, scheduling and quality.
Desktop Application Development Services: Desktop vs Web in 2026
When to choose desktop application development vs a web app in 2026: use cases, pros and cons, hybrid setups, and how AI fits into business software.
DOSS ERP: How to Evaluate It and Other Modern ERP Platforms
Considering DOSS ERP? How to evaluate newer, modern ERP platforms: fit, three-year cost, integrations, AI features and data ownership.
ERP Software: How to Choose the Right ERP and Where AI Fits
What ERP software does, the main types, a practical checklist for choosing one, how AI is changing ERPs, and when a custom operations system is the better fit.