artenis.alija
ende
Kubernetes · AWS EKS · MLOps9 min read18 January 2025

Kubernetes for ML Pipelines: Six Months on

Running batch ML workloads on EKS taught me things the documentation won't. Node pools, spot interruptions, GPU scheduling — the hard lessons.

Kubernetes is not the obvious choice for ML pipelines — managed solutions like SageMaker or Vertex AI take care of a lot of infrastructure complexity. But when you need full control over your stack, want to avoid vendor lock-in, and have workloads that span training, inference, and data prep, EKS with a well-configured node pool setup is hard to beat.

The critical decision is GPU node pool architecture. We ended up with three pools: a small on-demand pool for interactive inference (always warm, expensive), a spot pool for batch training (90% cheaper, needs interruption handling), and a CPU pool for preprocessing. Spot interruption handling with node affinity rules and graceful checkpointing was the engineering challenge that took longest.

Autoscaling takes time to tune. Cluster Autoscaler works well for CPU workloads; for GPU nodes, the scale-up latency (3–5 minutes for a g4dn instance to join) means you need to keep a warm floor. We used Karpenter in the end — its bin-packing and faster node provisioning made a meaningful difference.

If I were starting over, I'd containerise more aggressively from day one. Debugging Kubernetes issues where half your config lives in the container and half lives in Helm charts is a nightmare. Single-source-of-truth infrastructure, ideally Terraform managing EKS config and Helm charts from the same repo, is the sane path.

Work with me

Want this working in your business?

I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.

Or join the monthly newsletter and get my 6 n8n workflow templates free

One email at the start of each month. Nothing more.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.

Citation

Artenis Alija. "Kubernetes for ML Pipelines: Six Months on EKS." 2025. https://artenisalija.com/blog/kubernetes-ml-pipelines/

https://artenisalija.com/blog/kubernetes-ml-pipelines/
Free monthly newsletter

One email a month with an AI automation you can use

Sent at the start of each month, never more often. Real workflows I build for clients, new free tools first, and the n8n templates behind them. Sign up now and get my 6 n8n workflow templates straight away.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.