Where the money actually goes
A Kubernetes bill is mostly nodes, and nodes are mostly bought by two decisions nobody revisits: the instance types in the node group, and the resource requests in the deployment. Requests are what the scheduler bin-packs against — not usage. A pod requesting 2 CPU and using 200m reserves the two, and the cluster buys a node to hold them. Multiply that across a few hundred workloads and the gap between reserved and used is the overspend.
The usual response is a cost-cutting programme: dashboards, a spreadsheet, and a request that teams trim their manifests. It works briefly and decays, because it asks every team to keep caring about something none of them are measured on. The durable version changes the machinery so the default outcome is cheap.
Three changes, in order of payoff
1. Let the scheduler buy the hardware — Karpenter
Managed node groups make you guess: pick instance types, pick a size, pick min and max, then live with the rounding error forever. Karpenter inverts it. Unschedulable pods describe what they need, Karpenter provisions a node that fits those pods, and — the part that produces the savings — consolidation continuously asks whether the current pods would fit on something cheaper, then moves them and deletes the node.
Consolidation is the feature that matters and the one people leave off. Provisioning the right node once is worth something. Re-deciding it every few minutes as workloads come and go is worth much more, because clusters spend most of their lives in a shape nobody planned for.
2. Make eviction safe, so consolidation is allowed to work
Karpenter will not disrupt a node if doing so violates a PodDisruptionBudget, and it will hesitate around
workloads that cannot tolerate being moved. A cluster full of single-replica deployments with no PDBs, or
PDBs written as maxUnavailable: 0, is a cluster where consolidation quietly does nothing and
someone concludes Karpenter did not help.
So the prerequisite work is boring and unavoidable: real replica counts, PDBs that permit movement,
terminationGracePeriodSeconds that reflects how long the app actually takes to drain, and
topology spread so that moving one node does not take a service with it. Argo Rollouts helps here too
— canary and blue/green mean a shifting pod population is the normal case rather than an event.
3. Give ephemeral environments a lifecycle
Per-branch preview environments are the single best developer-experience change I have shipped and the single easiest way to leak money. Every open pull request holds a namespace; the branch merges; the namespace stays. Six months later a meaningful slice of the cluster is environments for work that shipped last quarter.
The fix is to make the environment's existence a consequence of the pull request's existence, rather than a side effect of a job that ran once. With FluxCD and ArgoCD app-of-apps driving Helm releases, the preview environment is generated from the set of open PRs — close the PR and the resource stops being generated, so it gets pruned. Plus a TTL for the ones that stay open forever.
Cleanup that depends on someone remembering is not cleanup. If deletion is not the automatic consequence of the thing that created it going away, you have built a leak with extra steps.
What I would tell someone starting this
- Instrument before you tune. Requested-versus-used per namespace is the only number that matters at the start, and it usually indicts a handful of workloads rather than everything.
- Do not touch limits first. Requests drive the bill. CPU limits mostly drive throttling and confusion; memory limits drive OOMKills. Fix requests, leave limits alone until you have a reason.
- Expect the second-order savings. Fewer, better-packed nodes means less per-node overhead — kubelet, daemonsets, and the agents that bill per host. That compounds quietly.
- Guard the floor. Consolidation is a disruption engine. Without PDBs and honest grace periods you are trading money for incidents, which is a bad trade and an obvious one in hindsight.
Outcome
- EKS spend down 30%, with no capacity freeze and no team asked to ration.
- Node lifecycle owned by Karpenter, including continuous consolidation.
- Isolated preview environments per branch, created and reaped automatically.
- Canary and blue/green via Argo Rollouts and Argo Image Updater on the same platform.
The 30% is the figure I can stand behind for the platform overall. Happy to walk through the breakdown and the before/after topology in conversation.