Selected work  /  Glorify  ·  2023–2024

Taking 30% off an EKS bill without a capacity freeze

Nobody was told to use less. The bill came down because the cluster stopped paying for capacity it was not using and stopped keeping environments alive after everyone had forgotten about them.

Result
30% off EKS spend
Scheduler
Karpenter
Delivery
FluxCD · ArgoCD · Helm
Rollouts
Canary & blue/green

Where the money actually goes

A Kubernetes bill is mostly nodes, and nodes are mostly bought by two decisions nobody revisits: the instance types in the node group, and the resource requests in the deployment. Requests are what the scheduler bin-packs against — not usage. A pod requesting 2 CPU and using 200m reserves the two, and the cluster buys a node to hold them. Multiply that across a few hundred workloads and the gap between reserved and used is the overspend.

The usual response is a cost-cutting programme: dashboards, a spreadsheet, and a request that teams trim their manifests. It works briefly and decays, because it asks every team to keep caring about something none of them are measured on. The durable version changes the machinery so the default outcome is cheap.

Three changes, in order of payoff

1. Let the scheduler buy the hardware — Karpenter

Managed node groups make you guess: pick instance types, pick a size, pick min and max, then live with the rounding error forever. Karpenter inverts it. Unschedulable pods describe what they need, Karpenter provisions a node that fits those pods, and — the part that produces the savings — consolidation continuously asks whether the current pods would fit on something cheaper, then moves them and deletes the node.

Consolidation is the feature that matters and the one people leave off. Provisioning the right node once is worth something. Re-deciding it every few minutes as workloads come and go is worth much more, because clusters spend most of their lives in a shape nobody planned for.

Node packing before and after consolidation Illustrative. Before: six uniform nodes, each roughly half occupied by pods, because the node group's instance size was fixed in advance. After: four nodes of varying size chosen to fit the same pods, each close to full. Before — fixed node group, one instance size After — Karpenter provisioning + consolidation 6 identical nodes · ~40% requested empty space is bought, reserved, and idle 4 nodes sized to the pods · ~85% requested consolidation re-decides this continuously Illustrative, not measured — the shape of the change, not the data.
Same pods, fewer and better-fitting nodes. The white space on the left is the bill. Note the second condition though: consolidation can only move a pod that is safe to move, which is why the next change is not optional.

2. Make eviction safe, so consolidation is allowed to work

Karpenter will not disrupt a node if doing so violates a PodDisruptionBudget, and it will hesitate around workloads that cannot tolerate being moved. A cluster full of single-replica deployments with no PDBs, or PDBs written as maxUnavailable: 0, is a cluster where consolidation quietly does nothing and someone concludes Karpenter did not help.

So the prerequisite work is boring and unavoidable: real replica counts, PDBs that permit movement, terminationGracePeriodSeconds that reflects how long the app actually takes to drain, and topology spread so that moving one node does not take a service with it. Argo Rollouts helps here too — canary and blue/green mean a shifting pod population is the normal case rather than an event.

3. Give ephemeral environments a lifecycle

Per-branch preview environments are the single best developer-experience change I have shipped and the single easiest way to leak money. Every open pull request holds a namespace; the branch merges; the namespace stays. Six months later a meaningful slice of the cluster is environments for work that shipped last quarter.

The fix is to make the environment's existence a consequence of the pull request's existence, rather than a side effect of a job that ran once. With FluxCD and ArgoCD app-of-apps driving Helm releases, the preview environment is generated from the set of open PRs — close the PR and the resource stops being generated, so it gets pruned. Plus a TTL for the ones that stay open forever.

Cleanup that depends on someone remembering is not cleanup. If deletion is not the automatic consequence of the thing that created it going away, you have built a leak with extra steps.

What I would tell someone starting this

Outcome

  • EKS spend down 30%, with no capacity freeze and no team asked to ration.
  • Node lifecycle owned by Karpenter, including continuous consolidation.
  • Isolated preview environments per branch, created and reaped automatically.
  • Canary and blue/green via Argo Rollouts and Argo Image Updater on the same platform.

The 30% is the figure I can stand behind for the platform overall. Happy to walk through the breakdown and the before/after topology in conversation.