Kubernetes cost visibility with Kubecost and VPA
You cannot optimize what you cannot see. This is how I built cost visibility into Kubernetes with Kubecost, rightsized workloads with the Vertical Pod Autoscaler, and consolidated node pools to cut node counts by about 20%.
Kubernetes cost is famously opaque. The cloud bill arrives as one number, but it is spread across namespaces, teams and workloads that share the same nodes, and nobody can say what anything actually costs. FinOps starts by fixing that: make the cost visible per workload, then act on what you see.
Step 1: cost visibility with Kubecost
I deployed Kubecost across the sandbox and production clusters, wired to the cloud billing export (via BigQuery) so its numbers reconcile with the real invoice rather than estimating. A few decisions that made it stick:
- Standalone Prometheus. I let Kubecost run its own metrics stack rather than bolting onto the existing Prometheus. Faster to deploy and guaranteed metric collection, with no fragile cross-service wiring.
- Internal ingress with TLS. Reachable by the team, not by the internet.
- Allocation by module. Configured cost allocation tags so the bill breaks down by module and team, not just by namespace.
The payoff is immediate: real-time cost by namespace and deployment, utilization vs. allocation, rightsizing recommendations, and billing reconciliation against the actual cloud invoice.
Step 2: rightsizing with the Vertical Pod Autoscaler
Visibility showed the usual story: workloads requesting far more CPU and memory than they ever use. I rolled out the Vertical Pod Autoscaler (VPA) across the non-production environments so requests track real usage instead of someone's cautious guess from a year ago. One caveat worth sharing: VPA is not free of edge cases. On one memory-hungry service it interacted badly with OOM behaviour, so I pulled VPA off that specific workload rather than let it fight the pod. Rightsizing is a default, not a religion.
Step 3: consolidation
With real utilization data, a lot of nodes turned out to be carrying almost-idle pods. I consolidated workloads onto shared node pools so the scheduler could bin-pack properly, which cut node counts by roughly 20% across environments with no impact on the workloads.
Step 4: make it a habit, not a project
One-off savings decay. The practice is what keeps the bill down:
- Daily cost watch. A standing habit of checking the cost reports and investigating any spike the same day, while the cause is still fresh.
- Committed-use review. Analyzed the steady-state compute fleet for committed-use discount coverage, so the baseline that is always running is also the baseline you pay least for.
- Cost analysis drives decisions. Baselined the cost of specific workloads so that decommissioning one came with a measured, validated saving, not a hopeful guess.
FinOps is a loop, not a cleanup
See the cost per workload, rightsize to real usage, consolidate what is idle, then watch for drift and repeat. The tooling (Kubecost, VPA, billing export) exists to make that loop cheap to run, so savings compound instead of eroding.
What I would tell someone starting this
- Buy visibility first. Per-namespace cost allocation reconciled with the real invoice is the foundation everything else stands on.
- Rightsize with VPA, but keep an eye on it. It is the right default; it is not right for every workload (memory-sensitive services especially).
- Consolidate on evidence. Real utilization data turns "we might be over-provisioned" into a specific, safe node-count cut.
- Make cost a daily glance, not a quarterly fire drill. Spikes are cheapest to fix the day they happen.