Finding the latency hidden in Kubernetes DNS

Trace the latencySeparate lookup and TCP time

Adjust resolutionTune ndots in the shared chart

Validate in productionRepeat the original measurement

The roughly 9× improvement describes the measured DNS lookup, not every API request or a sustained service-level guarantee.

Contribution at a glance

My responsibility

Layer-by-layer diagnosis, a shared Helm DNS setting and staged rollout validation.

Scope

Production DNS measurement and eight application/environment render checks.

Collaboration & boundaries

I changed the shared deployment configuration after separating DNS overhead from database connection time.

Outcome

DNS lookup p50 changed from ~18.7 ms to 2.06 ms in the recorded measurement; this is not a 9× API speedup.

A healthy service with an unstable latency signal

The API had healthy pods, normal resource usage and no observed 5xx errors, yet its P95 latency alert repeatedly opened and closed. Adding capacity would have treated the symptom before identifying where request time was spent.

My contribution

I compared DNS resolution, fully qualified lookup and direct TCP connection from inside a production pod. The database connection itself was fast. The Kubernetes resolver was trying local search domains before resolving an external database hostname.

The decision

The default ndots:5 treated the external hostname as relative and expanded it through the pod's search domains. I added a configurable ndots:2 setting to the shared Helm template, checked rendering across eight application/environment combinations, and promoted the change through development and staging to production.

Results and measurement

Both production deployments received the new DNS option. Repeating the lookup that opened the investigation showed a reduction from approximately 18.7 ms p50 to 2.06 ms, about nine times faster. Pods remained healthy with no restarts reported in the validation snapshot.

The closeout recorded API P95 around 0.12 seconds over the 20-minute post-rollout observation, and the alert stopped flapping. The DNS comparison and the API observation are different measurements; the lookup improvement should not be presented as a ninefold improvement in total API response time.

Trade-offs and follow-up

Changing resolver search behavior must preserve the short names used inside the cluster. The shared setting remained configurable. Cluster-wide DNS cache and resolver capacity were separate follow-ups, rather than results claimed for this per-application fix.

The lesson

A database-related latency symptom can originate before the database receives a query. Decompose the request path and repeat the same measurement after the change.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

Adding replicas or database capacity would not remove resolver search expansion. Tune resolution after isolating lookup time from TCP time.

Trade-off

A lower ndots value changes lookup behavior; validate internal names and external dependencies before generalizing the setting.

What I would improve next

Next improvement: extend the observation window and add resolver-regression checks for representative internal and external names.