How to debug "temporary failure in name resolution"

temporary failure in name resolution usually means the process asked the operating system to resolve a hostname, but the resolver path did not return a usable answer in time. It is common in Linux shells, Python, Go, package managers, containers, and system services.

The message is easy to misread. It does not automatically mean the hostname is wrong. It often means the client could not reach the configured resolver, the resolver timed out, search-domain expansion was slow, or the process used a different DNS configuration than the host shell.

The useful question is:

Which resolver did this exact process use, and did that resolver answer this exact name?

Do not change HTTP retries, database connection pools, or service startup order until the DNS phase is proven.

First separate resolution failure from connection failure

These errors are different:

If the application log contains both a DNS error and a later TCP error, preserve timestamps. The later error may come from retrying a cached IP, a different host, or a different dependency.

Check from the failing namespace

Start where the failing process runs, not where your terminal is convenient.

On a host:

hostname
cat /etc/resolv.conf
getent hosts api.example.com
resolvectl query api.example.com

Inside Docker:

docker exec -it <container> cat /etc/resolv.conf
docker exec -it <container> getent hosts api.example.com

Inside Kubernetes:

kubectl exec -it <pod> -- cat /etc/resolv.conf
kubectl exec -it <pod> -- getent hosts api.example.com
kubectl exec -it <pod> -- nslookup kubernetes.default.svc.cluster.local

Host DNS success does not prove container or pod DNS success. Each namespace can have a different resolver file, route table, network policy, and source IP.

Identify the resolver owner

Look at /etc/resolv.conf:

cat /etc/resolv.conf

Common patterns:

If the file is generated by NetworkManager, DHCP, VPN software, Docker, kubelet, or systemd-resolved, hand-editing it may disappear on restart. Treat it as evidence first.

Query the resolver directly

Use the resolver IP from the failing namespace:

dig @<resolver-ip> api.example.com +time=2 +tries=1
dig @<resolver-ip> api.example.com +tcp +time=2 +tries=1

Interpretation:

temporary failure often comes from timeout-like behavior. NXDOMAIN and SERVFAIL are different branches because the resolver did answer.

Check route and packet evidence

If direct queries time out:

ip route get <resolver-ip>
tcpdump -tttt -nn -i any host <resolver-ip> and port 53

Packet patterns:

Ping is not enough. DNS may work when ICMP is blocked, and ICMP may work while UDP/TCP 53 is blocked.

systemd-resolved branch

If /etc/resolv.conf points at 127.0.0.53, inspect the actual upstreams:

resolvectl status
resolvectl query api.example.com
journalctl -u systemd-resolved --since -30m

Common problems:

• the VPN installed stale per-link DNS servers;
• split DNS domains are assigned to the wrong interface;
• the stub resolver is healthy but upstream resolvers are unreachable;
• a link-specific DNS server answers only some domains;
• DNS cache hides the latest change.

Fix the configuration owner: NetworkManager, systemd-networkd, DHCP, or VPN. Restarting the resolver can clear a symptom, but it is not proof of root cause.

Containers and Kubernetes branch

For containers, compare host and container:

cat /etc/resolv.conf
docker exec -it <container> cat /etc/resolv.conf
docker exec -it <container> ip route

For Kubernetes:

kubectl -n kube-system get pods
kubectl -n kube-system logs deploy/coredns --tail=100
kubectl exec -it <pod> -- nslookup kubernetes.default.svc.cluster.local

Strong Kubernetes suspects:

• CoreDNS pods are not healthy;
• node-local DNS cache is failing on one node;
• NetworkPolicy blocks pod-to-DNS traffic;
• pods use an unexpected dnsPolicy;
• search domains and ndots create slow repeated lookups;
• only one node or namespace has the problem.

If host DNS works but pod DNS fails, stay in the pod path. Do not move straight to public DNS.

Search domains and ndots

Short names can trigger multiple lookup attempts through search domains.

Check:

cat /etc/resolv.conf
dig api
dig api.example.com
dig api.example.com.

If the fully qualified name with a trailing dot is fast but the short name is slow, search-domain expansion may be consuming the timeout budget.

Fix path:

• use fully qualified names for cross-namespace dependencies;
• avoid ambiguous short names in production configs;
• tune ndots only after measuring the impact;
• remove unnecessary search domains when you own the environment.

Fixes by evidence

Resolver unreachable

• restore route to the resolver subnet;
• allow UDP and TCP 53 from the failing namespace;
• check VPN, VPC, security group, NetworkPolicy, and host firewall;
• verify with dig @resolver name from the same namespace.

Wrong resolver

• fix DHCP, VPN, NetworkManager, Docker, kubelet, or systemd-resolved config;
• do not replace an internal resolver with public DNS if private zones are required;
• verify both public and internal names.

Resolver reachable but slow

• inspect resolver CPU, memory, query rate, and upstream latency;
• reduce retry storms;
• add resolver capacity or caching where appropriate;
• check whether search-domain expansion creates extra queries.

Name really missing

• if the resolver returns NXDOMAIN, debug zone ownership, record existence, search suffixes, and split DNS view;
• preserve the resolver IP that returned the answer.

What not to do

• Do not test only from your laptop.
• Do not assume public DNS can resolve private names.
• Do not edit generated /etc/resolv.conf as the final fix.
• Do not confuse resolver timeout with NXDOMAIN.
• Do not tune application retries before proving the DNS path.
• Do not ignore containers, pods, and system services using different resolver context.

Minimal incident note

error text:
failing process:
host/container/pod:
hostname being resolved:
/etc/resolv.conf from failing namespace:
resolver IP:
search domains and options:
dig @resolver result:
dig @resolver +tcp result:
route to resolver:
tcpdump evidence:
resolver logs:
confirmed failure layer:
fix:
verification command:

The issue is solved when the failing process can query the intended resolver and receive the expected answer within its DNS timeout budget.

References

• Linux resolv.conf(5) manual page
• systemd-resolved service documentation
• resolvectl manual page
• Kubernetes DNS for Services and Pods
• tcpdump manual page