kubectl logs: 502 Bad Gateway / timed out waiting for the condition¶

What you see¶

• kubectl get pods and other API calls work from your machine.
• kubectl logs <pod> fails with either:
• "timed out waiting for the condition", or
• "Error from server: Get \"https://:10250/containerLogs/...\": proxy error ... 502 Bad Gateway"

Cause¶

Logs are not read directly from the API server. The API server proxies the request to the kubelet on the node where the pod runs (port 10250). So:

Your machine → API server (e.g. 192.168.4.100:6443) → kubelet on pod’s node (e.g. 192.168.4.102:10250)

The 502 / timeout means the API server cannot successfully talk to that node’s kubelet. Common causes:

1. Kubelet TLS certificate on that node does not include the IP (or hostname) the API server uses to reach it. The API server validates the kubelet’s cert; if the cert is only for 127.0.0.1 or an old IP, the request fails (often reported as 502).
2. Network / firewall: API server cannot reach the node’s 10250 port (e.g. different subnet, firewall, or VPN only exposing the API).
3. Node in another network (e.g. Oracle node on 192.168.2.x, or WireGuard-only reachability). See WIREGUARD-ORACLE-NODE.md.

Quick check¶

# Verbose to see real error (instead of generic timeout)
kubectl logs <pod> -n <namespace> --tail=5 -v=6

• If you see 502 and an URL like https://192.168.4.102:10250/containerLogs/..., the API server is failing to connect to that node’s kubelet (cert or network).
• Logs often work for pods on the same node as the API server (e.g. control-plane node), and fail for pods on other nodes.

Fix: Kubelet certificate (node IP in cert)¶

On the node where logs fail (e.g. plumbus 192.168.4.102), the kubelet’s serving cert must include the IP (or name) the API server uses to reach it.

K3s server/agent¶

1. Set the node IP explicitly when starting K3s so the kubelet cert is issued for that IP.

Server node (e.g. plumbus if it runs the API):

# In the K3s service or install flags
--node-ip=192.168.4.102

Agent node (e.g. plumbus as worker):

# When joining or in the agent service
--node-ip=192.168.4.102

Use the node’s INTERNAL-IP from kubectl get nodes -o wide (the IP the control plane uses to reach the node).

1. Restart K3s on that node so the kubelet certificate is regenerated with the correct SANs.

sudo systemctl restart k3s    # server
# or
sudo systemctl restart k3s-agent

1. Optional: confirm cert SANs on that node:

   sudo openssl x509 -in /var/lib/rancher/k3s/agent/serving-kubelet.crt -noout -text | grep -A1 "Subject Alternative Name"
   

   You should see the node’s IP (and optionally hostname).

If the node has multiple IPs or changed IP¶

• Ensure one primary --node-ip is the one the API server uses (same subnet as control plane, or the address you see in kubectl get nodes -o wide).
• After changing --node-ip, restart K3s; avoid reusing certs from a different IP.

Workarounds from your machine¶

Until the kubelet cert (or network) is fixed on the target node:

1. Logs for pods on the API server node (e.g. yoda):

   kubectl get pods -A -o wide   # find which node the pod is on
   kubectl logs <pod> -n <ns>   # works if pod is on the same node as the API server
   

2. Logs for pods on other nodes: run logs from that node:

   ssh plumbus
   sudo crictl pods
   sudo crictl logs <container-id>
   

   Or use a logging/observability stack (e.g. Loki) that collects logs from the nodes.

Summary¶

After fixing, run again:

kubectl logs deployment/podinfo -n podinfo --tail=5