Kubernetes Debugging: The Commands I Actually Use When Things Break

I have been paged for Kubernetes incidents at all hours. The engineers who resolve them quickly are not the ones who know more - they are the ones who work through the problem systematically instead of randomly. Same cluster, same problem, very different time to resolution.

When something breaks, the instinct is to start randomly checking things. Look at pod logs. Restart the deployment. Hope it fixes itself. That sometimes works through luck, but it is slow and unreliable.

What actually works is systematic: start broad to find the scope of the problem, then narrow down to the specific cause. This article documents the exact commands I reach for at each stage.

Start Broad: Get the Full Picture

# Which pods are not Running or Completed?
kubectl get pods -A | grep -Ev "Running|Completed"

# What has happened recently in the cluster?
kubectl get events -A --sort-by=.lastTimestamp | tail -30

# Are any nodes unhealthy?
kubectl get nodes
kubectl describe nodes | grep -A 5 "Conditions:"

# What is resource usage on each node?
kubectl top nodes

The events command is underused. It shows scheduling failures, image pull errors, OOMKills, readiness probe failures, and everything else that has happened. Start here before diving into specific pods.

Diagnosing a Pod That Will Not Start

# The most useful command in Kubernetes debugging
kubectl describe pod POD_NAME -n NAMESPACE

# Events section at the bottom tells you exactly why it failed
# Common messages:
# "0/5 nodes are available: 2 Insufficient cpu"  → resource requests too high
# "Failed to pull image: not found"               → wrong image name or tag
# "Back-off restarting failed container"          → CrashLoopBackOff

# Current container logs
kubectl logs POD_NAME -n NAMESPACE

# Previous container's logs — use this after a crash/restart
# The current logs show the NEW container. --previous shows what crashed.
kubectl logs POD_NAME -n NAMESPACE --previous

# Follow logs in real time
kubectl logs POD_NAME -n NAMESPACE -f

# Specific container in a multi-container pod
kubectl logs POD_NAME -n NAMESPACE -c CONTAINER_NAME

# Last 100 lines
kubectl logs POD_NAME -n NAMESPACE --tail=100

Getting Inside a Container

# Open an interactive shell
kubectl exec -it POD_NAME -n NAMESPACE -- /bin/sh

# Run a single command without an interactive session
kubectl exec POD_NAME -n NAMESPACE -- env | grep -i database

# Test connectivity to another service from inside the container
kubectl exec -it POD_NAME -n NAMESPACE -- sh -c "nc -zv postgres-service 5432"
kubectl exec -it POD_NAME -n NAMESPACE -- sh -c "wget -qO- http://api-service:8080/health"

# If the container doesn't have networking tools, run a debug pod
kubectl run debug-pod   --image=nicolaka/netshoot   --rm -it --restart=Never   -n NAMESPACE   -- bash

# From inside the debug pod:
nslookup postgres-service.production.svc.cluster.local
dig +short postgres-service.production.svc.cluster.local
nc -zv postgres-service 5432

Diagnosing Networking Issues

# Is the service routing to any pods?
kubectl get endpoints SERVICE_NAME -n NAMESPACE
# Empty ENDPOINTS means selector doesn't match any ready pods

# Compare service selector to pod labels
kubectl describe service SERVICE_NAME -n NAMESPACE | grep Selector
kubectl get pods -n NAMESPACE --show-labels | grep app=my-app

# DNS resolution test from inside cluster
kubectl run dns-test --image=busybox --rm -it --restart=Never --   nslookup my-service.my-namespace.svc.cluster.local

Diagnosing Resource Issues

# Sort by memory to find hungry pods
kubectl top pods -n NAMESPACE --sort-by=memory
kubectl top pods -n NAMESPACE --containers --sort-by=memory

# Check configured limits vs actual usage
kubectl describe pod POD_NAME -n NAMESPACE | grep -A 4 "Limits:"

# Find OOMKilled pods
kubectl get pods -n NAMESPACE -o json |   jq '.items[] | select(
    .status.containerStatuses[]?.lastState.terminated.reason == "OOMKilled"
  ) | .metadata.name'

# Exit code 137 = OOMKill, 143 = SIGTERM, 1 = application error
kubectl describe pod POD_NAME -n NAMESPACE | grep "Exit Code"

Diagnosing Deployment Issues

# Check rollout status
kubectl rollout status deployment/my-app -n NAMESPACE

# See rollout history
kubectl rollout history deployment/my-app -n NAMESPACE

# Roll back to previous version
kubectl rollout undo deployment/my-app -n NAMESPACE

# Roll back to specific revision
kubectl rollout undo deployment/my-app -n NAMESPACE --to-revision=3

The Debugging Checklist

When paged about a problem, work through this in order:

1. kubectl get pods -A | grep -v Running — find what is broken
2. kubectl describe pod POD_NAME — read the Events section at the bottom
3. kubectl logs POD_NAME --previous — what did the crashed container say
4. Check endpoints if it is a networking issue
5. Check resource usage if it might be OOMKill or CPU throttling
6. Check recent changes — did something deploy around the time the alert fired?

The last point is the most important and easiest to forget. Most Kubernetes problems are caused by changes. Check kubectl rollout history and kubectl get events sorted by time. If something changed 20 minutes before the alert, start there.

For understanding why pods end up in these states — why OOMKill happens, why pods get stuck in Terminating, why readiness probes matter — see Kubernetes Pod Lifecycle.