What Years of Using Kubernetes Taught Me That the Documentation Never Could

I remember the first time I felt like I actually understood Kubernetes. I'd spent weeks reading the docs, deploying sample apps, and learning all the objects — Pods, Deployments, Services, Ingresses. I could look at a YAML file and know exactly what it would do.

Then we had our first real production incident. A deployment went out, requests started failing, and for twenty minutes nobody could figure out why. The app was running. The service existed. The ingress was configured correctly. But users were getting errors.

Turned out the new pods were receiving traffic before they were actually ready to handle it. We had no readiness probe. The docs had mentioned readiness probes. I'd read about them. But I hadn't understood why they mattered until that moment.

The Mental Shift That Changes Everything

Early on, I thought about Kubernetes as a tool for running containers. That's accurate but incomplete. The more useful mental model is: Kubernetes is a platform for running reliable systems. The difference changes every decision you make.

When you think about it as a container runner, you ask "how do I deploy this?" When you think about it as a reliability platform, you ask "what happens when this pod dies? When this node fails? When I deploy a bad version?"

Lesson 1: Readiness Probes Aren't Optional

A readiness probe is Kubernetes asking "is this pod actually ready to serve traffic?" — not just "is the process running?" Until a readiness probe passes, Kubernetes won't send traffic to the pod. This is the thing that would have prevented our incident.

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 10   # Give app time to start before first check
  periodSeconds: 5          # Check every 5 seconds
  failureThreshold: 3       # Fail 3 times before marking as not ready

Your /healthz/ready endpoint should only return 200 when your app has connected to the database, warmed its cache, and is genuinely ready to handle requests. A liveness probe is separate — it only checks whether the process is still functional, not whether dependencies are available.

# Test your probe endpoint locally before deploying
curl -v http://localhost:8080/healthz/ready
# Should return 200 only when actually ready

Lesson 2: Resource Requests Are a Promise to the Scheduler

I spent a long time treating resource requests as a formality. This caused two problems: pods scheduled onto nodes without enough capacity, and paying for resources doing nothing.

The right approach is to measure first:

# See actual resource usage after running under real traffic
kubectl top pods -n production
kubectl top pods -n production --containers

# Set requests at roughly the 90th percentile of what you measure
# Set limits at about 2x the requests

resources:
  requests:
    cpu: "250m"     # Based on actual measurement
    memory: "256Mi"
  limits:
    cpu: "500m"     # 2x the request for burst headroom
    memory: "512Mi"

Lesson 3: Applications Need to Handle SIGTERM

When Kubernetes terminates a pod — for a rolling update, node drain, or scale-down — it sends SIGTERM and waits 30 seconds. If your app ignores SIGTERM, it gets killed mid-request. Users get errors. This is completely avoidable:

process.on('SIGTERM', async () => {
  console.log('SIGTERM received — shutting down gracefully')

  // Stop accepting new connections
  server.close(async () => {
    // Close database connections cleanly
    await db.disconnect()
    process.exit(0)
  })

  // Safety net: don't wait longer than 25s (5s before SIGKILL)
  setTimeout(() => {
    console.log('Forcing exit')
    process.exit(1)
  }, 25000)
})

# Add a preStop hook to cover the propagation delay
lifecycle:
  preStop:
    exec:
      command: ["/bin/sleep", "5"]
terminationGracePeriodSeconds: 30

Lesson 4: Kubernetes Doesn't Make Apps Resilient — You Do

This took me longest to internalise. Kubernetes gives you the tools for resilience — replica sets, rolling deployments, health checks, automatic restarts. But none of those tools can fix an application that crashes when a database connection fails, or that doesn't retry on transient errors.

A pod that crashes and restarts is not resilient. It's a pod that restarts faster. True resilience is built into the application — retries with exponential backoff, circuit breakers, graceful degradation when upstream services are slow.

# Example: spread replicas across nodes for real redundancy
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            app: my-app

For a complete picture of what happens at each stage of a pod's life — from scheduling through to termination — see Kubernetes Pod Lifecycle. And for how resource requests and limits interact with scheduler decisions and eviction, see Kubernetes Resource Management.