The Operational Patterns Kubernetes Won't Teach You, but Production Will

These are the specific things I wish someone had put in front of me before I learned them through production incidents. None of this is secret knowledge. But there's a difference between reading about something and understanding when it matters — and that only comes from experience.

Pattern 1: Graceful Shutdown

Here's what happens when Kubernetes terminates a pod: it sends SIGTERM, waits 30 seconds, then sends SIGKILL. If your application doesn't handle SIGTERM, it gets forcibly killed mid-request. Every deployment becomes a reliability event.

There is also a timing gap. Kubernetes removes the pod from the load balancer endpoints and sends SIGTERM — but there's propagation delay between those two events. During that window, traffic might still reach the pod. A preStop sleep of 5 seconds covers it:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sleep", "5"]
terminationGracePeriodSeconds: 30

// Catch SIGTERM and drain requests cleanly
process.on('SIGTERM', async () => {
  server.close(async () => {
    await db.disconnect()
    process.exit(0)
  })
  // Force exit 5 seconds before SIGKILL
  setTimeout(() => process.exit(1), 25000)
})

Pattern 2: PodDisruptionBudgets

I learned this one the hard way. We ran a node upgrade and all three replicas of a critical service were evicted simultaneously. The service went down completely. A PodDisruptionBudget tells Kubernetes the minimum number of pods that must remain available during voluntary disruptions like upgrades and node drains.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: production
spec:
  minAvailable: 2   # Always keep at least 2 pods running
  selector:
    matchLabels:
      app: myapp

# Verify PDB is applied
kubectl get pdb -n production

# NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
# myapp-pdb   2               N/A               1

Add a PDB to every production workload. It takes five minutes and prevents an entire class of incidents.

Pattern 3: Pod Anti-Affinity

Three replicas on the same node is not redundancy — it's a single point of failure with extra steps. Use pod anti-affinity to spread replicas across nodes:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - myapp
        topologyKey: kubernetes.io/hostname

Use preferred rather than required. The required variant blocks scheduling when there aren't enough nodes with different hostnames, which trades one problem for another.

# Verify pods are on different nodes after applying
kubectl get pods -n production -o wide
# NAME          NODE
# myapp-abc     ip-10-0-1-xx...  ← different nodes
# myapp-def     ip-10-0-2-xx...
# myapp-ghi     ip-10-0-3-xx...

Pattern 4: Resource Quotas Stop Runaway Processes

Without quotas, one team running a load test can starve other services of resources. I've seen it happen. Add a quota per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "8"
    requests.memory: "16Gi"
    limits.cpu: "16"
    limits.memory: "32Gi"
    pods: "50"

Pair with a LimitRange so pods without resource specs get sensible defaults instead of BestEffort QoS:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "500m"
      memory: "256Mi"

kubectl apply -f quota.yaml
kubectl apply -f limitrange.yaml

# Verify
kubectl describe resourcequota team-quota -n team-a
kubectl describe limitrange default-limits -n team-a

The Daily kubectl Commands

These become muscle memory:

# Why is this pod not running?
kubectl describe pod POD_NAME -n NAMESPACE

# What did it say before it crashed?
kubectl logs POD_NAME -n NAMESPACE --previous

# How much is it actually using?
kubectl top pods -n NAMESPACE --sort-by=memory

# What happened recently in this namespace?
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp | tail -20

# Is my change safe? (dry run without applying)
kubectl apply -f manifest.yaml --dry-run=client

# Quickly see all non-running pods
kubectl get pods -A | grep -v Running | grep -v Completed

Understanding why each of these patterns matters is covered in Kubernetes Pod Lifecycle — specifically why graceful shutdown requires both the application and the preStop hook, and what the SIGTERM timing actually looks like.