What I Wish I'd Known About Kubernetes Reliability From Day One

Being on-call changes how you build things. When you are the person who gets paged at 2 AM, you start designing systems very differently than when you are just deploying them during business hours.

Define Reliability Before an Incident Does It For You

"Things not going down" is not a useful definition of reliability. It is too vague to act on and too broad to measure.

The shift that changed how I work was moving to SLIs and SLOs. An SLI (Service Level Indicator) is something you actually measure — request success rate, p99 latency, error rate per minute. An SLO (Service Level Objective) is a target — "99.5% of requests succeed over a rolling 30-day window."

Once you have a target, everything else follows. Your error budget tells you how much downtime is acceptable. Your alerting tells you when you are burning through that budget too fast. Your deployment risk tolerance reflects how much budget you have remaining. Without an SLO, you react to every alert as if it might be catastrophic.

# Example: SLO-based alert using Prometheus
# Alert when error budget is being burned 14x faster than expected
# (will exhaust 30-day budget in ~2 days at this rate)
- alert: HighErrorBudgetBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total[1h]))
    ) > 0.14 * 0.005   # 14x the allowed error rate (0.5% SLO)
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning too fast"

The Failure Modes That Keep Repeating

After enough incidents, patterns emerge. Most Kubernetes incidents fall into a small number of categories:

OOMKilled pods — memory exceeded the limit. Either the limit is too low for the actual workload, or there is a memory leak. Check actual usage first:

# See current memory usage vs configured limits
kubectl top pods -n production --containers
kubectl describe pod POD_NAME -n production | grep -A 5 "Limits\|Requests"

# Check if OOMKill was the reason for restarts
kubectl describe pod POD_NAME -n production | grep -A 3 "Last State"
# Last State:     Terminated
#   Reason:       OOMKilled
#   Exit Code:    137

Cascading restarts under load — an upstream dependency slows down, requests back up, memory grows, OOMKill hits, the pod restarts, the remaining pods absorb more traffic and the cycle repeats. Fix this with request timeouts, circuit breakers, and realistic connection pool limits in the application.

Rollout causing brief errors — new pods receiving traffic before they are ready (missing or misconfigured readiness probe), or existing pods not finishing in-flight requests before being terminated (no graceful shutdown). Both are covered in detail in Kubernetes Pod Lifecycle.

Node pressure evictions — the kubelet evicts pods when a node runs low on memory or disk. The eviction order depends on QoS class. If you have not set resource limits intentionally, your most important pods might be evicted first. BestEffort pods (no limits set at all) go first, then Burstable, then Guaranteed.

Things That Actually Prevent Incidents

Write runbooks before incidents, not after. For every alert, write a document that answers: what does this alert mean, what are the 2-3 most likely causes, what is the exact command to diagnose each one, what is the remediation, how do you verify it is fixed. Write this while you are calm. Incidents are not the time to think through process for the first time.

Test your alerts. An alert you have never seen fire is an alert of unknown quality. Deliberately trigger the condition in staging — force an OOMKill, kill a pod repeatedly, fill a disk. Verify the alert fires, routes through Alertmanager, and appears in Slack or PagerDuty. Then verify the runbook actually works.

Design dashboards for the next person, not for yourself. A dashboard that only makes sense if you already understand the system does not help during an incident. Label everything clearly. Add annotations. Link to runbooks directly from the panel. Assume the person looking at it has never seen this dashboard before.

# A quick health check that gives you the full picture in 60 seconds
kubectl get nodes                                          # Are nodes healthy?
kubectl get pods -A | grep -v Running | grep -v Completed # What is not running?
kubectl get events -A --sort-by=.lastTimestamp | tail -20 # What happened recently?
kubectl top nodes                                          # Is any node under pressure?

The Test of Good Operational Documentation

I have handed off Kubernetes clusters twice. What made the difference was never the code — it was whether the next engineer could operate the system without calling me.

The measure: can someone who has never seen this system navigate a production incident without help? If yes, you have built something operationally excellent. If no, the knowledge lives in your head, which is fragile and does not scale.

Reliable systems are understandable, operable, and recoverable — by anyone, at any time, under pressure. Getting there requires intentional design, not just good infrastructure.