Observability Stack on Kubernetes: Prometheus, Grafana, Loki and Alertmanager

Project Overview

A production-grade observability stack on Kubernetes using Helm. Single platform: from "alert fires" to "root cause identified" in under 3 minutes — metrics and logs correlated in one view.

Stack Components

• kube-prometheus-stack — Prometheus, Grafana, Alertmanager, Node Exporter in one chart
• Loki + Promtail — lightweight log aggregation (replaced EFK, 60% less resource usage)
• Dashboards-as-code — all Grafana dashboards provisioned via ConfigMaps
• Alertmanager → Slack — all critical alerts with context

Deploy the Stack

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack   --namespace monitoring --create-namespace   --set grafana.adminPassword=your-secure-password   --set prometheus.prometheusSpec.retention=15d

helm install loki grafana/loki-stack   --namespace monitoring   --set promtail.enabled=true

Custom Alerting Rules

groups:
  - name: cluster-health
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.pod }} is crash looping"
    - alert: NodeMemoryPressure
      expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
      for: 2m
      labels:
        severity: warning

Results

• Full stack deployed and operational: under 45 minutes
• MTTR: 15 min → under 3 min via log-metric correlation
• Alert latency: under 30 seconds to Slack