Full Observability Stack on Kubernetes: Prometheus, Grafana, Loki, and Alertmanager with Helm

We had an incident that took four hours to resolve. The actual fix took ten minutes once we found the problem. The other three hours and fifty minutes were spent guessing - looking at the wrong pods, waiting for log queries that timed out, not knowing which service was the actual culprit because we had no way to correlate a metric spike with a specific log entry at a specific time.

After that incident, we built a proper observability stack. Not to look sophisticated - to stop flying blind. This is exactly what we deployed and how.

What You Need

A running Kubernetes cluster with at least 8GB of available cluster memory for the full stack.

• Install kubectl
• Install Helm 3

helm version
# version.BuildInfo{Version:"v3.x.x"...}

kubectl get nodes
# Confirm your cluster is healthy before proceeding

We had an incident that took four hours to resolve. The actual fix took ten minutes once we found the problem. The other three hours and fifty minutes were spent guessing — looking at the wrong pods, waiting for log queries that timed out, not knowing which service was the actual culprit.

After that incident, we built a proper observability stack. This article covers exactly what we deployed.

The Stack

Prometheus (metrics) + Grafana (dashboards + visualisation) + Loki (logs) + Alertmanager (alerting). All deployed with Helm, all running inside Kubernetes.

Step 1: Add Helm Repositories

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Step 2: Install kube-prometheus-stack

This single chart installs Prometheus, Grafana, and Alertmanager with pre-built dashboards for Kubernetes:

kubectl create namespace monitoring

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack   --namespace monitoring   --set grafana.adminPassword=CHANGE_THIS_PASSWORD   --set prometheus.prometheusSpec.retention=15d   --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi   --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.resources.requests.storage=10Gi

# Verify all pods are running (takes 2-3 minutes)
kubectl get pods -n monitoring -w

# NAME                                                     READY   STATUS
# alertmanager-kube-prometheus-stack-alertmanager-0        2/2     Running
# kube-prometheus-stack-grafana-xxx                        3/3     Running
# kube-prometheus-stack-prometheus-node-exporter-xxx       1/1     Running
# prometheus-kube-prometheus-stack-prometheus-0            2/2     Running

Step 3: Access Grafana

# Port-forward to access Grafana locally
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

# Open http://localhost:3000
# Username: admin
# Password: CHANGE_THIS_PASSWORD (what you set above)

Navigate to Dashboards — you'll see pre-built dashboards for node resources, pod resource usage, and namespace overview. These give you immediate visibility without writing any PromQL.

Step 4: Install Loki for Log Aggregation

helm install loki grafana/loki-stack   --namespace monitoring   --set promtail.enabled=true   --set loki.persistence.enabled=true   --set loki.persistence.size=20Gi

# Verify Loki and Promtail are running
kubectl get pods -n monitoring | grep loki
# loki-0               1/1     Running
# loki-promtail-xxx    1/1     Running   (one per node)

Step 5: Connect Loki to Grafana

In Grafana: go to Connections → Data Sources → Add data source → Loki.

URL: http://loki:3100

Click Save & Test. You should see "Data source connected".

Now in any Grafana panel, you can switch between Prometheus metrics and Loki logs. Click a spike on a metrics graph, see the log lines from that exact timestamp. This is what changes incident response from 45 minutes to 5.

Step 6: Write Alerts That Matter

Don't alert on everything unusual. Alert on things that require human action:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: critical-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # Must match the Helm release label
spec:
  groups:
  - name: pods
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.pod }} keeps restarting"
        description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted {{ $value }} times in 15 minutes"
        runbook_url: "https://livinstone.dev/runbooks/pod-crash-looping"

  - name: nodes
    rules:
    - alert: NodeHighMemory
      expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.instance }} memory below 10%"

kubectl apply -f alerts.yaml

# Verify Prometheus picked up the rules
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
# Open http://localhost:9090/rules
# Your rules should appear in the list

Step 7: Configure Alertmanager to Send to Slack

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    global:
      slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
    route:
      receiver: 'slack-notifications'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    receivers:
    - name: 'slack-notifications'
      slack_configs:
      - channel: '#alerts-production'
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}
{{ end }}'
        send_resolved: true

kubectl apply -f alertmanager-config.yaml

# Test the config
kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093
# Open http://localhost:9093
# Click "Send Test Alert" to verify Slack receives it

After three months running this stack, our mean time to identify the root cause of incidents dropped from 45 minutes to under 5. Not because we became better engineers — because we stopped flying blind.

For connecting this alerting to structured runbooks, see Runbooks That Actually Get Used.