Kubernetes RBAC Hardening — Least Privilege, Kyverno Enforcement, and Automated Auditing

A personal project to simulate and fix one of the most common Kubernetes security problems — excessive cluster-admin access. I set up an EKS cluster deliberately misconfigured to mirror what is common in real clusters, hardened it systematically, then added Kyverno policies and an automated weekly audit to make sure it never drifts back.

Why I Built This

Cluster-admin gets granted because it is the easiest path. The CI pipeline needs to deploy — give it cluster-admin. The monitoring agent throws a permissions error — give it cluster-admin. The application service account fails — give it cluster-admin. Nobody goes back to clean it up because nothing is obviously broken. It is easy to create and easy to ignore. I wanted to build a complete, reproducible fix — not just the YAML, but the enforcement layer that prevents regression and the automated audit that catches it if something slips through.

Environment

The starting cluster was intentionally misconfigured to simulate what I commonly see — service accounts with far more access than they need:

AWS EKS — 12-node cluster
 └── production namespace
 │    ├── app-backend (service account)
 │    ├── app-frontend (service account)
 │    └── ci-pipeline (service account)
 ├── monitoring namespace
 │    └── monitoring-agent (service account)
 ├── break-glass (cluster-admin — emergency only)
 └── CloudWatch — audit log destination (90-day retention)

Implementation

Audit first, change nothing. Before touching any binding, I mapped out exactly what existed. Going straight to revoking permissions risks breaking something that silently depends on access that looks excessive on paper — and you only find out when it breaks. The audit is not optional. It is the only way to know what you are actually dealing with.

# Find every account with cluster-admin
kubectl get clusterrolebindings -o json | jq '
  .items[] |
  select(.roleRef.name == "cluster-admin") |
  {binding: .metadata.name, subjects: .subjects}'

# See exactly what a specific service account is allowed to do
kubectl auth can-i --list --as=system:serviceaccount:production:default

Replace broad bindings with namespace-scoped Roles. Each service account gets a Role covering only the resources and actions it actually uses. The CI pipeline needs to update deployments — it has no reason to touch secrets, other namespaces, or anything cluster-wide.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: app-deployer
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

Turn off automatic token mounting on application pods. This is the most invisible part of the project. Kubernetes defaults to mounting a credential into every pod — even the ones that never call the Kubernetes API. If a pod is compromised through an application vulnerability, the attacker gets that credential for free. Turning it off removes an attack surface most people do not know exists.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: production
automountServiceAccountToken: false

Verify the change, don't just assume it worked. kubectl-who-can makes this readable. Here is what the output actually looks like before and after:

kubectl krew install who-can

# Before — returns a long list including secrets, nodes, namespaces, cluster resources:
# get                  *                          []                    [*]
# create               *                          []                    [*]
# delete               *                          []                    [*]
# ... (dozens more lines)

# After — returns only what the Role permits in the production namespace:
# get                  deployments                [apps]                []
# update               deployments                [apps]                []
# get                  pods                       []                    []
# get                  pods/log                   []                    []

# Spot-check sensitive permissions
kubectl who-can delete pods --all-namespaces
kubectl who-can get secrets -n production

Key Decisions

Namespace-scoped Roles over ClusterRoles. A ClusterRole gives access across every namespace. If the CI pipeline service account is ever compromised, a namespace-scoped Role contains the damage to the production namespace. A ClusterRole means every namespace is exposed. Most service accounts do not need cluster-wide access — they just got it because it was easier to grant than to investigate.

Keep one break-glass account, not zero. Removing every cluster-admin binding sounds cleaner but creates a real operational problem — a genuine emergency with no fast path to respond. One dedicated break-glass account wired to alert CloudWatch the moment it is used is the practical middle ground. Security with an escape hatch beats security that locks you out.

Kyverno in Enforce mode, not Audit. Audit mode logs violations but does not stop them. For something as clear-cut as cluster-admin, Enforce mode is the right choice — the error message is explicit, the fix is well-documented, and the alternative is silent permission creep.

Automated audit over manual process. A manual audit relies on someone remembering to run it and caring enough to investigate the results. A weekly automated audit with a Slack notification removes both dependencies. If something is wrong, the team sees it Monday morning without anyone having to think about it.

Preventing Regression with Kyverno

Cleaning up permissions is only half the problem. Without enforcement, the next person who hits a permissions error will grant cluster-admin again and the work is undone. Kyverno runs as a Kubernetes admission webhook — it intercepts every API call before it is stored and blocks the ones that violate policy.

The block-cluster-admin policy blocks any attempt to create a ClusterRoleBinding for cluster-admin, with one exception for the break-glass account. The error message is specific — it tells the person exactly what to do instead:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-cluster-admin-binding
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-cluster-admin-bindings
      match:
        any:
          - resources:
              kinds: [ClusterRoleBinding]
      validate:
        message: >
          cluster-admin is not allowed. Use a namespace-scoped Role with
          only the permissions your workload actually needs. Contact the
          platform team if you need help scoping permissions correctly.
        deny:
          conditions:
            all:
              - key: "{{ request.object.roleRef.name }}"
                operator: Equals
                value: cluster-admin
              - key: "{{ request.object.subjects[0].name }}"
                operator: NotEquals
                value: break-glass

Two further policies are in place: require-no-automount blocks pods in application namespaces from running if they have token automounting enabled, and require-rbac-labels enforces ownership labels on all Roles and RoleBindings — without labels, RBAC resources accumulate silently with no record of who created them or why.

Automated Weekly Audit

Kyverno blocks future violations, but it does not catch anything that existed before the policy was applied. A GitHub Actions workflow runs every Monday at 8am, audits the cluster, and posts results to Slack. If the cluster-admin binding count exceeds 1, the job fails and the team gets an alert. Someone has to explain why.

name: Weekly RBAC Audit
on:
  schedule:
    - cron: "0 8 * * 1"   # every Monday at 8am UTC
  workflow_dispatch:       # also manually triggerable

jobs:
  audit:
    runs-on: ubuntu-latest
    permissions:
      id-token: write       # OIDC — no AWS credentials stored in GitHub
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Run audit
        run: |
          ADMIN_COUNT=$(kubectl get clusterrolebindings -o json |             jq '[.items[] | select(.roleRef.name == "cluster-admin")] | length')

          if [ "$ADMIN_COUNT" -gt 1 ]; then
            echo "status=FAILED" >> $GITHUB_OUTPUT
          else
            echo "status=PASSED" >> $GITHUB_OUTPUT
          fi

      - name: Post to Slack
        uses: slackapi/slack-github-action@v1.26.0
        # posts ✅ PASSED or 🚨 FAILED with binding count and run link

Here is what the Slack notification looks like — green when the posture is healthy, red when something needs investigating:

Results

Here is what changed across the three weeks of work:

MetricBeforeAftercluster-admin bindings81 — break-glass onlyservice disruptions—0audit loggingNoneCloudWatch — 90-day retentionregression protectionNoneKyverno — enforced at API levelaudit frequencyNeverWeekly — Slack notification

What I Learned

The hardest part was not writing the YAML — it was understanding what each service account actually needed before touching anything. Most had cluster-admin not because anything depended on it, but because it was the path of least resistance. Running kubectl auth can-i --list per service account before writing any Role is what makes this kind of cleanup safe.

The token automounting issue was the most surprising. Nobody makes a conscious decision to mount credentials into application pods — Kubernetes just does it quietly by default. It is completely invisible until you specifically look for it.

What I'd Do Differently

The Kyverno policies enforce at admission time but don't retroactively scan existing resources. Next time I'd run kubectl get policyreport --all-namespaces earlier in the process to surface existing violations alongside the new-resource blocking. I'd also add a policy that prevents application workloads from running in the default namespace entirely — another common shortcut that creates permission management problems further down the line.

This project started with a simple observation — cluster-admin gets handed out like candy because scoping permissions properly takes more effort than just granting everything. The fix is not just writing Roles. It is building an enforcement layer that prevents regression and an automated audit that catches it if something still slips through. That is the difference between a one-off cleanup and a sustainable security posture.

RBAC controls who can do what — but it doesn't control what containers can do once they're running, or which namespaces can talk to each other. Pod Security Standards and Network Policies close those gaps. That project is here.