Access Denied

intermediateForbidden

Objective

Escape Room: Access Denied

The application pod runs but crashes when it tries to talk to the cluster. It wants to do something but Kubernetes won't let it.

Your Mission

Identify what the application is trying to do
Understand why it's failing
Fix the configuration so the application can function

Success Criteria

The pod shows "SUCCESS: Pod listing completed!" in its logs
The pod remains in Running state (not failing/restarting)

Getting Started

# Check the pod status
kubectl get pods -n escape-room-access-denied

# Check the logs to see what's happening
kubectl logs -l app=escape-app -n escape-room-access-denied

Namespace

All resources are in the escape-room-access-denied namespace.

Good luck, engineer. The app knows what it wants to do, but Kubernetes won't let it.

Quick Start

Run this command in your terminal to set up the room:

$ make room-apply ROOM=room-access-denied

This creates the namespace escape-room-access-denied with the broken resources.

Other useful commands:

$ make room-test ROOM=room-access-denied

Verify the room is in the expected broken state

$ make room-escape-test ROOM=room-access-denied

Test if you have successfully fixed all issues

$ make room-reset ROOM=room-access-denied

Reset the room to try again

Useful Commands

Check pod status

$ kubectl get pods -n escape-room-access-denied

See the current state of pods in the namespace

View events

$ kubectl get events -n escape-room-access-denied --sort-by='.lastTimestamp'

Check recent events for error details

Describe pods

$ kubectl describe pods -n escape-room-access-denied

Get detailed information about pods

Check logs

$ kubectl logs -l app.kubernetes.io/part-of=K8sEscapeRoom -n escape-room-access-denied

View the application logs

Hints

0/4 revealed

Submit Proof

View Solution (Spoiler)

Solution preview locked

Complete the room to unlock the full solution here

Run this to see the full solution:

$ make room-solution ROOM=room-access-denied

Show solution anyway (spoiler)

Solution: RBAC Denied

Root Cause

The pod runs with a ServiceAccount (escape-sa) that has no RBAC permissions. When the application tries to list pods using the Kubernetes API, the API server denies the request:

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:escape-room-access-denied:escape-sa" cannot list resource "pods" in API group "" in the namespace "escape-room-access-denied"

In Kubernetes, ServiceAccounts have no permissions by default. You must explicitly grant permissions using Role/RoleBinding (namespace-scoped) or ClusterRole/ClusterRoleBinding (cluster-wide).

Diagnosis Steps

# Step 1: Check pod logs
kubectl logs -l app=escape-app -n escape-room-access-denied
# Shows: FAILED: Permission denied!

# Step 2: Identify the ServiceAccount
kubectl get deployment escape-app -n escape-room-access-denied -o jsonpath='{.spec.template.spec.serviceAccountName}'
# Output: escape-sa

# Step 3: Check current permissions
kubectl auth can-i list pods \
  --as=system:serviceaccount:escape-room-access-denied:escape-sa \
  -n escape-room-access-denied
# Output: no

# Step 4: Check if any roles/bindings exist
kubectl get roles,rolebindings -n escape-room-access-denied
# Output: No resources found

The Fix

Create a Role that allows reading pods and a RoleBinding that grants it to the ServiceAccount:

kubectl create role pod-reader \
  --verb=get,list,watch \
  --resource=pods \
  -n escape-room-access-denied

kubectl create rolebinding pod-reader-binding \
  --role=pod-reader \
  --serviceaccount=escape-room-access-denied:escape-sa \
  -n escape-room-access-denied

Then delete the pod so the Deployment recreates it with the new permissions:

kubectl delete pod -l app=escape-app -n escape-room-access-denied

Declarative Alternative

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: escape-room-access-denied
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: escape-room-access-denied
subjects:
  - kind: ServiceAccount
    name: escape-sa
    namespace: escape-room-access-denied
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Verification

# Check the RBAC resources exist
kubectl get roles,rolebindings -n escape-room-access-denied

# Verify permissions were granted
kubectl auth can-i list pods \
  --as=system:serviceaccount:escape-room-access-denied:escape-sa \
  -n escape-room-access-denied
# Output: yes

# Check the pod logs
kubectl logs -l app=escape-app -n escape-room-access-denied
# Should show: SUCCESS: Pod listing completed!

Lessons Learned

ServiceAccounts have no default permissions beyond basic API discovery
RBAC has two parts: Role (what's allowed) + RoleBinding (who gets it)
Namespace-scoped (Role/RoleBinding) vs cluster-scoped (ClusterRole/ClusterRoleBinding)
Use kubectl auth can-i to test permissions without running the actual workload
Pods must be restarted to pick up new RBAC permissions

Real-World Considerations

This commonly happens when:

Deploying apps that need to interact with the Kubernetes API
Operators and controllers that watch/manage resources
CI/CD tools running in-cluster
Monitoring tools that need to discover endpoints

Best practices:

Follow principle of least privilege - only grant what's needed
Use Roles (namespace-scoped) unless cluster-wide access is truly needed
Audit RBAC permissions regularly
Use kubectl auth can-i --list to see all permissions for a user/SA
Consider using pre-defined ClusterRoles like view, edit, admin where appropriate
Document required RBAC permissions in your application's deployment docs