The Vault is Empty

intermediateCreateContainerConfigError

Objective

Escape Room: The Vault is Empty

The application pod is stuck and cannot start. It's looking for something that should exist but doesn't.

Your Mission

Identify why the pod cannot start
Figure out what it needs
Fix the problem so the pod runs successfully

Success Criteria

The pod escape-app is in Running state
The pod shows "Application started successfully!" in its logs

Getting Started

# Check the pod status
kubectl get pods -n escape-room-vault-empty

# You'll see something like:
# NAME         READY   STATUS                       RESTARTS   AGE
# escape-app   0/1     CreateContainerConfigError   0          30s

Namespace

All resources are in the escape-room-vault-empty namespace.

Good luck, engineer. The application won't start without what it's looking for.

Quick Start

Run this command in your terminal to set up the room:

$ make room-apply ROOM=room-vault-empty

This creates the namespace escape-room-vault-empty with the broken resources.

Other useful commands:

$ make room-test ROOM=room-vault-empty

Verify the room is in the expected broken state

$ make room-escape-test ROOM=room-vault-empty

Test if you have successfully fixed all issues

$ make room-reset ROOM=room-vault-empty

Reset the room to try again

Useful Commands

Check pod status

$ kubectl get pods -n escape-room-vault-empty

See the current state of pods in the namespace

View events

$ kubectl get events -n escape-room-vault-empty --sort-by='.lastTimestamp'

Check recent events for error details

Describe pods

$ kubectl describe pods -n escape-room-vault-empty

Get detailed information about pods

Check logs

$ kubectl logs -l app.kubernetes.io/part-of=K8sEscapeRoom -n escape-room-vault-empty

View the application logs

Hints

0/4 revealed

Submit Proof

View Solution (Spoiler)

Solution preview locked

Complete the room to unlock the full solution here

Run this to see the full solution:

$ make room-solution ROOM=room-vault-empty

Show solution anyway (spoiler)

Solution: Secret Missing

Root Cause

The pod specification includes an environment variable that references a Secret:

env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

This Secret was never created, so Kubernetes cannot start the container. The CreateContainerConfigError status indicates that the container runtime couldn't configure the container environment because a required Secret is missing.

Diagnosis Steps

# Step 1: Check pod status
kubectl get pods -n escape-room-vault-empty
# Output: escape-app   0/1   CreateContainerConfigError   0   ...

# Step 2: Check events for details
kubectl describe pod escape-app -n escape-room-vault-empty
# Look at Events section - you'll see:
# Warning  Failed  ...  secret "db-credentials" not found

# Step 3: Verify Secret doesn't exist
kubectl get secrets -n escape-room-vault-empty
# Output: No resources found (or only default service account token)

# Step 4: Check what the pod expects
kubectl get pod escape-app -n escape-room-vault-empty -o yaml | grep -A5 secretKeyRef

The Fix

Option 1: Create Secret Imperatively

kubectl create secret generic db-credentials \
  --from-literal=password=supersecretpassword123 \
  -n escape-room-vault-empty

Option 2: Create Secret Declaratively

Create a file secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: escape-room-vault-empty
type: Opaque
stringData:
  password: supersecretpassword123

Apply it:

kubectl apply -f secret.yaml

Note: Using stringData allows plain text; using data requires base64 encoding.

Verification

# Check the Secret was created
kubectl get secret db-credentials -n escape-room-vault-empty

# Watch the pod recover
kubectl get pods -n escape-room-vault-empty -w

# Check the logs once running
kubectl logs escape-app -n escape-room-vault-empty
# Should show: "Application started successfully!"

Lessons Learned

CreateContainerConfigError often means a Secret or ConfigMap reference is broken
Secrets must exist before pods that reference them can start
The secretKeyRef must match both the Secret name AND the key within the Secret
Secrets are namespace-scoped - they must be in the same namespace as the pod

Real-World Considerations

This commonly happens when:

Deploying to a new environment without creating secrets first
Secret was deleted or expired
Typo in secret name or key (both are case-sensitive!)
Secret exists but in wrong namespace
CI/CD pipeline didn't provision secrets before deployment

Best practices:

Use external secret management (Vault, AWS Secrets Manager, etc.)
Use tools like External Secrets Operator to sync secrets
Consider optional: true on secretKeyRef only if truly optional
Never commit secrets to git - use sealed secrets or external references
Document required secrets in deployment documentation