K8s
EscapeRoom
Back to Level Select

Bad Gateway

intermediateIngressMisconfigured

Objective

Escape Room: Bad Gateway

Everything looks healthy internally, but requests from outside aren't making it to the app. The pod and service seem fine on their own.

Prerequisites

This room requires an Ingress controller to be installed in your cluster. If you're using kind, you can install nginx-ingress:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

If you don't have an ingress controller, you can still solve this room by fixing the Ingress configuration - the escape tests will verify the config even without testing actual connectivity.

Your Mission

  1. Verify the pod and service are working internally
  2. Investigate why the Ingress isn't routing traffic correctly
  3. Fix the Ingress configuration so external traffic can reach the app

Success Criteria

  • The pod escape-app is in Running state
  • Curling through the Ingress returns a successful response (HTTP 200)
  • You can access the nginx welcome page via the Ingress

Getting Started

# Check the pod and service
kubectl get pods,svc -n escape-room-bad-gateway

# Check the Ingress
kubectl get ingress -n escape-room-bad-gateway

# Try to reach the app via the ingress (if ingress controller is set up)
# Note: You may need to use port-forward or set up /etc/hosts
kubectl describe ingress escape-ingress -n escape-room-bad-gateway

Namespace

All resources are in the escape-room-bad-gateway namespace.

Good luck, engineer. The app is running, but no one outside can find it.

Quick Start

Run this command in your terminal to set up the room:

$ make room-apply ROOM=room-bad-gateway

This creates the namespace escape-room-bad-gateway with the broken resources.

Other useful commands:

$ make room-test ROOM=room-bad-gateway

Verify the room is in the expected broken state

$ make room-escape-test ROOM=room-bad-gateway

Test if you have successfully fixed all issues

$ make room-reset ROOM=room-bad-gateway

Reset the room to try again

Useful Commands

Check pod status

$ kubectl get pods -n escape-room-bad-gateway

See the current state of pods in the namespace

View events

$ kubectl get events -n escape-room-bad-gateway --sort-by='.lastTimestamp'

Check recent events for error details

Describe pods

$ kubectl describe pods -n escape-room-bad-gateway

Get detailed information about pods

Check logs

$ kubectl logs -l app.kubernetes.io/part-of=K8sEscapeRoom -n escape-room-bad-gateway

View the application logs

Hints

0/4 revealed

Submit Proof

Login to submit proof and track your progress.

Login with GitHub
View Solution (Spoiler)

Solution preview locked

Complete the room to unlock the full solution here

Run this to see the full solution:

$ make room-solution ROOM=room-bad-gateway
Show solution anyway (spoiler)

Solution: Ingress Misroute

Root Cause

The Ingress resource references a service named escape-svc, but the actual service is named escape-service:

# In the Ingress (WRONG):
backend:
  service:
    name: escape-svc      # Typo! Missing "-ervice"
    port:
      number: 80

# Actual service name:
metadata:
  name: escape-service    # Correct name

When the Ingress controller tries to route traffic to escape-svc, it can't find the service, resulting in a 404 or 503 error depending on the ingress controller.

Diagnosis Steps

# Step 1: Verify pod and service are working
kubectl get pods,svc -n escape-room-bad-gateway
# Pod is Running, Service exists

# Step 2: Check the Ingress
kubectl get ingress -n escape-room-bad-gateway
kubectl describe ingress escape-ingress -n escape-room-bad-gateway
# Look at the "Rules" section and "Backends"

# Step 3: Compare service names
kubectl get svc -n escape-room-bad-gateway -o name
# Output: service/escape-service

kubectl get ingress escape-ingress -n escape-room-bad-gateway -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}'
# Output: escape-svc  ← MISMATCH!

# Step 4: Test internal connectivity (should work)
kubectl run test-curl --rm -it --image=curlimages/curl --restart=Never \
  -n escape-room-bad-gateway -- curl -s http://escape-service
# Returns nginx welcome page - service works!

The Fix

Option 1: Patch the Ingress

kubectl patch ingress escape-ingress -n escape-room-bad-gateway \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/rules/0/http/paths/0/backend/service/name", "value": "escape-service"}]'

Option 2: Edit the Ingress Directly

kubectl edit ingress escape-ingress -n escape-room-bad-gateway

Change:

backend:
  service:
    name: escape-svc

To:

backend:
  service:
    name: escape-service

Option 3: Replace via YAML

kubectl get ingress escape-ingress -n escape-room-bad-gateway -o yaml | \
  sed 's/escape-svc/escape-service/g' | \
  kubectl apply -f -

Verification

# Check the Ingress now references the correct service
kubectl describe ingress escape-ingress -n escape-room-bad-gateway

# If you have the ingress controller and /etc/hosts configured:
curl -H "Host: escape.local" http://localhost/api

# Or use port-forward to the ingress controller and test
# Or run a curl pod with the Host header:
kubectl run test-ingress --rm -it --image=curlimages/curl --restart=Never \
  -n escape-room-bad-gateway -- \
  curl -s -H "Host: escape.local" http://ingress-nginx-controller.ingress-nginx/api

Lessons Learned

  1. Service names must match exactly - typos cause routing failures
  2. Ingress issues often result in 404/503 errors, not pod failures
  3. Use kubectl describe ingress to see the resolved backends
  4. Always verify the service name exists before referencing it in an Ingress
  5. Internal connectivity (Service) can work while external (Ingress) fails

Real-World Considerations

This commonly happens when:

  • Copy-pasting Ingress configs and forgetting to update service names
  • Service renamed but Ingress not updated
  • Different naming conventions across teams (svc vs service)
  • Auto-generated names don't match manual Ingress configs

Best practices:

  • Use Helm/Kustomize to template service names consistently
  • Validate Ingress backends exist before deployment
  • Monitor Ingress controller logs for routing errors
  • Use meaningful, consistent naming conventions
  • Consider using ExternalName services for external dependencies
  • Test Ingress rules in staging before production