Bad Gateway
Objective
Escape Room: Bad Gateway
Everything looks healthy internally, but requests from outside aren't making it to the app. The pod and service seem fine on their own.
Prerequisites
This room requires an Ingress controller to be installed in your cluster. If you're using kind, you can install nginx-ingress:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx \
--for=condition=ready pod \
--selector=app.kubernetes.io/component=controller \
--timeout=90s
If you don't have an ingress controller, you can still solve this room by fixing the Ingress configuration - the escape tests will verify the config even without testing actual connectivity.
Your Mission
- Verify the pod and service are working internally
- Investigate why the Ingress isn't routing traffic correctly
- Fix the Ingress configuration so external traffic can reach the app
Success Criteria
- The pod
escape-appis inRunningstate - Curling through the Ingress returns a successful response (HTTP 200)
- You can access the nginx welcome page via the Ingress
Getting Started
# Check the pod and service
kubectl get pods,svc -n escape-room-bad-gateway
# Check the Ingress
kubectl get ingress -n escape-room-bad-gateway
# Try to reach the app via the ingress (if ingress controller is set up)
# Note: You may need to use port-forward or set up /etc/hosts
kubectl describe ingress escape-ingress -n escape-room-bad-gateway
Namespace
All resources are in the escape-room-bad-gateway namespace.
Good luck, engineer. The app is running, but no one outside can find it.
Quick Start
Run this command in your terminal to set up the room:
$ make room-apply ROOM=room-bad-gatewayThis creates the namespace escape-room-bad-gateway with the broken resources.
Other useful commands:
$ make room-test ROOM=room-bad-gatewayVerify the room is in the expected broken state
$ make room-escape-test ROOM=room-bad-gatewayTest if you have successfully fixed all issues
$ make room-reset ROOM=room-bad-gatewayReset the room to try again
Useful Commands
Check pod status
$ kubectl get pods -n escape-room-bad-gatewaySee the current state of pods in the namespace
View events
$ kubectl get events -n escape-room-bad-gateway --sort-by='.lastTimestamp'Check recent events for error details
Describe pods
$ kubectl describe pods -n escape-room-bad-gatewayGet detailed information about pods
Check logs
$ kubectl logs -l app.kubernetes.io/part-of=K8sEscapeRoom -n escape-room-bad-gatewayView the application logs
Hints
Submit Proof
Login to submit proof and track your progress.
Login with GitHubView Solution (Spoiler)
Solution preview locked
Complete the room to unlock the full solution here
Run this to see the full solution:
$ make room-solution ROOM=room-bad-gatewayShow solution anyway (spoiler)
Solution: Ingress Misroute
Root Cause
The Ingress resource references a service named escape-svc, but the actual service is named escape-service:
# In the Ingress (WRONG):
backend:
service:
name: escape-svc # Typo! Missing "-ervice"
port:
number: 80
# Actual service name:
metadata:
name: escape-service # Correct name
When the Ingress controller tries to route traffic to escape-svc, it can't find the service, resulting in a 404 or 503 error depending on the ingress controller.
Diagnosis Steps
# Step 1: Verify pod and service are working
kubectl get pods,svc -n escape-room-bad-gateway
# Pod is Running, Service exists
# Step 2: Check the Ingress
kubectl get ingress -n escape-room-bad-gateway
kubectl describe ingress escape-ingress -n escape-room-bad-gateway
# Look at the "Rules" section and "Backends"
# Step 3: Compare service names
kubectl get svc -n escape-room-bad-gateway -o name
# Output: service/escape-service
kubectl get ingress escape-ingress -n escape-room-bad-gateway -o jsonpath='{.spec.rules[0].http.paths[0].backend.service.name}'
# Output: escape-svc ← MISMATCH!
# Step 4: Test internal connectivity (should work)
kubectl run test-curl --rm -it --image=curlimages/curl --restart=Never \
-n escape-room-bad-gateway -- curl -s http://escape-service
# Returns nginx welcome page - service works!
The Fix
Option 1: Patch the Ingress
kubectl patch ingress escape-ingress -n escape-room-bad-gateway \
--type='json' \
-p='[{"op": "replace", "path": "/spec/rules/0/http/paths/0/backend/service/name", "value": "escape-service"}]'
Option 2: Edit the Ingress Directly
kubectl edit ingress escape-ingress -n escape-room-bad-gateway
Change:
backend:
service:
name: escape-svc
To:
backend:
service:
name: escape-service
Option 3: Replace via YAML
kubectl get ingress escape-ingress -n escape-room-bad-gateway -o yaml | \
sed 's/escape-svc/escape-service/g' | \
kubectl apply -f -
Verification
# Check the Ingress now references the correct service
kubectl describe ingress escape-ingress -n escape-room-bad-gateway
# If you have the ingress controller and /etc/hosts configured:
curl -H "Host: escape.local" http://localhost/api
# Or use port-forward to the ingress controller and test
# Or run a curl pod with the Host header:
kubectl run test-ingress --rm -it --image=curlimages/curl --restart=Never \
-n escape-room-bad-gateway -- \
curl -s -H "Host: escape.local" http://ingress-nginx-controller.ingress-nginx/api
Lessons Learned
- Service names must match exactly - typos cause routing failures
- Ingress issues often result in 404/503 errors, not pod failures
- Use
kubectl describe ingressto see the resolved backends - Always verify the service name exists before referencing it in an Ingress
- Internal connectivity (Service) can work while external (Ingress) fails
Real-World Considerations
This commonly happens when:
- Copy-pasting Ingress configs and forgetting to update service names
- Service renamed but Ingress not updated
- Different naming conventions across teams (svc vs service)
- Auto-generated names don't match manual Ingress configs
Best practices:
- Use Helm/Kustomize to template service names consistently
- Validate Ingress backends exist before deployment
- Monitor Ingress controller logs for routing errors
- Use meaningful, consistent naming conventions
- Consider using ExternalName services for external dependencies
- Test Ingress rules in staging before production