Modern container security practices often promote using distroless images — minimal base images that contain only your application and its runtime dependencies.
No package manager, no shell, and definitely no debugging tools like bash, curl, or top.
That’s great for security, but what happens when something goes wrong inside your container?
You can’t just kubectl exec -it <pod> -- bash anymore.
This is where kubectl debug comes to the rescue — a built-in Kubernetes command that helps you troubleshoot distroless or restricted containers by attaching an ephemeral container for debugging purposes.
🧩 What Are Distroless Containers?
Distroless containers are built using Google’s Distroless project.
They contain only your application and runtime libraries, with no OS-level tools.
Example:
FROM gcr.io/distroless/nodejs20
COPY app.js .
CMD ["app.js"]
✅ Advantages:
Smaller image size
Reduced attack surface
No shell → more secure
🚫 But also:
- No shell → harder to debug
🧠 How kubectl debug Helps
kubectl debug lets you add a temporary debugging container into an existing Pod.
This new container can:
Share the same network and process namespace as the target container
Run your favorite debug tools (bash, curl, netcat, etc.)
Be removed automatically after you’re done
In short — it’s like getting a temporary “sidecar” to inspect your running Pod safely.
🧪 Sample Pod for Testing
Let’s create a simple distroless Pod you can use to test this.
Save the following as distroless-pod.yaml:
apiVersion: v1
kind: Pod
metadata:
name: distroless-demo
labels:
app: distroless-demo
spec:
containers:
- name: distroless-container
image: gcr.io/distroless/static:nonroot
command: ["/bin/sh", "-c", "while true; do echo 'App is running'; sleep 10; done"]
securityContext:
runAsNonRoot: true
Apply it:
kubectl apply -f distroless-pod.yaml
Check the pod status:
kubectl get pods
Try to exec into it:
kubectl exec -it distroless-demo -- sh
You’ll see:
error: Internal error occurred: error executing command in container: failed to exec in container: unable to start exec "sh": executable not found in $PATH
That’s because distroless images don’t have a shell.
⚡ Using kubectl debug
Now, let’s use the debug command:
kubectl debug -it distroless-demo --image=busybox --target=distroless-container
What this does:
Creates an ephemeral debug container inside the same pod
Mounts it with the same network and process namespaces
Gives you a shell environment to debug
Once attached, try commands like:
ps aux
netstat -tnlp
curl localhost:8080
Or inspect environment variables:
env
When you exit, the ephemeral container disappears — leaving your original Pod untouched.
🧰 Alternative Debug Images
Depending on your needs, you can use other debug containers:
🧾 nicolaka/netshoot — full of network debugging tools
🪄 ubuntu or debian — general-purpose shells
🔧 busybox — lightweight and minimal
Example:
kubectl debug -it distroless-demo --image=nicolaka/netshoot --target=distroless-container
🔒 Key Takeaways
Distroless containers are great for security but difficult to inspect directly.
kubectl debug provides a safe, ephemeral way to troubleshoot live Pods.
No restarts, no image rebuilds — just instant debugging access.
Always remember to remove or clean up debug sessions after use.
🧭 Wrapping Up
Using kubectl debug is a must-have skill for Kubernetes engineers adopting distroless or hardened container strategies.
It gives you full observability without compromising the immutability and minimalism of your container images.
Next time you face a “no shell found” error — remember, you don’t need to rebuild your image.
Just attach a debug container and fix the issue like a pro! ⚙️
Have you used kubectl debug in production yet?
Share your experience or favorite debug image in the comments 👇
#Kubernetes #DevOps #Containers #Distroless #kubectl #CloudNative #Troubleshooting
