🐳 Debugging Distroless Containers Using kubectl debug

Modern container security practices often promote using distroless images — minimal base images that contain only your application and its runtime dependencies.
No package manager, no shell, and definitely no debugging tools like bash, curl, or top.

That’s great for security, but what happens when something goes wrong inside your container?
You can’t just kubectl exec -it <pod> -- bash anymore.

This is where kubectl debug comes to the rescue — a built-in Kubernetes command that helps you troubleshoot distroless or restricted containers by attaching an ephemeral container for debugging purposes.

---

🧩 What Are Distroless Containers?

Distroless containers are built using Google’s Distroless project.
They contain only your application and runtime libraries, with no OS-level tools.

Example:

FROM gcr.io/distroless/nodejs20
COPY app.js .
CMD ["app.js"]

✅ Advantages:

• Smaller image size

• Reduced attack surface

• No shell → more secure

🚫 But also:

• No shell → harder to debug

---

🧠 How kubectl debug Helps

kubectl debug lets you add a temporary debugging container into an existing Pod.
This new container can:

• Share the same network and process namespace as the target container

• Run your favorite debug tools (bash, curl, netcat, etc.)

• Be removed automatically after you’re done

In short — it’s like getting a temporary “sidecar” to inspect your running Pod safely.

---

🧪 Sample Pod for Testing

Let’s create a simple distroless Pod you can use to test this.

Save the following as distroless-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: distroless-demo
  labels:
    app: distroless-demo
spec:
  containers:
    - name: distroless-container
      image: gcr.io/distroless/static:nonroot
      command: ["/bin/sh", "-c", "while true; do echo 'App is running'; sleep 10; done"]
      securityContext:
        runAsNonRoot: true

Apply it:

kubectl apply -f distroless-pod.yaml

Check the pod status:

kubectl get pods

Try to exec into it:

kubectl exec -it distroless-demo -- sh

You’ll see:

error: Internal error occurred: error executing command in container: failed to exec in container: unable to start exec "sh": executable not found in $PATH

That’s because distroless images don’t have a shell.

---

⚡ Using kubectl debug

Now, let’s use the debug command:

kubectl debug -it distroless-demo --image=busybox --target=distroless-container

What this does:

• Creates an ephemeral debug container inside the same pod

• Mounts it with the same network and process namespaces

• Gives you a shell environment to debug

Once attached, try commands like:

ps aux
netstat -tnlp
curl localhost:8080

Or inspect environment variables:

env

When you exit, the ephemeral container disappears — leaving your original Pod untouched.

---

🧰 Alternative Debug Images

Depending on your needs, you can use other debug containers:

• 🧾 nicolaka/netshoot — full of network debugging tools

• 🪄 ubuntu or debian — general-purpose shells

• 🔧 busybox — lightweight and minimal

Example:

kubectl debug -it distroless-demo --image=nicolaka/netshoot --target=distroless-container

---

🔒 Key Takeaways

• Distroless containers are great for security but difficult to inspect directly.

• kubectl debug provides a safe, ephemeral way to troubleshoot live Pods.

• No restarts, no image rebuilds — just instant debugging access.

• Always remember to remove or clean up debug sessions after use.

---

🧭 Wrapping Up

Using kubectl debug is a must-have skill for Kubernetes engineers adopting distroless or hardened container strategies.
It gives you full observability without compromising the immutability and minimalism of your container images.

Next time you face a “no shell found” error — remember, you don’t need to rebuild your image.
Just attach a debug container and fix the issue like a pro! ⚙️

---

Have you used kubectl debug in production yet?
Share your experience or favorite debug image in the comments 👇

#Kubernetes #DevOps #Containers #Distroless #kubectl #CloudNative #Troubleshooting

---