peter (admin) -

Intentionally Breaking Kubernetes Pods for Troubleshooting
Introduction

Now we will learn how to intentionally break Pods in a safe way so that we can understand common failure patterns and practice real Kubernetes troubleshooting skills.
For this demonstration, we are already logged into both the control-plane node and the worker node.
We will create two different failure scenarios:
-
A Pod that uses an invalid container image.
-
A Pod containing a container that intentionally crashes.
These examples allow us to examine how Kubernetes reports failures and how we can use Pod status, kubectl describe, and events to diagnose problems.
Simulating an Image Pull Failure
For our first demonstration, we create a Pod using an intentionally incorrect container image tag.
We create a Pod named:
bad-image
and attempt to run NGINX using an image tag that does not exist in the container registry.
We also specify that the Pod should not restart its container automatically.
For example:
kubectl run bad-image
--image=nginx:wrong-tag
--restart=Never
Because the image tag is invalid, Kubernetes will attempt to pull the image but will not be able to find it.
This creates a controlled failure scenario that allows us to examine how Kubernetes reports image pull errors.
Checking the Pod
Check the Pod status:
kubectl get pods
You may initially see a status such as:
NAME READY STATUS RESTARTS AGE
bad-image 0/1 ErrImagePull 0 10s
After Kubernetes has made additional attempts to retrieve the image, the status may change to:
NAME READY STATUS RESTARTS AGE
bad-image 0/1 ImagePullBackOff 0 30s
This immediately tells us that Kubernetes is having difficulty obtaining the required container image.
Describing the Pod
To investigate further, run:
kubectl describe pod bad-image
The output provides detailed information about the Pod and its container.
We can see that the container image is:
nginx:wrong-tag
This is the intentionally incorrect image tag.
Because this tag does not exist in the container registry, the container cannot be started.
The container state will normally show:
State: Waiting
Reason: ErrImagePull
or, after repeated attempts:
State: Waiting
Reason: ImagePullBackOff
This tells us that the container itself has not started because Kubernetes cannot obtain the required image.
Examining the Events
The Events section at the bottom of kubectl describe provides more information about what happened.
First, the scheduler successfully assigns the Pod to the worker node.
You may see an event similar to:
Scheduled
Kubernetes then attempts to pull:
nginx:wrong-tag
from the container registry.
The events may then report:
Pulling image "nginx:wrong-tag"
followed by a failure such as:
Failed to pull image "nginx:wrong-tag"
Eventually, you may see:
Back-off pulling image "nginx:wrong-tag"
The Pod status then becomes:
ImagePullBackOff
ImagePullBackOff means Kubernetes has failed to retrieve the image and is introducing a delay before making additional attempts.
The basic sequence is:
Pod created
|
v
Pod scheduled to worker
|
v
Kubelet attempts image pull
|
v
Image cannot be found
|
v
ErrImagePull
|
v
Further attempts fail
|
v
ImagePullBackOff
This confirms that the Pod failure occurred because of the invalid container image tag we intentionally specified.
Simulating a Container Crash
Next, we create another Pod to simulate a container that starts successfully but immediately crashes.
This time, we create a Pod named:
crash-loop
using the BusyBox image.
We specify that the container should execute a shell command that simply exits with status code 1.
For example:
kubectl run crash-loop
--image=busybox
--restart=Never
--command -- sh -c 'exit 1'
An exit code of:
1
indicates that the process terminated unsuccessfully.
Because the container immediately exits with an error, it terminates almost as soon as it starts.
This creates another controlled failure that allows us to observe how Kubernetes reports a container process that has failed.
Checking the Pod
Check its status:
kubectl get pods
You should eventually see something similar to:
NAME READY STATUS RESTARTS AGE
crash-loop 0/1 Error 0 10s
The important information is:
STATUS: Error
RESTARTS: 0
The container failed, and because we specified:
--restart=Never
it is not restarted.
Describing the Failed Pod
Now examine the Pod:
kubectl describe pod crash-loop
The output shows that the container uses the BusyBox image.
It also shows the command executed by the container:
sh -c exit 1
Because this command immediately returns an error, the container terminates shortly after it starts.
The container state should show information similar to:
State: Terminated
Reason: Error
Exit Code: 1
The exit code of 1 confirms that the container's main process terminated with an error.
You can also examine the:
Started
Finished
timestamps.
These should show that the container ran for only a very short period before terminating.
Examining the Events
At the bottom of the description, the Events section shows the lifecycle of the Pod.
The scheduler first assigns the Pod to a worker node:
Scheduled
The kubelet then obtains the BusyBox image:
Pulled
The container is created:
Created
and then started:
Started
The important difference from our previous example is that the container image was successfully obtained and the container successfully started.
The failure happened inside the container after startup when the process executed:
exit 1
The sequence is therefore:
Pod created
|
v
Pod scheduled
|
v
BusyBox image obtained
|
v
Container created
|
v
Container started
|
v
exit 1
|
v
Container terminated
|
v
Pod status: Error
This confirms that the Pod itself was created successfully, but its container failed because of the command we intentionally provided.
Comparing the Two Failure Scenarios
These two examples demonstrate two very different Kubernetes problems.
Invalid Image
With the bad-image Pod:
bad-image
|
v
Invalid image tag
|
v
Image cannot be pulled
|
v
Container never starts
|
v
ErrImagePull
|
v
ImagePullBackOff
The problem occurs before the container can start.
Container Crash
With the crash-loop Pod:
crash-loop
|
v
Valid BusyBox image
|
v
Container starts
|
v
exit 1
|
v
Container terminates
|
v
Error
In this case, Kubernetes can obtain the image and start the container successfully. The problem occurs after the container starts.
Troubleshooting the Failures
The first command to use is usually:
kubectl get pods
This provides a quick overview of Pod status.
Next, obtain detailed information:
kubectl describe pod <pod-name>
For example:
kubectl describe pod bad-image
or:
kubectl describe pod crash-loop
Pay particular attention to the container's:
State
Reason
Exit Code
Restart Count
and the Events section at the bottom.
You can also examine recent Kubernetes events:
kubectl get events --sort-by=.metadata.creationTimestamp
For containers that started successfully but subsequently failed, logs can provide additional information:
kubectl logs <pod-name>
In a real application, these logs may contain an error message explaining why the application terminated.
Conclusion
Intentionally creating controlled Pod failures is a useful way to develop Kubernetes troubleshooting skills.
An invalid image demonstrates what happens when Kubernetes cannot obtain the container image:
ErrImagePull
ImagePullBackOff
A container that starts and then exits unsuccessfully demonstrates a different type of failure:
Terminated
Reason: Error
Exit Code: 1
The important troubleshooting skill is learning to determine where the failure occurred.
Using:
kubectl get pods
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl get events --sort-by=.metadata.creationTimestamp
allows us to investigate the Pod from several different perspectives and determine whether the problem occurred while obtaining the image, starting the container, or running the application inside it.
By intentionally creating these failures in a safe test environment, we can learn to recognize common Kubernetes failure patterns and become more effective at diagnosing real problems.
Comments (0)
There are no comments yet. Be the first to comment.
Log in or register to leave a comment. all comments are subject to admin approval