peter (admin) -

Kubernetes Troubleshooting: Breaking and Fixing a Service
We will learn how to intentionally break a Kubernetes configuration and then fix it by troubleshooting the problem step by step.
The purpose of this exercise is to understand how to identify errors, analyze the configuration, and apply the correct fix to restore the application.
Let's get started.
Create an Intentionally Broken Service
For this, we are already logged into both the Master node and the Worker node.
Here we create a Service configuration using a YAML definition. A Service in Kubernetes is responsible for exposing Pods and allowing external users or other applications to access them through a stable network endpoint.
The purpose of this exercise is to intentionally create a misconfigured Service so that we can observe how troubleshooting works when an application becomes unreachable.
To send the configuration directly to Kubernetes without creating a separate file, we can use a here-document:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: broken-svc
spec:
type: NodePort
selector:
app: web-app
ports:
- port: 80
targetPort: 81
nodePort: 31607
EOF
At the beginning of the definition, we specify:
apiVersion: v1
kind: Service
The apiVersion specifies the API version for the core Kubernetes resource.
The kind: Service field tells Kubernetes that we want to create a networking component that routes traffic to a set of Pods.
Next, we assign a name to the Service:
metadata:
name: broken-svc
The name broken-svc becomes the identifier used to manage the Service inside the cluster.
Configure the Service Selector
Inside the Service specification, we define a selector:
selector:
app: web-app
This tells Kubernetes that the Service should route traffic to Pods with the following label:
app=web-app
Any Pod matching this selector can become an endpoint for the Service.
Configure an Incorrect Target Port
Next, we configure the Service ports:
ports:
- port: 80
targetPort: 81
nodePort: 31607
The Service listens on port 80 and forwards incoming traffic to the targetPort inside the selected Pods.
However, the targetPort has intentionally been configured as:
81
This does not match the port on which the NGINX container is listening.
By default, NGINX listens for HTTP traffic on port:
80
Therefore, the Service attempts to send traffic to port 81, where nothing is listening.
The purpose of this incorrect configuration is to deliberately make the application unreachable through the Service.
This simulates a troubleshooting scenario in which the Kubernetes resources exist and appear healthy, but users cannot access the application.
Verify the Broken Service
After creating the Service, retrieve its information using:
kubectl get service broken-svc -o wide
You should see that broken-svc has been created as a NodePort Service.
A NodePort Service exposes the application through a port on each Kubernetes node.
The output should show information such as:
NAME TYPE CLUSTER-IP PORT(S) SELECTOR
broken-svc NodePort 10.x.x.x 80:31607/TCP app=web-app
The Service uses NodePort:
31607
It also has the selector:
app=web-app
This confirms that the Service is configured to send traffic to Pods belonging to the web application.
Describe the Broken Service
We can retrieve more detailed information using:
kubectl describe service broken-svc
The output displays the complete configuration of the Service, including:
-
Service type
-
Cluster IP
-
Selector
-
Service port
-
Target port
-
NodePort
-
Endpoints
You should see that the Service port is 80, while the target port is:
TargetPort: 81/TCP
This means that traffic arriving at the Service is being forwarded to port 81 on the selected Pods.
However, the NGINX container is listening on port 80.
The incorrect targetPort therefore causes the Service to send traffic to the wrong port, preventing the application from responding correctly.
Fix the Service
Let's fix the problem by patching the Service configuration and correcting the port mapping.
We need to change the targetPort from 81 to 80.
Run:
kubectl patch service broken-svc -p '{"spec":{"ports":[{"port":80,"targetPort":80,"nodePort":31607}]}}'
This updates the Service so that the target port matches the port used by the NGINX container:
80
After pressing Enter, you should see a confirmation similar to:
service/broken-svc patched
This means that the configuration has been successfully updated.
The Service will now forward incoming traffic to the correct container port.
Verify the Corrected Service
Let's verify the updated Service configuration:
kubectl get service broken-svc -o wide
The NodePort Service should still be active with the same ClusterIP and NodePort.
It should also continue selecting Pods with:
app=web-app
To confirm that the targetPort has been corrected, run:
kubectl describe service broken-svc
The port configuration should now show:
Port: 80/TCP
TargetPort: 80/TCP
NodePort: 31607/TCP
This confirms that traffic arriving at the Service on port 80 is now correctly forwarded to port 80 on the selected Pods.
Verify the Application from the Worker Node
Let's move to the Worker node to verify that the Service is now working correctly.
Use curl with the Worker node's IP address and NodePort 31607:
curl -I http://<WORKER-NODE-IP>:31607
The request is sent to the NodePort Service, which then forwards the traffic to one of the selected application Pods.
After pressing Enter, you should receive a response similar to:
HTTP/1.1 200 OK
Server: nginx/1.26.x
The response:
HTTP/1.1 200 OK
confirms that the HTTP request successfully reached the application.
The NGINX server information confirms that the request reached the NGINX container.
Kubernetes Troubleshooting Process
This exercise demonstrates a simple Kubernetes troubleshooting workflow.
We started with a working application but intentionally created a Service with an incorrect targetPort:
NodePort :31607
↓
Service Port :80
↓
TargetPort :81
↓
NGINX listens on :80
↓
Connection Fails
We then inspected the Service configuration and identified that the Service was forwarding traffic to the wrong container port.
After changing the targetPort to 80, the traffic flow became:
NodePort :31607
↓
Service Port :80
↓
TargetPort :80
↓
NGINX :80
↓
HTTP 200 OK
The application became accessible again.
This demonstrates an important Kubernetes troubleshooting principle: when an application is running but cannot be reached through a Service, check the Service selector, endpoints, ports, targetPort, and the actual port on which the application is listening.
By systematically checking each layer, we can identify where communication is failing and correct the configuration.
Comments (0)
There are no comments yet. Be the first to comment.
Log in or register to leave a comment. all comments are subject to admin approval