peter (admin)   -  



 

Kubernetes Troubleshooting: Breaking and Fixing a Service

We will learn how to intentionally break a Kubernetes configuration and then fix it by troubleshooting the problem step by step.

The purpose of this exercise is to understand how to identify errors, analyze the configuration, and apply the correct fix to restore the application.

Let's get started.

Create an Intentionally Broken Service

For this, we are already logged into both the Master node and the Worker node.

Here we create a Service configuration using a YAML definition. A Service in Kubernetes is responsible for exposing Pods and allowing external users or other applications to access them through a stable network endpoint.

The purpose of this exercise is to intentionally create a misconfigured Service so that we can observe how troubleshooting works when an application becomes unreachable.

To send the configuration directly to Kubernetes without creating a separate file, we can use a here-document:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: broken-svc
spec:
  type: NodePort
  selector:
    app: web-app
  ports:
    - port: 80
      targetPort: 81
      nodePort: 31607
EOF

At the beginning of the definition, we specify:

apiVersion: v1
kind: Service

The apiVersion specifies the API version for the core Kubernetes resource.

The kind: Service field tells Kubernetes that we want to create a networking component that routes traffic to a set of Pods.

Next, we assign a name to the Service:

metadata:
  name: broken-svc

The name broken-svc becomes the identifier used to manage the Service inside the cluster.

Configure the Service Selector

Inside the Service specification, we define a selector:

selector:
  app: web-app

This tells Kubernetes that the Service should route traffic to Pods with the following label:

app=web-app

Any Pod matching this selector can become an endpoint for the Service.

Configure an Incorrect Target Port

Next, we configure the Service ports:

ports:
  - port: 80
    targetPort: 81
    nodePort: 31607

The Service listens on port 80 and forwards incoming traffic to the targetPort inside the selected Pods.

However, the targetPort has intentionally been configured as:

81

This does not match the port on which the NGINX container is listening.

By default, NGINX listens for HTTP traffic on port:

80

Therefore, the Service attempts to send traffic to port 81, where nothing is listening.

The purpose of this incorrect configuration is to deliberately make the application unreachable through the Service.

This simulates a troubleshooting scenario in which the Kubernetes resources exist and appear healthy, but users cannot access the application.

Verify the Broken Service

After creating the Service, retrieve its information using:

kubectl get service broken-svc -o wide

You should see that broken-svc has been created as a NodePort Service.

A NodePort Service exposes the application through a port on each Kubernetes node.

The output should show information such as:

NAME         TYPE       CLUSTER-IP      PORT(S)        SELECTOR
broken-svc   NodePort   10.x.x.x        80:31607/TCP   app=web-app

The Service uses NodePort:

31607

It also has the selector:

app=web-app

This confirms that the Service is configured to send traffic to Pods belonging to the web application.

Describe the Broken Service

We can retrieve more detailed information using:

kubectl describe service broken-svc

The output displays the complete configuration of the Service, including:

  • Service type

  • Cluster IP

  • Selector

  • Service port

  • Target port

  • NodePort

  • Endpoints

You should see that the Service port is 80, while the target port is:

TargetPort: 81/TCP

This means that traffic arriving at the Service is being forwarded to port 81 on the selected Pods.

However, the NGINX container is listening on port 80.

The incorrect targetPort therefore causes the Service to send traffic to the wrong port, preventing the application from responding correctly.

Fix the Service

Let's fix the problem by patching the Service configuration and correcting the port mapping.

We need to change the targetPort from 81 to 80.

Run:

kubectl patch service broken-svc -p '{"spec":{"ports":[{"port":80,"targetPort":80,"nodePort":31607}]}}'

This updates the Service so that the target port matches the port used by the NGINX container:

80

After pressing Enter, you should see a confirmation similar to:

service/broken-svc patched

This means that the configuration has been successfully updated.

The Service will now forward incoming traffic to the correct container port.

Verify the Corrected Service

Let's verify the updated Service configuration:

kubectl get service broken-svc -o wide

The NodePort Service should still be active with the same ClusterIP and NodePort.

It should also continue selecting Pods with:

app=web-app

To confirm that the targetPort has been corrected, run:

kubectl describe service broken-svc

The port configuration should now show:

Port:        80/TCP
TargetPort:  80/TCP
NodePort:    31607/TCP

This confirms that traffic arriving at the Service on port 80 is now correctly forwarded to port 80 on the selected Pods.

Verify the Application from the Worker Node

Let's move to the Worker node to verify that the Service is now working correctly.

Use curl with the Worker node's IP address and NodePort 31607:

curl -I http://<WORKER-NODE-IP>:31607

The request is sent to the NodePort Service, which then forwards the traffic to one of the selected application Pods.

After pressing Enter, you should receive a response similar to:

HTTP/1.1 200 OK
Server: nginx/1.26.x

The response:

HTTP/1.1 200 OK

confirms that the HTTP request successfully reached the application.

The NGINX server information confirms that the request reached the NGINX container.

Kubernetes Troubleshooting Process

This exercise demonstrates a simple Kubernetes troubleshooting workflow.

We started with a working application but intentionally created a Service with an incorrect targetPort:

NodePort :31607
      ↓
Service Port :80
      ↓
TargetPort :81
      ↓
NGINX listens on :80
      ↓
Connection Fails

We then inspected the Service configuration and identified that the Service was forwarding traffic to the wrong container port.

After changing the targetPort to 80, the traffic flow became:

NodePort :31607
      ↓
Service Port :80
      ↓
TargetPort :80
      ↓
NGINX :80
      ↓
HTTP 200 OK

The application became accessible again.

This demonstrates an important Kubernetes troubleshooting principle: when an application is running but cannot be reached through a Service, check the Service selector, endpoints, ports, targetPort, and the actual port on which the application is listening.

By systematically checking each layer, we can identify where communication is failing and correct the configuration.

 


Comments (0)

There are no comments yet. Be the first to comment.

Log in or register to leave a comment. all comments are subject to admin approval