Connecting to a Kubernetes service resulted in connection refused

Question

I'm trying to deploy my web application using Kubernetes. I used Minikube to create the cluster and successfully exposed my frontend react app using ingress. Yet when I attached the backend service's URL in "env" field in the frontend's deployment.yaml, it does not work. When I tried to connect to the backend service through the frontend pod, the connection refused.

frontend deployment yaml

kind: Deployment
apiVersion: apps/v1
metadata:
 name: frontend
spec:
 replicas: 1
 selector:
   matchLabels:
     app: frontend
 template:
   metadata:
     labels:
       app: frontend
   spec:
     containers:
       - name: frontend
         image: <image_name>
         imagePullPolicy: Never
         ports:
           - containerPort: 80
         env:
         - name: REACT_APP_API_V1_ENDPOINT
           value: http://backend-svc
     restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
 name: frontend-svc
spec:
 ports:
 - port: 80
   protocol: TCP
   targetPort: 80
 selector:
   app: frontend

Ingress for frontend

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: front-ingress
 namespace: default
 annotations:
   kubernetes.io/ingress.class: "nginx"
   nginx.ingress.kubernetes.io/rewrite-target: /
   nginx.ingress.kubernetes.io/proxy-read-timeout: "12h"
   nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
 rules:
   - host: front-testk.info
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: frontend-svc
               port:
                 number: 80

Backend deployment yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: backend
  labels:
    name: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: <image_name>
        ports:
        - containerPort: 80
        imagePullPolicy: Never
      restartPolicy: Always
---
kind: Service
apiVersion: v1
metadata:
  name: backend-svc
  labels:
    app: backend
spec:
  selector:
    app: backend
  ports:
    - name: http
      port: 80
      targetPort: 80

% kubectl get svc backend-svc -o wide
NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE   SELECTOR
backend-svc   ClusterIP   10.109.107.145   <none>        80/TCP    21h   app=backend

Connected inside frontend pod and try to connect to the backend using the ENV created during deploy:

? kubectl exec frontend-75579c8499-x766s -it sh
/app # apk update && apk add curl
OK: 10 MiB in 20 packages

/app # env
REACT_APP_API_V1_ENDPOINT=http://backend-svc

/app # curl $REACT_APP_API_V1_ENDPOINT
curl: (7) Failed to connect to backend-svc port 80: Connection refused

/app # nslookup backend-svc
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   backend-svc.default.svc.cluster.local
Address: 10.109.107.145

** server can't find backend-svc.cluster.local: NXDOMAIN

Exec into my backend pod

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1/node

# netstat -lnturp
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG        0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0

Answer 1

As I suspected your application listens on port 8080. If you closely at your output from netstat you will notice that Local Address is 0.0.0.0:8080:

# netstat -tulpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0     ?? 0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1/node

In order to fix that you have to correct your targetPort in your service:

kind: Service
apiVersion: v1
metadata:
  name: backend-svc
  labels:
    app: backend
spec:
  selector:
    app: backend
  ports:
    - name: http
      port: 80
      targetPort: 8080 # ?? change this to 8080

There is no need to change the port on the deployment side since the containerPort is primarily informational. Not specifying a port there does not prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible.