kubernetes - Ingress - default backend - 404 on TLS

Question

I'm using Rancher 2.4.7 to deploy applications on customer's environments. I successfully created a new cluster for a production environment however whenever I create an Ingress resource with a specific hostname (the certificate is valid) : I get the default backend - 404 message when accessing my applications.

If I create an Ingress resource using the embed xip.io hostname, I can access my applications like it's supposed to.

I tried creating my resource using kubectl and the Rancher UI.

When getting the default backend - 404 error, no logs are registered on my deployment.

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        workload.user.cattle.io/workloadselector: deployment-default-nginx
    spec:
      containers:
      - image: nginx:latest
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          name: nginx
          protocol: TCP
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities: {}
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: false
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Service :

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: nginx
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    workload.user.cattle.io/workloadselector: deployment-default-nginx
  sessionAffinity: None
  type: ClusterIP

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  namespace: default
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
        path: /
  tls:
  - hosts:
    - mydomain.com
    secretName: mycertificate

Any thoughts what is wrong?

EDIT:

I'm using nginx-ingress shipped with Rancher. I'm not using any Cloud Provider. I installed a RKE Cluster using the Rancher UI.

Ingress controller logs:

I0111 16:13:02.868872       6 flags.go:204] Watching for Ingress class: nginx 
W0111 16:13:02.869187       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false) 
W0111 16:13:02.869225       6 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work. 
I0111 16:13:02.869357       6 main.go:220] Creating API client for https://10.43.0.1:443 
------------------------------------------------------------------------------- 
NGINX Ingress controller 
  Release:       0.32.0 
  Build:         git-29dd9760f 
  Repository:    https://github.com/rancher/ingress-nginx.git 
  nginx version: nginx/1.17.10 
 
------------------------------------------------------------------------------- 
 
I0111 16:13:02.879836       6 main.go:264] Running in Kubernetes cluster version v1.18 (v1.18.6) - git (clean) commit dff82dc0de47299ab66c83c626e08b245ab19037 - platform linux/amd64 
I0111 16:13:02.884023       6 main.go:94] Validated ingress-nginx/default-http-backend as the default backend. 
I0111 16:13:03.022518       6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem 
I0111 16:13:03.023350       6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18 
E0111 16:13:03.024271       6 main.go:122] Unexpected error searching IngressClass: ingressclasses.networking.k8s.io "nginx" is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot get resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope 
W0111 16:13:03.024285       6 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used. 
I0111 16:13:03.060623       6 nginx.go:263] Starting NGINX Ingress controller 
I0111 16:13:03.077607       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"eb5e7318-e99c-428f-82a0-4321925fdc4e", APIVersion:"v1", ResourceVersion:"610", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services 
I0111 16:13:03.077660       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"91590425-b422-4c4b-829d-a0be04973a9d", APIVersion:"v1", ResourceVersion:"611", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services 
I0111 16:13:03.077677       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"20b82b4a-e536-4a63-9fd2-20e17489b1ab", APIVersion:"v1", ResourceVersion:"607", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration 
I0111 16:13:04.168739       6 backend_ssl.go:66] Adding Secret "default/mycertificate" to the local store 
I0111 16:13:04.261564       6 nginx.go:307] Starting NGINX process 
I0111 16:13:04.261672       6 leaderelection.go:242] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx... 
I0111 16:13:04.262436       6 controller.go:139] Configuration changes detected, backend reload required. 
I0111 16:13:04.363051       6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx 
I0111 16:13:04.363157       6 status.go:86] new leader elected: nginx-ingress-controller-5dwhm 
I0111 16:13:04.385580       6 controller.go:155] Backend successfully reloaded. 
I0111 16:13:04.385608       6 controller.go:164] Initial sync, sleeping for 1 second. 
E0111 16:14:14.950751       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:161: Failed to watch *v1.Pod: Get "https://10.43.0.1:443/api/v1/namespaces/ingress-nginx/pods?allowWatchBookmarks=true&labelSelector=app%3Dingress-nginx%2Ccontroller-revision-hash%3D55dd558cd5%2Cpod-template-generation%3D1&resourceVersion=1028734&timeout=9m32s&timeoutSeconds=572&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
E0111 16:14:14.950798       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:159: Failed to watch *v1.Service: Get "https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=1028611&timeout=5m1s&timeoutSeconds=301&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
E0111 16:14:14.950816       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:157: Failed to watch *v1.Secret: Get "https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=1028785&timeout=9m23s&timeoutSeconds=563&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
E0111 16:14:14.950847       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:182: Failed to watch *v1beta1.Ingress: Get "https://10.43.0.1:443/apis/networking.k8s.io/v1beta1/ingresses?allowWatchBookmarks=true&resourceVersion=1028611&timeout=8m58s&timeoutSeconds=538&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
E0111 16:14:14.951438       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:160: Failed to watch *v1.ConfigMap: Get "https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=1028928&timeout=9m58s&timeoutSeconds=598&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
E0111 16:14:14.951936       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:158: Failed to watch *v1.Endpoints: Get "https://10.43.0.1:443/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=1028926&timeout=9m48s&timeoutSeconds=588&watch=true": dial tcp 10.43.0.1:443: connect: connection refused 
I0112 09:02:55.888418       6 controller.go:139] Configuration changes detected, backend reload required. 
I0112 09:02:55.969141       6 controller.go:155] Backend successfully reloaded. 
I0112 09:22:55.591923       6 controller.go:139] Configuration changes detected, backend reload required. 
I0112 09:22:55.591920       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173601", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/nginx 
I0112 09:22:55.676508       6 controller.go:155] Backend successfully reloaded. 
I0112 09:23:04.372633       6 status.go:275] updating Ingress default/nginx status from [] to [{10.107.129.176 }] 
I0112 09:23:04.380700       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173626", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx 
I0112 09:23:04.388779       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173628", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx 
I0112 09:23:04.402679       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173629", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx 
I0112 09:23:07.714377       6 controller.go:139] Configuration changes detected, backend reload required. 
I0112 09:23:07.791810       6 controller.go:155] Backend successfully reloaded. 
10.117.131.141 - - [12/Jan/2021:09:23:11 +0000] "GET / HTTP/1.1" 200 414 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36" 504 0.006 [default-nginx-80] [] 10.42.0.168:80 612 0.004 200 0bb06b53bbc47c83854cf8e19fde2e4a 
I0112 09:24:06.033945       6 controller.go:139] Configuration changes detected, backend reload required. 
I0112 09:24:06.114217       6 controller.go:155] Backend successfully reloaded. 
E0112 09:35:32.853089       6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:161: Failed to watch *v1.Pod: Get "https://10.43.0.1:443/api/v1/namespaces/ingress-nginx/pods?allowWatchBookmarks=true&labelSelector=app%3Dingress-nginx%2Ccontroller-revision-hash%3D55dd558cd5%2Cpod-template-generation%3D1&resourceV

Answer 1

I am not sure if I am able to understand since it would require some RnD. Meanwhile, you can checkout the ingress config that I used on my project :

Note: What version of Ingress I used and also the annotations for it

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    nginx.ingress.kubernetes.io/default-backend: ingress-nginx-controller
    ## tells ingress to check for regex in the config file
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header Access-Control-Allow-Methods "POST, GET, OPTIONS";
      add_header Access-Control-Allow-Credentials true;
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"  ## IMPORTANT

spec:
  ## TLS config
  tls:
    - hosts:
        - www.mydomain.com
        - mydomain.com
      secretName: cultor-test-tls
  ## our custom routing rules
  rules:
    - host: www.mydomain.com
      http:
        paths:
          - path: /api/upload/?(.*)
            backend:
              serviceName: aws-srv
              servicePort: 3000
          - path: /api/project/?(.*)
            backend:
              serviceName: project-srv
              servicePort: 3000
          - path: /api/profile/?(.*)
            backend:
              serviceName: profile-srv
              servicePort: 3000
          - path: /api/users/?(.*)
            backend:
              serviceName: auth-srv
              servicePort: 3000
          - path: /?(.*)
            backend:
              serviceName: client-srv
              servicePort: 3000

Important

Make sure you've tried accessing the domain with and without WWW and add this in your config as shown above

    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"