How to expose custom ports on Istio ingress gateway

An interesting question popped up on one of my YouTube videos where I am explaining how to bring external traffic to the cluster using Istio and its ingress gateway.

The question (paraphrased) goes like this:

Note

I'd like to expose two ports on the ingress gateway (e.g. 5000 and 3000) and use any host (e.g. "*"). Can I do this?

Practically, here's what we want to happen. When a request is sent to GATEWAY_IP:3000 the response comes from one service within the cluster and when the request is sent to GATEWAY_IP:5000 the response will come from another service. The host stays the same (* in this case, but it could be any other hostname). The difference is in the port numbers.

Is it possible to do this?

The answer is YES and here are the high-level steps:

Install Istio and expose additional ports through the ingress gateway service.
Configure the Gateway resource to tell the Envoy proxy to listen to those ports.
Create the VirtualService resource to route traffic to the services.
Deploy the sample workload (httpbin).

Let's get started.

Install Istio & expose additional ports

We need to modify how the Istio ingress gateway gets installed to expose the additional ports. By default, the ingress gateway exposes ports 80, 443, and a couple of other ports (15021 for health checks, 15012 for xDS, etc.).

Another option is to create a separate ingress gateway that only exposes the additional ports.

Here's an example IstioOperator resource that exposes the two additional ports called http-custom-1 and http-custom-2:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-with-extra-ports
spec:
  profile: default
  components:
    ingressGateways:
      - namespace: istio-system
        name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            ports:
              - port: 15021
                targetPort: 15021
                name: status-port
                protocol: TCP
              - port: 80
                targetPort: 8080
                name: http2
                protocol: TCP
              - port: 443
                targetPort: 8443
                name: https
                protocol: TCP
              - port: 15012
                targetPort: 15012
                name: tcp-istiod
                protocol: TCP
              - port: 15443
                targetPort: 15443
                name: tls
                protocol: TCP
              - port: 3000
                targetPort: 3000
                name: http-custom-1
                protocol: TCP
              - port: 5000
                targetPort: 5000
                name: http-custom-2
                protocol: TCP

Once you deploy this (e.g. istioctl install -f istio-with-extra-ports.yaml) and list the services in the istio-system namespace, you'll notice the extra two ports listed:

$ kubectl get svc  -n istio-system
NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                                                                                                    AGE
istio-ingressgateway   LoadBalancer   10.96.8.13    [GATEWAY_IP]   15021:32415/TCP,80:31070/TCP,443:31578/TCP,15012:31223/TCP,15443:30192/TCP,3000:32333/TCP,5000:31199/TCP   10m
istiod                 ClusterIP      10.96.4.186   <none>         15010/TCP,15012/TCP,443/TCP,15014/TCP

This allows us to send requests to GATEWAY_IP:3000 and GATEWAY_IP:5000.

Configure the Gateway resource to listen on those ports

We can now create the Gateway resource that tells the ingress gateway (Envoy) to listen on port 3000 and port 5000.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 3000
        name: http
        protocol: HTTP
      hosts:
        - '*'
    - port:
        number: 5000
        name: http-second
        protocol: HTTP
      hosts:
        - '*'

Behind the scenes, this creates two listeners on the ingress gateway - http.3000 and http.5000:

$ istioctl proxy-config listener istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 3000  ALL   Route: http.3000
0.0.0.0 5000  ALL   Route: http.5000
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

In the next step, we need to configure the VirtualService resource. If we check the configured routes on the ingress gateway, you'll notice that the 404 in the VIRTUAL SERVICE column:

$ istioctl proxy-config routes  istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
NAME          DOMAINS     MATCH                  VIRTUAL SERVICE
http.5000     *           /*                     404
http.3000     *           /*                     404
              *           /stats/prometheus*
              *           /healthz/ready*

Create a VirtualService to define routes

We need the VirtualService resource to tell Envoys where to route the traffic. As a best practice, you'd probably create two separate VirtualService resources, but for brevity, I'll create only one.

In the end, it doesn't matter as Istio merges the resources, resulting in the same Envoy configuration.

Here's what the YAML could look like:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-virtualservice
spec:
  hosts:
    - '*'
  gateways:
    - my-gateway
  http:
    - match:
        - port: 3000
      route:
        - destination:
            host: httpbin-one.default.svc.cluster.local
            port:
              number: 3000
    - match:
        - port: 5000
      route:
        - destination:
            host: httpbin-two.default.svc.cluster.local
            port:
              number: 5000

We're specifying the host as *, which will match the host specified in the gateway. Next, we're attaching the my-gateway gateway to the VirtualService. When traffic hits the ingress gateway from any host, it will match to this VirtualService (because of the * in the hosts field on both resources). In a more typical scenario, you'd configure one or more hosts in the Gateway resource (e.g., mydomain.com) and then have hostname defined in the VirtualService where you configure the destination you want to expose to the world.

These two settings are insufficient as we need a way to differentiate or tell Istio where the traffic should go when it's received on port 3000 vs. port 5000. To do that, the match statement supports the port value and allows us to match on the port. We have the two match sections, and inside those, we have the route section where we specify the actual destination: httpbin-one for traffic on port 3000 and httpbin-two for traffic on port 5000.

To complete the example, we need two workloads - we'll use two deployments of httpbin:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin-one
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-one
  labels:
    app: httpbin-one
    service: httpbin-one
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 80
  selector:
    app: httpbin-one
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-one
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin-one
      version: v1
  template:
    metadata:
      labels:
        app: httpbin-one
        version: v1
    spec:
      serviceAccountName: httpbin-one
      containers:
        - image: docker.io/kennethreitz/httpbin
          imagePullPolicy: IfNotPresent
          name: httpbin-one
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin-two
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-two
  labels:
    app: httpbin-two
    service: httpbin-two
spec:
  ports:
    - name: http
      port: 5000
      targetPort: 80
  selector:
    app: httpbin-two
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-two
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin-two
      version: v1
  template:
    metadata:
      labels:
        app: httpbin-two
        version: v1
    spec:
      serviceAccountName: httpbin-two
      containers:
        - image: docker.io/kennethreitz/httpbin
          imagePullPolicy: IfNotPresent
          name: httpbin-two
          ports:
            - containerPort: 80

For simplicity sake, both deployments (and other resources) are in the same namespace (default):

$ kubectl get po
NAME                           READY   STATUS    RESTARTS   AGE
httpbin-one-5bb7696978-4jmpz   2/2     Running   0          14m
httpbin-two-56947cd774-j7hth   2/2     Running   0          14m

We can use the proxy-config command in the Istio CLI to see what the Envoy configuration looks like. Starting with the listeners (this should be the same as before):

$ istioctl proxy-config listener  istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 3000  ALL   Route: http.3000
0.0.0.0 5000  ALL   Route: http.5000
0.0.0.0 15021 ALL   Inline Route: /healthz/ready*
0.0.0.0 15090 ALL   Inline Route: /stats/prometheus*

The listeners on ports 3000 and 5000 correspond to the two servers we defined in the Gateway resource. You'll also notice two different routes are in the configuration - http.3000 and http.5000.

We can also see the routes using the routes sub-command:

$ istioctl proxy-config routes istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
NAME          DOMAINS     MATCH                  VIRTUAL SERVICE
http.3000     *           /*                     my-virtualservice.default
http.5000     *           /*                     my-virtualservice.default
              *           /stats/prometheus*
              *           /healthz/ready*

This time the two routes map to the VirtualService we deployed. Also, note the domains (*) - this corresponds to the hosts setting in the Gateway resource.

By the way, the name of the VirtualService used to generate the Envoy configuration gets stored in the filter metadata. Here's a snippet from the configuration:

...
"metadata": {
  "filter_metadata": {
    "istio": {
      "config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/my-virtualservice"
    }
  }
}
...

This is how Istio CLI knows which VirtualService to put in the above output.

If we list the clusters and filter by port number, we'll see which Kubernetes services the two ports are mapped to:

$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 3000
SERVICE FQDN                                            PORT     SUBSET     DIRECTION     TYPE     DESTINATION RULE
httpbin-one.default.svc.cluster.local                   3000     -          outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local     3000     -          outbound      EDS

$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 5000
SERVICE FQDN                                            PORT     SUBSET     DIRECTION     TYPE     DESTINATION RULE
httpbin-two.default.svc.cluster.local                   5000     -          outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local     5000     -          outbound      EDS

Finally, we can send the requests to GATEWAY_IP:3000 and GATEWAY_IP:5000 to verify we get back the response from the correct services:

$ curl GATEWAY_IP:3000/headers
{
  "headers": {
    "Accept": "*/*",
    "Host": "GATEWAY_IP:3000",
    "User-Agent": "curl/7.74.0",
    "X-B3-Parentspanid": "5af6344416c73bcc",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "fc079d9bc28baee1",
    "X-B3-Traceid": "954cffe5dda361e85af6344416c73bcc",
    "X-Envoy-Attempt-Count": "1",
    "X-Envoy-Internal": "true",
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-one;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}

$ curl GATEWAY_IP:5000/headers
{
  "headers": {
    "Accept": "*/*",
    "Host": "GATEWAY_IP:5000",
    "User-Agent": "curl/7.74.0",
    "X-B3-Parentspanid": "d9469dcbeed22f99",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "0f2d699ec78b914d",
    "X-B3-Traceid": "c58aa2e193746bd1d9469dcbeed22f99",
    "X-Envoy-Attempt-Count": "1",
    "X-Envoy-Internal": "true",
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-two;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}

Notice the SPIFFE URI in the X-Forwarded-Client-Cert is different in each request. One uses the httpbin-one service account, and the second one uses httpbin-two service account, proving the requests came from correct services.

Conclusion

This article explained how to expose custom ports on the Istio ingress gateway Kubernetes service. Additionally, we crafted a VirtualService configuration that matches the incoming ports and routes the traffic to appropriate backend services.