- Published on
How to expose custom ports on Istio ingress gateway
- Author
- Written by Peter Jausovec
- Name Peter Jausovec
- @pjausovec
SHARE THIS ARTICLE
Contents
An interesting question popped up on one of my YouTube videos where I am explaining how to bring external traffic to the cluster using Istio and its ingress gateway.
The question (paraphrased) goes like this:
Note
I'd like to expose two ports on the ingress gateway (e.g. 5000 and 3000) and use any host (e.g. "*"). Can I do this?
Practically, here's what we want to happen. When a request is sent to
GATEWAY_IP:3000 the response comes from one service within the cluster and when the request is sent to GATEWAY_IP:5000 the response will come from another service. The host stays the same (* in this case, but it could be any other hostname). The difference is in the port numbers.Is it possible to do this?
The answer is YES and here are the high-level steps:
- Install Istio and expose additional ports through the ingress gateway service.
- Configure the Gateway resource to tell the Envoy proxy to listen to those ports.
- Create the VirtualService resource to route traffic to the services.
- Deploy the sample workload (
httpbin).
Let's get started.
Install Istio & expose additional ports
We need to modify how the Istio ingress gateway gets installed to expose the additional ports. By default, the ingress gateway exposes ports
80, 443, and a couple of other ports (15021 for health checks, 15012 for xDS, etc.).Another option is to create a separate ingress gateway that only exposes the additional ports.
Here's an example IstioOperator resource that exposes the two additional ports called
http-custom-1 and http-custom-2:apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-with-extra-ports
spec:
profile: default
components:
ingressGateways:
- namespace: istio-system
name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- port: 15021
targetPort: 15021
name: status-port
protocol: TCP
- port: 80
targetPort: 8080
name: http2
protocol: TCP
- port: 443
targetPort: 8443
name: https
protocol: TCP
- port: 15012
targetPort: 15012
name: tcp-istiod
protocol: TCP
- port: 15443
targetPort: 15443
name: tls
protocol: TCP
- port: 3000
targetPort: 3000
name: http-custom-1
protocol: TCP
- port: 5000
targetPort: 5000
name: http-custom-2
protocol: TCP
Once you deploy this (e.g.
istioctl install -f istio-with-extra-ports.yaml) and list the services in the istio-system namespace, you'll notice the extra two ports listed:$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.96.8.13 [GATEWAY_IP] 15021:32415/TCP,80:31070/TCP,443:31578/TCP,15012:31223/TCP,15443:30192/TCP,3000:32333/TCP,5000:31199/TCP 10m
istiod ClusterIP 10.96.4.186 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP
This allows us to send requests to
GATEWAY_IP:3000 and GATEWAY_IP:5000.Configure the Gateway resource to listen on those ports
We can now create the Gateway resource that tells the ingress gateway (Envoy) to listen on port 3000 and port 5000.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 3000
name: http
protocol: HTTP
hosts:
- '*'
- port:
number: 5000
name: http-second
protocol: HTTP
hosts:
- '*'
Behind the scenes, this creates two listeners on the ingress gateway -
http.3000 and http.5000:$ istioctl proxy-config listener istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
ADDRESS PORT MATCH DESTINATION
0.0.0.0 3000 ALL Route: http.3000
0.0.0.0 5000 ALL Route: http.5000
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
In the next step, we need to configure the VirtualService resource. If we check the configured routes on the ingress gateway, you'll notice that the 404 in the VIRTUAL SERVICE column:
$ istioctl proxy-config routes istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
NAME DOMAINS MATCH VIRTUAL SERVICE
http.5000 * /* 404
http.3000 * /* 404
* /stats/prometheus*
* /healthz/ready*
Create a VirtualService to define routes
We need the VirtualService resource to tell Envoys where to route the traffic. As a best practice, you'd probably create two separate VirtualService resources, but for brevity, I'll create only one.
In the end, it doesn't matter as Istio merges the resources, resulting in the same Envoy configuration.
Here's what the YAML could look like:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-virtualservice
spec:
hosts:
- '*'
gateways:
- my-gateway
http:
- match:
- port: 3000
route:
- destination:
host: httpbin-one.default.svc.cluster.local
port:
number: 3000
- match:
- port: 5000
route:
- destination:
host: httpbin-two.default.svc.cluster.local
port:
number: 5000
We're specifying the host as
*, which will match the host specified in the gateway. Next, we're attaching the my-gateway gateway to the VirtualService. When traffic hits the ingress gateway from any host, it will match to this VirtualService (because of the * in the hosts field on both resources). In a more typical scenario, you'd configure one or more hosts in the Gateway resource (e.g., mydomain.com) and then have hostname defined in the VirtualService where you configure the destination you want to expose to the world.These two settings are insufficient as we need a way to differentiate or tell Istio where the traffic should go when it's received on port 3000 vs. port 5000. To do that, the
match statement supports the port value and allows us to match on the port. We have the two match sections, and inside those, we have the route section where we specify the actual destination: httpbin-one for traffic on port 3000 and httpbin-two for traffic on port 5000.To complete the example, we need two workloads - we'll use two deployments of
httpbin:apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin-one
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-one
labels:
app: httpbin-one
service: httpbin-one
spec:
ports:
- name: http
port: 3000
targetPort: 80
selector:
app: httpbin-one
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-one
spec:
replicas: 1
selector:
matchLabels:
app: httpbin-one
version: v1
template:
metadata:
labels:
app: httpbin-one
version: v1
spec:
serviceAccountName: httpbin-one
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin-one
ports:
- containerPort: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin-two
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-two
labels:
app: httpbin-two
service: httpbin-two
spec:
ports:
- name: http
port: 5000
targetPort: 80
selector:
app: httpbin-two
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-two
spec:
replicas: 1
selector:
matchLabels:
app: httpbin-two
version: v1
template:
metadata:
labels:
app: httpbin-two
version: v1
spec:
serviceAccountName: httpbin-two
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin-two
ports:
- containerPort: 80
For simplicity sake, both deployments (and other resources) are in the same namespace (
default):$ kubectl get po
NAME READY STATUS RESTARTS AGE
httpbin-one-5bb7696978-4jmpz 2/2 Running 0 14m
httpbin-two-56947cd774-j7hth 2/2 Running 0 14m
We can use the
proxy-config command in the Istio CLI to see what the Envoy configuration looks like. Starting with the listeners (this should be the same as before):$ istioctl proxy-config listener istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
ADDRESS PORT MATCH DESTINATION
0.0.0.0 3000 ALL Route: http.3000
0.0.0.0 5000 ALL Route: http.5000
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
The listeners on ports 3000 and 5000 correspond to the two servers we defined in the Gateway resource. You'll also notice two different routes are in the configuration -
http.3000 and http.5000.We can also see the routes using the
routes sub-command:$ istioctl proxy-config routes istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
NAME DOMAINS MATCH VIRTUAL SERVICE
http.3000 * /* my-virtualservice.default
http.5000 * /* my-virtualservice.default
* /stats/prometheus*
* /healthz/ready*
This time the two routes map to the VirtualService we deployed. Also, note the domains (
*) - this corresponds to the hosts setting in the Gateway resource.By the way, the name of the VirtualService used to generate the Envoy configuration gets stored in the filter metadata. Here's a snippet from the configuration:
...
"metadata": {
"filter_metadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/my-virtualservice"
}
}
}
...
This is how Istio CLI knows which VirtualService to put in the above output.
If we list the clusters and filter by port number, we'll see which Kubernetes services the two ports are mapped to:
$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 3000
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
httpbin-one.default.svc.cluster.local 3000 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 3000 - outbound EDS
$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 5000
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
httpbin-two.default.svc.cluster.local 5000 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 5000 - outbound EDS
Finally, we can send the requests to GATEWAY_IP:3000 and GATEWAY_IP:5000 to verify we get back the response from the correct services:
$ curl GATEWAY_IP:3000/headers
{
"headers": {
"Accept": "*/*",
"Host": "GATEWAY_IP:3000",
"User-Agent": "curl/7.74.0",
"X-B3-Parentspanid": "5af6344416c73bcc",
"X-B3-Sampled": "1",
"X-B3-Spanid": "fc079d9bc28baee1",
"X-B3-Traceid": "954cffe5dda361e85af6344416c73bcc",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Internal": "true",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-one;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
$ curl GATEWAY_IP:5000/headers
{
"headers": {
"Accept": "*/*",
"Host": "GATEWAY_IP:5000",
"User-Agent": "curl/7.74.0",
"X-B3-Parentspanid": "d9469dcbeed22f99",
"X-B3-Sampled": "1",
"X-B3-Spanid": "0f2d699ec78b914d",
"X-B3-Traceid": "c58aa2e193746bd1d9469dcbeed22f99",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Internal": "true",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-two;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
Notice the SPIFFE URI in the
X-Forwarded-Client-Cert is different in each request. One uses the httpbin-one service account, and the second one uses httpbin-two service account, proving the requests came from correct services.Conclusion
This article explained how to expose custom ports on the Istio ingress gateway Kubernetes service. Additionally, we crafted a VirtualService configuration that matches the incoming ports and routes the traffic to appropriate backend services.
