How to expose custom ports on Istio ingress gateway
This article explains how to expose custom ports on the Istio ingress and how can you use the same host name, but different port, and route the traffic to two (or more) Kubernetes services.
Note
I'd like to expose two ports on the ingress gateway (e.g. 5000 and 3000) and use any host (e.g. "*"). Can I do this?
GATEWAY_IP:3000
the response comes from one service within the cluster and when the request is sent to GATEWAY_IP:5000
the response will come from another service. The host stays the same (* in this case, but it could be any other hostname). The difference is in the port numbers.- Install Istio and expose additional ports through the ingress gateway service.
- Configure the Gateway resource to tell the Envoy proxy to listen to those ports.
- Create the VirtualService resource to route traffic to the services.
- Deploy the sample workload (
httpbin
).
Install Istio & expose additional ports
80
, 443
, and a couple of other ports (15021
for health checks, 15012
for xDS, etc.).http-custom-1
and http-custom-2
:apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-with-extra-ports
spec:
profile: default
components:
ingressGateways:
- namespace: istio-system
name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- port: 15021
targetPort: 15021
name: status-port
protocol: TCP
- port: 80
targetPort: 8080
name: http2
protocol: TCP
- port: 443
targetPort: 8443
name: https
protocol: TCP
- port: 15012
targetPort: 15012
name: tcp-istiod
protocol: TCP
- port: 15443
targetPort: 15443
name: tls
protocol: TCP
- port: 3000
targetPort: 3000
name: http-custom-1
protocol: TCP
- port: 5000
targetPort: 5000
name: http-custom-2
protocol: TCP
istioctl install -f istio-with-extra-ports.yaml
) and list the services in the istio-system
namespace, you'll notice the extra two ports listed:$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.96.8.13 [GATEWAY_IP] 15021:32415/TCP,80:31070/TCP,443:31578/TCP,15012:31223/TCP,15443:30192/TCP,3000:32333/TCP,5000:31199/TCP 10m
istiod ClusterIP 10.96.4.186 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP
GATEWAY_IP:3000
and GATEWAY_IP:5000
.Configure the Gateway resource to listen on those ports
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 3000
name: http
protocol: HTTP
hosts:
- '*'
- port:
number: 5000
name: http-second
protocol: HTTP
hosts:
- '*'
http.3000
and http.5000
:$ istioctl proxy-config listener istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
ADDRESS PORT MATCH DESTINATION
0.0.0.0 3000 ALL Route: http.3000
0.0.0.0 5000 ALL Route: http.5000
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
$ istioctl proxy-config routes istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system
NAME DOMAINS MATCH VIRTUAL SERVICE
http.5000 * /* 404
http.3000 * /* 404
* /stats/prometheus*
* /healthz/ready*
Create a VirtualService to define routes
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-virtualservice
spec:
hosts:
- '*'
gateways:
- my-gateway
http:
- match:
- port: 3000
route:
- destination:
host: httpbin-one.default.svc.cluster.local
port:
number: 3000
- match:
- port: 5000
route:
- destination:
host: httpbin-two.default.svc.cluster.local
port:
number: 5000
*
, which will match the host specified in the gateway. Next, we're attaching the my-gateway
gateway to the VirtualService. When traffic hits the ingress gateway from any host, it will match to this VirtualService (because of the *
in the hosts field on both resources). In a more typical scenario, you'd configure one or more hosts in the Gateway resource (e.g., mydomain.com
) and then have hostname defined in the VirtualService where you configure the destination you want to expose to the world.match
statement supports the port
value and allows us to match on the port. We have the two match sections, and inside those, we have the route
section where we specify the actual destination: httpbin-one
for traffic on port 3000 and httpbin-two
for traffic on port 5000.httpbin
:apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin-one
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-one
labels:
app: httpbin-one
service: httpbin-one
spec:
ports:
- name: http
port: 3000
targetPort: 80
selector:
app: httpbin-one
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-one
spec:
replicas: 1
selector:
matchLabels:
app: httpbin-one
version: v1
template:
metadata:
labels:
app: httpbin-one
version: v1
spec:
serviceAccountName: httpbin-one
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin-one
ports:
- containerPort: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin-two
---
apiVersion: v1
kind: Service
metadata:
name: httpbin-two
labels:
app: httpbin-two
service: httpbin-two
spec:
ports:
- name: http
port: 5000
targetPort: 80
selector:
app: httpbin-two
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin-two
spec:
replicas: 1
selector:
matchLabels:
app: httpbin-two
version: v1
template:
metadata:
labels:
app: httpbin-two
version: v1
spec:
serviceAccountName: httpbin-two
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin-two
ports:
- containerPort: 80
default
):$ kubectl get po
NAME READY STATUS RESTARTS AGE
httpbin-one-5bb7696978-4jmpz 2/2 Running 0 14m
httpbin-two-56947cd774-j7hth 2/2 Running 0 14m
proxy-config
command in the Istio CLI to see what the Envoy configuration looks like. Starting with the listeners (this should be the same as before):$ istioctl proxy-config listener istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
ADDRESS PORT MATCH DESTINATION
0.0.0.0 3000 ALL Route: http.3000
0.0.0.0 5000 ALL Route: http.5000
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
http.3000
and http.5000
.routes
sub-command:$ istioctl proxy-config routes istio-ingressgateway-6668f9548d-mrtp5 -n istio-system
NAME DOMAINS MATCH VIRTUAL SERVICE
http.3000 * /* my-virtualservice.default
http.5000 * /* my-virtualservice.default
* /stats/prometheus*
* /healthz/ready*
*
) - this corresponds to the hosts
setting in the Gateway resource....
"metadata": {
"filter_metadata": {
"istio": {
"config": "/apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/my-virtualservice"
}
}
}
...
$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 3000
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
httpbin-one.default.svc.cluster.local 3000 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 3000 - outbound EDS
$ istioctl proxy-config clusters istio-ingressgateway-9f6bc6bd7-szd5k -n istio-system --port 5000
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
httpbin-two.default.svc.cluster.local 5000 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 5000 - outbound EDS
$ curl GATEWAY_IP:3000/headers
{
"headers": {
"Accept": "*/*",
"Host": "GATEWAY_IP:3000",
"User-Agent": "curl/7.74.0",
"X-B3-Parentspanid": "5af6344416c73bcc",
"X-B3-Sampled": "1",
"X-B3-Spanid": "fc079d9bc28baee1",
"X-B3-Traceid": "954cffe5dda361e85af6344416c73bcc",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Internal": "true",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-one;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
$ curl GATEWAY_IP:5000/headers
{
"headers": {
"Accept": "*/*",
"Host": "GATEWAY_IP:5000",
"User-Agent": "curl/7.74.0",
"X-B3-Parentspanid": "d9469dcbeed22f99",
"X-B3-Sampled": "1",
"X-B3-Spanid": "0f2d699ec78b914d",
"X-B3-Traceid": "c58aa2e193746bd1d9469dcbeed22f99",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Internal": "true",
"X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin-two;Hash=b957e7eef3880984bbf97b7443f5f4afdced8d1082dc56bd21b25ae2949b6a88;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
X-Forwarded-Client-Cert
is different in each request. One uses the httpbin-one
service account, and the second one uses httpbin-two
service account, proving the requests came from correct services.