Deploy and Operate Multiple Istio Meshes in one Kubernetes Cluster
The rising demand for hard kubernetes multitenancy, either for customers of a SaaS offering or to support disparate internal teams within an organization, coupled with mass adoption of service-meshes (Istio being the more popular of the choices), we are starting to notice a need for supporting multiple meshes within a single Kubernetes cluster.
Introduction
istio-proxy
sidecars associate and communicate with their respective control planes.Note on Automatic Sidecar Injection
MutatingWebhookConfigurations
to inject the istio-proxy
sidecar container and the istio-init
init-container (unless you are using the CNI option) into the selected workload pods. By default, istio requires istio-injection: enabled
label to be applied to namespaces (when selecting all pods in a namespace) or to specific deployments
or individual pods
to enable automatic sidecar injection.MutatingWebhookConfiguration
to watch for (non-default) unique/combo labels.Instructions
Modifying the manifests
Cluster Scoped Resource
ClusterRole
, ClusterRoleBindings
and MutatingWebhookConfiguration
& ValidatingWebhookConfiguration
resources unique names.ClusterRoles & ClusterRoleBindings
ClusterRole
and ClusterRoleBinding
is entirely optional since they can be re-used across namespaces if needed. The changes are required if other vendors modify these resource definitions to apply more stringent RBAC policies.ClusterRole and ClusterRoleBindings
--- cr_crb_clean.yaml 2020-03-13 12:25:02.000000000 -0400
+++ cr_crb_affirmed.yaml 2020-03-13 12:23:55.000000000 -0400
@@ -56,7 +56,7 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
- name: kiali
+ name: kiali-affirmed
labels:
app: kiali
chart: kiali
@@ -125,7 +125,7 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
- name: kiali-viewer
+ name: kiali-viewer-affirmed
labels:
app: kiali
chart: kiali
@@ -333,7 +333,7 @@
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
- name: istio-reader
+ name: istio-reader-affirmed
rules:
- apiGroups: ['']
resources: ['nodes', 'pods', 'services', 'endpoints', "replicationcontrollers"]
@@ -377,7 +377,7 @@
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
- name: kiali
+ name: kiali-affirmed
subjects:
- kind: ServiceAccount
name: kiali-service-account
@@ -490,13 +490,13 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
- name: istio-multi
+ name: istio-multi-affirmed
labels:
chart: istio-1.5.0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
- name: istio-reader
+ name: istio-reader-affirmed
subjects:
- kind: ServiceAccount
name: istio-multi
@@ -505,14 +505,16 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
- name: istio-reader
+ name: istio-reader-affirmed
labels:
chart: istio-1.5.0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
- name: istio-reader
+ name: istio-reader-affirmed
subjects:
- kind: ServiceAccount
name: istio-reader
Mutating/Validation Webhook Configurations
Webhooks
(and their relatedConfigMap
)
--- webhook_clean.yaml 2020-03-13 12:37:56.000000000 -0400
+++ webhook_affirmed.yaml 2020-03-13 12:37:56.000000000 -0400
@@ -1,18 +1,17 @@
-
---
# Source: istio/charts/sidecarInjectorWebhook/templates/mutatingwebhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
- name: istio-sidecar-injector
+ name: istio-affirmed-sidecar-injector
labels:
app: sidecarInjectorWebhook
chart: sidecarInjectorWebhook
heritage: Tiller
release: istio
webhooks:
- - name: sidecar-injector.istio.io
+ - name: affirmed-sidecar-injector.istio.io
clientConfig:
service:
name: istio-sidecar-injector
@@ -27,7 +26,7 @@
failurePolicy: Fail
namespaceSelector:
matchLabels:
- istio-injection: enabled
+ istio-injection: enabled-affirmed
---
# Source: istio/charts/galley/templates/configmap.yaml
@@ -47,7 +46,7 @@
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
- name: istio-galley
+ name: istio-affirmed-galley
labels:
app: galley
chart: galley
@@ -55,7 +54,7 @@
release: istio
istio: galley
webhooks:
- - name: pilot.validation.istio.io
+ - name: affirmed-pilot.validation.istio.io
clientConfig:
service:
name: istio-galley
@@ -121,7 +120,7 @@
# endpoint is ready.
failurePolicy: Ignore
sideEffects: None
- - name: mixer.validation.istio.io
+ - name: affirmed-mixer.validation.istio.io
clientConfig:
service:
name: istio-galley
@@ -149,12 +148,13 @@
failurePolicy: Ignore
sideEffects: None
+---
# Source: istio/charts/galley/templates/validatingwebhookconfiguration.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
- name: istio-galley
+ name: istio-affirmed-galley
labels:
app: galley
chart: galley
@@ -162,7 +162,7 @@
release: istio
istio: galley
webhooks:
- - name: pilot.validation.istio.io
+ - name: affirmed-pilot.validation.istio.io
clientConfig:
service:
name: istio-galley
@@ -228,7 +228,7 @@
# endpoint is ready.
failurePolicy: Ignore
sideEffects: None
- - name: mixer.validation.istio.io
+ - name: affirmed-mixer.validation.istio.io
clientConfig:
service:
name: istio-galley
Loadbalancer/NodePort port numbers
Service
that requests a NodePort. Since two Service
s cannot use the same NodePort
and we will need to modify the default values for each of the NodePorts associated with the Gateway
deployments.---
# Source: istio/charts/gateways/templates/service.yaml
--- nodeport_svc_clean.yaml 2020-03-13 12:47:02.000000000 -0400
+++ nodeport_svc_affirmed.yaml 2020-03-13 12:47:03.000000000 -0400
@@ -4,20 +4,20 @@
apiVersion: v1
kind: Service
metadata:
- name: istio-ingressgateway
+ name: istio-affirmed-ingressgateway
namespace: istio-affirmed
annotations:
labels:
chart: gateways
heritage: Tiller
release: istio
- app: istio-ingressgateway
+ app: istio-affirmed-ingressgateway
istio: ingressgateway
spec:
type: LoadBalancer
selector:
release: istio
- app: istio-ingressgateway
+ app: istio-affirmed-ingressgateway
istio: ingressgateway
ports:
Deployments - Container command and arguments
Note
CAVEAT: A significant limitation that was uncovered during this effort was that we must statically specify the namespaces that we wish to include in our mesh. To achieve this I have added flag to the citadel container -- --listened-namespaces=istio-affirmed,istio-affirmed-workspace,kube-system
--- deployment_clean.yaml 2020-03-13 13:10:34.000000000 -0400
+++ deployment_affirmed.yaml 2020-03-13 13:10:28.000000000 -0400
# Source: istio/charts/galley/templates/deployment.yaml
@@ -53,6 +53,7 @@
- --enable-reconcileWebhookConfiguration=true
- --monitoringPort=15014
- --log_output_level=default:info
+ - --webhook-name=istio-affirmed-galley
volumeMounts:
- name: certs
mountPath: /etc/certs
# Source: istio/charts/security/templates/deployment.yaml
@@ -177,6 +178,7 @@
- --monitoring-port=15014
- --self-signed-ca=true
- --workload-cert-ttl=2160h
+ - --listened-namespaces=istio-affirmed,istio-affirmed-workspace,kube-system
env:
- name: CITADEL_ENABLE_NAMESPACES_BY_DEFAULT
value: "true"
# Source: istio/charts/sidecarInjectorWebhook/templates/deployment.yaml
@@ -265,6 +267,8 @@
- --healthCheckInterval=2s
- --healthCheckFile=/tmp/health
- --reconcileWebhookConfig=true
+ - --webhookName=affirmed-sidecar-injector.istio.io
+ - --webhookConfigName=istio-affirmed-sidecar-injector
volumeMounts:
- name: config-volume
mountPath: /etc/istio/config
Generate, deploy and verify
Requirements
Installation
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.5.0
Generate the manifests
- Add the istio helm repo
helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.5.0/charts/
- Istio [Clean] - istio-clean.yaml
helm template install/kubernetes/helm/istio --name istio --namespace istio-affirmed --values install/kubernetes/helm/istio/values-istio-demo.yaml > istio-clean.yaml
- Istio [Modified] - istio-affirmed.yaml
# Apply the patch file to the clean istio yaml and also backup the original
patch -b istio-clean.yaml istio.patch
# Rename istio-clean.yaml to the new modified version istio-affirmed.yaml
mv istio-clean.yaml istio-affirmed.yaml
# Rename the backup generated by the patch function to istio-clean.yaml
mv istio-clean.yaml.orig istio-clean.yaml
Kubectl apply the generated manifests to your Kubernetes Cluster
Deploy the control plane
- Create the mesh control plane Namespace
kubectl create namespace istio-affirmed
- Deploy all CRDs to the namespace
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-affirmed | kubectl apply -f -
- Apply manifest to the istio namespace
kubectl create -f istio-affirmed.yaml
Verify sidecar injection works with a sample NGINX deployment (See nginx-affirmed.yaml)
- Create a workload namespace and label it to enable sidecar injection
kubectl create namespace nginx-affirmed
kubectl label namespace nginx-affirmed istio-injection=enabled-affirmed
- Deploy NGINX application to labeled namespace
kubectl create -f nginx-affirmed.yaml
- Verify the deployment
kubectl get pods -n nginx-affirmed
NAME READY STATUS RESTARTS AGE
nginx-deployment-54f57cf6bf-h9n9t 2/2 Running 0 10s
- Ensure that the
istio-proxy
sidecar is up and running and is configured to use the right (istio-affirmed) control plane services.
kubectl describe pod nginx-deployment-54f57cf6bf-h9n9t
# file truncated
Name: nginx-deployment-54f57cf6bf-h9n9t
Namespace: nginx-affirmed
...
Init Containers:
istio-init:
...
Containers:
nginx:
...
istio-proxy:
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
...
# Verification checkpoint. MUST point to deployed istio control-plane
# This acknowledges that the mutatingwebhook did the right thing.
--discoveryAddress
istio-pilot.istio-affirmed:15010
...
--zipkinAddress
zipkin.istio-affirmed:9411
...
Caveats and Gotchas
Owned/managed namespaces must be specified in advance
ServiceAccount
Secrets
generated by the API server
as new ServiceAccount
are created. In return citadel's secretcontroller
generates new Secrets
of type istio.io/key-and-cert
with the cert-chain.pem
, key.pem
and root-cert.pem
. This Secret
is then used by the mutating webhook
to authenticate itself with the API server
. secretcontroller
leverages spiffe
to generate a unique identity for every Service
in the mesh, which is embedded in the x509
URI of the cert-chain.pem
file mounted in the istio-sidecar-injection
pod.citadel
watches ServiceAccounts
and Secrets
across all namespaces in the Cluster. However, this may cause contention in a multi control-plane cluster, as one citadel instance from one mesh might step-on, and modify, another mesh's Secret
and embed an incorrect spiffe
URI, leading to errors during the certificate authentication between the API Server and the MutatingWebhook endpoint.citadel
container takes in command-line arguments that can be used to configure a list of namespaces to be watched by citadel's secretcontroller
, for events. This, however, is entirely static and requires the administrator to know what namespaces will be part of the service-mesh apriori.citadel
nor a way to provide regex-based command line args to citadel
.# Source: istio/charts/security/templates/deployment.yaml
@@ -177,6 +178,7 @@
- --monitoring-port=15014
- --self-signed-ca=true
- --workload-cert-ttl=2160h
+ - --listened-namespaces=istio-affirmed,istio-affirmed-workspace,kube-system
env:
- name: CITADEL_ENABLE_NAMESPACES_BY_DEFAULT
value: "true"
Istio configuration resources are applied to all istio mesh sidecars
VirtualServices
, Gateways
, DestinationRules
, etc. to a specific Istio control plane (however, this might change with the introduction of label/annotation-based tagging in the upcoming istio-1.6.0 release).istio-proxy
sidecars across multiple meshes will receive envoy xDS
updates for all deployed Service
s and Istio configuration resources from every namespace they are deployed to. However, there is a way to limit the amount of configurations a sidecar receives using Istio's Sidecar
resource. All namespaces owned by a single mesh must deploy this additional resource to each of their namespaces to avoid syncing with the other control-planes.