How to apply Istio DestinationRule across the mesh?

An interesting question came up today in Istio Slack where someone asked if and how to apply DestinationRules globally to all workloads inside the cluster. The short answer is yes, you can do that, and this article will explain how to do it.

The DestinationRule resource allows us to configure policies that get applied to traffic once routing has occurred. Under the hood, the Istio translates this config to the Envoy config's clusters section.

Typically you use DestinationRule to configure different subsets (e.g., v1, v2 or prod, dev, and so on) and traffic policy settings, such as the load balancing policy, connection pool sizes, and outlier detection.

If you're using only subsets, you'll probably have one DestinationRule per host, as those typically don't make sense to apply globally across multiple workloads.

On the other hand, settings such as outlier detection or load balancer policies might make sense to apply globally.

For example:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ?
  namespace: ?
spec:
  host: ?
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    outlierDetection:
      consecutive5xxErrors: 10
      interval: 2m
      baseEjectionTime: 10m

It would be best if you looked first at the Istio's global mesh options. The global mesh options are the place for any settings and configurations Istio applies globally across the whole mesh. However, in the case of the outlier detection, load balancer policy, or other configuration that can be set in the DestinationRule, we can't define those through the global options.

Looking at the above YAML, we know we need to answer two questions -- where to create the DestinationRule (name and namespace), and what do we set the host field to?

Istio has a concept of a root namespace (rootNamespace field in the global mesh options). So let's say we're running a workload in ns1; when processing the configuration, Istio looks in ns1 first, and if there's no configuration there, it will look in the root namespace. If not explicitly set, the root namespace defaults to istio-system. The answer to the first question is that we must deploy the resource to the istio-system namespace.

But how can we target multiple hosts? The host field in the DestinationRule expects the name of a service that exists in the service registry - for example, product-page or product-page.svc.cluster.local. Note that using a fully qualified name is the preferred way to address the service, so you can avoid misinterpretation and know precisely which service the rule applies to.

The "feature" that will allow us to apply the rule globally is the support for wildcards (*) in the service name. Therefore, to apply a destination rule globally, we could use the notation *.cluster.local or just leave the host field off all together.

This notation, coupled with the resource being in the root namespace, will apply the configuration across all services in the mesh:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: global-dr
  namespace: istio-system
spec:
  # Note we don't specify the `host` field at all.
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    outlierDetection:
      consecutive5xxErrors: 15
      interval: 2m
      baseEjectionTime: 10m

We can try this out quickly by deploying workloads into two separate namespaces and then applying this global DestinationRule.

Then, we can use the istioctl pc cluster command to check the settings are applied globally to all Envoy clusters in the mesh:

istioctl pc cluster deploy/httpbin.httpbin -o yaml | grep -B3 consecutive5xx

 name: outbound|8000||httpbin.httpbin.svc.cluster.local
  outlierDetection:
    baseEjectionTime: 900s
    consecutive5xx: 15
--
  name: outbound|80||istio-ingressgateway.istio-system.svc.cluster.local
  outlierDetection:
    baseEjectionTime: 900s
    consecutive5xx: 15
--
  name: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
  outlierDetection:
    baseEjectionTime: 900s
    consecutive5xx: 15
...

Another option we have for targeting workloads is by using the workloadSelector and specifying the labels. Note that having a mesh or namespace destination rule together with the workloadSelector isn't going to work. You'll have to specify the exact host name.

Here's how you can apply a destination rule for v1 versions of the workloads:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sleep-dr
  namespace: default
spec:
  host: "sleep.default.svc.cluster.local"
  workloadSelector:
    matchLabels:
      version: v1
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    outlierDetection:
      consecutive5xxErrors: 15
      interval: 2m
      baseEjectionTime: 10m

How about merging multiple DestinationRules?

Another interesting scenario is having a global DestinationRule and having workload or host specific DestinationRules. In this case we'd expect Istio to merge the two DestinationRules.

However, that only happens if you set the PILOT_ENABLE_DESTINATION_RULE_INHERITANCE environment variable to true when installing Istio. By default, the destination rule inheritance is turned off.

So let's apply a global destination rule (without a host set) to the istio-system namespace:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: global-dr
  namespace: istio-system
spec:
  # Note we don't specify the `host` field at all.
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    outlierDetection:
      consecutive5xxErrors: 15
      interval: 2m
      baseEjectionTime: 10m

Just like before, these settings will be applied to all services in the mesh. If we want to customize the settings per namespace, we can deploy another DestinationRule and only modify the specific values we want to change.

For example, let's say we want to increase the consecutive5xxErrors to 25, for the sleep host:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sleep-dr
  namespace: default
spec:
  host: "sleep.default.svc.cluster.local"
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 25

If we run the istioctl pc cluster command, we'll see that the sleep workload inherited the settings from the global DestinationRule that got merged with the local DestinationRule where we changed the consecutive5xxErrors value:

...
         name: httpbin
          namespace: default
  name: outbound|8000||httpbin.default.svc.cluster.local
  outlierDetection:
    baseEjectionTime: 600s
    consecutive5xx: 15
    enforcingConsecutive5xx: 100
    enforcingSuccessRate: 0
    interval: 120s
  transportSocketMatches:
  - match:
--
...
          name: sleep
          namespace: default
  name: outbound|80||sleep.default.svc.cluster.local
  outlierDetection:
    baseEjectionTime: 600s
    consecutive5xx: 25
    enforcingConsecutive5xx: 100
    enforcingSuccessRate: 0
    interval: 120s
  transportSocketMatches:
  - match:
...