What is circuit breaking?
A circuit breaker is an automatically operated electrical switch designed to protect an electrical circuit from damage caused by excess current from an overload or short circuit. How does that apply to your services and Istio service mesh?
- number of consecutive errors
- scan interval
- base ejection time
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: greeter-service
spec:
host: greeter-service.default.svc.cluster.local
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
outlierDetection:
consecutiveErrors: 1
interval: 1s
baseEjectionTime: 10m
maxEjectionPercent: 100
http1MaxPendingRequests
) to a single request. This means if there is more than one request queued for connection, the circuit breaker will trip. Similarly, the circuit breaker trips if there is more than one request per connection (maxRequestsPerConnection
setting).baseEjectionTime
setting). If the pod is ejected from the load balancing pool, no requests will be able to reach it. The second setting in play with the ejection is the maxEjectionPercent
. This setting represents a threshold that, if reached, causes the circuit breaker to load balance across all pods again.maxEjectionPercent
is set to 50%. If pods are failing, circuit breaker keeps ejecting them from the load balancing pool. With failing pods ejected from the load balancing pool, only healthy pods receive trafficconsecutiveErrors
setting. In the above example, if there's more than 1 error (an HTTP 5xx is considered an error), the pod gets ejected from the load balancing pool. Finally, the interval
is a time interval between checking if new pods need to be ejected or brought back into the load balancing pool.cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: greeter-service
spec:
host: greeter-service
trafficPolicy:
connectionPool:
http:
http1MaxPendingRequests: 1
maxRequestsPerConnection: 1
outlierDetection:
consecutiveErrors: 1
interval: 1s
baseEjectionTime: 10m
maxEjectionPercent: 100
EOF
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: fortio
labels:
app: fortio
spec:
ports:
- port: 8080
name: http
selector:
app: fortio
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fortio-deploy
spec:
replicas: 1
selector:
matchLabels:
app: fortio
template:
metadata:
annotations:
sidecar.istio.io/statsInclusionPrefixes: cluster.outbound,cluster_manager,listener_manager,http_mixer_filter,tcp_mixer_filter,server,cluster.xds-grpc
labels:
app: fortio
spec:
containers:
- name: fortio
image: fortio/fortio:latest_release
imagePullPolicy: Always
ports:
- containerPort: 8080
name: http-fortio
- containerPort: 8079
name: grpc-ping
EOF
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: greeter-service-v1
labels:
app: greeter-service
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: greeter-service
version: v1
template:
metadata:
labels:
app: greeter-service
version: v1
spec:
containers:
- image: learnistio/greeter-service:1.0.0
imagePullPolicy: Always
name: svc
ports:
- containerPort: 3000
---
kind: Service
apiVersion: v1
metadata:
name: greeter-service
labels:
app: greeter-service
spec:
selector:
app: greeter-service
ports:
- port: 3000
name: http
EOF
export FORTIO_POD=$(kubectl get pod | grep fortio | awk '{ print $1 }')
kubectl exec -it $FORTIO_POD -c fortio /usr/bin/fortio -- load -curl http://greeter-service:3000/hello
HTTP/1.1 200 OK
x-powered-by: Express
content-type: application/json; charset=utf-8
content-length: 43
etag: W/"2b-DdO+hdtaORahq7JZ8niOkjoR0XQ"
date: Fri, 04 Jan 2019 00:53:19 GMT
x-envoy-upstream-service-time: 7
server: envoy
{"message":"hello 👋 ","version":"1.0.0"}
kubectl exec -it $FORTIO_POD -c fortio /usr/bin/fortio -- load -c 2 -qps 0 -n 20 -loglevel Warning http://greeter-service:3000/hello
...
Code 200 : 19 (95.0 %)
Code 503 : 1 (5.0 %)
kubectl exec -it $FORTIO_POD -c fortio /usr/bin/fortio -- load -c 3 -qps 0 -n 50 -loglevel Warning http://greeter-service:3000/hello
...
Code 200 : 41 (82.0 %)
Code 503 : 9 (18.0 %)
$ kubectl exec -it $FORTIO_POD -c istio-proxy -- sh -c 'curl localhost:15000/stats' | grep greeter-service | grep pending
cluster.outbound|3000||greeter-service.default.svc.cluster.local.upstream_rq_pending_active: 0
cluster.outbound|3000||greeter-service.default.svc.cluster.local.upstream_rq_pending_failure_eject: 107
cluster.outbound|3000||greeter-service.default.svc.cluster.local.upstream_rq_pending_overflow: 9
cluster.outbound|3000||greeter-service.default.svc.cluster.local.upstream_rq_pending_total: 2193
istioctl dashboard prometheus
greeter-service
and group them by response code, flags, and source app:sum(istio_requests_total{destination_app="greeter-service}) by (response_code, response_flags, source_app)