备考ICA----Istio实验9---熔断Circuit Breaking 实验

备考ICA----Istio实验9—熔断Circuit Breaking 实验

1. 环境准备

创建httpbin环境

kubectl apply -f istio/samples/httpbin/httpbin.yaml
kubectl get svc httpbin

2. 创建测试用客户端

kubectl apply -f istio/samples/httpbin/sample-client/fortio-deploy.yaml

3. 创建Httpbin的Circuit Breaking DR

circuit/httpbin-dr-Circuit.yaml

kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

部署dr

kubectl apply -f circuit/httpbin-dr-Circuit.yaml

4. 访问测试

4.1 maxConnections=1

kubectl exec deploy/fortio-deploy -- /usr/bin/fortio curl -quiet http://httpbin:8000/get

当并发访问时会有一部分请求失败

kubectl exec deploy/fortio-deploy -- /usr/bin/fortio load -c 2 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get

因为2个并发,20次,不一定是50%的503.主要还是看同一时间内多少如果大于1个连接那么这个请求结束前其他请求就被拒绝.

放大请求数和并发数,被503熔断的请求比例会更多一点

kubectl exec deploy/fortio-deploy -- /usr/bin/fortio load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get

可以查看 75 该 upstream_rq_pending_overflow 值，这意味着 75 到目前为止的调用已被标记为 Circuit

kubectl exec deploy/fortio-deploy -c istio-proxy -- pilot-agent request GET stats | grep httpbin | grep pending

2.2 调大maxConnections值

将maxConnections和http1MaxPendingRequests,maxRequestsPerConnection值一并调大后
circuit/httpbin-dr-Circuit.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 2
      http:
        http1MaxPendingRequests: 2
        maxRequestsPerConnection: 2
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

部署dr

kubectl apply -f  circuit/httpbin-dr-Circuit.yaml

再次用3并发测试30个连接

kubectl exec deploy/fortio-deploy -- /usr/bin/fortio load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get

可见503的比例明显变小

即使将并发4请求40个连接也只有40%的被503

实际工作中我们可以根据请求量和服务性能来调整maxConnections,http1MaxPendingRequests,maxRequestsPerConnection的值通过熔断,牺牲掉一部分请求来保证其余请求的正常使用

5. 环境清理

kubectl delete -f istio/samples/httpbin/httpbin.yaml -f istio/samples/httpbin/sample-client/fortio-deploy.yaml
kubectl delete -f circuit/httpbin-dr-Circuit.yaml

至此Circuit Breaking 熔断实验完成