金丝雀部署是一种渐进式的应用升级部署方式,可以把应用升级对业务的影响降到最低。

一般情况下,Argo rollouts实现金丝雀部署,是通过控制新老应用的pod数量来进行流量分配的。这种方式也存在缺点:pod数量比较多,资源占用比较多,流量比例和pod数量绑定,无法实现精细的流量比例控制。

通过使用 Ingress-nginx,Argo rollouts可以把调节流量的功能放到Ingress上面,流量比例可以控制得更加精细,对于新老应用的pod数量也没有要求。

操作系统 Debian 12, Kubernetes 1.28.4,argo rollouts 1.6.6。

一、安装软件

在Kubernetes集群的master主机上,运行下列命令,安装argo rollouts

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

运行下列命令,安装kubectl插件

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
kubectl argo rollouts version

kubernetets集群应当安装过 Ingress-nginx Controller ,如果没有,运行下列命令安装

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml

二、使用案例

用nginx升级作为案例,把nginx从1.24升级到1.25.2。

1.部署原有的稳定版本

新建文件rollouts-nginx.yaml,内容如下

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 1
  strategy:
    canary:
      canaryService: rollouts-demo-canary
      stableService: rollouts-demo-stable
      trafficRouting:
        nginx:
          stableIngress: rollouts-demo-stable
      steps:
      - setWeight: 5
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: atomhub.openatom.cn/library/nginx:1.24
        ports:
        - name: http
          containerPort: 80
          protocol: TCP

新建文件service-nginx.yaml,内容如下

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-canary
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo


---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-stable
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo

新建文件ingress-nginx.yaml,内容如下

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rollouts-demo-stable
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: rollouts-demo.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field
            name: rollouts-demo-stable
            port:
              number: 80

运行

kubelctl apply -f rollouts-nginx.yaml
kubelctl apply -f service-nginx.yaml
kubelctl apply -f ingress-nginx.yaml

把nginx1.24、service和ingress部署上去。

运行

kubectl argo rollouts get rollout rollouts-demo

输出类似下面

ame:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          atomhub.openatom.cn/library/nginx:1.24 (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  94s
└──# revision:1
   └──⧉ rollouts-demo-5f7bff5859           ReplicaSet  ✔ Healthy  64s  stable
      └──□ rollouts-demo-5f7bff5859-ck6fn  Pod         ✔ Running  64s  ready:1/1

可以看到nginx1.24有一个pod,ActualWeight是100,100%用户流量都在它上面。

2. 部署新版本

运行

kubectl argo rollouts set image rollouts-demo rollouts-demo=atomhub.openatom.cn/library/nginx:1.25.2

把nginx1.25.2同时部署上去。

运行

kubectl argo rollouts get rollout rollouts-demo

可以看到当前应用的容器部署情况。输出类似下面

Name:            rollouts-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     5
  ActualWeight:  5
Images:          atomhub.openatom.cn/library/nginx:1.24 (stable)
                 atomhub.openatom.cn/library/nginx:1.25.2 (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS     AGE    INFO
⟳ rollouts-demo                            Rollout     ॥ Paused   2m47s
├──# revision:2
│  └──⧉ rollouts-demo-5cbc85b74c           ReplicaSet  ✔ Healthy  11s    canary
│     └──□ rollouts-demo-5cbc85b74c-pvzhk  Pod         ✔ Running  11s    ready:1/1
└──# revision:1
   └──⧉ rollouts-demo-5f7bff5859           ReplicaSet  ✔ Healthy  2m17s  stable
      └──□ rollouts-demo-5f7bff5859-ck6fn  Pod         ✔ Running  2m17s  ready:1/1

现在nginx1.24和nginx1.25.2各有一个pod在运行 ,ActualWeight是5,有5%的用户流量分配到了nginx1.25.2上面。

现在可以通过service rollouts-demo-canary对新版本的应用进行测试,看看是否运行正常。

3. 增加新版本的流量比例

如果测试没有问题,运行

kubectl argo rollouts promote rollouts-demo

继续部署过程。

运行

kubectl argo rollouts get rollout rollouts-demo --watch

进行观察。

根据rollouts-nginx.yaml的定义,每隔10秒钟,ActualWeight就会从5开始,不断增加,Nginx1.25.2上的用户流量比例一步步变成40%,60%,80%,最后100%。

如果接下来运行正常,这个过程就结束了,金丝雀部署就此完成,nginx从1.24顺利升级到了1.25.2。

4. 中断升级,回退到老版本

在部署过程中,如果发现新版本运行出现问题,可以运行

kubectl argo rollouts abort rollouts-demo

中断部署。

运行

kubectl argo rollouts get rollout rollouts-demo

可以看到如下输出

Name:            rollouts-demo
Namespace:       default
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 2
Strategy:        Canary
  Step:          0/8
  SetWeight:     0
  ActualWeight:  0
Images:          atomhub.openatom.cn/library/nginx:1.24 (stable)
                 atomhub.openatom.cn/library/nginx:1.25.2 (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS      AGE    INFO
⟳ rollouts-demo                            Rollout     ✖ Degraded  5m33s
├──# revision:2
│  └──⧉ rollouts-demo-5cbc85b74c           ReplicaSet  ✔ Healthy   2m57s  canary,delay:25s
│     └──□ rollouts-demo-5cbc85b74c-pvzhk  Pod         ✔ Running   2m57s  ready:1/1
└──# revision:1
   └──⧉ rollouts-demo-5f7bff5859           ReplicaSet  ✔ Healthy   5m3s   stable
      └──□ rollouts-demo-5f7bff5859-ck6fn  Pod         ✔ Running   5m3s   ready:1/1

现在ActualWight是0,所有的用户流量又回到了Nginx1.24上面。

如果应用升级已经完成,所有的用户流量都已经到了nginx1.25.2上面,这个时候才发现新版本出现问题。可以运行

kubectl argo rollouts undo rollouts-demo

进行回滚,所有的用户流量又会回到nginx1.24上面。

参考:

https://argo-rollouts.readthedocs.io/en/stable/installation

https://argo-rollouts.readthedocs.io/en/stable/getting-started/nginx

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注