
金丝雀部署是一种渐进式的应用升级部署方式,可以把应用升级对业务的影响降到最低。
一般情况下,Argo rollouts实现金丝雀部署,是通过控制新老应用的pod数量来进行流量分配的。这种方式也存在缺点:pod数量比较多,资源占用比较多,流量比例和pod数量绑定,无法实现精细的流量比例控制。
通过使用 Ingress-nginx,Argo rollouts可以把调节流量的功能放到Ingress上面,流量比例可以控制得更加精细,对于新老应用的pod数量也没有要求。
操作系统 Debian 12, Kubernetes 1.28.4,argo rollouts 1.6.6。
一、安装软件
在Kubernetes集群的master主机上,运行下列命令,安装argo rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
运行下列命令,安装kubectl插件
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
kubectl argo rollouts version
kubernetets集群应当安装过 Ingress-nginx Controller ,如果没有,运行下列命令安装
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml
二、使用案例
用nginx升级作为案例,把nginx从1.24升级到1.25.2。
1.部署原有的稳定版本
新建文件rollouts-nginx.yaml,内容如下
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollouts-demo
spec:
replicas: 1
strategy:
canary:
canaryService: rollouts-demo-canary
stableService: rollouts-demo-stable
trafficRouting:
nginx:
stableIngress: rollouts-demo-stable
steps:
- setWeight: 5
- pause: {}
- setWeight: 40
- pause: {duration: 10}
- setWeight: 60
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollouts-demo
template:
metadata:
labels:
app: rollouts-demo
spec:
containers:
- name: rollouts-demo
image: atomhub.openatom.cn/library/nginx:1.24
ports:
- name: http
containerPort: 80
protocol: TCP
新建文件service-nginx.yaml,内容如下
apiVersion: v1
kind: Service
metadata:
name: rollouts-demo-canary
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-demo
---
apiVersion: v1
kind: Service
metadata:
name: rollouts-demo-stable
spec:
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-demo
新建文件ingress-nginx.yaml,内容如下
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rollouts-demo-stable
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: rollouts-demo.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
# Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field
name: rollouts-demo-stable
port:
number: 80
运行
kubelctl apply -f rollouts-nginx.yaml
kubelctl apply -f service-nginx.yaml
kubelctl apply -f ingress-nginx.yaml
把nginx1.24、service和ingress部署上去。
运行
kubectl argo rollouts get rollout rollouts-demo
输出类似下面
ame: rollouts-demo
Namespace: default
Status: ✔ Healthy
Strategy: Canary
Step: 8/8
SetWeight: 100
ActualWeight: 100
Images: atomhub.openatom.cn/library/nginx:1.24 (stable)
Replicas:
Desired: 1
Current: 1
Updated: 1
Ready: 1
Available: 1
NAME KIND STATUS AGE INFO
⟳ rollouts-demo Rollout ✔ Healthy 94s
└──# revision:1
└──⧉ rollouts-demo-5f7bff5859 ReplicaSet ✔ Healthy 64s stable
└──□ rollouts-demo-5f7bff5859-ck6fn Pod ✔ Running 64s ready:1/1
可以看到nginx1.24有一个pod,ActualWeight是100,100%用户流量都在它上面。
2. 部署新版本
运行
kubectl argo rollouts set image rollouts-demo rollouts-demo=atomhub.openatom.cn/library/nginx:1.25.2
把nginx1.25.2同时部署上去。
运行
kubectl argo rollouts get rollout rollouts-demo
可以看到当前应用的容器部署情况。输出类似下面
Name: rollouts-demo
Namespace: default
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 1/8
SetWeight: 5
ActualWeight: 5
Images: atomhub.openatom.cn/library/nginx:1.24 (stable)
atomhub.openatom.cn/library/nginx:1.25.2 (canary)
Replicas:
Desired: 1
Current: 2
Updated: 1
Ready: 2
Available: 2
NAME KIND STATUS AGE INFO
⟳ rollouts-demo Rollout ॥ Paused 2m47s
├──# revision:2
│ └──⧉ rollouts-demo-5cbc85b74c ReplicaSet ✔ Healthy 11s canary
│ └──□ rollouts-demo-5cbc85b74c-pvzhk Pod ✔ Running 11s ready:1/1
└──# revision:1
└──⧉ rollouts-demo-5f7bff5859 ReplicaSet ✔ Healthy 2m17s stable
└──□ rollouts-demo-5f7bff5859-ck6fn Pod ✔ Running 2m17s ready:1/1
现在nginx1.24和nginx1.25.2各有一个pod在运行 ,ActualWeight是5,有5%的用户流量分配到了nginx1.25.2上面。
现在可以通过service rollouts-demo-canary对新版本的应用进行测试,看看是否运行正常。
3. 增加新版本的流量比例
如果测试没有问题,运行
kubectl argo rollouts promote rollouts-demo
继续部署过程。
运行
kubectl argo rollouts get rollout rollouts-demo --watch
进行观察。
根据rollouts-nginx.yaml的定义,每隔10秒钟,ActualWeight就会从5开始,不断增加,Nginx1.25.2上的用户流量比例一步步变成40%,60%,80%,最后100%。
如果接下来运行正常,这个过程就结束了,金丝雀部署就此完成,nginx从1.24顺利升级到了1.25.2。
4. 中断升级,回退到老版本
在部署过程中,如果发现新版本运行出现问题,可以运行
kubectl argo rollouts abort rollouts-demo
中断部署。
运行
kubectl argo rollouts get rollout rollouts-demo
可以看到如下输出
Name: rollouts-demo
Namespace: default
Status: ✖ Degraded
Message: RolloutAborted: Rollout aborted update to revision 2
Strategy: Canary
Step: 0/8
SetWeight: 0
ActualWeight: 0
Images: atomhub.openatom.cn/library/nginx:1.24 (stable)
atomhub.openatom.cn/library/nginx:1.25.2 (canary)
Replicas:
Desired: 1
Current: 2
Updated: 1
Ready: 2
Available: 2
NAME KIND STATUS AGE INFO
⟳ rollouts-demo Rollout ✖ Degraded 5m33s
├──# revision:2
│ └──⧉ rollouts-demo-5cbc85b74c ReplicaSet ✔ Healthy 2m57s canary,delay:25s
│ └──□ rollouts-demo-5cbc85b74c-pvzhk Pod ✔ Running 2m57s ready:1/1
└──# revision:1
└──⧉ rollouts-demo-5f7bff5859 ReplicaSet ✔ Healthy 5m3s stable
└──□ rollouts-demo-5f7bff5859-ck6fn Pod ✔ Running 5m3s ready:1/1
现在ActualWight是0,所有的用户流量又回到了Nginx1.24上面。
如果应用升级已经完成,所有的用户流量都已经到了nginx1.25.2上面,这个时候才发现新版本出现问题。可以运行
kubectl argo rollouts undo rollouts-demo
进行回滚,所有的用户流量又会回到nginx1.24上面。
参考:
https://argo-rollouts.readthedocs.io/en/stable/installation
https://argo-rollouts.readthedocs.io/en/stable/getting-started/nginx