[K8S] SW Version & Cluster Upgrade

 

controlplane ~ ➜  k get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   61m   v1.28.0
node01         Ready    <none>          61m   v1.28.0

v1.11.3
v1: major version
11: minor version
-- Features, Functionalities (monthly releases)
3: patch version
-- Bug Fixes

• 쿠버네티스는 3개의 마이너 버전까지만 지원해준다.
  • ex. 최신 버전이 v1.12라면, 1.12, 1.11, 1.10까지만 서포트한다.
• 모든 Bug가 fix되고나서는 alpha release, 그 뒤에 beta release.
• 업그레이드는 한 번에 하나의 마이너 버전씩 하는 것을 권고한다.
  • 1.10 -> 1.11 -> 1.12->1.13

 

# kubeadm upgrade

- kubernetes와 같은 버전을 가져간다.

controlplane ~ ➜  kubectl describe node | grep Taints
Taints:             <none>
Taints:             <none>
# 둘다 application을 host할 수 있다는 의미다.

controlplane ~ ➜  kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.0
I0428 05:15:40.520516   16879 version.go:256] remote version is much newer: v1.30.0; falling back to: stable-1.28
[upgrade/versions] Target version: v1.28.9
[upgrade/versions] Latest version in the v1.28 series: v1.28.9

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     2 x v1.28.0   v1.28.9        

Upgrade to the latest version in the v1.28 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.28.0   v1.28.9
kube-controller-manager   v1.28.0   v1.28.9
kube-scheduler            v1.28.0   v1.28.9
kube-proxy                v1.28.0   v1.28.9
CoreDNS                   v1.10.1   v1.10.1
etcd                      3.5.9-0   3.5.9-0

You can now apply the upgrade by executing the following command:    # available stable version

        kubeadm upgrade apply v1.28.9

Note: Before you can perform this upgrade, you have to update kubeadm to v1.28.9.

 

 

# Cluster Upgrade

• 업그레이드 중에 마스터가 Down되었다고 해서 worker nodes와 application이 영향을 받는 것은 아니다.
• 마스터가 down되면, 모든 management functions이 down돼서 kubectl 을 사용해서 cluster나 kube-API에 접근할 수 없다.
  • 그냥 새로운 application을 deploy할 수 없고, 수정/삭제할 수 없다.
  • controller-manager도 작동을 안하기 때문에 자동으로 새로운 파드가 생기지도 않는다.
• 워커노드를 업그레이드하는 방법은 총 3가지가 있다.
• 첫째, 마스터가 업그레이드 되고 나면 모든 워커노드가 업그레이드되는데, 이때는 application에 접근할 수 없다.
  • 노드의 업그레이드가 완성되면, nodes가 백업되고, 새로운 pod가 스케줄링된다.
• 둘째, 노드 하나씩만 업그레이드가 될 수 도 있다.
  • 첫 번째 노드가 업그레이드 되면, workloads가 전부 2~3번으로 옮겨간다. (user는 2~3번에서 서비스 제공받음)
• 셋째, 새로운 버전의 워커노드를 하나씩 추가한다.

# 업그레이드 순서

$ kubeadm upgrade plan
$ kubeadm upgrade apply

# 한 번에 하나의 minor version만 upgrade할 수 있음을 기억하기
$ apt-get upgrade -y kubeadm=1.12.0-00

$ kubeadm upgrade apply v1.12.0

$ kubectl get nodes 
# API serverf에 등록된 kubelet의 버전을 보여주는 거라 예전 버전을 보여준다.
# 그렇다고 API server의 version은 아닌 것을 명심한다.

# 마스터 노드에 있는 kubelet 업그레이드하기
$ apt-get upgrade -y kubelet=1.12.0-00
$ systemctl restart kubelet

$ kubectl get nodes 
# master가 업그레이드 된 것 확인 가능할 것이다.


# node 업그레이드
$ kubectl drain node-1
$ apt-get upgrade -y kubeadm=1.12.0-00
$ apt-get upgrade -y kubelet=1.12.0-00
$ kubeadm upgrade node config --kubelet-version v1.12.0
$ systemctl restart kubelet
$ kubectl uncordon node-1

 

# 실습

1. controlane 먼저 drain 시키기 (v1.28.0 -> v1.28.9)

# We will be upgrading the controlplane node first. Drain the controlplane node of workloads and mark it UnSchedulable
controlplane ~ ✖ k drain controlplane  --ignore-daemonsets
node/controlplane already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-h7wnv, kube-system/kube-proxy-79jkp
evicting pod kube-system/coredns-5dd5756b68-fd7dn
evicting pod default/blue-667bf6b9f9-qfwmr
evicting pod kube-system/coredns-5dd5756b68-7kpjt
evicting pod default/blue-667bf6b9f9-gtjl6
pod/blue-667bf6b9f9-qfwmr evicted
pod/blue-667bf6b9f9-gtjl6 evicted
pod/coredns-5dd5756b68-fd7dn evicted
pod/coredns-5dd5756b68-7kpjt evicted
node/controlplane drained

controlplane ~ ➜  kubectl get nodes
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready,SchedulingDisabled   control-plane   70m   v1.28.0
node01         Ready                      <none>          70m   v1.28.0

controlplane ~ ➜  k get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
blue-667bf6b9f9-8hg4c   1/1     Running   0          9m30s   10.244.1.3    node01   <none>           <none>
blue-667bf6b9f9-9vmng   1/1     Running   0          47s     10.244.1.11   node01   <none>           <none>
blue-667bf6b9f9-lh47h   1/1     Running   0          47s     10.244.1.8    node01   <none>           <none>
blue-667bf6b9f9-w8cxh   1/1     Running   0          9m30s   10.244.1.2    node01   <none>           <none>
blue-667bf6b9f9-z5l6c   1/1     Running   0          9m30s   10.244.1.4    node01   <none>           <none>
# node01로 모두 옮겨왔다.

2. Upgrade the controlplane components to exact version v1.29.0

controlplane ~ ➜  cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal

# 어떤 버전으로 upgrade할 지 찾아보기
controlplane ~ ➜  sudo apt update
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                                  
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease                                
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease           
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease   
Hit:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
78 packages can be upgraded. Run 'apt list --upgradable' to see them.

controlplane ~ ➜  sudo apt-cache madison kubeadm
   kubeadm | 1.28.9-2.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.8-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages
   kubeadm | 1.28.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.28/deb  Packages


controlplane ~ ➜ apt-get update && apt-get install -y --allow-change-held-packages kubeadm=1.28.9-2.1
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                                                              
Hit:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease                     
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease                 
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease     
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following held packages will be changed:
  kubeadm
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
Need to get 10.1 MB of archives.
After this operation, 2,490 kB disk space will be freed.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  kubeadm 1.28.9-2.1 [10.1 MB]
Fetched 10.1 MB in 0s (31.6 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 20477 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.28.9-2.1_amd64.deb ...
Unpacking kubeadm (1.28.9-2.1) over (1.28.0-1.1) ...
Setting up kubeadm (1.28.9-2.1) ...

controlplane ~ ➜  kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.9", GitCommit:"587f5fe8a69b0d15b578eaf478f009247d1c5d47", GitTreeState:"clean", BuildDate:"2024-04-16T15:04:37Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}

controlplane ~ ✖ kubeadm upgrade apply v1.28.9
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.28.9"
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.9
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.9". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

 

3. Upgrade kubelet and kubectl

controlplane ~ ➜  sudo apt-mark unhold kubelet kubectl && \
> sudo apt-get update && sudo apt-get install -y kubelet='1.28.9-*' kubectl='1.28.9-*' && \
> sudo apt-mark hold kubelet kubectl
kubelet was already not hold.
kubectl was already not hold.
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                    
Hit:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease
Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease                  
Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease                      
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
...
kubelet set on hold.
kubectl set on hold.

controlplane ~ ➜  sudo systemctl daemon-reload

controlplane ~ ➜  sudo systemctl restart kubelet

controlplane ~ ➜  kubectl get nodes
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready,SchedulingDisabled   control-plane   90m   v1.28.9 # 업그레이드 된 것 확인
node01         Ready                      <none>          90m   v1.28.0

controlplane ~ ➜  k uncordon controlplane
node/controlplane uncordoned

controlplane ~ ➜  kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   91m   v1.28.9
node01         Ready    <none>          91m   v1.28.0

 

4. Drain worker node

controlplane ~ ✖ k drain node01 --ignore-daemonsets
node/node01 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-pbfbp, kube-system/kube-proxy-wqhq6
evicting pod kube-system/coredns-5dd5756b68-lwwmk
evicting pod default/blue-667bf6b9f9-lh47h
evicting pod default/blue-667bf6b9f9-9vmng
evicting pod default/blue-667bf6b9f9-8hg4c
evicting pod default/blue-667bf6b9f9-w8cxh
evicting pod kube-system/coredns-5dd5756b68-5gg5l
evicting pod default/blue-667bf6b9f9-z5l6c
pod/blue-667bf6b9f9-9vmng evicted
pod/blue-667bf6b9f9-lh47h evicted
pod/blue-667bf6b9f9-w8cxh evicted
pod/blue-667bf6b9f9-8hg4c evicted
pod/blue-667bf6b9f9-z5l6c evicted
pod/coredns-5dd5756b68-5gg5l evicted
pod/coredns-5dd5756b68-lwwmk evicted
node/node01 drained

controlplane ~ ➜  k get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
blue-667bf6b9f9-gtf7s   1/1     Running   0          64s   10.244.0.7    controlplane   <none>           <none>
blue-667bf6b9f9-gtn4v   1/1     Running   0          63s   10.244.0.8    controlplane   <none>           <none>
blue-667bf6b9f9-h7mmg   1/1     Running   0          63s   10.244.0.10   controlplane   <none>           <none>
blue-667bf6b9f9-t5xxr   1/1     Running   0          63s   10.244.0.12   controlplane   <none>           <none>
blue-667bf6b9f9-t9cpt   1/1     Running   0          63s   10.244.0.11   controlplane   <none>           <none>

 

5. Upgrade worker nodes

controlplane ~ ✖ ssh node01

root@node01 ~ ➜  sudo apt-mark unhold kubeadm && \
> sudo apt-get update && sudo apt-get install -y kubeadm='1.28.9-*' && \
> sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Get:2 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]                      
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease [1189 B]
...

root@node01 ~ ➜  sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2567781811/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

root@node01 ~ ➜  sudo apt-mark unhold kubelet kubectl && \
> sudo apt-get update && sudo apt-get install -y kubelet='1.28.9-*' kubectl='1.28.9-*' && \
> sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
...
(Reading database ... 14854 files and directories currently installed.)
Preparing to unpack .../kubectl_1.28.9-2.1_amd64.deb ...
Unpacking kubectl (1.28.9-2.1) over (1.28.0-1.1) ...
Preparing to unpack .../kubelet_1.28.9-2.1_amd64.deb ...
Unpacking kubelet (1.28.9-2.1) over (1.28.0-1.1) ...
Setting up kubectl (1.28.9-2.1) ...
Setting up kubelet (1.28.9-2.1) ...
kubelet set on hold.
kubectl set on hold.

root@node01 ~ ➜  sudo systemctl daemon-reload
root@node01 ~ ➜  sudo systemctl restart kubelet

root@node01 ~ ➜  exit
logout
Connection to node01 closed.

controlplane ~ ➜  k get nodes 
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready                      control-plane   99m   v1.28.9
node01         Ready,SchedulingDisabled   <none>          98m   v1.28.9

controlplane ~ ➜  k uncordon node01
node/node01 uncordoned

controlplane ~ ➜  k get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   99m   v1.28.9
node01         Ready    <none>          99m   v1.28.9