1. This lab tests your skills on upgrading a kubernetes cluster. We have a production cluster with applications running on it. Let us explore the setup first. What is the current version of the cluster?
controlplane ~ ➜ k describe pod kube-controller-manager-controlplane -n kube-system
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 23m kubelet Container image "registry.k8s.io/kube-controller-manager:v1.28.0" already present on machine
Normal Created 23m kubelet Created container kube-controller-manager
Normal Started 22m kubelet Started container kube-controller-manager
answer : v.1.28.0
2. How many nodes are part of this cluster? Including controlplane and worker nodes
controlplane ~ ✖ k get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 24m v1.28.0
node01 Ready <none> 23m v1.28.0
answer : 2
3, How many nodes can host workloads in this cluster? Inspect the applications and taints set on the nodes.
controlplane ~ ➜ k describe node controlplane | grep Taints
Taints: <none>
controlplane ~ ➜ k describe node node01 | grep Taints
Taints: <none>
두 노드 모두 Taints 항목이 없으므로 모두 pod 가 host 될수 있다.
answer : 2
4. How many applications are hosted on the cluster? Count the number of deployments in the default namespace.
controlplane ~ ➜ k get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
blue 5/5 5 5 15m
answer : 1
5. What nodes are the pods hosted on?
controlplane ~ ✖ k get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
blue-667bf6b9f9-4dtf4 1/1 Running 0 17m 10.244.0.4 controlplane <none> <none>
blue-667bf6b9f9-bs7fw 1/1 Running 0 17m 10.244.1.4 node01 <none> <none>
blue-667bf6b9f9-sgthr 1/1 Running 0 17m 10.244.1.3 node01 <none> <none>
blue-667bf6b9f9-wf4hz 1/1 Running 0 17m 10.244.0.5 controlplane <none> <none>
blue-667bf6b9f9-xcr75 1/1 Running 0 17m 10.244.1.2 node01 <none> <none>
answer : controlplane, node01
6. You are tasked to upgrade the cluster. Users accessing the applications must not be impacted, and you cannot provision new VMs. What strategy would you use to upgrade the cluster?
노드 하나를 업그레이드 한 후에 옮기는 전략을 사용할 때 무중단으로 업그레이드가 가능하다.
answer : Upgrade one node at time while moving the workloads to the other
7. What is the latest version available for an upgrade with the current version of the kubeadm tool installed?
Use the kubeadm tool
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/
kubeadm upgrade
kubeadm upgrade is a user-friendly command that wraps complex upgrading logic behind one command, with support for both planning an upgrade and actually performing it. kubeadm upgrade guidance The steps for performing an upgrade using kubeadm are outlined
kubernetes.io
controlplane ~ ➜ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
...
COMPONENT CURRENT TARGET
kubelet 2 x v1.28.0 v1.28.11
Upgrade to the latest version in the v1.28 series:
...
answer : v1.28.11
8. We will be upgrading the controlplane node first. Drain the controlplane node of workloads and mark it UnSchedulable
controlplane ~ ✖ k drain controlplane --ignore-daemonsets
node/controlplane already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-2xrgh, kube-system/kube-proxy-7xnq6
evicting pod kube-system/coredns-5dd5756b68-glf86
evicting pod default/blue-667bf6b9f9-wf4hz
evicting pod default/blue-667bf6b9f9-4dtf4
evicting pod kube-system/coredns-5dd5756b68-4xq7s
pod/blue-667bf6b9f9-4dtf4 evicted
pod/blue-667bf6b9f9-wf4hz evicted
pod/coredns-5dd5756b68-glf86 evicted
pod/coredns-5dd5756b68-4xq7s evicted
node/controlplane drained
9. Upgrade the controlplane components to exact version v1.29.0
Upgrade the kubeadm tool (if not already), then the controlplane components, and finally the kubelet. Practice referring to the Kubernetes documentation page.
노드 업그레이드 절차
9-1. kubernetes 옵션에 적힌 버전 변경
vim /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /
9-2. apt update (debian 기준)
controlplane ~ ➜ apt update
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb InRelease [1,189 B]
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:7 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb Packages [10.2 kB]
Fetched 11.4 kB in 1s (10.2 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
87 packages can be upgraded. Run 'apt list --upgradable' to see them.
9-3. apt-cache 적용
apt-cache madison 명령어는 Debian 기반 시스템(예: Ubuntu)에서 패키지의 가용한 버전 목록을 확인할 때 유용한 도구입니다. Kubernetes 클러스터를 업그레이드하거나 특정 버전의 kubeadm을 설치하려는 경우, apt-cache madison kubeadm 명령어를 사용하여 사용할 수 있는 kubeadm 패키지의 버전 목록을 확인할 수 있습니다.
controlplane ~ ➜ apt-cache madison kubeadm
kubeadm | 1.29.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.4-2.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
9-4. apt를 통해 받은 kubeadm 업그레이드 진행
controlplane ~ ✖ apt-get install kubeadm=1.29.0-1.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.
Need to get 10.1 MB of archives.
After this operation, 2,511 kB disk space will be freed.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb kubeadm 1.29.0-1.1 [10.1 MB]
Fetched 10.1 MB in 0s (37.7 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 20477 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.29.0-1.1_amd64.deb ...
Unpacking kubeadm (1.29.0-1.1) over (1.28.0-1.1) ...
Setting up kubeadm (1.29.0-1.1) ...
9-5. kubeadm upgrade 진행 확인
plan 을 통해 업그레이드 할 버전을 확인한 후 apply 로 적용한다.
controlplane ~ ✖ kubeadm upgrade plan v1.29.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.29.0
[upgrade/versions] Target version: v1.29.0
[upgrade/versions] Latest version in the v1.28 series: v1.29.0
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.28.0 v1.29.0
Upgrade to the latest version in the v1.28 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.28.0 v1.29.0
kube-controller-manager v1.28.0 v1.29.0
kube-scheduler v1.28.0 v1.29.0
kube-proxy v1.28.0 v1.29.0
CoreDNS v1.10.1 v1.11.1
etcd 3.5.9-0 3.5.10-0
...
controlplane ~ ➜ kubeadm upgrade apply v1.29.0
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.29.0"
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.29.0
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
...
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.29.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
9-7. kubelet 재기동.
controlplane ~ ➜ apt-get install kubelet=1.29.0-1.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubelet
1 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.
Need to get 19.8 MB of archives.
After this operation, 1,044 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb kubelet 1.29.0-1.1 [19.8 MB]
Fetched 19.8 MB in 0s (45.8 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 20477 files and directories currently installed.)
Preparing to unpack .../kubelet_1.29.0-1.1_amd64.deb ...
Unpacking kubelet (1.29.0-1.1) over (1.28.0-1.1) ...
Setting up kubelet (1.29.0-1.1) ...
controlplane ~ ➜ systemctl daemon-reload
controlplane ~ ➜ systemctl restart kubelet
controlplane ~ ➜ kubectl uncordon controlplane
node/controlplane uncordoned
10. Mark the controlplane node as "Schedulable" again
- 위 예제에서 uncordon 을 통해 스케쥴링 되도록 수정되었다.
11. Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable
ontrolplane ~ ✖ k drain node01 --ignore-daemonsets
node/node01 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-w5d5j, kube-system/kube-proxy-zhwcw
evicting pod kube-system/coredns-76f75df574-5w7hd
evicting pod default/blue-667bf6b9f9-tp6gn
evicting pod kube-system/coredns-76f75df574-5v5dw
evicting pod default/blue-667bf6b9f9-bs7fw
evicting pod default/blue-667bf6b9f9-xcr75
evicting pod default/blue-667bf6b9f9-sgthr
evicting pod default/blue-667bf6b9f9-csppb
pod/blue-667bf6b9f9-tp6gn evicted
pod/blue-667bf6b9f9-csppb evicted
pod/blue-667bf6b9f9-bs7fw evicted
pod/blue-667bf6b9f9-xcr75 evicted
I0711 10:26:16.928123 26119 request.go:697] Waited for 1.046296325s due to client-side throttling, not priority and fairness, request: GET:https://controlplane:6443/api/v1/namespaces/kube-system/pods/coredns-76f75df574-5w7hd
pod/blue-667bf6b9f9-sgthr evicted
pod/coredns-76f75df574-5v5dw evicted
pod/coredns-76f75df574-5w7hd evicted
node/node01 drained
controlplane ~ ➜ k get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 68m v1.29.0
node01 Ready,SchedulingDisabled <none> 67m v1.28.0
12. Upgrade the worker node to the exact version v1.29.0
worker 그룹 업그레이드를 위해서는 node01 로 ssh 접속후에 10 에서 진행한 내용을 동일하게 진행해준다.
root@node01 ~ ➜ apt update
root@node01 ~ ➜ apt-cache madison kubeadm
kubeadm | 1.29.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.4-2.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
kubeadm | 1.29.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb Packages
root@node01 ~ ➜ apt-get install kubeadm=1.29.0-1.1
Reading package lists... Done
...
root@node01:~# kubeadm upgrade node
root@node01:~# apt-get install kubelet=1.29.0-1.1
root@node01:~# systemctl daemon-reload
root@node01:~# systemctl restart kubelet
13. Remove the restriction and mark the worker node as schedulable again.
root@node01 ~ ➜ exit
logout
Connection to node01 closed.
controlplane ~ ➜ k uncordon node01
node/node01 uncordoned
'IT 기술 > k8s' 카테고리의 다른 글
| [cka] Backup and Restore Methods 2 (1) | 2024.07.12 |
|---|---|
| [cka] Backup and Restore Methods (0) | 2024.07.12 |
| [cka] OS Upgrades (0) | 2024.07.10 |
| [cka] Init Containers (0) | 2024.07.06 |
| [cka] Multi Container PODs (0) | 2024.07.05 |
댓글