[cka] TroubleShooting - Application Failure

1. Troubleshooting Test 1: A simple 2 tier application is deployed in the alpha namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed.

 

Troubleshoot and fix the issue. Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

controlplane ~ ✖ k get svc -n alpha 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql         ClusterIP   10.43.158.104   <none>        3306/TCP         10m
web-service   NodePort    10.43.26.166    <none>        8080:30081/TCP   10m

 

주어진 구성도로 보았을 때 mysql 의 서비스 명이 잘못되어있다. 

 

서비스 명 변경하기

---
apiVersion: v1
kind: Service
metadata:
  name: mysql-service
  namespace: alpha
spec:
    ports:
    - port: 3306
      targetPort: 3306
    selector:
      name: mysql

 

2.

Troubleshooting Test 2: The same 2 tier application is deployed in the beta namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

 

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

웹페이지 접속시 DB 접근이 잘 안되는 것으로 확인.

mysql/webap po 와 mysql-service 서비스 를 확인해 봄

pod 확인시에 따로 문제될 사항이 확인되지 않음.

서비스 확인시 아래와 같이 확인된다.

controlplane ~ ➜  k get svc -n beta mysql-service -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-08-25T09:46:46Z"
  name: mysql-service
  namespace: beta
  resourceVersion: "1019"
  uid: 76bd89ad-c2df-4c1b-a808-802c97ec3cb0
spec:
  clusterIP: 10.43.52.149
  clusterIPs:
  - 10.43.52.149
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 8080
  selector:
    name: mysql
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

ClusterIp 로 할당된 서비스에서 targetPort 가 3306 이 아닌 8080 으로 설정되어 있다. 

targetPort 를 8080 -> 3306 으로 수정해준다.

 

3. Troubleshooting Test 3: The same 2 tier application is deployed in the gamma namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed or unresponsive. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

 

이번에도 웹 접속시 디비 접근 애러가 발생한다.

 

먼저 webapp deployment 확인했을때 아무 이상이 없는것으로 보인다.

mysql po 설정도 문제가 없다.

 

그럼 service check.

 

mysql-service 를 먼저 체크해본다.

controlplane ~ ➜  k get svc -n gamma -o yaml mysql-service
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-08-25T09:52:47Z"
  name: mysql-service
  namespace: gamma
  resourceVersion: "1223"
  uid: 40735a66-bfd4-468d-b5a7-768a4a9dfc4c
spec:
  clusterIP: 10.43.196.138
  clusterIPs:
  - 10.43.196.138
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    name: sql00001
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

 

yaml 값으로 보았을때 spec.selector.name 이 잘못된 것이 확인되었다.

selector 값은 msyql 의 값의 name 으로 설정해주어야 해당 mysql po 가 서비스에 연결될 수 있다. 

 

controlplane ~ ➜  k get pods -n gamma mysql -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-08-25T09:52:47Z"
  labels:
    name: mysql
  name: mysql
  namespace: gamma
  resourceVersion: "1252"
  uid: 40735dd2-0f82-46bd-9023-31014592b98c
spec:
...

mysql 의 labels:name 은 mysql  이다.

 

k edit 를 이용하여 svc의 selector 를 수정해주자.

controlplane ~ ➜  k edit svc -n gamma mysql-service
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-08-25T09:52:47Z"
  name: mysql-service
  namespace: gamma
  resourceVersion: "1223"
  uid: 40735a66-bfd4-468d-b5a7-768a4a9dfc4c
spec:
  clusterIP: 10.43.196.138
  clusterIPs:
  - 10.43.196.138
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    name: sql00001 ### -> mysql 로 수정
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

 

 

 

4.

Troubleshooting Test 4: The same 2 tier application is deployed in the delta namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

 

웹페이지 접속시 아래와 같은 메시지가 출력되어 있다.

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=sql-user; DB_Password=paswrd; 1045 (28000): Access denied for user 'sql-user'@'10.42.0.15' (using password: YES)

 

주어진 환경에서는 root 유저로 접속해야 하는데 sql-user 로 웹에서 접근하고 있다.

 

이로서 우리는 웹 po 가 배포되어진 deployment 를 확인해야 한다.

 

controlplane ~ ➜  k get deployments.apps -o yaml -n delta webapp-mysql 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2024-08-25T09:59:55Z"
  generation: 1
  labels:
    name: webapp-mysql
  name: webapp-mysql
  namespace: delta
  resourceVersion: "1475"
  uid: 79a89341-9100-487d-95b6-c15d4b4cacf4
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: webapp-mysql
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: webapp-mysql
      name: webapp-mysql
    spec:
      containers:
      - env:
        - name: DB_Host
          value: mysql-service
        - name: DB_User
          value: sql-user
        - name: DB_Password
          value: paswrd
        image: mmumshad/simple-webap

 

여기서 continers 의 env 값에서 유저명이 잘못들어가 있는것을 확인하였다. 이를 edit 를 이용하여 수정해주자.

...
	spec:
      containers:
      - env:
        - name: DB_Host
          value: mysql-service
        - name: DB_User
          value: root # 수정
        - name: DB_Password
          value: paswrd

 

 

 

5. Troubleshooting Test 5: The same 2 tier application is deployed in the epsilon namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

이번에도 웹페이지 확인시에 sql-user 로 접속되어 있다. 이를 먼저 4번처럼 수정한다.

 

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd; 1045 (28000): Access denied for user 'root'@'10.42.0.20' (using password: YES)

 

유저명은 변경되었으나 여전히 접근되지 않는다. 그럼 이제 다른 pod 와 svc 를 체크해보자.

 

mysql pod 체크시 아래와 같이 확인이 된다.

controlplane ~ ➜  k get pods -n epsilon mysql -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-08-25T10:03:25Z"
  labels:
    name: mysql
  name: mysql
  namespace: epsilon
  resourceVersion: "1665"
  uid: 1c25deb2-02fb-4774-886d-52cb4dc9ecee
spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      value: passwooooorrddd
    image: mysql:5.6
    imagePullPolicy: IfNotPresent
    name: mysql
    ports:
    - containerPort: 3306
...

 패스워드가 잘못 적혀있다 . 이또한 수정해준다.

 

해당 pod 는 edit 로 수정시 바로 수정이 되어지지 않는다. 

tmp 에 저장된 edit 된 yaml 파일을 --force 를 통해 적용시켜 준다.

controlplane ~ ➜  k edit po -n epsilon mysql
error: pods "mysql" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-419945046.yaml"
error: Edit cancelled, no valid changes were saved.

controlplane ~ ✖ k apply -f /tmp/kubectl-edit-419945046.yaml --force
Warning: resource pods/mysql is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
pod/mysql configured

 

 

 

6.

Troubleshooting Test 6: The same 2 tier application is deployed in the zeta namespace. It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

Stick to the given architecture. Use the same names and port numbers as given in the below architecture diagram. Feel free to edit, delete or recreate objects as necessary.

 

 

 

이번엔 웹페이지 접근시 nginx 502 에러가 표시된다.

서비스에서 nodePort 정보를 확인하여 제대로 된 포트로 포워딩 되었는지 확인이 필요하다.

controlplane ~ ➜  k get svc -n zeta
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql-service   ClusterIP   10.43.108.118   <none>        3306/TCP         108s
web-service     NodePort    10.43.119.178   <none>        8080:30088/TCP   108s

 

서비스 확인시 주어진 환경에서는 30081 로 포워딩 해야 하나 30088 로 포워딩 되어있다. 이를 먼저 수정한다.

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd; 1045 (28000): Access denied for user 'root'@'10.42.0.20' (using password: YES)

수정후 웹페이지는 접근 되나 또 다시 디비 접속 에러가 나타난다. 위 상황들과 동일하게 msyql  po 와 deployment를 확인하여 제대로된 디비 정보가 들어갔는지 확인해보자.

 

controlplane ~ ✖ k get po -n zeta mysql -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-08-25T10:10:07Z"
  labels:
    name: mysql
  name: mysql
  namespace: zeta
  resourceVersion: "1940"
  uid: 31457c3d-5a8e-46e2-9245-5e251ee23f67
spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      value: passwooooorrddd

 

mysql po 의 비번정보가 잘못되어 있다. 수정하자.

controlplane ~ ➜  k edit po -n zeta mysql
error: pods "mysql" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-3131677932.yaml"
error: Edit cancelled, no valid changes were saved.

controlplane ~ ✖ k apply -f /tmp/kubectl-edit-3131677932.yaml --force
Warning: resource pods/mysql is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
pod/mysql configured

 

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=sql-user; DB_Password=paswrd; 2003: Can't connect to MySQL server on 'mysql-service:3306' (111 Connection refused)

 

수정후 접근시 또 sql-user 로 접속중이다. deployments 의 유저명을 변경하고 다시 적용한다.

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd;

From webapp-mysql-9b567db7f-2gx6v!

 

접속 성공.