본문 바로가기
Container/Kubernetes

[K8S] TS - Application Failure

융또융 2024. 4. 28. 03:22

🛠️Pod가 계속 Waiting 상태인 경우 

→ worker node에 스케줄링은 되었지만 해당 노드에서 실행할 수 없음을 의미한다. (일반적으로 이미지 풀링(pulling)에 실패)

🛠️ Pod는 Running 상태이지만 해야 할 일을 하지 않고 있을 경우

→ 명령어 오타 확인

# --validate 옵션으로 yaml 확인 가능
$ kubectl apply --validate -f mypod.yaml

🛠️ Service 디버깅 (svc는 파드 집합에 대한 로드 밸런싱 기능을 제공)

→ Service의 Endpoint 존재하는지 확인 (apiserver는 endpoints 리소스를 생성하고 available 상태로 만든다)

→ Endpoint 수가 서비스에 속하는 파드 수와 일치하는 지 확인

→ Service에 Endpoint가 없다면, 서비스가 사용중인 Label 확인 및 파드 목록을 조회해본다.

→ 파드의 containerPort가 서비스의 targetPort와 일치하는지 확인

참고 :https://kubernetes.io/ko/docs/tasks/debug/debug-application/

강의: certified-kubernetes-administrator-with-practice-tests 294강

 

  1. curl을 이용해서 application에 접근 가능한 지 확인한다.   ex. curl http://web-service-ip:node-port
  2. Service status를 확인한다.
  3. Pod를 확인한다.
  4. Dependent Service와 Application을 확인한다.

 

Troubleshooting Test 1:
 A simple 2 tier application is deployed in the alpha namespace.
It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

 

30081 NodePort

 

# 상태 확인

# Current namespace 변경
controlplane ~ ➜  kubectl config set-context --current --namespace=alpha
Context "default" modified.

controlplane ~ ➜  kubectl get pods -
NAME                           READY   STATUS    RESTARTS   AGE
webapp-mysql-b68bb6bc8-x98qm   1/1     Running   0          6m16s
mysql                          1/1     Running   0          6m16s

controlplane ~ ➜  kubectl get deploy
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
webapp-mysql   1/1     1            1           7m7s

controlplane ~ ➜  kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql         ClusterIP   10.43.124.40    <none>        3306/TCP         7m17s
web-service   NodePort    10.43.194.139   <none>        8080:30081/TCP   7m17s

# 노드포트에 접근 가능한지 확인
controlplane ~ ➜  curl http://localhost:30081
<!doctype html>
<title>Hello from Flask</title>
<body style="background: #ff3f3f;"></body>
<div style="color: #e4e4e4;
    text-align:  center;
    height: 90px;
    vertical-align:  middle;">
    <img src="/static/img/failed.png">
    <!-- <h1> DATABASE CONNECTION FAILED !!</h1> -->
    <h2> Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd; 2003: Can&#39;t connect to MySQL server on &#39;mysql-service:3306&#39; (-2 Name does not resolve) </h2>
    <p> From webapp-mysql-b68bb6bc8-x98qm!</p>
</div>

 

  • Deployment와 svc가 제대로 연결되어 있는지 확인
controlplane ~ ➜  kubectl describe deploy webapp-mysql 
Name:                   webapp-mysql
Namespace:              alpha
CreationTimestamp:      Sun, 28 Apr 2024 03:42:26 +0000
Labels:                 name=webapp-mysql
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               name=webapp-mysql
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  name=webapp-mysql
  Containers:
   webapp-mysql:
    Image:      mmumshad/simple-webapp-mysql
    Port:       8080/TCP
    Host Port:  0/TCP
    Environment:
      DB_Host:      mysql-service       # 해당 이름으로 접근하기 때문에 svc와 이름이 동일해야한다.
      DB_User:      root
      DB_Password:  paswrd
    Mounts:         <none>
  Volumes:          <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   webapp-mysql-b68bb6bc8 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  9m16s  deployment-controller  Scaled up replica set webapp-mysql-b68bb6bc8 to 1

controlplane ~ ➜  kubectl get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql         ClusterIP   10.43.124.40    <none>        3306/TCP         10m    # 이 친구가 매치되어야 한다.
web-service   NodePort    10.43.194.139   <none>        8080:30081/TCP   10m
# 애플리케이션이 데이터베이스에 연결할 수 없다.

controlplane ~ ➜  kubectl edit svc mysql
A copy of your changes has been stored to "/tmp/kubectl-edit-2834438716.yaml"
error: At least one of apiVersion, kind and name was changed


controlplane ~ ➜ cat /tmp/kubectl-edit-2834438716.yaml
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.


# apiVersion, kind, metadata.name와 같은 필드는 쿠버네티스 리소스를 식별하는 데 필수적 요소라 kubectl edit으로 변경되지 않는다.
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-04-28T03:42:26Z"
  name: mysql-service  
  namespace: alpha
  resourceVersion: "890"
  uid: 4fd070d8-ba3e-4521-94e7-ebf5dd0e5555
spec:
  clusterIP: 10.43.124.40
  clusterIPs:
  - 10.43.124.40
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    name: mysql
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

controlplane ~ ➜  kubectl delete svc mysql 
service "mysql" deleted

controlplane ~ ➜  kubectl create -f /tmp/kubectl-edit-2834438716.yaml
service/mysql-service created

controlplane ~ ➜  kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
web-service     NodePort    10.43.194.139   <none>        8080:30081/TCP   13m
mysql-service   ClusterIP   10.43.124.40    <none>        3306/TCP         7s

# Success로 바뀜!
controlplane ~ ➜  curl http://localhost:30081
<!doctype html>
<title>Hello from Flask</title>
<body style="background: #39b54b;"></body>
<div style="color: #e4e4e4;
    text-align:  center;
    height: 90px;
    vertical-align:  middle;">
    <!-- <h1> DATABASE CONNECTION SUCCESSFUL !</h1> -->
    <img src="/static/img/success.jpg">
    <h2> Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd;  </h2>
    <p> From webapp-mysql-b68bb6bc8-x98qm!</p>
</div>

 

Troubleshooting Test 2:
The same 2 tier application is deployed in the beta namespace.
It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.

 

# namespace 변경
controlplane ~ ➜ kubectl config set-context --current --namespace=beta
Context "default" modified.

controlplane ~ ➜  kubectl describe svc mysql-service 
Name:              mysql-service
Namespace:         beta
Labels:            <none>
Annotations:       <none>
Selector:          name=mysql
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.118.105
IPs:               10.43.118.105
Port:              <unset>  3306/TCP
TargetPort:        8080/TCP
Endpoints:         10.42.0.12:8080 # <mysql IP>:<mysql의 port>
Session Affinity:  None
Events:            <none>
# 8080은 DB의 Port가 아니다.

controlplane ~ ➜  kubectl get pods -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
mysql                          1/1     Running   0          16m   10.42.0.12   controlplane   <none>           <none> # IP
webapp-mysql-b68bb6bc8-g4sj9   1/1     Running   0          16m   10.42.0.11   controlplane   <none>           <none>

controlplane ~ ➜  kubectl edit svc mysql-service 
service/mysql-service edited
# Port를 3306으로 바꿔준다.

controlplane ~ ➜  kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
web-service     NodePort    10.43.126.248   <none>        8080:30081/TCP   18m
mysql-service   ClusterIP   10.43.118.105   <none>        3306/TCP         18m

controlplane ~ ➜  kubectl describe svc mysql-service 
Name:              mysql-service
Namespace:         beta
Labels:            <none>
Annotations:       <none>
Selector:          name=mysql
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.118.105
IPs:               10.43.118.105
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.42.0.12:3306  # 바뀐 부분!
Session Affinity:  None
Events:            <none>

 

  • Port: 서비스 자체에서 외부로 노출되는 포트이다. 클라이언트(다른 Pod 또는 외부 요청)가 이 포트를 통해 서비스에 요청을 보낸다. → 즉, 클라이언트가 접근하는 서비스 포트 !
  • Target Port: 서비스가 실제로 트래픽을 라우팅할 때 사용하는 Pod 내 컨테이너의 포트
    • 서비스는 클라이언트로부터 받은 요청을 targetPort로 지정된 포트로 전달한다.
    • Pod 내 컨테이너가 수신하는 포트
Troubleshooting Test 3:
The same 2 tier application is deployed in the gamma namespace.
It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed or unresponsive. Troubleshoot and fix the issue.

 

controlplane ~ ➜  kubectl config set-context --current --namespace=gamma
Context "default" modified.

controlplane ~ ➜  kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql-service   ClusterIP   10.43.78.129    <none>        3306/TCP         4m5s
web-service     NodePort    10.43.121.145   <none>        8080:30081/TCP   4m5s

controlplane ~ ➜  kubectl describe svc mysql-service 
Name:              mysql-service
Namespace:         gamma
Labels:            <none>
Annotations:       <none>
Selector:          name=sql00001 # Label 확인해보자
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.78.129
IPs:               10.43.78.129
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         <none> # 왜 없는지 확인해보자.
Session Affinity:  None
Events:            <none>


controlplane ~ ➜  kubectl describe pod mysql 
Name:             mysql
Namespace:        gamma
Priority:         0
Service Account:  default
Node:             controlplane/192.4.146.3
Start Time:       Sun, 28 Apr 2024 04:16:47 +0000
Labels:           name=mysql      # Label이 mysql이다. Endpoints가 없던 이유.
Annotations:      <none>
Status:           Running
IP:               10.42.0.13
IPs:
  IP:  10.42.0.13
# ->  webapp이 mysql-service에 접근할 수 있어야하는데 service가 not responding이었음  

controlplane ~ ➜  kubectl edit svc mysql-service 
service/mysql-service edited  
  
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-04-28T04:16:47Z"
  name: mysql-service
  namespace: gamma
  resourceVersion: "1697"
  uid: d7665711-4437-459c-b75f-f0a68d852b95
spec:
  clusterIP: 10.43.78.129
  clusterIPs:
  - 10.43.78.129
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    name: mysql       # 여기를 바꿔주면 된다.
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
  
  
  
controlplane ~ ➜  kubectl describe svc mysql-service 
Name:              mysql-service
Namespace:         gamma
Labels:            <none>
Annotations:       <none>
Selector:          name=mysql
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.78.129
IPs:               10.43.78.129
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.42.0.13:3306          # 생긴 것 확인 가능
Session Affinity:  None
Events:            <none>

 

Troubleshooting Test 4:
The same 2 tier application is deployed in the delta namespace.
It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue

Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=sql-user; DB_Password=paswrd; 1045 (28000): Access denied for user 'sql-user'@'10.42.0.16' (using password: YES)

 

controlplane ~ ➜  kubectl config set-context --current --namespace=delta
Context "default" modified.

controlplane ~ ➜  kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
mysql                          1/1     Running   0          98s
webapp-mysql-785cd8f94-b8ln7   1/1     Running   0          98s

controlplane ~ ➜  kubectl get svc
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
mysql-service   ClusterIP   10.43.143.93   <none>        3306/TCP         102s
web-service     NodePort    10.43.91.160   <none>        8080:30081/TCP   102s

controlplane ~ ➜  kubectl describe deployments.apps webapp-mysql 
Name:                   webapp-mysql
Namespace:              delta
CreationTimestamp:      Sun, 28 Apr 2024 04:29:22 +0000
Labels:                 name=webapp-mysql
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               name=webapp-mysql
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  name=webapp-mysql
  Containers:
   webapp-mysql:
    Image:      mmumshad/simple-webapp-mysql
    Port:       8080/TCP
    Host Port:  0/TCP
    Environment:
      DB_Host:      mysql-service
      DB_User:      sql-user  # 여기가 root 여야 한다.
      DB_Password:  paswrd
    Mounts:         <none>
  Volumes:          <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   webapp-mysql-785cd8f94 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3m28s  deployment-controller  Scaled up replica set webapp-mysql-785cd8f94 to 1
  
controlplane ~ ➜  kubectl edit deployments.apps webapp-mysql 
deployment.apps/webapp-mysql edited



controlplane ~ ➜  kubectl describe deployments.apps webapp-mysql 
Name:                   webapp-mysql
Namespace:              delta
CreationTimestamp:      Sun, 28 Apr 2024 04:29:22 +0000
Labels:                 name=webapp-mysql
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               name=webapp-mysql
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  name=webapp-mysql
  Containers:
   webapp-mysql:
    Image:      mmumshad/simple-webapp-mysql
    Port:       8080/TCP
    Host Port:  0/TCP
    Environment:
      DB_Host:      mysql-service
      DB_User:      root # 바꿔주니 정상 동작 가능.
      DB_Password:  paswrd
    Mounts:         <none>
  Volumes:          <none>
Conditions:

 

Troubleshooting Test 5:
The same 2 tier application is deployed in the epsilon namespace.
It must display a green web page on success. Click on the App tab at the top of your terminal to view your application. It is currently failed. Troubleshoot and fix the issue.


Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=sql-user; DB_Password=paswrd; 1045 (28000): Access denied for user 'sql-user'@'10.42.0.19' (using password: YES)
controlplane ~ ➜  kubectl edit deploy webapp-mysql 
deployment.apps/webapp-mysql edited

# 잘못된 유저를 바꿔주니 에러가 바꼈다
Environment Variables: DB_Host=mysql-service; DB_Database=Not Set; DB_User=root; DB_Password=paswrd; 1045 (28000): Access denied for user 'root'@'10.42.0.20' (using password: YES)

controlplane ~ ➜  kubectl describe pod mysql 
Name:             mysql
Namespace:        epsilon
Priority:         0
Service Account:  default
Node:             controlplane/192.5.179.9
Start Time:       Sun, 28 Apr 2024 04:44:52 +0000
Labels:           name=mysql
Annotations:      <none>
Status:           Running
IP:               10.42.0.17
IPs:
  IP:  10.42.0.17
Containers:
  mysql:
    Container ID:   containerd://6e0a4479da2f8c5eaa95a85ca305c177b8b12c4923196222d243a2be77bc23e6
    Image:          mysql:5.6
    Image ID:       docker.io/library/mysql@sha256:20575ecebe6216036d25dab5903808211f1e9ba63dc7825ac20cb975e34cfcae
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 28 Apr 2024 04:44:54 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      MYSQL_ROOT_PASSWORD:  passwooooorrddd # 오류 발견
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-szjp6 (ro)
      
controlplane ~ ➜  kubectl edit pod mysql 
error: pods "mysql" is invalid
A copy of your changes has been stored to "/tmp/kubectl-edit-2226943367.yaml"
error: Edit cancelled, no valid changes were saved.

controlplane ~ ✖ kubectl replace --force -f /tmp/kubectl-edit-2226943367.yaml
pod "mysql" deleted
pod/mysql replaced

controlplane ~ ➜  kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
webapp-mysql-b68bb6bc8-wrj4v   1/1     Running   0          2m1s
mysql                          1/1     Running   0          11s

 

반응형

'Container > Kubernetes' 카테고리의 다른 글

[K8S] TS - Worker Node Failure  (0) 2024.04.28
[K8S] TS - Control Plane Failure  (0) 2024.04.28
[K8S] Multiple Container  (0) 2024.04.28
[K8S] Cluster Upgrade  (0) 2024.04.26
[K8S] Logging & Monitoring  (1) 2024.04.26

'Container/Kubernetes' Related Articles

티스토리툴바