Question:
There aren’t any other pods deployed. I have 2 nodes with zero pods. Its a t3.small instance type.
Is there any other commands to debug and see why it is still stuck in pending state?
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
kubectl get pods NAME READY STATUS RESTARTS AGE mysql-6cc489dcd5-5jc8t 0/1 Pending 0 91s kubectl describe pod mysql-6cc489dcd5-5jc8t Name: mysql-6cc489dcd5-5jc8t Namespace: default Priority: 0 Node: Labels: app=mysql pod-template-hash=6cc489dcd5 Annotations: kubernetes.io/psp: eks.privileged Status: Pending IP: IPs: Controlled By: ReplicaSet/mysql-6cc489dcd5 Containers: mysql: Image: mysql:5.6 Port: 3306/TCP Host Port: 0/TCP Environment: MYSQL_ROOT_PASSWORD: dbpassword11 Mounts: /docker-entrypoint-initdb.d from usermanagement-dbcreation-script (rw) /var/lib/mysql from mysql-persistent-storage (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-nlx4p (ro) Volumes: mysql-persistent-storage: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: ebs-mysql-pv-claim ReadOnly: false usermanagement-dbcreation-script: Type: ConfigMap (a volume populated by a ConfigMap) Name: usermanagement-dbcreation-script Optional: false default-token-nlx4p: Type: Secret (a volume populated by a Secret) SecretName: default-token-nlx4p Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: |
Deployments shows that it has scaled and nothing wrong there.
|
1 2 3 4 5 |
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 5m34s deployment-controller Scaled up replica set mysql-6cc489dcd5 to 1 |
cat 01-storage-class.yml
|
1 2 3 4 5 6 7 8 |
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer |
cat 02-persistent-volume-claim.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ebs-mysql-pv-claim spec: accessModes: - ReadWriteOnce storageClassName: ebs-sc # Metadata name of the storage class resources: requests: storage: 4Gi |
cat 03-UserManagement-ConfigMap.yml
|
1 2 3 4 5 6 7 8 9 10 |
apiVersion: v1 kind: ConfigMap metadata: name: usermanagement-dbcreation-script data: mysql_usermgt.sql: |- DROP DATABASE IF EXISTS usermgmt; CREATE DATABASE usermgt; |
cat 04-mysql-deployment.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: replicas: 1 selector: matchLabels: app: mysql strategy: type: Recreate template: metadata: name: mysql-deployment labels: app: mysql spec: containers: - name: mysql image: mysql:5.6 env: - name: MYSQL_ROOT_PASSWORD value: dbpassword11 ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage # name from the volumes listed below mountPath: /var/lib/mysql - name: usermanagement-dbcreation-script # name from the volumes listed below mountPath: /docker-entrypoint-initdb.d volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: ebs-mysql-pv-claim - name: usermanagement-dbcreation-script configMap: name: usermanagement-dbcreation-script |
cat 05-mysql-clusterip-service.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
apiVersion: v1 kind: Service metadata: name: mysql spec: #type: # Defaults to cluster IP service selector: app: mysql ports: - name: mysql port: 3306 # If name isn't provided, port can start as a list `- port: 3306` #targetPort # Defaults to containerPort clusterIP: None # Use POD IP directly |
kubectl describe nodes
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
Name: ip-192-168-14-238.ap-south-1.compute.internal Roles: Labels: alpha.eksctl.io/cluster-name=eksdemo1 alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=t2.small beta.kubernetes.io/os=linux eks.amazonaws.com/capacityType=ON_DEMAND eks.amazonaws.com/nodegroup=eksdemo1-ng-public1 eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25 eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5 eks.amazonaws.com/sourceLaunchTemplateVersion=1 failure-domain.beta.kubernetes.io/region=ap-south-1 failure-domain.beta.kubernetes.io/zone=ap-south-1a kubernetes.io/arch=amd64 kubernetes.io/hostname=ip-192-168-14-238.ap-south-1.compute.internal kubernetes.io/os=linux node.kubernetes.io/instance-type=t2.small topology.ebs.csi.aws.com/zone=ap-south-1a topology.kubernetes.io/region=ap-south-1 topology.kubernetes.io/zone=ap-south-1a Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0b3699bcac4e70d31"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 25 Nov 2020 05:05:47 -0500 Taints: Unschedulable: false Lease: HolderIdentity: ip-192-168-14-238.ap-south-1.compute.internal AcquireTime: RenewTime: Thu, 26 Nov 2020 01:41:17 -0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:05:47 -0500 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 26 Nov 2020 01:40:40 -0500 Wed, 25 Nov 2020 05:06:17 -0500 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.14.238 ExternalIP: 15.206.94.127 Hostname: ip-192-168-14-238.ap-south-1.compute.internal InternalDNS: ip-192-168-14-238.ap-south-1.compute.internal ExternalDNS: ec2-15-206-94-127.ap-south-1.compute.amazonaws.com Capacity: attachable-volumes-aws-ebs: 39 cpu: 1 ephemeral-storage: 20959212Ki hugepages-2Mi: 0 memory: 2039136Ki pods: 11 Allocatable: attachable-volumes-aws-ebs: 39 cpu: 940m ephemeral-storage: 18242267924 hugepages-2Mi: 0 memory: 1551712Ki pods: 11 System Info: Machine ID: 10f1040dde9f47289c7e10ea3fb9f040 System UUID: EC2C067A-2E0E-8A49-76C1-7F5FFA16E7ED Boot ID: d2638886-d5b3-4fa6-a15b-e95e046397f5 Kernel Version: 4.14.203-156.332.amzn2.x86_64 OS Image: Amazon Linux 2 Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.6 Kubelet Version: v1.18.9-eks-d1db3c Kube-Proxy Version: v1.18.9-eks-d1db3c ProviderID: aws:///ap-south-1a/i-0b3699bcac4e70d31 Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system aws-node-zn9v5 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system coredns-6b4cdc67b4-pzrl7 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h kube-system ebs-csi-controller-7bf8974dc8-cth9n 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system ebs-csi-node-2vr6j 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system kube-proxy-dd4wd 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 210m (22%) 0 (0%) memory 70Mi (4%) 170Mi (11%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) attachable-volumes-aws-ebs 0 0 Events: Name: ip-192-168-57-231.ap-south-1.compute.internal Roles: Labels: alpha.eksctl.io/cluster-name=eksdemo1 alpha.eksctl.io/nodegroup-name=eksdemo1-ng-public1 beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=t2.small beta.kubernetes.io/os=linux eks.amazonaws.com/capacityType=ON_DEMAND eks.amazonaws.com/nodegroup=eksdemo1-ng-public1 eks.amazonaws.com/nodegroup-image=ami-01cfab2ec59e55c25 eks.amazonaws.com/sourceLaunchTemplateId=lt-02527e7be817e66a5 eks.amazonaws.com/sourceLaunchTemplateVersion=1 failure-domain.beta.kubernetes.io/region=ap-south-1 failure-domain.beta.kubernetes.io/zone=ap-south-1b kubernetes.io/arch=amd64 kubernetes.io/hostname=ip-192-168-57-231.ap-south-1.compute.internal kubernetes.io/os=linux node.kubernetes.io/instance-type=t2.small topology.ebs.csi.aws.com/zone=ap-south-1b topology.kubernetes.io/region=ap-south-1 topology.kubernetes.io/zone=ap-south-1b Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0317d2c406ad6034d"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 25 Nov 2020 05:05:49 -0500 Taints: Unschedulable: false Lease: HolderIdentity: ip-192-168-57-231.ap-south-1.compute.internal AcquireTime: RenewTime: Thu, 26 Nov 2020 01:41:17 -0500 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:05:49 -0500 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 26 Nov 2020 01:40:39 -0500 Wed, 25 Nov 2020 05:06:20 -0500 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.57.231 ExternalIP: 52.66.246.122 Hostname: ip-192-168-57-231.ap-south-1.compute.internal InternalDNS: ip-192-168-57-231.ap-south-1.compute.internal ExternalDNS: ec2-52-66-246-122.ap-south-1.compute.amazonaws.com Capacity: attachable-volumes-aws-ebs: 39 cpu: 1 ephemeral-storage: 20959212Ki hugepages-2Mi: 0 memory: 2039136Ki pods: 11 Allocatable: attachable-volumes-aws-ebs: 39 cpu: 940m ephemeral-storage: 18242267924 hugepages-2Mi: 0 memory: 1551712Ki pods: 11 System Info: Machine ID: a14b97ab355f485294e0e6bd736775af System UUID: EC2D6C5A-497D-E6A4-F235-CF3055FCB546 Boot ID: c37c79e9-4bb1-4ffb-810f-80a4405cb403 Kernel Version: 4.14.203-156.332.amzn2.x86_64 OS Image: Amazon Linux 2 Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.6 Kubelet Version: v1.18.9-eks-d1db3c Kube-Proxy Version: v1.18.9-eks-d1db3c ProviderID: aws:///ap-south-1b/i-0317d2c406ad6034d Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system aws-node-qnjzl 10m (1%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system coredns-6b4cdc67b4-whszf 100m (10%) 0 (0%) 70Mi (4%) 170Mi (11%) 20h kube-system ebs-csi-controller-7bf8974dc8-cs6p8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system ebs-csi-node-2qwjq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 20h kube-system kube-proxy-pkxc9 100m (10%) 0 (0%) 0 (0%) 0 (0%) 20h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 210m (22%) 0 (0%) memory 70Mi (4%) 170Mi (11%) ephemeral-storage 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) attachable-volumes-aws-ebs 0 0 Events: |
Answer:
In this setup, I would first suspect something wrong with the persistent storage. Do a describe on the Persistent Volume Claim and Storage Class to see whether they are alright.
Also, if this is an EKS cluster, you can use the preinstalled EBS storage class.