All DevOps engineers have or will come across this error during their time managing kubernetes clusters. I have compiled my findings and solution to save you all from the many hours of debugging.

Recently we were trying to restart a deployment using kubectl because we wanted to update its image. To our dismay, we were shocked by this error message.

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-08-03T10:05:37+03:00 is after 2023-08-02T09:03:12Z Kubernetes

From just reading that error it's clear it's a certificate problem or a Public Key Infrastructure (PKI) issue. To understand this problem let's first understand why Kubernetes needs PKI.

Kubernetes requires PKI for the following operations:

• Client certificates for the kubelet to authenticate to the API server

• Kubelet server certificates for the API server to talk to the kubelets

• Server certificate for the API server endpoint

• Client certificates for administrators of the cluster to authenticate to the API server.

• Client certificates for the API server to talk to the kubelets

• Client certificate for the API server to talk to etcd

• Client certificate/kubeconfig for the controller manager to talk to the API server

• Client certificate/kubeconfig for the scheduler to talk to the API server.

• Client and server certificates for the front-proxy

The error encountered above suggests that a certificate is expired which used for establishing a secure connection to the api-server .

The solution to this problem is to renew the certificates that are used by kubernetes, restart the control plane pods and update our kubernetes config usually located at ~/.kube/config with the updated certificates to renew access to the cluster.

Solution !!

Here is the solution step by step together with the commands needed to resolve this issue.

1. First of all ssh into the control plane and make sure you can execute commands with sudo.

2. Execute this command to renew the certificates that are expired.

   sudo kubeadm certs renew all

3. This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.

   After running the command you should restart the control plane Pods. This is required since dynamic certificate reload is currently not supported for all components and certificates. Static Pods are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them. To restart a static Pod you can temporarily remove its manifest file from /etc/kubernetes/manifests/ and wait for 20 seconds (see the fileCheckFrequency value in KubeletConfiguration struct. The kubelet will terminate the Pod if it's no longer in the manifest directory. You can then move the file back and after another fileCheckFrequency period, the kubelet will recreate the Pod and the certificate renewal for the component can complete.

4. Run the following commands to update the contents of your ~/.kube/config file to load the new certificates.

    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
   

5. After running these commands you should now be able to run kubectl commands again with no errors about expired certificates.

Conclusion

Kubernetes requires PKI certificates for authentication over TLS. The renewal of certificates can be done automatically and you can find how to do it on this page.
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal