Issue#

About a year after you build a cluster with kubeadm, kubectl stops working. The API server also stops accepting connections from the scheduler and the controller-manager:

Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-11-06T21:33:10Z is after 2026-11-01T09:12:45Z

The kubelet logs (journalctl -u kubelet) can show the same x509 errors.

Cause#

By default, kubeadm signs the control plane certificates (apiserver, etcd, front-proxy, and the client certs inside admin.conf, controller-manager.conf and scheduler.conf) for 1 year. The CA certificates last 10 years.

kubeadm renews these certificates automatically during kubeadm upgrade. If you never upgraded the cluster within the year, they expire.

Solution#

Run these steps on every control plane node.

  1. Check which certificates have expired:
sudo kubeadm certs check-expiration
  1. Back up the current certificates and kubeconfig files:
sudo cp -a /etc/kubernetes /root/kubernetes-backup-$(date +%F)
  1. Renew all certificates, including the ones embedded in the kubeconfig files:
sudo kubeadm certs renew all
  1. Restart the control plane static pods so they load the new certificates. The kubelet stops a static pod when its manifest is removed and starts it again when the manifest comes back:
sudo mkdir -p /tmp/manifests sudo mv /etc/kubernetes/manifests/*.yaml /tmp/manifests/ sleep 20 sudo mv /tmp/manifests/*.yaml /etc/kubernetes/manifests/
  1. Replace your local kubeconfig with the renewed admin config:
mkdir -p $HOME/.kube sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
  1. If kubelet still logs x509 errors, restart it:
sudo systemctl restart kubelet

Tip: Any other machine (a CI runner, your laptop) that uses a kubeconfig copied from the old admin.conf also needs the new file.

Verify#

sudo kubeadm certs check-expiration

Every certificate in the EXPIRES column should now show a date about one year from today, and RESIDUAL TIME should show about 364d.

kubectl get nodes kubectl get pods -n kube-system

The nodes should be Ready, and kube-apiserver, kube-controller-manager, kube-scheduler and etcd should be Running. Their AGE should show they just restarted.

Tip: To stop this from happening again, upgrade the cluster at least once a year, or schedule kubeadm certs check-expiration to warn you early.