How To Backup And Restore ETCD On Kubernetes Cluster?

In this article, I am going to give a step-by-step guide to taking backup and restore of the ETCD on the Kubernetes cluster.

For this, I am taking the Kubeadm v1.28.0 version cluster but you can do it in any version.

How To Take ETCD Backup?

It is the responsibility of a Kubernetes admin to take regular backups of the ETCD because all Kubernetes objects are stored on ETCD.

Step 1: Find The ETCD Manifest

Before we perform the backup we need some file path information such as truster-ca-file, cert-file and key-file.

Ensure you are in the Master Node (controlplane) if not, first, SSH into the Master Node.

ETCD manifest will be available in the /etc/kubernetes/manifests directory.

Let’s open the manifest.

sudo cat /etc/kubernetes/manifests/etcd.yaml

Our required information will be obtained from the following part of the file.

Step 2: ETCD Snapshot

To take a snapshot, use the following command.

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Here, /opt/etcd-backup.db is the path where I want to store the backup, you can choose your own destination and change the name as you wish but the .db extension is necessary.

The remaining information is the default for almost every cluster, so you can also use the same information.

If the backup is successfully done, you can also see the same output.

To verify the snapshot, use the following command.

ETCDCTL_API=3 etcdctl --write-out=table snapshot status /opt/etcd-backup.db

How to Restore ETCD?

Step 1: Restore ETCD From Snapshot

First, you need to know on which path the snapshot is, if you are preparing for the CKA exam, you might be asked to restore the ETCD and for that, they will provide the path of the snapshot.

For restoration, we need one information, which is data-dir, that we can take from the ETCD manifest.

sudo cat /etc/kubernetes/manifests/etcd.yaml

To restore the ETCD, use the following command.

ETCDCTL_API=3 etcdctl --data-dir /var/lib/etcd snapshot restore /opt/etcd-backup.db

If the restoring process is properly done, you will get this same output.

To ensure everything is properly running, check the Pods on the kube-system namespace.

kubectl get pods -n kube-system

Possible Errors & Troubleshoot

Error: Error: data-dir “/var/lib/etcd” not empty or could not be read

When you try to restore the ETCD, you might probably get this error.

Error: data-dir "/var/lib/etcd" not empty or could not be read

Solution:

Remove the /var/lib/etcd and perform the restore again.

Conclusion

For testing purposes, if you want to know the changes after the restore, then create a Pod on the cluster before performing the restore so that after the restore if the process is successfully done, you won’t able to see the Pod.

Etcd backup is a very important concept in CKA certification. If you are preparing for CKA certification, make use of the CKA coupon form the Linux Foundation offer page.