Kubernetes Day-to-Day Administration Manual

Upgrading a Kubernetes cluster

For cluster upgrades, follow the official documentation: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

Maintaining or removing a Kubernetes node

If a node needs maintenance or must be removed from the cluster entirely, follow these steps.

• 1. Mark the Node as unschedulable

root@linux-node1:~# kubectl cordon linux-node3.example.com

• 1. Check the Node status and confirm SchedulingDisabled appears

root@linux-node1:~# kubectl get node
NAME                      STATUS                     ROLES           AGE     VERSION
linux-node1.example.com   Ready                      control-plane   2d17h   v1.31.4
linux-node2.example.com   Ready                      worker          2d17h   v1.31.4
linux-node3.example.com   Ready,SchedulingDisabled   worker          2d17h   v1.31.4

• 1. Evict all Pods on that node

kubectl drain linux-node3.example.com --force --ignore-daemonsets

You can now maintain, reboot, or upgrade the node.

• 1. Delete the node if required

root@linux-node1:~# kubectl delete node linux-node3.example.com

Etcd backup and restore

Back up Etcd daily with cron. Note that the etcd Pod has no ls or date commands.

• Etcd backup example

kubectl exec -it etcd-k8s-node1.opsany.com -n kube-system -- /bin/sh
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/etcd-snapshot.db

• Etcd restore example

kubectl exec -it etcd-k8s-node1.opsany.com -n kube-system -- /bin/sh
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /var/lib/etcd/etcd-snapshot.db

Rotating Kubernetes certificates

kubeadm renews all certificates when the control plane is upgraded. If the Kubernetes version has not been updated for more than a year, renew the certificates manually.

1. Check certificate expiry

This command shows the expiry and remaining time of the client certificates in /etc/kubernetes/pki and of the client certificates embedded in the kubeconfig files kubeadm uses (admin.conf, controller-manager.conf, scheduler.conf).

kubeadm certs check-expiration

1. Renew all certificates

kubeadm certs renew all

1. Update the kubeconfig

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Getting the real client IP with the Nginx Ingress Controller

kubectl edit configmap ingress-nginx-controller -n ingress-nginx
# Add the following below that line
allow-snippet-annotations: "true"
# Content to add
compute-full-forwarded-for: 'true'
forwarded-for-header: X-Forwarded-For
use-forwarded-headers: 'true'
# Other useful settings
max-worker-connections: '65536'
proxy-body-size: 20m
proxy-connect-timeout: '10'
reuse-port: 'true'
server-tokens: 'false'
ssl-redirect: 'false'
upstream-keepalive-timeout: '900'
use-forwarded-headers: 'true'
worker-cpu-affinity: auto

Adding a DNS record to CoreDNS manually

1. Back up the ConfigMap first, so you can roll back if the edit goes wrong.

kubectl get cm coredns -n kube-system -o yaml > coredns-backup.yaml

1. Add the DNS record.

   kubectl edit cm coredns -n kube-system
   
   # Edit the ConfigMap and add
   
   .:53 {
       errors
       health {
           lameduck 5s
       }
       ready
       hosts {
           192.168.1.100 www.example.com
           fallthrough
       }
       kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
   }
   

Kubernetes detects ConfigMap changes and reloads the CoreDNS configuration automatically, but to be certain the change takes effect immediately you may want to restart the CoreDNS Pods by deleting them.