Deploying Kubernetes Manually with kubeadm¶
Since Kubernetes v1.13, kubeadm has been GA and is ready for production. For production environments we strongly recommend kubeadm rather than other non-official open-source tools. Mirror sites for the Kubernetes image registry exist in China (Alibaba Cloud), which makes kubeadm clusters much simpler to build. This guide walks you through deploying Kubernetes v1.31 with kubeadm.
Kubernetes overview¶
Lab environment¶
| Hostname | IP address (NAT) | Description |
|---|---|---|
| linux-node1.example.com | eth0:192.168.56.11 | Kubernetes Master/Etcd node |
| linux-node2.example.com | eth0:192.168.56.12 | Kubernetes Node |
| linux-node3.example.com | eth0:192.168.56.13 | Kubernetes Node |
| Service CIDR | 10.1.0.0/16 | Service IP range |
| Pod CIDR | 10.2.0.0/16 | Pod IP range |
| Note | Ubuntu 22.04, Rocky Linux, and AlmaLinux are the recommended operating systems. |
Kubernetes architecture¶
-
Master node:
- kube-apiserver: the API server is the control plane component that exposes the Kubernetes API and handles incoming requests. It is the front end of the control plane.
- etcd: a consistent, highly available key-value store used as the backing database for all cluster data.
- kube-scheduler: a control plane component that watches for newly created Pods with no assigned node and selects a node for them to run on.
- kube-controller-manager: a control plane component that runs the controller processes.
-
Worker node:
- kubelet: runs on every node in the cluster and ensures containers are running in Pods.
- kube-proxy: the network proxy running on every node, implementing part of the Kubernetes Service concept.
- Container runtime: the component that lets Kubernetes run containers efficiently, managing container execution and lifecycle.
Preparing the runtime environment¶
First configure the package repositories used to install dependencies such as kubeadm, kubelet, and containerd.
1. System initialization¶
This must be done on every node. If the hostname cannot be resolved, Kubernetes installation will fail.
- Set the hostname
[root@linux-node1 ~]# hostnamectl set-hostname linux-node1.example.com
[root@linux-node2 ~]# hostnamectl set-hostname linux-node2.example.com
[root@linux-node3 ~]# hostnamectl set-hostname linux-node3.example.com
- Configure /etc/hosts so the hostnames resolve
[root@linux-node1 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.11 linux-node1 linux-node1.example.com
192.168.56.12 linux-node2 linux-node2.example.com
192.168.56.13 linux-node3 linux-node3.example.com
- Disable swap completely
[root@linux-node1 ~]# swapoff -a
[root@linux-node1 ~]# vim /etc/fstab
# Delete the swap entry, then reboot the host.
[root@linux-node1 ~]# reboot
2. Configure the Kubernetes repository¶
# Step1: install the required system tools
sudo apt-get -y update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
# Step2: configure the Kubernetes repository
mkdir -p /etc/apt/keyrings/
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.31/deb/ /" | tee /etc/apt/sources.list.d/kubernets.list
# Step3: configure the Docker repository
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get -y update
# Step4: install the container runtime, kubeadm, kubectl, and kubelet
sudo apt-get -y install containerd.io kubeadm kubectl kubelet socat
# Step1: configure the Kubernetes repository
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core%3A/stable%3A/v1.31/rpm/
enabled=1
gpgcheck=0
EOF
# Step2: configure the Docker repository
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step3: install the container runtime, kubeadm, kubectl, and kubelet
yum install -y containerd.io kubeadm kubelet kubectl socat
# Step4: check the installed version — you will need it when editing kubeadm.yaml.
root@kube-node1:~# kubeadm version
# Step1: configure the Kubernetes repository
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core%3A/stable%3A/v1.31/rpm/
enabled=1
gpgcheck=0
EOF
# Step2: configure the Docker repository
dnf config-manager --add-repo=http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step3: install the container runtime, kubeadm, kubectl, and kubelet
dnf -y install containerd.io kubeadm kubelet kubectl iproute-tc socat
# Step4: set the kernel parameters
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
# Step5: check the installed version — you will need it when editing kubeadm.yaml.
root@kube-node1:~# kubeadm version
3. Set the kernel parameters¶
[root@linux-node1 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
Apply them
[root@linux-node1 ~]# sysctl --system
4. Enable kubelet and set it to start on boot¶
Note that kubelet cannot start successfully yet — /var/log/messages shows errors. This is expected; it starts normally after the cluster is initialized.
systemctl enable kubelet && systemctl start kubelet
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's#registry.k8s.io/pause:3.8#registry.aliyuncs.com/google_containers/pause:3.10#g' /etc/containerd/config.toml
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
systemctl enable containerd && systemctl restart containerd
5. Load balancing with IPVS¶
Kube-Proxy uses iptables for load balancing by default; production environments should use ipvs. Enable the ipvs modules on all nodes.
# Enable the kernel modules
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
modprobe overlay
modprobe br_netfilter
[root@linux-node1 ~]# cat <<EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
overlay
br_netfilter
EOF
[root@linux-node1 ~]# cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
modprobe overlay
modprobe br_netfilter
EOF
[root@linux-node1 ~]# chmod +x /etc/sysconfig/modules/ipvs.modules
[root@linux-node1 ~]# source /etc/sysconfig/modules/ipvs.modules
Check that the modules loaded correctly
[root@linux-node1 ~]# lsmod | grep -e ip_vs -e nf_conntrack
ip_vs_sh 12688 0
ip_vs_wrr 12697 0
ip_vs_rr 12600 0
ip_vs 145497 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4 15053 15
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
nf_conntrack 133095 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
Initializing the cluster and deploying the Master¶
Once everything is installed on all nodes, initialize the cluster on the Master node linux-node1.
1. Create the configuration file¶
Generate a default kubeadm configuration file, then edit it.
[root@linux-node1 ~]# kubeadm config print init-defaults > kubeadm.yaml
Edit the file
[root@linux-node1 ~]# vim kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.56.11 #Change to the API Server address
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: linux-node1.example.com #Change to the Master node's hostname
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers #Change to the Alibaba Cloud registry
kind: ClusterConfiguration
kubernetesVersion: v1.31.1 #Change to the exact version
networking:
dnsDomain: cluster.local
serviceSubnet: 10.1.0.0/16 #Change the Service network
podSubnet: 10.2.0.0/16 #Add the Pod network
scheduler: {}
--- # Three extra lines below make KubeProxy use LVS
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
2. Deploy the Master node¶
[root@linux-node1 ~]# kubeadm init --config kubeadm.yaml
[init] Using Kubernetes version: v1.31.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR Swap]: running with swap on is not supported. Please disable swa
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
--ignore-preflight-errors=.
--ignore-preflight-errors=: ignores runtime errors. For the [ERROR NumCPU] and [ERROR Swap] above, add --ignore-preflight-errors=NumCPU and --ignore-preflight-errors=Swap.
Run initialization again:
[root@linux-node1 ~]# kubeadm init --config kubeadm.yaml \
--ignore-preflight-errors=NumCPU
[init] Using Kubernetes version: v1.31.1
[preflight] Running pre-flight checks
[WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
After that it pauses while downloading the Kubernetes component images. Depending on your network this takes one to five minutes; you can also watch the images arrive with docker images. Once the images are down, initialization proceeds:
The output below is omitted; initialization goes through these 15 steps, each line prefixed with the stage name in brackets:
- [init]: initializes with the specified version
- [preflight]: pre-initialization checks and image downloads.
- [kubelet-start]: generates kubelet's configuration file "/var/lib/kubelet/config.yaml"; without it kubelet cannot start, which is why kubelet failed before initialization.
- [certificates]: generates the certificates Kubernetes uses, stored in /etc/kubernetes/pki.
- [kubeconfig]: generates the KubeConfig files in /etc/kubernetes, used for component-to-component communication.
- [control-plane]: installs the Master components from the YAML files in /etc/kubernetes/manifests.
- [etcd]: installs Etcd from /etc/kubernetes/manifests/etcd.yaml.
- [wait-control-plane]: waits for the control-plane Master components to start.
- [apiclient]: checks the status of the Master component services.
- [uploadconfig]: uploads the configuration.
- [kubelet]: configures kubelet through a ConfigMap.
- [patchnode]: records the CNI information on the Node as an annotation.
- [mark-control-plane]: labels the node with the Master role and an unschedulable taint, so Pods are not scheduled here by default.
- [bootstrap-token]: generates and records the token used later by
kubeadm join. - [addons]: installs the CoreDNS and kube-proxy addons.
On success you will see output like this:
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.56.11:6443 --token 19fhhl.3mzkyk16tcgp6vga --discovery-token-ca-cert-hash sha256:76a88c38b673d3b2ac73e33127a809688cb3e58c533512ac6d92ecb66aa57a45
If it fails, something earlier was wrong. Check in this order:
- Base environment
- Whether the hostname resolves, and whether SELinux and iptables are disabled.
- Whether swap is still enabled — check with
free -m. -
Whether the kernel parameters and IPVS settings were applied (these do not cause failures at this stage).
-
Base software
- Whether Docker is installed and running.
- Whether kubelet is installed and running.
- Whether
kubeadmprinted other errors you ignored. systemctl status kubeletto check whether kubelet is running.-
If kubelet will not start, read its logs and fix the error.
-
Once all of the above is resolved, re-initialize
kubeadm reset(never run this in production — it destroys the cluster).- Follow the
kubeadm resetprompts to clear the iptables and IPVS rules.
Configure kubectl to access the cluster as the output above instructs.
3. Configure the kube config¶
kubectl looks for the config file in the .kube directory of the executing user's home directory. Copy the admin.conf generated during the [kubeconfig] step to .kube/config.
[root@linux-node1 ~]# mkdir -p $HOME/.kube
[root@linux-node1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@linux-node1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config
4. Check component status¶
[root@linux-node1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
Get Node information with kubectl — there is one node so far, role Master, status NotReady.
[root@linux-node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
linux-node1.example.com NotReady master 14m v1.30.2
5. Deploy the Flannel network plugin¶
The Master is NotReady because no network plugin is installed, so the Node and Master cannot communicate yet. The most popular Kubernetes network plugins are Flannel, Calico, and Canal; the examples below cover Canal and Flannel — pick one. With the base cluster in place, adding components is almost always a matter of kubectl plus one YAML file.
Deploying Flannel requires changing the Pod CIDR to match the one you initialized with. Download the Flannel YAML, edit it, then apply it. Because the images in the official YAML are unreachable, use the copy the author has prepared.
# Step1: deploy Flannel
[root@linux-node1 ~]# git clone --depth 1 https://gitee.com/unixhot/salt-kubeadm.git
[root@linux-node1 ~]# cp salt-kubeadm/salt/k8s/templates/flannel/kube-flannel.yml.template /opt/kube-flannel.yml
[root@linux-node1 Documentation]# vim /opt/kube-flannel.yml +91
# Change "Network": "{{ POD_CIDR }}", to "Network": "10.2.0.0/16",
74 net-conf.json: |
75 {
76 "Network": "10.2.0.0/16",
77 "Backend": {
78 "Type": "vxlan"
79 }
80 }
Deploy Flannel
[root@linux-node1 Documentation]# kubectl create -f /opt/kube-flannel.yml
Check Flannel status — everything must be Running
root@linux-node1:~# kubectl get pod -n kube-flannel
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-btlhh 1/1 Running 0 74s
kube-flannel-ds-bzwph 1/1 Running 0 74s
kube-flannel-ds-j7ft5 1/1 Running 0 74s
Check the Pod status.
[root@linux-node1 Documentation]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-58cc8c89f4-cjhdv 0/1 Pending 0 41m
coredns-58cc8c89f4-vdfn2 0/1 Pending 0 41m
etcd-linux-node1.example.com 1/1 Running 0 41m
kube-apiserver-linux-node1.example.com 1/1 Running 0 40m
kube-controller-manager-linux-node1.example.com 1/1 Running 1 40m
kube-flannel-ds-amd64-bwsxl 0/1 Init:0/1 0 20s
kube-proxy-5qrmh 1/1 Running 0 41m
kube-scheduler-linux-node1.example.com 1/1 Running 1 41m
CoreDNS is Pending; once the Flannel Pods reach Running, CoreDNS becomes healthy too. When every Pod is Running, query the Node again and it reports Ready.
[root@linux-node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
linux-node1.example.com Ready master 29m v1.30.3
Deploying Node nodes¶
With the Master deployed, you can deploy the Node nodes. First install Docker, kubeadm, and kubelet on each Node following the earlier section; that is not repeated here.
1. Generate the join command¶
Print the command for adding nodes on the Master
[root@linux-node1 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.56.11:6443 --token isggqa.xjwsm3i6nex91d2x --discovery-token-ca-cert-hash sha256:718827895a9a5e63dfa9ff54e16ad6dc0c493139c9c573b67ad66968036cd569
2. Join the nodes to the cluster¶
Note that if a node has swap enabled, add --ignore-preflight-errors=Swap.
Deploy linux-node2
[root@linux-node2 ~]# kubeadm join 192.168.56.11:6443 --token isggqa.xjwsm3i6nex91d2x --discovery-token-ca-cert-hash sha256:718827895a9a5e63dfa9ff54e16ad6dc0c493139c9c573b67ad66968036cd569 --ignore-preflight-errors=Swap
Deploy linux-node3
[root@linux-node3 ~]# kubeadm join 192.168.56.11:6443 --token isggqa.xjwsm3i6nex91d2x --discovery-token-ca-cert-hash sha256:718827895a9a5e63dfa9ff54e16ad6dc0c493139c9c573b67ad66968036cd569 --ignore-preflight-errors=Swap
Kubernetes then deploys flannel and kube-proxy on every node as DaemonSets. Once that completes, the nodes are done.
[root@linux-node1 ~]# kubectl get daemonset --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system canal 2 2 1 2 1 beta.kubernetes.io/os=linux 17m
kube-system kube-proxy 2 2 2 2 2 <none> 47m
View all nodes
[root@linux-node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
linux-node1.example.com Ready master 49m v1.30.3
linux-node2.example.com Ready <none> 4m48s v1.30.3
linux-node3.example.com Ready <none> 4m48s v1.30.3
How to add a Roles label to a Node
-
View the node's labels
[root@linux-node1 ~]# kubectl get nodes --show-labels -
Add the label
[root@linux-node1 ~]# kubectl label nodes linux-node2.example.com node-role.kubernetes.io/worker= node/linux-node2.example.com labeled -
Check the result
[root@linux-node1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION linux-node1.example.com Ready control-plane 20m v1.30.5 linux-node2.example.com Ready worker 10m v1.30.5 linux-node3.example.com Ready worker 10m v1.30.5
Testing the Kubernetes cluster¶
In the steps above we built a cluster with one Master and two Nodes. Production environments also need Master high availability; that is covered later.
1. Create a single-Pod Nginx application¶
- Create a Deployment named nginx
[root@linux-node1 ~]# kubectl create deployment nginx --image=registry.cn-beijing.aliyuncs.com/opsany/nginx:1.26-perl
deployment.apps/nginx created
[root@linux-node1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-54458cd494-9j7ql 0/1 ContainerCreating 0 10s
- Check the Pod status
Once the Pod is Running you can get its IP, allocated from the --pod-network-cidr=10.2.0.0/16 range configured when initializing the Master.
[root@linux-node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-54458cd494-9j7ql 1/1 Running 0 59s 10.2.1.2 linux-node2.example.com
- Test Nginx access
[root@linux-node1 ~]# curl --head http://10.2.1.2
HTTP/1.1 200 OK
Server: nginx/1.15.8
Date: Sun, 13 Jan 2019 01:16:36 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 26 Dec 2018 23:21:49 GMT
Connection: keep-alive
ETag: "5c240d0d-264"
Accept-Ranges: bytes
2. Test Deployment scaling¶
Now scale the Nginx application to two Pod replicas
[root@linux-node1 ~]# kubectl scale deployment nginx --replicas=2
deployment.extensions/nginx scaled
[root@linux-node1 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-54458cd494-9j7ql 1/1 Running 0 2m13s
nginx-54458cd494-vnm4f 1/1 Running 0 5s
3. Expose Nginx with a Service¶
Adding a Service to Nginx creates a ClusterIP allocated from the --service-cidr=10.1.0.0/16 range configured on the Master, and NodePort maps a port on each node for external access.
- Create a NodePort Service
[root@linux-node1 ~]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
[root@linux-node1 ~]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 88m
nginx NodePort 10.1.147.204 <none> 80:30599/TCP 67m
- Test the Service Cluster IP
[root@linux-node1 ~]# curl --head http://10.1.147.204/
HTTP/1.1 200 OK
Server: nginx/1.15.8
Date: Sun, 13 Jan 2019 01:26:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 26 Dec 2018 23:21:49 GMT
Connection: keep-alive
ETag: "5c240d0d-264"
Accept-Ranges: bytes
You now have a Kubernetes cluster. From here you can explore Kubernetes and OpsAny further.
