kubernetes-cluster-on-raspberry-pi

With Kubernetes Cluster on Raspberry Pi, you may orchestrate and manage your Docker containers with full control

Kubernetes Cluster

(Total Setup Time: 70 mins)

 

Similar to the previous MicroK8s cluster setup, I am using Ubuntu Server 20.04 LTS (64-bit) as my OS. Having a Kubernetes Cluster on Raspberry Pi, you will have more control over how the cluster configured.

After burning the image onto my 64GB SD card, an empty file named ssh is created at d:/boot for a headless setup.

 

Powering Up

(5 mins)

 

After powering up the Pi, update the OS to the latest by:

sudo apt update
sudo apt upgrade

 

Change its hostname by running:

sudo nano /etc/hostname
sudo nano /etc/hosts

 

Enable memory cgroup, by opening the file:

sudo nano /boot/firmware/cmdline.txt

 

Next, add the following at the end of the line:

cgroup=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1

 

Add the following /boot/firmware/usercfg.txt to disable WiFi and Bluetooth:

dtoverlay=disable-wifi
dtoverlay=disable-bt

 

For iptables to see bridged traffic, perform these:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl --system

 

Perform a reboot:

sudo reboot

 

Installing Docker

(5 mins)

 

First, referencing the previous post on external storage, storage setup are summarized in these steps:

# Get UUID of external HDD
sudo blkid

# Create the required folder
sudo mkdir /mnt/hdd
sudo nano /etc/fstab 

# Add line to /etc/fstab
UUID=b89ca96d-5ff9-403e-8c7c-82a1a49e5f0d /mnt/hdd ext4 defaults 0 1

# Test mount without reboot
sudo mount -a

 

Second, from the previous preparation section on docker and gitea, these commands set docker up:

# Install docker-compose
sudo apt install docker-compose
# Check version
docker-compose version
# Add ubuntu user to docker group
sudo usermod -aG docker ubuntu
sudo mkdir -p /mnt/hdd/master1k8s/docker/images

 

Third, add the –data-root option to the ExecStart:

sudo nano /lib/systemd/system/docker.service

 

Fourth, add the –data-root option to the ExecStart, similar to this:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --data-root=/mnt/hdd/master1k8s/docker/images

 

Last, to setup docker daemon, run this command:

# Run su commnad as root
sudo su -

# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

 

Installing kudeadm

(25 mins)

 

Add Kubernetes repository:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

 

Next, install kubeadm and perform a reboot:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo reboot

 

Creating Kubernetes Cluster

(20 mins)

 

Create the kubernetes cluster by running:

# Pass the container runtime sockets:
sudo kubeadm init --control-plane-endpoint master1k8s --service-cidr 10.96.0.0/12

# Reset the cluster creation when things went wrong and try init again
sudo kubeadm reset

 

[init] Using Kubernetes version: v1.18.5
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master1k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1k8s] and IPs [10.96.0.1 192.168.100.51]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master1k8s localhost] and IPs [192.168.100.51 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master1k8s localhost] and IPs [192.168.100.51 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0710 11:09:46.065710   24688 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0710 11:09:46.069293   24688 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 31.009879 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1k8s as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master1k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: cnifgh.ptp6is3fyb60to81
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
    --discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5 \
    --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
    --discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5

 

If you plan to add nodes as in my case, please keep a copy of the above kubeadm init

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

The Kubernetes Cluster on Raspberry Pi status is verified with:

kubectl cluster-info

kubernetes-cluster-info

 

To allow the control-plane node to schedule pods, this command can be run (by default, for security reasons the kubernetes cluster will not schedule pods on the control-plane node):

kubectl taint nodes --all node-role.kubernetes.io/master-

 

Installing Networking Addons

(5 mins)

 

You may choose from any of the networking addons listing. For me, I use Calio:

curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml
watch kubectl get pods --all-namespaces
kubectl get nodes -o wide

 

Without installing the networking addons, the nodes status will be NotReady and pods status will be Pending.

 

Adding Leaf Nodes

(10 mins)

 

Firstly, install kubectl, kubelet and kubeadm, following the same steps as the above section for the master node.

 

Secondly, install docker:

sudo apt install docker.io
sudo usermod -aG docker ubuntu
su - ubuntu
sudo systemctl enable docker.service

 

Thirdly, add hostname of the master to /etc/hosts:

192.168.100.51 master1k8s

 

Lastly, join to the newly created kubernetes cluster:

sudo kubeadm join master1k8s:6443 --token sxddbo.ouonf7ehvkkso3b2 \
  --discovery-token-ca-cert-hash sha256:130451a73b3d4ff65fc4dbbfba4e78b648c7b75e90621ebc1fdba789135957b5

 

Troubleshooting

 

[ERROR FileExisting-conntrack]: conntrack not found in system path

From the troubeshooting guide, install conntrack by:

sudo apt -y install conntrack

 

[WARNING FileExisting-socat]: socat not found in system path

Install conntrack by:

sudo apt -y install socat