Highly Available Kubernetes Pi Cluster II

Posted on August 17, 2020June 5, 2021 by dojocube

highly-available-kubernetes-pi-cluster

By having a Highly Available Kubernetes Pi Cluster, you will have full control over your production grade environment on-premise

HA Kubernetes Pi Cluster II

(Total Setup Time: 20 mins)

In this Part 2 guide, I will configure the HA Kubernetes Cluster using the previous external etcd setup.

Installing Docker

(5 mins)

Firstly, installs docker container runtimes to all master nodes:

sudo apt install docker.io
sudo usermod -aG docker ubuntu
su - ubuntu
sudo systemctl enable docker.service

Secondly, sets up the docker daemon:

# Run su commnad as root
sudo su -

# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

Thirdly, enables and restarts Docker

systemctl daemon-reload
systemctl enable docker
systemctl restart docker

Installing kudeadm

(10 mins)

Starts by adding Kubernetes repository to all master nodes:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

Next, installs kubeadm:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo reboot

Initializing Kubernetes Master Nodes

(5 mins)

First, prepares the certs by following kubeadm HA setup:

sudo mkdir -p /etc/kubernetes/pki/etcd/
sudo cp /srv/etcd-certs/cacert.pem /etc/kubernetes/pki/etcd/
sudo cp /srv/etcd-certs/etcd.pem /etc/kubernetes/pki
sudo cp /srv/etcd-certs/etcd-key.pem /etc/kubernetes/pki

Second, inserts the following into kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "cluster-endpoint:6443"
etcd:
  external:
    endpoints:
    - https://192.168.100.119:2379
    - https://192.168.100.173:2379
    - https://192.168.100.100:2379
    caFile: /etc/kubernetes/pki/etcd/cacert.pem
    certFile: /etc/kubernetes/pki/etcd.pem
    keyFile: /etc/kubernetes/pki/etcd-key.pem

This is my /etc/hosts for all master nodes:

# cluster-endpoint points to first master node
127.0.0.1 localhost
192.168.100.119 master1
192.168.100.173 master2
192.168.100.100 master3
192.168.100.119 cluster-endpoint

Third, initializes the master node by:

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Fourth, installs Calio from the list of networking addons:

curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico-etcd.yaml

Fifth, prepares TLS for calico by following config options:

cat /srv/etcd-certs/cacert.pem | base64 -w 0
cat /srv/etcd-certs/etcd.pem | base64 -w 0
sudo cat /srv/etcd-certs/etcd-key.pem | base64 -w 0

# Searches for the Secret section in calico.yaml and paste cat output from the cacert.pem, etcd.pem and etcd-key.pem
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following files with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # This self-hosted install expects three files with the following names.  The values
  # should be base64 encoded strings of the entire contents of each file.
  etcd-key: LS0tLS1CRUdJTiB...VZBVEUgS0VZLS0tLS0=
  etcd-cert: LS0tLS1...ElGSUNBVEUtLS0tLQ==
  etcd-ca: LS0tLS1CRUdJTiBD...JRklDQVRFLS0tLS0=
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://cluster-endpoint:2379"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: "/calico-secrets/etcd-ca"
  etcd_cert: "/calico-secrets/etcd-cert"
  etcd_key: "/calico-secrets/etcd-key"
  
  
kubectl apply -f calico.yaml
watch kubectl get pods --all-namespaces
kubectl get nodes -o wide

Lastly, runs this on the other 2 master nodes:

sudo kubeadm join cluster-endpoint:6443 --token puuck4.kjgqjif1xyy2397f \
    --discovery-token-ca-cert-hash sha256:50bb31b6f26ee6e98228f098fb9f50eaf7f1db67a011c90d6c764f7331cb90e1 \
    --control-plane --certificate-key 28ad20c49a7c6b8d06b2faa305fdf0c98d9763b932b929deb4704d57ec8ff959

This is my modified /etc/hosts:

# cluster-endpoint points to the loadbalancer IP
127.0.0.1 localhost
192.168.100.119 master1
192.168.100.173 master2
192.168.100.100 master3
192.168.100.200 cluster-endpoint

And that’s all to it! You now own a Highly Available Kubernetes Pi Cluster!

Troubleshooting

The connection to the server localhost:8080 was refused – did you specify the right host or port?

After kubeadm init, you need to run these commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Secret “kubeadm-certs” was not found in the “kube-system” Namespace. This Secret might have expired

If the certificate-key expires, generates a new one as below:

sudo kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml

network: stat /var/lib/calico/nodename: no such file or directory

After resetting kubeadm, my kubernetes cluster runs successfully

sudo kubeadm reset
sudo rm -rf /etc/cni/net.d
sudo iptables --flush
sudo ipvsadm --clear
rm $HOME/.kube/config
sudo rm -rf /var/lib/cni

# External etcd clean up
sudo etcdctl del "" --prefix --cacert /srv/etcd-certs/cacert.pem --cert /srv/etcd-certs/etcd.pem  --key /srv/etcd-certs/etcd-key.pem

Re-joining cluster

When your SD failed and you wish to re-configure the faulty PI, you need to perform:

# Cordon the faulty node
kubectl cordon master3

# Drain the faulty one
kubectl drain master3 -delete-emptydir-data --ignore-daemonsets
(if it hungs, you may have to open a separate terminal and to delete the relevant pod)

# Delete the node
kubectl delete node master3 

# Retrieve bootstrap token
kubeadm token list

# Retrieve token cacert hash (sha256)
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
    | openssl rsa -pubin -outform der 2>/dev/null \
    | openssl dgst -sha256 -hex \
    | sed 's/^.* //'

# Generate new key
kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml

# Forms the join as master command based on above results
kubeadm join 192.168.100.180:6443 --token 0abut7.guksf3dfp1k67osq --discovery-token-ca-cert-hash sha256:a263dfc32ddd3d16cc72f2f91a0786e869f9310646614dbdf6a103be5d42b9c7 --v=5 --control-plane --certificate-key 97e86ed12f1e1b4f97d51d1a1ff92485d678d5fb8725c3c73e66b17a660c68af

Jul

Building a CI/CD pipeline on a Raspberry PI Cluster (Part 4), with SonarQube integrating into Jenkins and Gitlab Building a CI/CD pipeline on a Raspberry PI Cluster (Part 4) (Total Setup Time: 15 mins) In continuation from part 3 of this guide, I will add SonarQube into my CI/CD pipeline. This will enahnce our […]

Jul

Building a CI/CD pipeline on a Raspberry PI Cluster (Part 3), with JFrog Container Registry supporting our Docker containers and Helm Chart repositories Building a CI/CD pipeline on a Raspberry PI Cluster (Part 3) (Total Setup Time: 15 mins) Continue from part 2 of this guide, I will add JFrog Container Registry to my […]

Jun

Building a CI/CD pipeline part 2 on a Raspberry PI Cluster, with JFrog Artifactory as the repository manager Building a CI/CD pipeline on a Raspberry PI Cluster (Part 2) (Total Setup Time: 40 mins) Continuing from part 1 of this guide, I will add JFrog Artifactory to my CI/CD pipeline. Preparation (1 mins) […]

Jun

Building a CI/CD pipeline on a Raspberry PI Cluster, with 3 master and 1 worker nodes, enclosed in a custom-made LEGO structure. Building a CI/CD pipeline on a Raspberry PI Cluster (Part 1) (Total Setup Time: 40 mins) In this series, I will build my own CI/CD pipeline, with tools that are configured to […]

Jun

By running Raspberry PI Cluster with Longhorn, you will have a simplified, easy to deploy cloud-native persistent block storage. Raspberry PI Cluster and Longhorn (Total Setup Time: 30 mins) In this section, I will install Longhorn, a highly available persistance storage for Kubernetes. This guide assumes that you have a working Raspberry PI Cluster. […]

May

By provisioning a Kubernetes PI Cluster with Ansible, you can easily spin off a Raspberry PI cluster Kubernetes Cluster with Ansible (Total Setup Time: 50 mins) In this guide, I will configure a Kubernetes Cluster using Ansible. This guide follows closely to the Raspbernetes Cluster Installation. I will be using 3x Raspberry Pi 4 […]

Aug

By having a Highly Available Kubernetes Pi Cluster, you will have full control over your production grade environment on-premise HA Kubernetes Pi Cluster II (Total Setup Time: 20 mins) In this Part 2 guide, I will configure the HA Kubernetes Cluster using the previous external etcd setup. Installing Docker (5 mins) Firstly, […]

Aug

serial # Root certificate openssl genrsa -aes256 -out private/cakey.pem 4096 openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 3650 -set_serial 0 -subj ‘/C=SG/ST=SG/O=Dojocube/CN=master1′ Second, create certs for all master nodes: # Create master nodes’ certificate cd /root/ca/requests/ openssl genrsa -out etcd-key.pem openssl req -new -key etcd-key.pem -out etcd.csr -subj ‘/C=SG/ST=SG/O=Dojocube/CN=master1,master2,master3,localhost,cluster-endpoint’ openssl ca -in […]