Kubernetes PI cluster with Ansible

Posted on by dojocube

kubernetes-pi-cluster-ansible

By provisioning a Kubernetes PI Cluster with Ansible, you can easily spin off a Raspberry PI cluster

Kubernetes Cluster with Ansible

(Total Setup Time: 50 mins)

 

In this guide, I will configure a Kubernetes Cluster using Ansible. This guide follows closely to the Raspbernetes Cluster Installation. I will be using 3x Raspberry Pi 4 Model B 8GB as the master nodes and 1x Raspberry Pi 3 Model B as the only worker node.

 

You need to install ansible, flash and kubectl. I installed both Ansible and Flash onto my elementary OS host by:

#Installing Ansible
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible

#Installing Flash
curl -LO https://github.com/hypriot/flash/releases/download/2.7.0/flash
cdmod +x flash
sudo mv flash /usr/local/bin/flash
sudo apt-get install -y pv curl python-pip unzip hdparm
sudo pip install awscli

 

Preparation

(20 mins)

 

Firstly, downloads the Ubuntu 20.04 image and unzips it:

curl -L "http://cdimage.ubuntu.com/releases/20.04.2/release/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz" -o ~/Downloads/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz
unxz -T 0 ~/Downloads/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img.xz

 

Secondly, clones the Raspbernetes repository and makes the necessary changes (hostname, users.name, ssh_authorized_keys, etho0.addresses and gateway4) to the cloud-init file. I have created 4x cloud-config.yml for each of the 64GB SD card:

git clone https://github.com/raspbernetes/k8s-cluster-installation.git
cd k8s-cluster-installation
vi setup/cloud-config.yml

#Create the 4x SD cards ubuntu image using Flash
sudo flash --userdata cloud-config.yml ~/Downloads/ubuntu-20.04.2-preinstalled-server-arm64+raspi.img

 

Thirdly, ensures that Ansible inventory is changed according to hostname, username and IP addresses for each SD card.

vi ansible/inventory

#My setup as reference
[masters]
master1 hostname=master1 ansible_host=192.168.100.180 ansible_user=ubuntu
master2 hostname=master2 ansible_host=192.168.100.181 ansible_user=ubuntu
master3 hostname=master3 ansible_host=192.168.100.182 ansible_user=ubuntu
[workers]
worker1 hostname=worker1 ansible_host=192.168.100.188 ansible_user=ubuntu

#Ensures that you are able to ping all nodes
env ANSIBLE_CONFIG=ansible/ansible.cfg ansible all -m ping

 

Fourthly, prior to the cluster setup, I ssh into each of the nodes and run the following.

ssh ubuntu@master1

sudo -i
systemctl stop apt-daily.timer
systemctl disable apt-daily.timer
systemctl mask apt-daily.service
systemctl daemon-reload
reboot

 

These are some of my customizations:

# group_vars/all.yml
26: keepalived_vip: '192.168.100.200'
32: cri_plugin: docker
38: cni_plugin: 'calico'

# group_var/masters.yml
16: cluster_kube_proxy_enabled:true
23: cni_plugin: 'calico'

# roles/cluster/defaults/main/main.yml
101: docker: 'unix:///run/containerd/containerd.sock'

# roles/clusters/templates/kubeadm-config.yaml.j2
25: criSocket: unix:///run/containerd/containerd.sock

# roles/clusters/templates/kubeadm-join.yaml.j2
26: criSocket: unix:///run/containerd/containerd.sock

# roles/cni/defaults/main.yml
3: cni_bgp_peer_address: 192.168.100.1

# roles/common/tasks/main.yml (added to line 1)
- name: Bugfix for Ubuntu 20.04
  set_fact:
    ansible_os_family: "debian"

# roles/container-runtime/containerd/defaults/main.yml
9: containerd_use_docker_repo: true

# roles/container-runtime/containerd/defaults/main.yml (added to line 1)
# Instructions: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
- name: Add Docker’s official GPG key
  apt_key:
    url: https://download.docker.com/linux/ubuntu/gpg
	state: present
	
- name: Add apt repository for Docker
  apt_repository:
    repo: deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu focal stable  
	state: present
  regiser: docker_repository
  until: docker_repository is success
  
# roles/container-runtime/defaults/main.yml
2: cri_plugin: 'docker'

# roles/container-runtime/docker/defaults/main.yml 
12: #- docker-ce
13: #- docker-ce-cli
14: #- containerd.io
15: - docker.io

# roles/container-runtime/vars/main.yml
23: docker: 'unix:///run/containerd/containerd.sock'

# roles/haproxy/templates/haproxy.cfg.j2
22: server {{hostvars[host]['hostname'] }} {{hostvars[host]['ansible_host'] }}:6433 check

 

Installation

(30 mins)

 

First, you may start the Kubernetes cluster setup by running Ansible playbook:

env ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook ansible/playbooks/all.yml

 

Second, when installation completes, you may copy contents of ansible/playbooks/output/k8s-config.yaml to your local machine:

# Create a kube folder in host machine
mkdir ~/.kube

# Paste copied content into config file
vi ~/.kube/config

kubectl get nodes

kubectl-get-nodes

Finally, the Kubernetes PI Cluster with Ansible is ready for play! If you are interested in creating a HA cluster, you may refer to my previous post

 

Troubleshooting

 

if BLKRRPART failed: Device or resource busy

If you face similar error as above, you may comment out line 410 in flash script:

vi /usr/local/bin/flash

#sudo hdparm -z "$1"
sleep 1m

 

msg: Only python3 is supported, you’re running 2.7.17 locally

If you encouter this issue, you may install Python 3 and try re-running the playbook with the interpreter setting:

env ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook ansible/playbooks/all.yml -e ansible_python_interpreter=/usr/bin/python3

 

Please ensure that:
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.

If you face this issue during ansible setup, you may need to ssh ubuntu@master1

sudo -i
vi /etc/kubernetes/kubeadm-join.yaml

#modify apiServerEndpoint to
192.168.100.180:6443