Building a CI/CD pipeline on a Raspberry PI Cluster

Posted on by dojocube

raspberry-pi-enclosed-in-lego-structure

Building a CI/CD pipeline on a Raspberry PI Cluster, with 3 master and 1 worker nodes, enclosed in a custom-made LEGO structure.

Building a CI/CD pipeline on a Raspberry PI Cluster (Part 1)

(Total Setup Time: 40 mins)

 

In this series, I will build my own CI/CD pipeline, with tools that are configured to run on Raspberry PI Cluster with Longhorn, a HA Raspberry PI Cluster. By end of this guide, you will have a self-hosted Git service, working hand-in-hand with Jenkins.

 

Preparation

(5 mins)

Metallb is a load-balancer implementation for bare metal Kubernetes clusters. You may refer to the previous guide on Metallb. Let’s install it by with these commands:

mkdir ~/metallb
cd ~/metallb

wget https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml -O metallb-namespace.yaml
kubectl apply -f metallb-namespace.yaml

wget https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml -O metallb.yaml
kubectl apply -f metallb.yaml

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

 

This is my layer2 configuration, in metallb-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.100.200-192.168.100.250

kubectl apply -f metallb-config.yaml

 

I create 4 volumes, namely mysql, gitea, jenkins and maven-agent, specific to each of the tools.

longhorn-volumes

 

Installing Mysql

(5 mins)

Referencing to Gitea on Kubernetes PI Cluster, these are the steps for installing Mysql.

kubectl create namespace dojocube
docker pull mysql/mysql-server:latest

mkdir ~/mysql
cd ~/mysql

vi mysql-deployment.yaml
# Insert below into mysql-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace:dojocube
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace:dojocube
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      hostname: mysql
      containers:
      - image: mysql/mysql-server:latest
        imagePullPolicy: IfNotPresent
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc

kubectl apply -f mysql-deployment.yaml

 

You may check on the pod ID and run the following scripts:

kubectl get pods -n dojocube
kubectl exec --namespace=dojocube -it mysql-6587f996b5-qrcr6 -- /bin/bash
mysql -u root -p
 
# Execute these within the docker prompt, with username=gitea, password=gitea, database=gitedb
CREATE USER 'gitea'@'%' IDENTIFIED BY 'gitea';
CREATE DATABASE giteadb;
GRANT ALL PRIVILEGES ON giteadb.* TO 'gitea';
FLUSH PRIVILEGES;

 

Installing Gitea

(5 mins)

First, installs Gitea by running thse commands:

apt install docker-compose

mkdir ~/gitea
cd ~/gitea

vi docker-compose.yml
# Insert below into docker-compose.yml

version: "2"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=mysql
      - DB_HOST=mysql:3306
      - DB_NAME=giteadb
      - DB_USER=gitea
      - DB_PASSWD=gitea
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

# Builds, starts and stops container
docker-compose up -d
docker-compose down

 

Second, creates the deployment file for Gitea:

vi gitea-deployment.yaml
# Insert below into gitea-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: gitea
  namespace: dojocube
  annotations:
    metallb.universe.tf/allow-shared-ip: home-net
spec:
  ports:
  - port: 3000
    targetPort: 3000
    nodePort: 30000
    name: gitea-http
  - port: 2222
    targetPort: 2222
    nodePort: 32222
    name: gitea-ssh
  selector:
    app: gitea
  type: LoadBalancer
  loadBalancerIP: 192.168.100.250
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitea
  namespace: dojocube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitea
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: gitea
    spec:
      hostname: gitea
      containers:
      - image: gitea/gitea:latest
        imagePullPolicy: IfNotPresent
        name: gitea
        ports:
        - containerPort: 3000
          name: gitea-http
        - containerPort: 22
          name: gitea-ssh
        volumeMounts:
        - name: gitea-persistent-storage
          mountPath: /data
      volumes:
      - name: gitea-persistent-storage
        persistentVolumeClaim:
          claimName: gitea-pvc

kubectl apply -f gitea-deployment.yaml
kubectl get svc -n dojocube

 

Third, access to the configured url, i.e. 192.168.100.250:3000 and follows the gitea installation. For my case, the parameters are:

username: gitea
password: gitea
database: giteadb

 

Installing Jenkins

(5 mins)

By referencing Jenkins for Kuberenetes Cluster, the installation steps are:

mkdir ~/jenkins
cd ~/jenkins

vi Dockerfile
# Insert below into Dockerfile

FROM balenalib/raspberrypi4-64-debian-openjdk:11-bullseye

ENV JENKINS_HOME /var/jenkins_home
ENV JENKINS_SLAVE_AGENT_PORT 50000

RUN apt-get update \
  && apt-get install -y --no-install-recommends curl git
  
RUN curl -fL -o /opt/jenkins.war http://updates.jenkins-ci.org/download/war/2.295/jenkins.war

VOLUME ${JENKINS_HOME}
WORKDIR ${JENKINS_HOME}

EXPOSE 8080 ${JENKINS_SLAVE_AGENT_PORT}

CMD ["/bin/bash","-c","java -jar /opt/jenkins.war"]

# Builds, tags the image
docker build -t dojocube/jenkins:1.0 .

 

Next, creates the Jenkins deployment file and deploys it to the cluster.

vi jenkins-deployment.yaml
# Insert below into jenkins-deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: dojocube
  annotations:
    metallb.universe.tf/allow-shared-ip: home-net
spec:
  ports:
    - port: 8080
      name: jenkins-http
      targetPort: 8080
      nodePort: 30080
    - port: 50000
      name: jenkins-slave
      targetPort: 50000
  selector:
    app: jenkins
  type: LoadBalancer
  loadBalancerIP: 192.168.100.250
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: dojocube
spec:
  selector:
    matchLabels:
      app: jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      hostname: jenkins
      containers:
      - name: jenkins
        image: dojocube/jenkins:1.0
        ports:
        - containerPort: 8080
        volumeMounts:
          - name: jenkins-persistent-storage
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-persistent-storage
          persistentVolumeClaim:
            claimName: jenkins-pvc

kubectl apply -f jenkins-deployment.yaml
kubectl get po -n dojocube

# Gets the pod ID, replace the following accordingly and retrieves the password
kubectl logs -n dojocube jenkins-6c6cb7d48c-pnnr5

 

Installing Maven-agent

(5 mins)

By referencing to Jenkins Agent on Kubernetes Cluster, the steps to create a maven-agent are:

# Source
# https://hub.docker.com/r/jenkins/slave/dockerfile
FROM balenalib/raspberrypi4-64-debian-openjdk:11-bullseye

ARG VERSION=4.7
ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000

# Modified for debian-bullseye
RUN groupadd -g ${gid} ${group}
RUN useradd -d /home/${user} -u ${uid} -g ${group} -m ${user}
LABEL Description="This is a base image, which provides the Jenkins agent executable (slave.jar)" Vendor="Jenkins project" Version="${VERSION}"

ARG AGENT_WORKDIR=/home/${user}/agent

# Modified for debian-bullseye
RUN apt-get update && apt-get install -y curl bash git git-lfs openssh-client openssl procps \
  && curl --create-dirs -fsSLo /usr/share/jenkins/agent.jar https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar \
  && chmod 755 /usr/share/jenkins \
  && chmod 644 /usr/share/jenkins/agent.jar \
  && ln -sf /usr/share/jenkins/agent.jar /usr/share/jenkins/slave.jar \
  && apt-get remove -y curl

USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}

VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}

# Source
# https://hub.docker.com/r/jenkins/inbound-agent/dockerfile
USER ${user}
COPY jenkins-agent /usr/share/jenkins/jenkins-agent

USER root
COPY jenkins-agent /usr/local/bin/jenkins-agent
RUN chmod +x /usr/local/bin/jenkins-agent \
  && ln -s /usr/local/bin/jenkins-agent /usr/local/bin/jenkins-slave

USER ${user}

ENTRYPOINT ["jenkins-agent"]

 

Next, copies the content of jenkins-agent and pastes it into a file (COPY command will copy this file into the docker image).

docker build -t dojocube/jenkins-agent:1.0 .

# Checks Java version
docker run -it dojocube/jenkins-agent:1.0 /bin/bash
java -version

 

Configuring Jenkins – Kubernetes

(5 mins)

First, navigates to the configured url, i.e. 192.168.100.250:8080. Installs the recommended commonly used plugins.

installing-common-plugins

 

Second, installs Kuberenetes and Gitea plugin

install-kubernetes-plugin

 

Third, navigates to Manage Jenkins -> Manage Nodes and Cloud -> Configure Clouds. Selects Kuberenetes, clicks on the Kuberenetes Cloud Details and tests the connection. Enters Jenkins url:

test-kubernetes-cloud-connection

 

You may setup the Jenkins URL with:

Jenkins URL: http://jenkins:8080
jenkins tunnel: jenkins:50000

configure-cloud-jenkins-url

 

Fourth, configures the pod template:

configure-pod-template

 

Lastly, navigates to Manage Jenkins -> Configure Global Security. Enters the Agent values:

configure-global-security-agent

 

Configuring Jenkins – Gitea

(5 mins)

First, navigates to Manage Jenkins -> Configure System. Adds Gitea server:

configre-system-gitea-server

 

Second, creates a new Dojocube Gitea Organization. Configures the Jenkins Credentials and Projects under the General tab.

create-jenkin-gitea-organization

add-jenkins-credentials

jenkins-project-setup-for-gitea

 

Integrating Jenkins and Gitea

(5 mins)

By referencing Integration between Jenkins and Gitea, adds a new Dojocube organization. Then, creates a Jenkins user in Gitea and adds as collaborators.

add-jenkins-user-gitea-collaborators

 

Next, adds Gitea webhooks to Jenkins and tests the delivery.

add-gitea-webhook

test-webhook-delivery

 

In Action

Finally, if you followed through the steps diligently, building a CI/CD pipeline on a Raspberry PI Cluster is completed. When you commit any code, it triggers the CI pipeline. Maven-agent will build and compile your code.

jenkins-built-success

 

Troubleshooting

 

Error testing connection : Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn’t have access. Service account may have been revoked.

When testing connection at Kuberenetes Cloud setting, the above error appears. You may fix by:

cd ~/jenkins

vi jenkins-rbac.yaml
# Insert below into jenkins-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-rbac
  namespace: dojocube
subjects:
  - kind: ServiceAccount
    name: default
    namespace: dojocube
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
  
kubectl apply -f jenkins-rbac.yaml

 

Cannot find Jenkins password in /var/jenkins/home/secrets/initialAdminPassword

Depending on your configuration, you may find the initialAdminPassword

docker ps
docker logs 90494de7c46d (Container ID)

kubectl get po -n dojocube
kubectl logs -n dojocube jenkins-57fd4d8dd4-mwbpr (Pod Name)

 

Upgrading Jenkins version

For upgrading Jenkins version, the permanent fix is to modify jenkins/Dockerfile

# Find the jenkins.war file and modify the version to the latest
RUN curl -fL -o /opt/jenkins.war http://updates.jenkins-ci.org/download/war/2.297/jenkins.war

docker build -t dojocube/jenkins:1.0 .

 

/home/jenkins/agent/workspace/dojocube_hello-world_master@tmp/durable-5d2c75ea/script.sh: 1: ./mvnw: Permission denied

You may check the mvnw file in your project and changes the file permission accordingly.

chmod +x mvnw

# Adds and pushes change to gitea
git update-index --chmod=+x mvnw