Setting up the Kubernetes cluster

k3s + Raspberry Pi + Ansible

---

In the previous blog post I wrote about what made build a 3-node Raspberry Pi Kubernetes cluster (1 control plane + 2 worker nodes) and today I’m writing about how to set up the Pi’s to work with Kubernetes, I will not cover how to install Linux on them as there is enough information on the Raspberry Pi Foundation website, like this guide.

The software I chose to install on the Pi’s is called k3s because it is designed to run in IoT devices as it is very lightweight. A full Kubernetes software would not run well on a Raspberry Pi 4 (or not run at all).

Prerequisites

You are going to be running commands from your PC (deployment environment) and need to have passwordless SSH access to every Raspberry Pi (control plane and workers). To have this, copy the content of your ~/.ssh/id_rsa.pub file from your deployment environment to the target file ~/.ssh/authorized_keys on all your Raspberry Pi’s.

Deployment environment must have Ansible 2.4.0+ installed.

k3s installation

We are going to take the easiest and best way of installing it, which is using Ansible. We will use the oficial playbook that k3s has uploaded to GitHub (made by itwars).

Download the repository to your computer and create a new folder within the inventory directory with a name for your cluster (in the example below is called k3s-cluster):

$ cp -R inventory/sample inventory/k3s-cluster

Edit the file inventory/k3s-cluster/hosts.ini to match the system information gathered above (your Raspberry Pi IP’s):

[master]
192.16.35.12

[node]
192.16.35.[10:11]

[k3s_cluster:children]
master
node

Edit the file inventory/my-cluster/group_vars/all.yml to match your environment. In my case I had to change the variable ansible_user y k3s_version

$ cat all.yml
---
k3s_version: v1.23.6+k3s1
ansible_user: pi
systemd_dir: /etc/systemd/system
master_ip: "{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}"
extra_server_args: ""
extra_agent_args: ""

Linux cgroups

Cgroups need to be enabled (will be enabled when running the playbook unless you have the /boot partition mounted as Read-Only). To do so, mount the partition as Read-Write:

Check your partition identifier with:

$ mount -v | grep "^/" | awk '{print "\nPartition identifier: " $1  "\n Mountpoint: "  $3}'

Example:

$ sudo mount -o remount,rw /dev/sda1 /boot

Then edit /boot/cmdline.txt and add to the end of the first line:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Run playbook

At this point we are all set up to run the playbook and have Ansible do the rest:

$ ansible-playbook site.yml -i inventory/k3s-cluster/hosts.ini

It will take some time to run al the tasks, but it everything goes well you should have k3s installed in all the Raspberry Pi’s with special configuration applied for the control plane and the worker nodes.

Kubeconfig

To get access to your Kubernetes cluster just type:

$ scp pi@control_plane_IP:~/.kube/config ~/.kube/config-k3s-cluster

Installing Kubectl

Lastly you need the software to communicate with the Kubernetes API, that will be kubectl, probably the most widely used software to communicate with k8s. The k3s and kubectl version that you install can’t have a difference of +/-1 version, this means that if you have “1.23 k3s version” you can’t have 1.21 or less or 1.25 or more of kubectl. This works the other way around too, so both softwares need to be on at most +1/-1 version difference.

If you needed to change the k3s version, remember to update inventory/my-cluster/group_vars/all.yml with the new version before running the playbook again.

To install kubectl I will follow the official guide and since I have Raspbian installed I will follow the Debian-based steps:

1. Update the apt package index and install packages needed to use the Kubernetes apt repository:

       sudo apt-get update
       sudo apt-get install -y apt-transport-https ca-certificates curl
   

2. Download the Google Cloud public signing key:

       sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
   

3. Add the Kubernetes apt repository:

       echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
   

4. Update apt package index with the new repository and install kubectl:

       sudo apt-get update
       sudo apt-get install -y kubectl
   

Test that it is indeed installed checking its version:

$ kubectl version

To use kubectl you need to export the variable KUBECONFIG with the location of your kube-config file:

$ export KUBECONFIG=~/.kube/config-k3s-cluster

If everything worked you should now see your nodes:

$ kubectl get nodes
NAME         STATUS   ROLES                  AGE    VERSION
master-k3s   Ready    control-plane,master   230d   v1.23.6+k3s1
worker-02    Ready    <none>                 230d   v1.23.6+k3s1
worker-01    Ready    <none>                 230d   v1.23.6+k3s1

Congratulations! You are now the owner of your very own k3s cluster that is up and running and ready to have pods running on it.

Wrapping up

In this blog post I didn’t cover what can you do with your cluster once it is up and running. Perhaps install and configure an ingress controller (such as Traefik or Nginx Reverse Proxy) and run a simple webserver using a DNS resolution. Also it is a good idea to install a monitoring tool such as Prometheus + Grafana. I would recommend the kube-prometheus-stack helm chart for it’s easiness to install.