Kubernetes 1.9 on a Raspberry Pi Cluster

First, this post (and my Raspberry Pi Kubernetes Cluster) stands on the shoulders of giants: work that’s already been done and explored by others. This post serves pretty much as a “here’s what I did so it worked for me” post based on their work.


To that end, before proceeding on with this you should read these posts by Cody Bunch, Chris Short, and Scott Hanselman on building their own clusters:

You can see references to other people’s work in their posts but if I follow all the turtles down I’ll end up at Charles Babbage. These are good starting points if you’re trying to do this yourself.

Shopping List

Having built a RetroPie for gaming in the past I was familiar with what was needed for at least ONE Raspberry Pi system, but I didn’t really know what to get for several in a cluster. I mostly went with Chris’s shopping list, including the Black Box switch from Scott’s post.

You may notice I screwed up already. The switch is USB powered, and I have six Raspberry Pi nodes, and a 6-Port USB Wall Charger. That’s 7 USB cords that need to plug into 6 ports. Lucky me: the switch also came with a AC power cord.

Scott recommends picking up a router as well, but I just plugged the switch into my home router that manages all my other home internet things. What could go wrong?

Delivery Day!

All the components were delivered, and my nephew and I put it together:

Building the Image

Using Cody’s work as a base, I created a arm image using Packer and an arm image plugin: https://github.com/solo-io/packer-builder-arm-image. Like Cody, I used the Vagrant VM included in the Packer plugin repository:

git clone https://github.com/solo-io/packer-builder-arm-image
cd packer-builder-arm-image
vagrant up && vagrant ssh

My main changes to the disk files Cody made were using an SSH key instead of a password and using wired, static IP addresses over using the Raspberry Pi’s Wifi. Since I ran all the nodes off of a switch and my router I was not using the master as a network gateway. I also did not install kubeadm here, because I wanted to use Ansible for that later. The files I used are here, and they need to be in your /vagrant/ directory in the VM (it shares your project directory automatically).

Inside the Vagrant VM, I ran the same commands. I then shut down the VM:

sudo packer build /vagrant/kubernetes_base.json
rsync --progress --archive /home/vagrant/output-arm-image/image /vagrant/raspbian-stretch-modified.img

Burning the Image

I used Hypriot’s flash utility to burn the image on the SDCard. Since I was using static addressing I needed 6 user-data.yml files. I had to do the master by itself but for-loop’d the workers:

flash --hostname master-00 --userdata ./master-user-data.yml ./raspbian-stretch-modified.img
for i in {0..4}; do flash --hostname worker-0$i --userdata ./worker-user-data-0$i.yml ./raspbian-stretch-modified.img; done

Ansible Time

I switched gears at this point from Cody’s work to Chris Short’s setup. Since I’m a recent Red Hatter wanted to re-learn some Ansible and this seemed like a great excuse to dive back in. Chris open sourced his work as the rak8s project (pronounced “rackets”).

I powered on my Raspberry Pi cluster, let cloud-init do it’s thing with my networking setup and SSH key and watched them reboot. They were back online and ready for Ansible.

I updated my inventory to reflect my node names and ran it:

ansible-playbook cluster.yml

…and it didn’t work for me. Shocking, I know.


I had to tweak only a few things to get Ansible to be successful for me. First, I had to add some missing APT keys.

- name: Add an apt key-1
    keyserver: keyserver.ubuntu.com
    id: 8B48AD6246925553

- name: Add an apt key-2
    keyserver: keyserver.ubuntu.com
    id: 9D6D8F6BC857C906

Next, Ansible was set to reboot the nodes after changing the /boot/config.txt and /boot/cmdline.txt files but skipped the task. To fix it, I changed the “when” line. Maybe it’s because I’m using Ansible 2.4? Anyway, here’s that diff:

diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml
index f7585b7..d09fb75 100644
--- a/roles/common/tasks/main.yml
+++ b/roles/common/tasks/main.yml
- name: Reboot
   shell: sleep 2 && shutdown -r now "Ansible Reboot for /boot/cmdline.txt Change"
   async: 1
   poll: 0
   ignore_errors: True
-  when: cmdline|changed
+  when: cmdline.changed 

Last but not least, I didn’t add the master to /etc/hosts and Ansible wasn’t picking it up. Since it was a static address I updated the kubeadm join command run on the workers to just use the master IP address. Here’s the diff:

diff --git a/roles/workers/tasks/main.yml b/roles/workers/tasks/main.yml
 - name: Join Kubernetes Cluster
-  shell: kubeadm join --ignore-preflight-errors=all --token {{ token }} {{ groups['master'][0] }}:6443 --discovery-token-unsafe-skip-ca-verification
+  shell: kubeadm join --ignore-preflight-errors=all --token {{ token }} --discovery-token-unsafe-skip-ca-verification
   when: kubeadm_reset|succeeded
   register: kubeadm_join

I made a few other small changes but I think those are the most important to call out here. You can see everything I changed in my fork of rak8s here.

Once I made my changes and re-ran Ansible, everything was ALL GOOD. Hooray!

Next Steps

Now that I did an initial install, I want to get some Let’s Encrypt going in this cluster for Kubernetes, and start playing with some services at home. This cluster, in all honesty, will probably be broken most of the time. But that’s OK - it’s part of the fun!