Compiling your own WSL2 Kernel for Cilium

UPDATE Jan 3, 2023: https://wsl.dev/wslcilium/ has a much better, more recent walkthrough to do this. Check it out.

Recently I joined Isovalent as a Senior Solution Architect (yay!) and wanted to play with their CNI, Cilium, on my laptop with Kubernetes-in-Docker (KinD). I run Windows 11 with WSL2 (Ubuntu 20.04), where all my code-related stuff happens.

Right away I ran into Linux kernel issues, as Cilium needs some Netfliter things enabled to work in the kernel, which are not enabled by default. Luckily Microsoft makes it pretty easy to compile your own kernel image for WSL2. At the time of this post, these steps worked on the latest kernel image, version 5.10.60.1.

The steps are:

1. Run a Docker container to build the image
2. Copy the image out of the container to C:\Users\youruser
3. Tell WSL to use your new custom kernel image

Run a Docker container to build the image

Open up a PowerShell terminal:

docker run --name wsl-kernel-builder --rm -it ubuntu@sha256:9d6a8699fb5c9c39cf08a0871bd6219f0400981c570894cd8cbea30d3424a31f bash

from inside the container (now bash):

WSL_COMMIT_REF=linux-msft-wsl-5.10.60.1
apt update && apt install -y git build-essential flex bison libssl-dev libelf-dev bc

mkdir src
cd src
git init
git remote add origin https://github.com/microsoft/WSL2-Linux-Kernel.git
git config --local gc.auto 0
git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +${WSL_COMMIT_REF}:refs/remotes/origin/build/linux-msft-wsl-5.10.y
git checkout --progress --force -B build/linux-msft-wsl-5.10.y refs/remotes/origin/build/linux-msft-wsl-5.10.y

# adds support for clientIP-based session affinity
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_RECENT is not set/CONFIG_NETFILTER_XT_MATCH_RECENT=y/' Microsoft/config-wsl

# required modules for Cilium
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_CT is not set/CONFIG_NETFILTER_XT_TARGET_CT=y/' Microsoft/config-wsl
sed -i 's/# CONFIG_NETFILTER_XT_TARGET_TPROXY is not set/CONFIG_NETFILTER_XT_TARGET_TPROXY=y/' Microsoft/config-wsl

# build the kernel
make -j2 KCONFIG_CONFIG=Microsoft/config-wsl

Copy the image out of the container

Open up a new PowerShell terminal:

docker cp wsl-kernel-builder:/src/arch/x86/boot/bzImage .

This should put it in your %UserProfile% directory (C:\Users\youruser). You can exit out of the first container and delete it now. We're done with that.

Tell Windows Subsystem for Linux to use your kernel

Edit C:\Users\something\.wslconfig

[wsl2]
kernel=C:\\Users\\<your_user>\\bzImage

Keep the double slashes. Shut down WSL (in PowerShell: wsl --shutdown) and restart Docker Desktop. From there restart everything, start up KinD and install Cilium like the install instructions say. Hooray!

NOTE: this worked for me to install the open source version of Cilium. I'm still working on installing Cilium Enterprise, a product of Isovalent, in a WSL-based KinD cluster. Also of note, none of this is supported by Isovalent or me or anyone.

Credit goes to the TKG Community Edition team for easy-to-follow Windows Subsystem for Linux kernel instructions. Those are here and obviously I cribbed it all for this.