Networking

K3s uses Flannel to allow pod to pod communication between different hosts, Flannel is a lightweight provider of layer 3 network fabric that implements the Kubernetes Container Network Interface (CNI). It is what is commonly referred to as a CNI Plugin.

Flannel supports multiple backends for encapsulating packets. By default K3s uses Virtual Extensible LAN (VXLAN), which runs a Layer 2 network on top of a Layer 3 infrastructure. VXLAN uses in-kernel VXLAN to encapsulate the packets using UDP on port 8472.

During one of our HA setups (PLTF-157) we noticed after running tcpdump -leni any -w output.pcap the UDP packets were not arriving at the destination host and we had to change the Flannel backend from VXLAN to host-gw which uses IP routes to pod subnets via node IPs.

To use host-gw backend you need to execute the following steps in all the nodes:

1. Uninstall K3s

2. Restart VM

3. You will need to slightly change the command you use to configure K3s into master nodes (you don’t need to change the command for workers nodes as they’ll read the configuration form the master ones) by appending --flannel-backend=host-gw , example:

curl -sfL https://$URL/k3s.sh | INSTALL_K3S_VERSION="v1.26.10+k3s1" K3S_KUBECONFIG_MODE="644" sh -s - server --node-name=master1 --cluster-init --flannel-backend=host-gw