Spinning out a High Available Kubernetes cluster with kubeadm

Deploying a production-ready Kubernetes cluster locally in virtual machines with kubeadm

Motivation

In most cases, a local cluster of one server is sufficient for application development. And in some implementations it is not a virtual machine but a container. There are a lot of tools k3d and minikube just few of them. Yes, in this case there may be no abstraction layers in the form of its own file system and interaction between nodes. But it's not always required.

But if you need to prepare for deploying a production cluster on your own, or to test the fault tolerance of distributed storage, or to prepare for the CNCF certification. In such cases I have found it useful to have a cluster on my laptop as similar as possible to the one in production.

It is also possible to deploy something similar to VPS servers, but in this case you may encounter difficulties characteristic only for a particular hosting. Therefore, I consider it a good practice to deploy a local cluster with instruments for deploying a production ready cluster.

Originally I used the hobby-kube tutorial. But many things have changed since then, for example kubeadm started to support deployment of high available etcd clusters. I found that some CNIs can encrypt traffic and don't work well with external firewall.

Running virtual machines

I settled on a layout where there are three control-plane nodes. And additionally there is an external load balancer, which is also located on a separate virtual machine. But it is possible to do without the balancer, but in this case more editing of "hosts" will be required.

I use Vagrantfile with kvm driver to deploy virtual machines. There are probably other ways, but I'm happy with this one.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu2204"

  config.vm.provider "libvirt" do |libvirt|
    libvirt.driver = "kvm"
    libvirt.management_network_name = 'vagrant-libvirt-rancher-academy'
    libvirt.management_network_address = '172.23.0.0/24'
  end

  config.vm.define "ha-proxy" do |config|
    config.vm.hostname = "ha-proxy"
    config.vm.provider :libvirt do |libvirt|
      libvirt.cpus = 1
      libvirt.memory = 2048
    end
  end

  config.vm.define "master-01" do |config|
    config.vm.hostname = "master-01"
    config.vm.provider :libvirt do |libvirt|
      libvirt.cpus = 3
      libvirt.memory = 4096
    end
  end

  config.vm.define "master-02" do |config|
    config.vm.hostname = "master-02"
    config.vm.provider :libvirt do |libvirt|
      libvirt.cpus = 3
      libvirt.memory = 4096
    end
  end

  config.vm.define "master-03" do |config|
    config.vm.hostname = "master-03"
    config.vm.provider :libvirt do |libvirt|
      libvirt.cpus = 3
      libvirt.memory = 4096
    end
  end

end

The same file but with comments at github. A few notes to this file:

  1. If ip address range 172.23.0.0.0/24 used by other network including docker. It is better to specify another network to avoid conflicts.
  2. The number of allocated resources is specified for virtual machines(libvirt.cpus, libvirt.memory). It's probably will work with much lower resources but it could work much slower.
  3. Ubuntu 22.04 is specified, but you can use some other operating system. I choose Ubuntu LTS because it is widely available distro across hosting providers.

After creating a Vagrant needed to spin-up those machines with command

vagrant up
# few more useful commands for Vagrant
# to make an ssh connection to specific node
vagrant ssh master-01
vagrant ssh master-02
vagrant ssh master-03
# turn off all machines from Vagrantfile
vagrant halt

HAProxy

HAProxy will know about all nodes and will balance load between them.

Installing proxy and text editor:

sudo apt-get update ; sudo apt-get install -y haproxy vim

After that editing configuration file and set correct addresses of each node but comment all nodes besides first. Check ip address possible via command ip addr

defaults
        log global #<-- Edit these three lines, starting around line 21
        mode tcp
        option tcplog

....

frontend proxynode #<-- Add the following lines to bottom of file
        bind *:80
        bind *:6443
        stats uri /proxystats
        default_backend k8sServers

backend k8sServers
        balance roundrobin
        server master1 172.23.0.254:6443 check #<-- Edit these with your IP addresses, port, and hostname
        # server master2 172.23.0.219:6443 check #<-- Comment out until ready
        # server master3 172.23.0.104:6443 check #<-- Comment out until ready


listen stats
        bind :9999
        mode http
        stats enable
        stats hide-version
        stats uri /stats

After that enable and restart haproxy

sudo systemctl enable haproxy
sudo systemctl restart haproxy

If configuration correct and haproxy works then it's possible to review state via browser http://172.23.0.135:9999/stats . Where the ip is address of ha-proxy.

Nodes

Installing a container runtime

It will be containerd from docker repo. It should be installed on each node of Kubernetes cluster.

# Add Docker's official GPG key:
sudo apt-get update
#sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Installing containerd
sudo apt-get update && sudo apt-get install -y containerd.io

After installation, reconfigure the containerd.

# reset configuration of containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
# update for systemd
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
# restart
sudo systemctl restart containerd

It's possible to double check that /etc/containerd/config.toml contain configuration for Cgroups that used by operation system on our nodes.

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

And check status of containerd service.

sudo systemctl status containerd

Install Kubernetes packages

Its should be installed on each node of Kubernetes cluster.

```bash
sudo apt-get update

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# sudo systemctl enable --now kubelet # optional step to speedup join and initialization of cluster

Initializing cluster

Before initializing cluster it's good idea to check that load balancer address could be resolved on each node. To achieve that we could simply add line to /etc/hosts file with next content

172.23.0.135 k8smaster
  • Where ip is address of virtual machine with HAProxy on it.
  • And k8smaster is control-plane-endpoint. It's good practice to use domain name instead of IP address in case if in future ip addresses of machine will be changed.
  • If access from host machine required this step should be done on host too.

Initialization of cluster should be made just once on the first control plane node. It will be master-01 in our case.

sudo kubeadm init --cri-socket=unix:///var/run/containerd/containerd.sock --pod-network-cidr 10.42.0.0/16 --control-plane-endpoint k8smaster --upload-certs

If the command runs successfully, it should end with two commands that can be used to connect nodes to the cluster. Let's keep the command for connecting control plane nodes(ie write down in text file with notes). And these commands have an expiration date, and by default they will not work after 24 hours and other commands will have to be executed to connect nodes to the cluster.

Next, let's copy the certificate for connecting to the cluster and check the number of nodes in it.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

The are should be one node and could be status "Not ready" it's okay for now.

Setup local access to cluster(optional)

In order to not connect to node on cluster and use ha-proxy to connect control plane nodes. It's possible to set up connection from host machine to kubernetes cluster via ha-proxy.

So commands that we need installing scp plugin for vagrant. Copy certificate from initialized node via that command. Well, one option is to set in the KUBECONFIG environment variable the path to this configuration.

vagrant plugin install vagrant-scp
vagrant scp master-01:/home/vagrant/.kube/config ~/.kube/local-kubeconfig.yaml
export KUBECONFIG=~/.kube/local-kubeconfig.yaml

Setup network

Will use Calico CNI since it's simple in installation and support a lot of things such as network policy and node to node traffic encryption with WireGuard.

Next command should be called from connected to cluster machine with installed helm and kubectl. There's tutorial of how to install Calico CNI with HELM.

helm repo add projectcalico https://docs.tigera.io/calico/charts
kubectl create namespace tigera-operator
helm install calico projectcalico/tigera-operator --version v3.27.3 --namespace tigera-operator

Before next steps it's important to wait when container network interface fully initialized.

watch kubectl get pods -n calico-system

Join Control Plane Nodes

Just run command that we get during cluster initialization. Command should be with flag `--control-plane` for Control Plane nodes. In my case it was command

sudo kubeadm join k8smaster:6443 --token s134l2.etz5wimavlwh4y5s \
        --discovery-token-ca-cert-hash sha256:19ceef551f7f9181e0260d44b86361bbd2070e8cf5d20117bbd56c87bc4cae21 \
        --control-plane --certificate-key 25b963211b4a216aaa1a4dc1d9f330e00de7e3ffd7d9fca041d37b82fc9f09e8

What's next

If everything went well. We should have a highly available kubernetes cluster(that's good) on a single physical machine(that's not so good, since it's a single point of failure). But we can move on. Try to deploy some applications. For example, ingress and storageclass are still missing here. In my opinion ingress-nginx and longhorn are not bad candidates for next steps. Deploy some applications in containers. If no additional nodes will be added, you should remove taint

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
Posted on May 7, 2024
With tags: kubernetes kubeadm vagrant tutorial