Highly Available RKE2 on Immutable openSUSE MicroOS

This is how I moved my Kubernetes cluster to three RKE2 servers on openSUSE MicroOS — an immutable OS that installs every update into a new snapshot and rolls back on its own if a node comes up broken. The article follows the build from an empty VPS to a dual-stack, highly available cluster with Calico, WireGuard-encrypted pod traffic, Gateway API ingress, coordinated reboots and etcd backups, and it includes the problems I ran into on the real nodes, with the commands to spot them on yours.

Motivation

I already run my projects on a Kubernetes cluster — three control-plane nodes on Ubuntu, built with kubeadm much like I described in Spinning out a High Available Kubernetes cluster with kubeadm. A few things pushed me to rebuild it.

Cost. In four years the price of the existing cluster doubled — for the same hardware, which is already seven years old. Moving to new servers came out cheaper.

Less manual upkeep. With kubeadm, every Kubernetes update is manual work — or automation I would have to build and maintain myself. Ubuntu adds a major release upgrade once in a while on top of that; I wrote about one of them in An upgrade to Jammy Jellyfish.

openSUSE. My workstation runs openSUSE Tumbleweed. It is a rolling release that follows new kernels as they come out: there is no major upgrade every few years, just a steady stream of smaller updates. I wanted the same model on the servers.

Something new. I had never run an immutable OS. openSUSE MicroOS is built from Tumbleweed and is immutable: a read-only /usr, a Btrfs snapshot for every change, and an automatic rollback if a node comes up broken after an update.

What this article builds

A three-server, highly available RKE2 cluster on plain KVM VPS instances. The base OS is openSUSE MicroOS, and RKE2 is installed as a proper RPM inside the same transactional update stream as the OS. There are no separate agents: RKE2 servers are schedulable by default, so all three run etcd, the control plane and workloads.

flowchart TB
    dns["k8s.example.com<br/>fixed registration address<br/>round-robin DNS: 3× A + 3× AAAA"]
    subgraph dc["One datacenter / region"]
        n1["k8s-1<br/>rke2-server<br/>etcd + control plane + workloads"]
        n2["k8s-2<br/>rke2-server<br/>etcd + control plane + workloads"]
        n3["k8s-3<br/>rke2-server<br/>etcd + control plane + workloads"]
    end
    dns --> n1
    dns --> n2
    dns --> n3

Target topology: three RKE2 servers behind one DNS name, no load balancer.

Everything below assumes a generic KVM/QEMU VPS provider whose panel gives you a console and the ability to boot a custom ISO (most do). Placeholders used throughout:

I built this in August 2026 on the MicroOS snapshot of that time, with RKE2 v1.35.7+rke2r1 from the stable channel.

Part 1. Why MicroOS and RKE2

The last row is the answer to the kubeadm problem from the motivation: Kubernetes patch releases arrive the same way as kernel updates, with nothing extra to run.

The trade-off: you cannot zypper install into the running system, and every OS-level change needs a reboot. In an HA cluster that is a feature, not a problem — but only if reboots are serialised, which is what Part 14 is about.

Why exactly three servers

etcd needs a quorum of (n/2)+1 members:

Three is the number.

Part 2. What you need

Per node (×3):

• CPU/RAM: RKE2's minimum is 2 vCPU / 4 GB; 4 vCPU / 8 GB is comfortable when servers also run workloads (see RKE2 requirements).
• Disk: 40 GB+ SSD/NVMe. etcd is latency-sensitive; spinning or throttled storage shows up as etcdserver: request timed out under load.
• Boot mode: the DVD boots on both legacy BIOS and UEFI. Prefer UEFI if the panel offers it.
• Disk device: virtio disks show up as /dev/vda in the installer, SATA/SCSI as /dev/sda.

Cluster-wide:

• All three in the same datacenter/region. etcd replicates every write synchronously; a cross-region member turns every API write into a WAN round-trip. Check the RTT between nodes before committing: ping -c 20 <IP2> from k8s-1 — single-digit milliseconds is what you want.
• Distinct hostnames. The node name comes from the hostname and must be unique.
• A global IPv6 address per node. The cluster is dual-stack, so each node needs working IPv6 (a /64 per VPS is the usual allocation) alongside its IPv4.
• DNS: one name for the cluster (k8s.example.com) with an A and an AAAA record per node — see Part 3.
• Time sync active on all three — etcd will not tolerate drift (timedatectl shows it).

The installer ISO lives at https://download.opensuse.org/tumbleweed/iso/. MicroOS is rolling, but openSUSE-MicroOS-DVD-x86_64-Current.iso always points at the latest snapshot.

Part 3. One name in front of the cluster — round-robin DNS on Cloudflare

Servers 2 and 3, every agent, and your kubectl all need one stable endpoint in front of the three servers. RKE2 accepts a load balancer, a virtual IP, or plain round-robin DNS (HA docs). I use the last one — there is no infrastructure to run.

Create k8s.example.com in the Cloudflare dashboard as six records under the name k8s: an A record for each node's IPv4 address (<IP1>, <IP2>, <IP3>) and an AAAA record for each node's IPv6 address (<IPV6_1>, <IPV6_2>, <IPV6_3>) — the API listens on both families. Set every one to DNS only (grey cloud) with a TTL of 1 min (or Auto).

Add a per-node name as well — k8s-1, k8s-2, k8s-3, each pointing at one node's A/AAAA pair, and also DNS only. These go into tls-san (Part 7), so that reaching one specific server never depends on an address baked into a certificate.

Or with the API — the token needs Zone → DNS → Edit on the zone:

CF_TOKEN=<TOKEN>; ZONE_ID=<ZONE_ID>
add() {  # add <A|AAAA> <address>
  curl -sS -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
    -H "Authorization: Bearer $CF_TOKEN" -H "Content-Type: application/json" \
    -d "{\"type\":\"$1\",\"name\":\"k8s\",\"content\":\"$2\",\"ttl\":60,\"proxied\":false}" \
    | jq -r '.success, (.errors[]?.message)'
}
for ip in <IP1> <IP2> <IP3>;          do add A    "$ip"; done
for ip in <IPV6_1> <IPV6_2> <IPV6_3>; do add AAAA "$ip"; done

The one thing that will break this: the orange cloud

The record must be proxied: false / DNS only. Cloudflare's proxy only forwards a fixed set of HTTP/HTTPS ports — 9345, 6443 and 2379/2380 are not among them — and even on 443 it terminates TLS at Cloudflare, which is fatal for a Kubernetes API that authenticates clients by certificate. A proxied record hands out Cloudflare's anycast IPs instead of your nodes, so joins fail with connection timeouts or x509 errors that look like a certificate problem and are not.

This is per-record, so it does not affect the rest of the zone: application hostnames can stay orange-clouded, and cert-manager's HTTP-01 solver keeps working through the proxy. Only the cluster endpoint is grey.

Verify before installing anything — this is the check that catches the mistake:

dig +short k8s.example.com A    @1.1.1.1
dig +short k8s.example.com AAAA @1.1.1.1

Expect exactly your three node IPs in each family. If you see 104.x.x.x or 172.67.x.x, the record is proxied: switch it to DNS only and re-check. Run it a few times to watch the answer order rotate.

What round-robin DNS does and does not give you

It has no health checking: a dead node's IP stays in rotation, and roughly one connection in three to k8s.example.com will hang until it times out. That is acceptable here because of where the address is actually used:

flowchart LR
    laptop["kubectl on your laptop"] -->|"k8s.example.com:6443<br/>retries if it hits a dead node"| rr["k8s.example.com<br/>round-robin, no health checks"]
    rr --> servers["k8s-1 · k8s-2 · k8s-3"]
    joiner["server that is joining"] -->|"server: k8s-1.example.com:9345<br/>one specific live server"| n1["k8s-1"]
    joined["node that has joined"] -->|"rke2 agent load-balances across<br/>the server addresses it learned"| servers

Who uses the round-robin name, and who deliberately does not.

• Joining a node — but point server: at a specific node instead (Part 8): the round-robin set contains the joining node itself, and a server that asks itself for the etcd member list loops instead of failing over.
• Your kubectl — kubectl retries; if a node is down long enough to annoy you, target a live one directly with --server=https://k8s-2.example.com:6443.
• Running cluster traffic — does not use it. Once a node has joined, rke2 agent load-balances across the real server addresses it learned from the cluster, so a dead server does not interrupt anything.

Keep the TTL low so replacing a node propagates quickly, and remember to update the records whenever a node's IP changes — nothing does that for you. (Cloudflare Load Balancing would add health checks and automatic removal, but it is a paid add-on and not needed for this.)

Publishing all three node IPs is inherent to this approach; the firewall rules in Part 6 are what keep the exposed ports down to 22/80/443 plus your admin IPs.

Part 4. Install MicroOS from the ISO — on all three nodes

The panel console is only needed twice per node: for a few seconds to edit the boot entry, and once after the install to put an SSH key in place. Everything else happens over SSH.

sequenceDiagram
    actor You
    participant Console as Panel console
    participant Installer as ISO installer
    participant Node as Installed node
    You->>Console: attach ISO, boot, edit the Installation entry
    Console->>Installer: boot with ssh=1 sshpassword=...
    Installer-->>You: linuxrc reports sshd is running
    You->>Installer: ssh root@IP1, then yast.ssh
    Installer->>Node: install, reboot, remove the ISO
    You->>Console: log in once, write authorized_keys, restorecon
    You->>Node: ssh root@IP1 with the key
    You->>Node: turn password authentication off

Where the console is unavoidable and where SSH takes over.

1. In the panel, attach openSUSE-MicroOS-DVD-x86_64-Current.iso (upload it or give the URL):

   https://download.opensuse.org/tumbleweed/iso/openSUSE-MicroOS-DVD-x86_64-Current.iso
   

2. Boot from it and run the installer over SSH instead of the panel's console. In the GRUB menu highlight Installation, press e, and append to the linux line:

   ssh=1 sshpassword=<PASSWORD>
   

   linuxrc (the ISO's pre-installer) then brings up the network, starts sshd and waits. If the VPS has no DHCP, also add ifcfg=eth0=<IP1>/24,<GATEWAY>,<DNS>.

3. Press F10/Ctrl-X to boot, wait for the console to report that sshd is running, then:

   ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@<IP1>
   

   You land on a plain shell; start the installer with yast.ssh. It is the ncurses YaST — Tab/arrows to move, Enter to activate, Alt+the highlighted letter for shortcuts. The throwaway host key is why the known_hosts flags are there: the installed system will present a different one.

4. In the installer: accept the license, then pick the MicroOS system role. MicroOS Container Host would only add Podman, which RKE2 does not need — it ships its own containerd.

   Accept the proposed partitioning. It puts the read-only Btrfs root and its snapshots on one partition (20 GB on my nodes) and /var on a separate Btrfs partition with the rest of the disk. Container images, etcd and volumes therefore never compete with the snapshots for space.

5. Set the root password, and on the installation summary screen make sure SSH service is enabled and the SSH port is open — this is about the installed system, not the installer's temporary sshd. Getting it wrong means finishing the job over the panel console.

6. Install, reboot, remove the ISO from the virtual drive.

Do this three times, with a different root password on each node.

Getting your SSH key onto the nodes

ssh-copy-id root@<IPn> does not work: openSUSE ships sshd with upstream's PermitRootLogin prohibit-password, so root may authenticate by key but never by password. (The distro is explicit about it — there is a separate package, openssh-server-config-rootlogin, whose entire content is a drop-in reading PermitRootLogin yes.) ssh-copy-id needs a password login to place the key, so the first key has to arrive another way.

Use the panel console once per node — console logins are not affected by that setting — and write root's authorized_keys by hand:

# on the node's console, as root
mkdir -p /root/.ssh && chmod 700 /root/.ssh
printf '%s\n' 'ssh-ed25519 AAAA...your key... you@laptop' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
restorecon -R /root/.ssh       # SELinux is enforcing; a wrong label fails auth silently

Do not skip restorecon: a file created in the right place with the wrong SELinux label is invisible to sshd, and the rejection looks exactly like a wrong key. Check it before you leave the console:

ls -Z /root/.ssh/authorized_keys     # expect ...:ssh_home_t:...

Typing a public key into a laggy console is unpleasant. Two ways to make it less so:

• Use a short key — ssh-keygen -t ed25519 produces a one-line key roughly a quarter the length of an RSA one.
• Or fetch it instead of typing it, if the key is already published somewhere you trust: curl -fsSL https://github.com/<user>.keys >> /root/.ssh/authorized_keys (then chmod/ restorecon as above, and read the file back to confirm you got only the keys you expect).

Now confirm from your laptop, before changing anything else:

ssh root@<IP1> hostnamectl

Once key login works, turn passwords off. Not because root is exposed today — prohibit-password already covers that — but because it makes the safe setting the default for whatever comes later: a user account added next month, or an openssh-server-config-rootlogin dependency quietly flipping PermitRootLogin to yes. It also means brute-force bots fail at method negotiation instead of PAM, which keeps the logs readable.

/etc/ssh/sshd_config.d/*.conf is included by the shipped sshd_config, and /etc is writable on MicroOS, so this needs no transactional-update:

echo 'PasswordAuthentication no' > /etc/ssh/sshd_config.d/60-no-passwords.conf
sshd -t && systemctl reload sshd

Keep one console session open until you have confirmed a new SSH login works.

Part 5. First boot sanity checks — on every node

hostnamectl set-hostname k8s-1        # k8s-2 / k8s-3 on the others; node name comes from here
getenforce                            # Enforcing
findmnt -no OPTIONS /                 # ro,...   -> root is read-only, as intended
snapper list                          # transaction history
timedatectl                           # NTP must be active; etcd hates clock drift

getenforce should say Enforcing with nothing done to arrange it: the MicroOS installer selects SELinux in enforcing mode and makes AppArmor unselectable (control.MicroOS.xml), and the pre-built images boot with security=selinux selinux=1 and rewrite /etc/selinux/config to match. The wiki's older transactional-update setup-selinux instruction is a Kubic-era leftover and is not needed. If a node does report Disabled, stop and fix that first — the restorecon steps in Part 4 and the rke2-selinux policy both assume it.

Network

Current MicroOS uses NetworkManager. Confirm it before configuring anything, because a wicked-based system needs different commands (and the forwarding sysctls in Part 6 become mandatory rather than merely recommended):

systemctl is-active NetworkManager wicked
nmcli device status                      # device names: ens3, eth0, ...
nmcli -g NAME,DEVICE connection show     # connection names, and which device each one drives

IPv4 on a KVM VPS is normally DHCP and needs nothing.

IPv6 must work on every node — the cluster is dual-stack. Check what the provider already did:

ip -6 addr show ens3 | grep 'scope global'   # your NIC name; a global address means SLAAC/DHCPv6 worked
ip -6 route show default                     # ... and a default route via fe80::1 or similar
ping -6 -c3 2606:4700:4700::1111             # ... and it actually reaches the internet

If all three succeed, NetworkManager has done the work already. Otherwise the provider expects static configuration — and given a /64 such as 2001:db8:1234:5678::/64, pick an address inside it. My servers use the :: address of the block itself, and that has worked fine; strictly, RFC 4291 reserves the all-zeros interface ID as the subnet-router anycast address, so ::1 is the more conventional choice.

To set it statically, note that nmcli con mod takes a connection name, not the device name — with a NIC called ens3 the connection is usually called ens3 too, but on a cloud image it is just as often Wired connection 1. Ask for it instead of guessing:

CON=$(nmcli -g GENERAL.CONNECTION device show ens3); echo "$CON"

nmcli con mod "$CON" \
  ipv6.method manual \
  ipv6.addresses <IPV6_1>/64 \
  ipv6.gateway fe80::1
nmcli con up "$CON"

Confirm the gateway against the panel before applying — fe80::1 is a common convention, not a rule, and a wrong gateway gives you an address with no route. Then re-run the three checks.

Finally, make sure every node can reach the other two on both families, and that all three agree on the time (timedatectl from the checks above):

for ip in <IP1> <IP2> <IP3>;       do ping  -c 3 -q "$ip"; done
for ip in <IPV6_1> <IPV6_2> <IPV6_3>; do ping6 -c 3 -q "$ip"; done

How transactional updates work

Every change to the OS goes into a new snapshot that only becomes active after a reboot. If the node does not come up healthy, health-checker rolls it back on its own:

flowchart LR
    run["Running system<br/>snapshot N, read-only /usr"] -->|"transactional-update<br/>dup / pkg install"| next["Snapshot N+1<br/>prepared, not active yet"]
    next -->|reboot| boot["Boot into N+1"]
    boot --> hc{"health-checker:<br/>node healthy?"}
    hc -->|yes| ok["N+1 is the running system"]
    hc -->|no| rb["Automatic rollback to N"]
    keep[("/var and /etc changes<br/>are not rolled back")]

One update cycle on MicroOS.

The commands you will use:

transactional-update dup                 # update the whole system into a new snapshot
transactional-update pkg install <pkg>   # install a package into a new snapshot
transactional-update -c pkg install <p>  # ... stacked on top of the pending snapshot
transactional-update shell               # interactive shell inside a new snapshot
transactional-update apply               # activate without rebooting (services only)
transactional-update rollback last       # go back
reboot                                   # every change needs one

Part 6. Prepare each node for RKE2

Everything in this part is per-node — run it on all three.

NetworkManager must ignore CNI interfaces

Otherwise it fights Calico over routes:

cat > /etc/NetworkManager/conf.d/rke2-canal.conf <<'EOF'
[keyfile]
unmanaged-devices=interface-name:flannel*;interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
EOF
systemctl reload NetworkManager

Nothing installs this file for you — rpm -ql rke2-common lists no NetworkManager drop-in, and neither does the install script. The content is copied verbatim from RKE2's known issues → NetworkManager; the file name refers to canal, RKE2's default CNI (Flannel for the datapath plus Calico for policy).

With cni: calico there is no Flannel on the node at all — Calico does both jobs — so interface-name:flannel* matches nothing. Leave it in anyway: it costs nothing and keeps the file byte-identical to the documented one if you ever switch CNI. The entries doing the real work on a Calico node are cali*, tunl* (IPIP), vxlan*.calico (VXLAN) and wireguard.cali / wg-v6.cali (encryption).

Sysctls

Forwarding plus inotify headroom, applied at the node level rather than through a privileged DaemonSet:

cat > /etc/sysctl.d/90-rke2.conf <<'EOF'
net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
EOF
sysctl --system

One package: semanage

policycoreutils-python-utils provides semanage for the next step. MicroOS does not ship it:

transactional-update pkg install policycoreutils-python-utils
reboot

Label Calico's state directory for SELinux

Calico keeps state in /var/lib/calico, which is outside RKE2's data directory — and rke2-selinux only labels /var/lib/rancher/rke2, /opt/cni and /etc/cni. Left with the default var_lib_t, the tigera-operator cannot read the MTU file that calico-node writes there. calico-node runs privileged and is not affected, but the operator runs as an ordinary container (container_t) and reads the directory through a hostPath mount, so every reconcile fails:

calico  False  ...  error reading network MTU: open /var/lib/calico/mtu: permission denied

On a fresh install everything downstream waits on it — tigerastatus shows gatewayapi: Waiting for Installation Variant to be set. On a running cluster the same error can appear days later: the operator is a single pod that can be rescheduled to any node, and as soon as it lands on one without the label, calico goes degraded. kubectl -n tigera-operator get pods -o wide shows where it runs, and that node's /var/log/audit/audit.log has the denial (comm="operator" name="mtu"). (tiers: Waiting for Tigera API server to be ready is unrelated and expected with RKE2 — see Part 18.)

Fix it with a policy rule rather than a one-off chcon, so it also survives a full relabel after a selinux-policy update and the directory being recreated:

mkdir -p /var/lib/calico
semanage fcontext -a -t container_file_t '/var/lib/calico(/.*)?'
restorecon -RFv /var/lib/calico
ls -Zd /var/lib/calico            # ...:container_file_t:...
restorecon -nv /var/lib/calico    # silence: a future relabel leaves it alone

# on a cluster that is already running: no restart needed, the operator retries every 5 minutes
kubectl get tigerastatus          # calico: AVAILABLE True, DEGRADED False

firewalld must not run

It periodically reloads and flushes chains it does not recognise, taking Calico's and kube-proxy's rules with them; RKE2 says to disable it. If it is absent — which it may well be, depending on the system role — that is the finished state, not something to install. The check below only catches a node where something pulled it in:

rpm -q firewalld && systemctl disable --now firewalld

Disabling it does not leave the node unfiltered at the kernel level — Calico and kube-proxy still program iptables/nftables directly. It does mean the ports table below has to be enforced somewhere, in this order of preference:

1. The provider's firewall, if it offers one. Filtering before the packet reaches the NIC cannot be flushed by anything running on the node.
2. Calico host-endpoint policy — HostEndpoint plus GlobalNetworkPolicy, which is what RKE2's note means by "custom firewall rules can be implemented through Calico resources". Rules live in Git with everything else. Two things to know before treating it as the perimeter: it only filters while Felix is running (so not during boot, a crash, or a Calico upgrade), and Felix's failsafe ports are allowed no matter what your policy says — by default tcp:22,udp:68,tcp:179,tcp:2379,tcp:2380,tcp:5473,tcp:6443,tcp:6666,tcp:6667 inbound, which leaves etcd and the API reachable from anywhere. Narrowing failsafeInboundHostPorts is possible, but a mistake there locks you out of all three nodes at once.
3. Plain nftables in /etc/nftables.conf — fine, but add your own table and never flush the ruleset, or you reproduce the firewalld problem by hand.

Ports

Three public VPS means the control plane talks over the internet. Everything RKE2 opens between servers is TLS with mutual certificate auth (etcd peers, the supervisor, the apiserver, kubelet), so the risk is exposure and abuse rather than eavesdropping — but the cluster ports still have no business being reachable from anywhere except the other two nodes.

Restrict at the provider's firewall: allow the peer IPs to everything below, the world to nothing but 22/80/443, and your admin IPs to 6443. Write every rule twice — once for IPv4, once for IPv6. The cluster is dual-stack and the API listens on IPv6 too; an IPv4-only rule set filters nothing on IPv6.

Source: RKE2 inbound network rules. Pod traffic on 4789 is not encrypted by default — that is what Part 12 fixes.

Part 7. Install RKE2 on the first server

Install the RPM on all three nodes now — only the config files differ.

The repository, not the install script

get.rke2.io is a wrapper around the two supported methods, and on MicroOS it picks the wrong one: the docs state plainly that "SLE Micro and MicroOS default to a tarball installation". A tarball lands in /opt/rke2 (because /usr/local is read-only) and sits outside the update stream. So add the repository directly, exactly as the MicroOS tab of the install docs describes:

export RKE2_MINOR=35
cat << EOF > /etc/zypp/repos.d/rancher-rke2-1-${RKE2_MINOR}-stable.repo
[rancher-rke2-common-stable]
name=Rancher RKE2 Common Stable
baseurl=https://rpm.rancher.io/rke2/stable/common/microos/noarch
enabled=1
autorefresh=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key

[rancher-rke2-1-${RKE2_MINOR}-stable]
name=Rancher RKE2 1.${RKE2_MINOR} Stable
baseurl=https://rpm.rancher.io/rke2/stable/1.${RKE2_MINOR}/microos/x86_64
enabled=1
autorefresh=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key
EOF

MicroOS is immutable, so the install goes through transactional-update and needs a reboot:

transactional-update --no-selfupdate -d run zypper --gpg-auto-import-keys install -y rke2-server
reboot

(rke2-agent instead of rke2-server on agent nodes — see Part 11.)

Three deliberate differences from the docs' snippet:

• stable instead of latest. The docs use the latest channel; stable is the channel Rancher recommends running in production, and it is what get.rke2.io defaults to.
• The minor is pinned in the repo URL (1.35). That is what keeps transactional-update dup on patch releases only — a minor upgrade becomes a deliberate act of editing this file. Use the same minor on all three nodes.
• autorefresh=1. A .repo file without it defaults to no auto-refresh, and transactional-update dup does not refresh by itself — the node keeps resolving against the metadata it cached at install time and never sees a new patch release, while the OS repos (which do auto-refresh) keep updating and kured keeps rebooting. Nothing looks broken; RKE2 just silently stays on the version you installed.

Verify before moving on — an RPM install answers rpm -q, a tarball install leaves /opt/rke2:

rpm -q rke2-server rke2-common rke2-selinux
zypper lr -u | grep rancher            # both repos, pointing at .../microos/...
ls /opt/rke2 2>/dev/null && echo "tarball install — not what we want on MicroOS"

rke2-selinux pulling in as a dependency is the sign the RPM path worked; the tarball never installs the policy.

The shared token

Generate it once, on your laptop, and use the same value on all three servers:

openssl rand -hex 32

(If you skip token:, RKE2 generates one on the first server. Either way the value in use ends up in /var/lib/rancher/rke2/server/token, in the secure K10<CA hash>::<credentials> format that lets a joining node verify the cluster before sending credentials.)

Back it up with your etcd snapshots. The server token doubles as the PBKDF2 passphrase for the bootstrap data in the datastore, so a snapshot can only be restored with the token that was in use when it was taken.

If a node ends up with the wrong one:

• Server 1, already started — nothing is broken; the token is a random secret, not a name. Either adopt what it generated (cat /var/lib/rancher/rke2/server/token) and use that on the other two, or rotate to the value you meant: rke2 token rotate --token <OLD> --new-token <NEW>, then update config.yaml and restart every server and agent — the old token stops working (docs).

• A joining server — the join simply fails, nothing reached the cluster. Fix config.yaml, then clear the state it cached before retrying:

  rke2-killall.sh                 # stops containers and processes; deletes nothing
  rm -rf /var/lib/rancher/rke2    # db, certs, token, images — /etc/rancher/rke2 is untouched
  systemctl start rke2-server
  

  A node that never joined holds no unique state, so the whole data directory can go — but never run this on a surviving etcd member, where that directory is the cluster.

  If it got as far as appearing in kubectl get nodes, kubectl delete node <name> first — the etcd member goes with it.

Config for server 1

mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<'EOF'
token: <SHARED_TOKEN>

write-kubeconfig-mode: "0600"

# names — not addresses — that clients use to reach the API
tls-san:
  - k8s.example.com
  - k8s-1.example.com
  - k8s-2.example.com
  - k8s-3.example.com

# dual-stack: identical on all three servers
cluster-cidr: "10.42.0.0/16,fd00:42::/56"
service-cidr: "10.43.0.0/16,fd00:43::/112"

# this node's own addresses, v4 first
node-ip: "<IP1>,<IPV6_1>"
node-external-ip: "<IP1>,<IPV6_1>"

cni: calico
ingress-controller: none          # ingress comes from a Gateway API gateway (Part 13)

selinux: true                     # containerd labels containers; not set by the RPM

etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 20
EOF
chmod 600 /etc/rancher/rke2/config.yaml

Notes on the choices:

• cni: calico — the CNI my previous cluster already used, and it earns its place twice over: NetworkPolicy enforcement (RKE2's default canal gets that from Calico anyway, but Calico alone does routing and policy in one component), and Gateway API — the Tigera operator ships a GatewayAPI CR that stands up an Envoy data plane, so the ingress layer comes from the CNI instead of being a separate controller to install and upgrade. Valid values are none, calico, canal (default), cilium, flannel.

• ingress-controller: none — RKE2 v1.35 still packages ingress-nginx, and v1.36 switches the default to Traefik. Turn it off if you route with Gateway API.

• disable: accepts rke2-coredns, rke2-metrics-server, rke2-snapshot-controller, rke2-snapshot-controller-crd, rke2-snapshot-validation-webhook. Only disable the snapshot trio if you don't do CSI volume snapshots.

• No image-GC tuning. kubelet's defaults (evict images above 85% of the image filesystem, down to 80%) are fine. Images live under /var, which is a separate partition from the root and its snapshots (Part 4), so a full image filesystem cannot starve transactional-update. Check the layout and the usage on your nodes:

  df -h / /var
  crictl imagefsinfo
  

  If you do want different thresholds, set them as a config key — kubelet-arg: ["image-gc-high-threshold=70", "image-gc-low-threshold=60"] — which survives reboots and upgrades, unlike editing /var/lib/kubelet/config.yaml by hand.

• Critical configuration values. cluster-cidr, service-cidr, cluster-dns, cluster-domain, egress-selector-mode, disable-kube-proxy and disable-cloud-controller become critical configuration values once you override them, and they must then be identical on all three servers or the join fails with critical configuration value mismatch. The dual-stack config above overrides cluster-cidr and service-cidr, so copy those two lines byte for byte.

• tls-san holds names, not IPs. Each server already puts its own addresses in its serving certificate automatically — 127.0.0.1, ::1, localhost, the node name and short name, every node-ip, every node-external-ip, and the apiserver Service IP (k3s pkg/cli/server/server.go, the codebase RKE2 shares). Listing node IPs would duplicate that and pin the certificate to addresses you may want to change — see "Replacing a node" in Part 14.

• node-external-ip mirrors node-ip because these addresses are public. It is what puts ExternalIP in the node's status.addresses, and it is the value that stays correct if node-ip ever moves to a private network.

Dual-stack: pod and Service IPv6

Dual-stack is a create-time decision — RKE2 cannot enable it later on a cluster that started IPv4-only — so it goes in the config now, on all three servers.

cluster-cidr and service-cidr take a comma-separated pair; node-ip and node-external-ip take each node's own pair:

cluster-cidr: "10.42.0.0/16,fd00:42::/56"
service-cidr: "10.43.0.0/16,fd00:43::/112"
node-ip: "<IP1>,<IPV6_1>"
node-external-ip: "<IP1>,<IPV6_1>"

Do not set bind-address: "::". RKE2 polls its own API server at whatever bind address you configure, and no certificate can carry the unspecified address as a SAN, so the server never reaches ready:

Polling for API server readiness: GET /readyz failed: ... x509: certificate is valid for
127.0.0.1, ::1, <IP1>, <IPV6_1>, 10.43.0.1, not ::

Left unset, the poll uses loopback — which is in the cert — and you get the IPv6 listener anyway: both the API and the supervisor come up on dual-stack sockets by default, serving IPv4 and IPv6 from one socket each. Nothing needs configuring for that; see the check after the first start.

The pod and Service ranges are ULA (fd00::/8) with natOutgoing, so they need no cooperation from the provider; keep the /56 and /112 sizes, which are the shapes RKE2's own example uses.

Pin VXLAN on both pools — the operator rejects IPIP for IPv6 but accepts VXLAN, and explicit pools beat inherited defaults. Drop this on the first server before its first start. It becomes a cluster object, so servers 2 and 3 join with it already in place and need no copy; keep the file on that one server and make every later change there (Part 12 adds WireGuard to it):

mkdir -p /var/lib/rancher/rke2/server/manifests
cat > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-calico
  namespace: kube-system
spec:
  valuesContent: |-
    installation:
      calicoNetwork:
        bgp: Disabled
        ipPools:
          - cidr: 10.42.0.0/16
            encapsulation: VXLAN
            natOutgoing: Enabled
            blockSize: 26
          - cidr: fd00:42::/56
            encapsulation: VXLAN
            natOutgoing: Enabled
            blockSize: 122
EOF

First start

You should already be rebooted into the snapshot that contains RKE2 (the install step ends with reboot, and rke2-server must not start before the SELinux policy is loaded). Start the first server:

systemctl enable --now rke2-server.service
journalctl -u rke2-server -f

The first start takes a few minutes: it extracts the runtime image, brings up etcd, the static control-plane pods, then Calico and CoreDNS. Wait for the node to go Ready:

export PATH=$PATH:/var/lib/rancher/rke2/bin KUBECONFIG=/etc/rancher/rke2/rke2.yaml
kubectl get nodes

Check what the API is listening on:

ss -lntp '( sport = :6443 or sport = :9345 )'
curl -k https://[<IPV6_1>]:6443/version

Read the address column carefully — it tells you the family without any guessing:

On my nodes both 6443 and 9345 show up as *.

Check the IP pools before joining anything else — two pools, both VXLAN, and pods with an address from each:

kubectl get ippools.crd.projectcalico.org \
  -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,VXLAN:.spec.vxlanMode,NAT:.spec.natOutgoing'

kubectl -n kube-system get pods -l k8s-app=kube-dns \
  -o custom-columns='NAME:.metadata.name,IPS:.status.podIPs[*].ip'

Only one pool, or a pool with vxlanMode: Never, means the HelmChartConfig did not apply — kubectl -n kube-system get helmchartconfig rke2-calico should exist, and its name must match the chart exactly. Fix it and rebuild this node: the CIDRs cannot be changed once the cluster is running, so this is far cheaper now than after servers 2 and 3 have joined.

Part 8. Join servers 2 and 3

Same repo file, same transactional-update install, same reboot — no Calico pool manifest, it is already in the cluster — plus one extra line in the config: server:, pointing at a specific running server, not at k8s.example.com.

The round-robin name resolves to all three nodes, including the one that is joining. If it picks itself, the new server asks itself for the etcd member list, is not a member yet, and loops:

Sending HTTP/1.1 500 response to [<its own address>]: failed to get etcd MemberList: context deadline exceeded
Waiting to retrieve etcd cluster member list: ...

sequenceDiagram
    participant J as k8s-2, joining
    participant RR as k8s.example.com
    participant S1 as k8s-1, running
    alt server points at the round-robin name
        J->>RR: resolve
        RR-->>J: any of the three addresses, sometimes its own
        J->>J: ask itself for the etcd member list
        J-->>J: 500 failed to get etcd MemberList, retry forever
    else server points at k8s-1.example.com
        J->>S1: join with the shared token
        S1-->>J: etcd member list, k8s-2 becomes a member
    end

The self-join trap, and the one-line fix.

Point server: at k8s-1.example.com (or node 1's literal address, which its certificate covers) and the problem cannot occur. Once a node has joined, server: is ignored on later starts — etcd data on disk decides — so there is nothing to change back.

Do them one at a time, waiting for each to reach Ready before starting the next — etcd should only ever add one member at a time.

On k8s-2 (and then k8s-3, with its own IP):

mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<'EOF'
server: https://k8s-1.example.com:9345    # a specific live server, NOT the round-robin name
token: <SHARED_TOKEN>

write-kubeconfig-mode: "0600"

tls-san:
  - k8s.example.com
  - k8s-1.example.com
  - k8s-2.example.com
  - k8s-3.example.com

cluster-cidr: "10.42.0.0/16,fd00:42::/56"
service-cidr: "10.43.0.0/16,fd00:43::/112"

node-ip: "<IP2>,<IPV6_2>"
node-external-ip: "<IP2>,<IPV6_2>"

cni: calico
ingress-controller: none
selinux: true

etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 20
EOF
chmod 600 /etc/rancher/rke2/config.yaml

systemctl enable --now rke2-server.service
journalctl -u rke2-server -f

Everything except server:, node-ip and node-external-ip is identical across the three files — cluster-cidr and service-cidr especially, since a mismatch there is what produces critical configuration value mismatch. Diff them if a join misbehaves.

Because server: is ignored once a node has etcd data on disk, re-pointing a node at a different cluster requires wiping /var/lib/rancher/rke2/server/db.

When a join gets stuck

The join is a sequence, and each step fails differently. Read the joining node's log first, then the server it is joining.

For the third case, the 500s in the log name the address being asked. Compare it with the node's own node-ip:

getent hosts k8s.example.com            # does the round-robin set include this node itself?
grep ^server: /etc/rancher/rke2/config.yaml

If it is asking itself, that is the round-robin trap above. If it is asking another node, check that node's etcd health and the peer ports:

# on the node being joined
curl -s http://127.0.0.1:2381/metrics | grep '^etcd_server_has_leader'
journalctl -u rke2-server | grep -iE 'member|learner|adding' | tail

No member-add line there means the joining node never asked. Meanwhile, on the joiner, check whether its own etcd was ever started:

export PATH=$PATH:/var/lib/rancher/rke2/bin CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
ls /var/lib/rancher/rke2/agent/pod-manifests/     # etcd.yaml written?
pgrep -af etcd                                    # process running?
crictl logs $(crictl ps -aq --name etcd | head -1) 2>&1 | tail -40

Whatever the cause, the recovery is the same: wipe the node (the commands under "The shared token" in Part 7) and start again with server: pointing at a member you know is healthy. Delete a half-registered node object from the cluster first, so its etcd member goes with it.

Part 9. Verify the cluster is actually HA

kubectl get nodes -o wide                              # three nodes, all Ready
kubectl get nodes -l node-role.kubernetes.io/etcd=true # three etcd members
kubectl get pods -A                                    # nothing stuck outside Running/Completed

etcd health, on each server — RKE2's etcd serves metrics on port 2381, so no client certs and no etcdctl are needed:

curl -s http://127.0.0.1:2381/metrics | grep -E '^etcd_server_has_leader|^etcd_server_leader_changes_seen_total'

etcd_server_has_leader 1 on all three is the answer you want. A node reporting 0, or a leader-changes counter that keeps climbing, means the members cannot talk to each other reliably — check the peer port (2380) and the RTT between nodes before going further.

Dual-stack across nodes — the test that actually exercises IPv6 VXLAN, since a single node proves nothing about encapsulation:

kubectl run n1 --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"k8s-1"}}' -- sleep 3600
kubectl run n2 --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"k8s-2"}}' -- sleep 3600
kubectl exec n1 -- ping6 -c3 "$(kubectl get pod n2 -o jsonpath='{.status.podIPs[1].ip}')"
kubectl delete pod n1 n2

The real test is a failure test, and it is worth doing before you put anything important on the cluster. Reboot one node and watch from another:

# on k8s-3
kubectl drain k8s-3 --ignore-daemonsets --delete-emptydir-data
reboot
# from k8s-1, meanwhile:
kubectl get nodes -w        # k8s-3 goes NotReady, the API keeps answering
# once it is back:
kubectl uncordon k8s-3

If the API stops answering while one node is down, you do not have an HA cluster — most likely your kubeconfig or registration address points at that single node.

Part 10. Access the cluster

On the nodes

RKE2 extracts kubectl, crictl and ctr into /var/lib/rancher/rke2/bin and puts its kubeconfig in /etc/rancher/rke2/rke2.yaml, so none of them work out of the box on a node. crictl additionally needs CRI_CONFIG_FILE to find RKE2's containerd socket. One drop-in on each node fixes all three for login shells:

cat > /etc/profile.d/rke2.sh <<'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
EOF

From your laptop

Point the kubeconfig at the registration address, not at a single node — otherwise losing that one node locks you out of a perfectly healthy cluster:

ssh root@<IP1> cat /etc/rancher/rke2/rke2.yaml > ~/.kube/prod.yaml
sed -i 's|127.0.0.1|k8s.example.com|' ~/.kube/prod.yaml
KUBECONFIG=~/.kube/prod.yaml kubectl get nodes

That works because k8s.example.com is in tls-san. If you would rather not expose 6443 to the internet at all, drop it from the provider's firewall and tunnel through one of the nodes: ssh -L 6443:127.0.0.1:6443 root@<IP1>.

Part 11. Optional: agent (worker) nodes

Three servers already run workloads, so agents are only about capacity or isolation. I added one later. Same OS build, same Part 6 prep, then:

# same repo file as Part 7, then:
transactional-update --no-selfupdate -d run zypper --gpg-auto-import-keys install -y rke2-agent
reboot

mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<'EOF'
server: https://k8s.example.com:9345
token: <SHARED_TOKEN>
node-ip: "<IP4>,<IPV6_4>"
node-external-ip: "<IP4>,<IPV6_4>"
selinux: true
EOF
chmod 600 /etc/rancher/rke2/config.yaml

systemctl enable --now rke2-agent.service     # note: agent, not server

"Same Part 6 prep" means all of it — including the semanage label on /var/lib/calico. calico-node runs on agents too, and the tigera-operator can be scheduled onto an agent like any other pod; an unlabelled directory there puts calico into the degraded permission denied state from Part 6. That is exactly what happened on my agent: the label was missed, nothing showed until the operator pod was rescheduled onto it, and from then on calico stayed degraded for eight days.

Everything node-level applies to agents as well: REBOOT_METHOD=kured (Part 14), and the Longhorn packages (Part 16) if volumes will land there. The unprivileged-port sysctl (Part 13) is needed only where Envoy runs; with the node affinity shown in Part 13, that is the three servers. Add the agent's addresses, both families, to the node-to-node firewall rules, and to the app hostnames' A/AAAA records if it serves traffic.

The round-robin name is fine for server: here: an agent is not behind that name and never asks for the etcd member list, so the self-join loop from Part 8 cannot happen.

If you later want servers to stop running workloads, taint all three with node-taint: ["CriticalAddonsOnly=true:NoExecute"] — but only after agents exist to take over.

Part 12. Encrypt pod traffic between nodes

Control-plane traffic between servers is already mutually authenticated TLS. Pod traffic is not: with VXLAN it is plain UDP 4789 across the public internet. Calico closes that with WireGuard — below, with the evidence at each step.

flowchart LR
    subgraph before["Before"]
        a1["pod n1 on k8s-1"] -->|"VXLAN, UDP 4789<br/>payload readable on the wire"| a2["pod n2 on k8s-2"]
    end
    subgraph after["After wireguardEnabled + wireguardEnabledV6"]
        b1["pod n1 on k8s-1"] -->|"WireGuard, UDP 51820 v4 / 51821 v6<br/>nothing readable"| b2["pod n2 on k8s-2"]
    end

Pod-to-pod traffic between nodes, before and after enabling WireGuard in Calico.

Tools first; MicroOS ships neither:

transactional-update pkg install tcpdump wireguard-tools && reboot   # on every node

Show that pod traffic is readable

Two pods on different nodes, and a payload you can grep for. ping -p takes pad bytes in hex and repeats them to fill the ICMP payload, so feed it the ASCII bytes of a word — 574754455354 is W G T E S T (printf 'WGTEST' | xxd -p). The packet body becomes WGTESTWGTEST…, which tcpdump -A prints as text and grep can find:

kubectl run n1 --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"k8s-1"}}' -- sleep 3600
kubectl run n2 --image=nicolaka/netshoot --overrides='{"spec":{"nodeName":"k8s-2"}}' -- sleep 3600
kubectl wait --for=condition=Ready pod/n1 pod/n2

P4=$(kubectl get pod n2 -o jsonpath='{.status.podIPs[0].ip}')
P6=$(kubectl get pod n2 -o jsonpath='{.status.podIPs[1].ip}')

Capture on k8s-1 into files, bounded by time rather than packet count — tcpdump -c N waits for N matching packets and a pipe buffers, so a live | grep usually just appears to hang:

# on k8s-1 — both captures run for 30s in the background
timeout 30 tcpdump -ni ens3 -A -l 'ip  and udp port 4789' >/tmp/v4.txt 2>/dev/null &
timeout 30 tcpdump -ni ens3 -A -l 'ip6 and udp port 4789' >/tmp/v6.txt 2>/dev/null &

# from your laptop, immediately
kubectl exec n1 -- ping  -c 20 -p 574754455354 "$P4"
kubectl exec n1 -- ping6 -c 20 -p 574754455354 "$P6"

# on k8s-1, once the captures exit
grep -c WGTEST /tmp/v4.txt /tmp/v6.txt

Both counts are non-zero: the marker is coming straight out of the VXLAN payload, on both families, on a public network.

Zero counts mean the experiment is not measuring what you think. Check, in order: that any VXLAN is flowing at all (timeout 15 tcpdump -ni ens3 -c 5 'udp port 4789' while pinging), that WireGuard is not already enabled (kubectl get felixconfiguration default -o jsonpath='{.spec.wireguardEnabled}'), and that the two pods really landed on different nodes (kubectl get pod n1 n2 -o wide).

Enable WireGuard

Open UDP 51820 (v4) and 51821 (v6) between all nodes at the provider's firewall first — Calico uses one port per family — and remember the v6 rules, so the tunnels can come up as soon as the keys appear. wireguard.cali and wg-v6.cali are already in the NetworkManager unmanaged-devices list from Part 6.

modinfo wireguard >/dev/null && echo ok      # in-tree since 5.6; MicroOS is well past that

Turn it on through the same HelmChartConfig that pins the IP pools (Part 7) — on k8s-1, where that file lives — and not with kubectl patch:

# on k8s-1
cat > /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml <<'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-calico
  namespace: kube-system
spec:
  valuesContent: |-
    felixConfiguration:
      wireguardEnabled: true
      wireguardEnabledV6: true
    installation:
      calicoNetwork:
        bgp: Disabled
        ipPools:
          - cidr: 10.42.0.0/16
            encapsulation: VXLAN
            natOutgoing: Enabled
            blockSize: 26
          - cidr: fd00:42::/56
            encapsulation: VXLAN
            natOutgoing: Enabled
            blockSize: 122
EOF

kubectl get felixconfiguration default \
  -o jsonpath='v4={.spec.wireguardEnabled} v6={.spec.wireguardEnabledV6}{"\n"}'   # v4=true v6=true

The reason for the detour: the rke2-calico chart renders its own FelixConfiguration default, and its values set wireguardEnabled: false. On every RKE2 upgrade Helm re-applies the chart and takes that field back, so a kubectl patch lasts only until the next upgrade. Values under felixConfiguration in the HelmChartConfig go into that same rendered object, so the chart itself keeps WireGuard on. The managed fields show who owns what:

kubectl get felixconfiguration default -o yaml --show-managed-fields | sed -n '/managedFields/,/^  name:/p'

I learned this the hard way: I had enabled WireGuard with kubectl patch, and the next RKE2 upgrade quietly set wireguardEnabled back to false. Nothing broke and nothing warned — IPv4 pod traffic simply went back to crossing the internet unencrypted. IPv6 stays encrypted only because the chart does not set wireguardEnabledV6, which makes the half-off state even easier to miss.

Each node should now publish a key per family:

kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,WG-V4:.metadata.annotations.projectcalico\.org/WireguardPublicKey,WG-V6:.metadata.annotations.projectcalico\.org/WireguardPublicKeyV6'

Same commands, different result

Repeat the capture exactly — same pods, same pings, same filters:

# on k8s-1 — same two captures as before, plus one on the tunnel ports
timeout 30 tcpdump -ni ens3 -A -l 'ip  and udp port 4789' >/tmp/v4-after.txt 2>/dev/null &
timeout 30 tcpdump -ni ens3 -A -l 'ip6 and udp port 4789' >/tmp/v6-after.txt 2>/dev/null &
timeout 30 tcpdump -ni ens3 -l 'udp port 51820 or udp port 51821' >/tmp/wg.txt 2>/dev/null &

# from your laptop, immediately
kubectl exec n1 -- ping  -c 20 -p 574754455354 "$P4"
kubectl exec n1 -- ping6 -c 20 -p 574754455354 "$P6"

# on k8s-1, once the captures exit
grep -c WGTEST /tmp/v4-after.txt /tmp/v6-after.txt   # both 0
wc -l /tmp/wg.txt                                    # the traffic moved here

Two more confirmations, from a node:

ip route get "$P4"      # dev wireguard.cali
ip -6 route get "$P6"   # dev wg-v6.cali

wg show wireguard.cali  # the other nodes as peers, recent handshake, non-zero transfer
wg show wg-v6.cali

A peer with no handshake means 51820/51821 is blocked between those two nodes. Clean up the test pods with kubectl delete pod n1 n2.

What this does not cover: wireguardEnabled protects pod traffic. Node-to-node control-plane traffic — etcd, the supervisor, kubelet — keeps its own mutual TLS and does not ride the tunnel.

If your provider offers a private network between instances, using it for node-ip is the other valid answer, and cheaper on CPU. It is also a bigger change: every node-ip/tls-san value and firewall rule moves to the private range, and the registration address has to stay reachable.

Part 13. Ingress: the Calico Gateway API data plane

There is no cloud load balancer here, so traffic has to arrive on the nodes themselves. The shape I use — and the reason ingress-controller: none is in the server config — is Calico's Gateway API support running Envoy as a host-network DaemonSet that binds 80/443 on each server. With three nodes that is also an HA win: any of them can serve any hostname.

flowchart LR
    client["Client"] --> cf["Cloudflare proxy<br/>app hostnames, orange cloud"]
    cf -->|"A + AAAA records<br/>for all three nodes"| e1["k8s-1<br/>Envoy, hostNetwork :80 / :443"]
    cf --> e2["k8s-2<br/>Envoy, hostNetwork :80 / :443"]
    cf --> e3["k8s-3<br/>Envoy, hostNetwork :80 / :443"]
    e1 --> pods["Service and pods<br/>on any node"]
    e2 --> pods
    e3 --> pods

Request path: any node that receives the request has an Envoy listening.

The bundled Calico is new enough to do this without installing Envoy Gateway yourself. RKE2 v1.35 ships Calico v3.32 (charts/chart_versions.yaml), and the v3.32 operator includes the gatewayapis.operator.tigera.io CRD:

kubectl get crd gatewayapis.operator.tigera.io

Let non-root Envoy bind 80/443 — on every node that runs it

Envoy runs as uid 65532 with all capabilities dropped, so binding privileged ports needs a node sysctl rather than a capability grant:

cat > /etc/sysctl.d/99-unprivileged-ports.conf <<'EOF'
net.ipv4.ip_unprivileged_port_start=0
EOF
sysctl --system
sysctl net.ipv4.ip_unprivileged_port_start   # expect: ... = 0

Despite the ipv4 in its name, this one setting covers IPv6 sockets too — there is no net.ipv6 counterpart. sysctl net.ipv6.ip_unprivileged_port_start answers cannot stat /proc/sys/net/ipv6/ip_unprivileged_port_start: No such file or directory, and a line for it in the file only makes sysctl report that error. With just the IPv4 line, Envoy on my nodes listens on *:443, which is both families.

Also make sure nothing else already owns those ports — an old ingress-nginx, or a stray web server:

ss -lntp '( sport = :80 or sport = :443 )'    # expect no output before the gateway exists

If something holds them, the Envoy pods CrashLoop with address already in use.

Enable Gateway API in the operator

Two steps, in this order. The first creates the tigera-gateway namespace and installs the Gateway API and Envoy Gateway CRDs; only then does the EnvoyProxy kind exist for the second step:

kubectl apply -f - <<'EOF'
apiVersion: operator.tigera.io/v1
kind: GatewayAPI
metadata:
  name: default
EOF

kubectl wait --for=condition=established --timeout=120s \
  crd/envoyproxies.gateway.envoyproxy.io crd/gatewayclasses.gateway.networking.k8s.io
kubectl get ns tigera-gateway

Apply the host-network data plane

gateway.yaml holds both pieces: an EnvoyProxy that turns the deployment into a host-network DaemonSet, plus the GatewayAPI update that points tigera-gateway-class at it:

# gateway.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: host-network
  namespace: tigera-gateway
spec:
  # Without this every listener binds 0.0.0.0 only, and IPv6 clients get "Connection refused".
  ipFamily: DualStack
  provider:
    type: Kubernetes
    kubernetes:
      # Bind 80/443 as the container port instead of the default offset (80 -> 10080).
      useListenerPortAsContainerPort: true
      # Run Envoy as a DaemonSet, one per node.
      envoyDaemonSet:
        # No native hostNetwork/dnsPolicy fields exist on EnvoyProxy, so merge them
        # straight into the generated DaemonSet pod template.
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  hostNetwork: true
                  dnsPolicy: ClusterFirstWithHostNet
                  # Envoy stays non-root (uid 65532, no capabilities); the
                  # ip_unprivileged_port_start sysctl is what lets it bind 80/443.
                  # Run Envoy only on the nodes that are in the app hostnames' DNS.
                  affinity:
                    nodeAffinity:
                      requiredDuringSchedulingIgnoredDuringExecution:
                        nodeSelectorTerms:
                          - matchExpressions:
                              - key: kubernetes.io/hostname
                                operator: In
                                values: ["k8s-1", "k8s-2", "k8s-3"]
      # No cloud LB; hostNetwork carries the traffic.
      envoyService:
        type: ClusterIP
---
# Updates the existing GatewayAPI CR named `default`: render tigera-gateway-class
# as a DaemonSet with the EnvoyProxy customization above.
apiVersion: operator.tigera.io/v1
kind: GatewayAPI
metadata:
  name: default
spec:
  gatewayClasses:
    - name: tigera-gateway-class
      gatewayKind: DaemonSet
      envoyProxyRef:
        name: host-network
        namespace: tigera-gateway

What the pieces do:

• hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet put Envoy on the node's own network while keeping cluster DNS.
• useListenerPortAsContainerPort: true — otherwise Envoy listens on 10080/10443.
• ipFamily: DualStack — without it, every listener silently binds 0.0.0.0 only (the CRD's own default), no error anywhere, and IPv6 clients get a clean Connection refused. Easy to miss: ss -lntp on the node looks completely normal, just IPv4-only, and the failure only shows up when someone actually tests curl -6.
• The node affinity keeps Envoy on the nodes that are in the application DNS records. Other nodes, such as an agent, still run workloads; the Envoys reach their pods over the cluster network. A new node gets no Envoy until it is added both here and to DNS.

kubectl apply -f gateway.yaml

Verify what exists at this stage — the controller and the class, not a data plane:

kubectl get tigerastatus
kubectl get gatewayclass tigera-gateway-class     # ACCEPTED = True
kubectl -n tigera-gateway get envoyproxy          # host-network, from the file above
kubectl -n tigera-gateway get pods                # envoy-gateway-... Running (the controller)

There is no DaemonSet yet, and nothing is listening on 80/443. Envoy Gateway creates a proxy DaemonSet per Gateway resource, so the data plane appears only once a Gateway using tigera-gateway-class exists — which is the next step.

The Gateway resource

Only the operator-level pieces above are applied by hand. The Gateway itself — listeners, hostnames, TLS secrets — lives in Git and arrives with everything else once Flux takes over (Part 17). Trimmed to one application, it looks like this:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main
  namespace: tigera-gateway
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: tigera-gateway-class
  listeners:
    - name: http                 # plain HTTP and ACME HTTP-01 challenges
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: app                  # one HTTPS listener per hostname
      hostname: app.example.com
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: app-tls
      allowedRoutes:
        namespaces:
          from: All

Once it exists, the data plane shows up and the host ports are bound:

kubectl -n tigera-gateway get ds -o wide          # envoy-tigera-gateway-<name>-..., DESIRED = 3
kubectl -n tigera-gateway get pods -o wide        # one per node
ss -lntp '( sport = :80 or sport = :443 )'        # envoy, on each node
kubectl -n tigera-gateway get gateway main        # PROGRAMMED = True

Certificates are minted by cert-manager's gateway-shim from the cert-manager.io/cluster-issuer annotation on the Gateway, solving HTTP-01 over the same port-80 listener.

DNS for the hostnames you serve

Unlike the cluster endpoint in Part 3, application hostnames are plain HTTP/HTTPS, so Cloudflare's proxy is fine here — orange cloud, WAF and caching included. Point each hostname at all three nodes, A and AAAA: whichever node receives the request has an Envoy listening, and ACME HTTP-01 challenges resolve the same way.

The caveat is the same one round-robin DNS always has, and it applies to proxied records too: neither plain DNS nor the Cloudflare proxy health-checks your origins, so while a node is down a share of requests get connection errors or 5xx. Cloudflare Load Balancing (paid) adds origin health checks; without it, remove a node's A record while it is out for maintenance if you care.

Part 14. Day 2: updates and coordinated reboots

MicroOS runs transactional-update.timer daily: it builds a new snapshot and then wants a reboot. Because the RKE2 RPM repo is enabled, RKE2 patch releases come with the same cycle — a node reboot is also an RKE2 upgrade.

In a three-node cluster the danger is obvious: if all three reboot at 03:30, quorum is gone, so the reboots have to take turns. I use kured for that. Another option, with no cluster component, is rebootmgr with a different maintenance window on each node — but it does not drain, so pods are killed rather than evicted.

kured

transactional-update has a built-in reboot method for exactly this. Setting REBOOT_METHOD=kured makes it touch /var/run/reboot-required instead of rebooting, which is kured's default sentinel file — kured then takes a cluster-wide lock, cordons and drains the node, reboots it, and only releases the lock when the node is back.

sequenceDiagram
    participant TU as transactional-update
    participant Node as k8s-2
    participant Kured as kured
    participant Lock as cluster-wide lock
    TU->>Node: build new snapshot
    TU->>Node: touch /var/run/reboot-required
    Kured->>Node: notice the sentinel
    Kured->>Lock: acquire, one node at a time
    Kured->>Node: cordon and drain
    Kured->>Node: reboot into the new snapshot
    Node->>Node: health-checker, roll back if broken
    Node-->>Kured: back and Ready
    Kured->>Node: uncordon
    Kured->>Lock: release

One coordinated reboot. The other nodes wait for the lock.

On every node — and not by editing the file you would expect. The shipped config lives in /usr/etc/transactional-update.conf, openSUSE's vendor-config location on read-only /usr; /etc/transactional-update.conf is the admin override and does not exist until you create it. So write it rather than sed it:

echo 'REBOOT_METHOD=kured' > /etc/transactional-update.conf
systemctl disable --now rebootmgr.service     # otherwise two things want to reboot the node

One line is enough: the vendor file is sourced first and yours second, so every other default survives. One further trap — only /usr/etc/transactional-update.conf.d/ is scanned for drop-ins, not the /etc equivalent, so a drop-in under /etc is silently ignored (sbin/transactional-update.in). Valid values are auto, rebootmgr, notify, systemd, kured and none; kexec is deprecated.

Then deploy kured into the cluster. I run it as a Flux HelmRelease from the https://kubereboot.github.io/charts repository. The defaults already match MicroOS — sentinel /var/run/reboot-required, one node at a time — and these are the values I set on top:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kured
  namespace: kured
spec:
  chart:
    spec:
      chart: kured
      sourceRef:
        kind: HelmRepository
        name: kubereboot
        namespace: flux-system
      version: '>=6.0.0 <7.0.0'
  interval: 1h0m0s
  targetNamespace: kured
  values:
    configuration:
      # Default sentinel; matches what transactional-update's kured method writes.
      rebootSentinel: /var/run/reboot-required
      period: 30m
      # Release the lock only after the node is back and Ready, and never let a
      # stuck node hold it forever.
      lockTtl: 60m
      lockReleaseDelay: 5m
      # Give workloads (and Longhorn replicas) time to move, but do not hang.
      drainTimeout: 10m
      drainGracePeriod: "60"
      # Do not reboot past a drain failure - an unevictable pod is a decision for
      # a human, not for 03:00.
      forceReboot: false
      # Stop other rebooting nodes from scheduling onto this one mid-cycle.
      preferNoScheduleTaint: weave.works/kured-node-reboot
      annotateNodes: true

Verify the wiring end to end without waiting for an update: touch /var/run/reboot-required on one node, then watch kubectl get nodes -w — the node should be cordoned, drained and rebooted, and nothing else should follow it.

Check that every node is still updating

A node whose nightly update fails does not complain anywhere you would see it: it keeps running, it just stops getting OS and RKE2 updates. Compare versions across nodes now and then:

kubectl get nodes -o wide          # same RKE2 version and kernel on every node?
# on a node that lags behind:
journalctl -u transactional-update.service --since -2d | tail -3
snapper list | tail -3             # the newest snapshot should be recent

My agent fell behind exactly this way: from October 2 its nightly update failed every day on Signature verification failed for repomd.xml (Part 18), and it stayed on an older RKE2 and kernel for twelve days before I noticed.

Manual control

transactional-update dup && reboot          # update one node, wait for Ready, then the next
zypper addlock rke2-server                  # freeze RKE2 while still updating the OS
transactional-update rollback last && reboot
snapper list

A rollback reverts /usr (kernel, RKE2 binary, SELinux policy) but not /var — etcd data, container images and Longhorn volumes stay as they are, and /etc changes are kept. That is what you want for a bad kernel. It also means an RKE2 minor downgrade is not a supported path: etcd schema migrations are one-way.

RKE2 minor upgrades

Minor upgrades are deliberate, and done one node at a time, waiting for Ready in between. Point the repo file at the next minor and let dup do the rest:

rm /etc/zypp/repos.d/rancher-rke2-1-35-stable.repo
# re-run the repo heredoc from Part 7 with RKE2_MINOR=36, then:
transactional-update dup && reboot

Check the release notes for skipped minors first — Kubernetes does not support jumping two.

Snapshot disk usage

Keep an eye on how much space the snapshots take, and cap how many are kept if it gets tight:

snapper -c root list
btrfs filesystem usage /
sed -i 's/^NUMBER_LIMIT=.*/NUMBER_LIMIT="4-6"/' /etc/snapper/configs/root

Replacing a node

Keeping addresses out of tls-san is what makes a node swap cheap. Each node has a stable name (k8s-1.example.com …) alongside the round-robin k8s.example.com (Part 3), so the certificate does not care which IP is behind the name.

flowchart TB
    a["kubectl drain k8s-3<br/>kubectl delete node k8s-3"] --> b["etcd member goes with the node<br/>quorum 3 → 2, still a quorum"]
    b --> c["DNS: new address for k8s-3.example.com<br/>and for k8s-3's records in k8s.example.com"]
    c --> d["Build the new VPS through Parts 4–8<br/>same hostname, own node-ip, same tls-san"]
    d --> e["It joins: quorum back to 3<br/>no changes on the two survivors"]

Swapping k8s-3 for a new VPS.

Swapping k8s-3 for a new VPS then means:

1. Remove the old member cleanly — kubectl drain k8s-3 --ignore-daemonsets --delete-emptydir-data then kubectl delete node k8s-3. RKE2's etcd controller removes the etcd member with the node, so quorum goes 3 → 2 (still a quorum) and back to 3 when the replacement joins.
2. Update DNS: the A/AAAA pair for k8s-3.example.com, and the node's records inside k8s.example.com. Nothing else references the old address.
3. Build the replacement through Parts 4–8 with the same hostname, its own node-ip pair, and an unchanged tls-san. No certificate work, no edits on the surviving servers.

The certificates on the two survivors never mention k8s-3's addresses, so there is nothing stale to clean up. Had you listed all six IPs in tls-san, this step would instead mean editing three config files and restarting all three servers.

If you do need to add a SAN later, edit tls-san on every server and restart rke2-server; force it with rke2 certificate rotate if the serving cert does not pick the change up.

Part 15. Backups and quorum loss

Every server writes its own etcd snapshots to /var/lib/rancher/rke2/server/db/snapshots on the schedule set in config.yaml — every six hours, keeping 20. Three copies on three nodes is not a backup; push them off-box.

flowchart LR
    snap["etcd snapshot on each server<br/>every 6 hours"] -->|"etcd-s3: true<br/>credentials from a Secret"| s3[("S3-compatible bucket")]
    keep["Kept outside the cluster:<br/>server token + S3 credentials"]
    s3 --> restore["cluster-reset on the survivor"]
    keep -->|"--etcd-s3-* flags on the command line"| restore

What a restore needs, and where each piece lives.

Keep the S3 credentials out of config.yaml — RKE2 reads them from a Secret instead, which means they can live in Git as a SealedSecret next to everything else Flux manages, and be rotated without restarting anything. Two lines on each server, and only these two: any other etcd-s3-* flag makes RKE2 ignore the Secret.

# /etc/rancher/rke2/config.yaml, on all three servers
etcd-s3: true
etcd-s3-config-secret: rke2-etcd-snapshot-s3-config

# sealed into Git; the Secret need not exist when RKE2 starts
apiVersion: v1
kind: Secret
metadata:
  name: rke2-etcd-snapshot-s3-config
  namespace: kube-system
type: etcd.k3s.cattle.io/s3-config-secret
stringData:
  etcd-s3-endpoint: <S3_ENDPOINT>
  etcd-s3-access-key: <KEY>
  etcd-s3-secret-key: <SECRET>
  etcd-s3-bucket: <BUCKET>
  etcd-s3-folder: rke2
  etcd-s3-region: <REGION>

Any S3-compatible target works. Cloudflare R2 fits this cluster well — 10 GB and zero egress on the free tier, against snapshots of tens of MB — and takes etcd-s3-endpoint: <ACCOUNT_ID>.r2.cloudflarestorage.com (host only) with etcd-s3-region: auto, authenticated by an R2 API token scoped to the bucket. Backblaze B2, Storj and Scaleway have comparable free tiers if you would rather not keep DNS and backups with one vendor.

The R2 dashboard may show the endpoint with the bucket path appended (https://<ACCOUNT_ID>.r2.cloudflarestorage.com/<bucket>). Pasted as is into etcd-s3-endpoint, it fails only at snapshot time:

FATA[0000] Error: see server log for details: failed to initialize S3 client: Endpoint url cannot have fully qualified paths.

The fix is to strip everything from the bucket name onward — scheme and path both — leaving just the bare host; the bucket belongs solely in etcd-s3-bucket.

Keep a copy of those credentials outside the cluster, next to the server token. The Secret is read through the apiserver, which does not exist during a restore — RKE2 says so explicitly — so restoring from S3 means passing --etcd-s3-access-key and friends on the command line. A backup whose credentials only exist inside the cluster you are restoring is not a backup.

Check the snapshots with kubectl get etcdsnapshotfile (RKE2 tracks snapshots as cluster resources) or rke2 etcd-snapshot list. Don't wait out etcd-snapshot-schedule-cron to find out whether the S3 upload actually works — rke2 etcd-snapshot save --name manual-test runs (and uploads) immediately, on demand. And don't stop at the CRD/CLI output either: both report what RKE2 attempted, not confirmation the object exists — check the bucket itself (R2 dashboard, rclone, aws s3 ls pointed at the R2 endpoint) at least once.

Losing quorum

If you lose two nodes at once, the survivor cannot elect a leader and the API goes read-only-ish until you rebuild the cluster from one member — the documented procedure in short:

# on all three: systemctl stop rke2-server
# on the survivor:
rke2 server --cluster-reset --cluster-reset-restore-path=<SNAPSHOT>   # omit the path to keep current data
systemctl start rke2-server
# on the other two: rm -rf /var/lib/rancher/rke2/server/db && systemctl start rke2-server

Practise this once on throwaway nodes. The first time should not be during an outage.

Part 16. Optional: prerequisites for Longhorn

Longhorn needs an iSCSI initiator on every node, plus an NFS client for ReadWriteMany volumes. Neither is in the MicroOS image — and with three nodes, Longhorn's default 3 replicas finally have somewhere to live:

transactional-update pkg install open-iscsi nfs-client
reboot
systemctl enable --now iscsid.service

If any StorageClass sets encrypted: "true", every node also needs cryptsetup — Longhorn nsenters into the host mount namespace to run it directly, so it must exist on the host, not just inside the longhorn-manager container. Missing it doesn't fail fast: the PVC provisions, but the replica sits in attaching/unknown forever, and the only clue is FailedStarting … cryptsetup --version … exit status 127 in kubectl -n longhorn-system get events.

transactional-update pkg install cryptsetup
reboot

Longhorn stores data in /var/lib/longhorn (writable, and outside the snapshot/rollback set). It also needs to know the kubelet root directory; RKE2 uses the default, which you can confirm with ps -ef | grep '[k]ubelet' | tr ' ' '\n' | grep root-dir — no output means /var/lib/kubelet and no Helm override is needed. If volumes fail to attach with permission errors, check journalctl -t audit before reaching for permissive mode.

Longhorn and kured interact: a drained node's replicas rebuild afterwards. Keep kured's lock TTL (lockTtl in Part 14) longer than a replica rebuild, or Longhorn will still be catching up when the next node goes down.

Part 17. Next: hand the cluster to Flux

Once all three nodes are Ready, bootstrap GitOps as usual:

flux bootstrap github \
  --components-extra=image-reflector-controller,image-automation-controller \
  --owner=<github-user> \
  --repository=<repository> \
  --branch=main \
  --path=clusters/<cluster> \
  --read-write-key \
  --personal

From here on, the Gateway (Part 13), kured (Part 14), the etcd S3 Secret (Part 15) and Longhorn (Part 16) come from Git.

Part 18. Troubleshooting

Results

The cluster has been running since August 23 — about 50 days at the time of writing — and it has kept itself up to date the whole time:

• Nightly updates with reboots that take turns. MicroOS builds a new snapshot every night and kured reboots the nodes one at a time. Each server has booted more than 40 times since it was installed; on October 13, for example, the three servers rebooted at 00:31, 02:01 and 02:36.
• Kubernetes upgrades itself. RKE2 arrived as v1.35.7 and moved to v1.35.8 on September 19 and to v1.35.9 on October 2, both times in the middle of the night, through the same update cycle as the OS. The kubeadm-era chore from the motivation is gone.
• The bill is smaller than for the old cluster.

Automatic does not mean unattended, though. Three things failed silently along the way, and each is covered above: the missing SELinux label on the agent (Part 11), the agent's nightly update blocked by an untrusted repository key (Part 14), and WireGuard for IPv4 reverted by an RKE2 upgrade (Part 12). None of them took anything down; all of them needed someone to look.

Next time I would not install each node from the ISO by hand. MicroOS can configure a node on its first boot with Ignition or Combustion, and that is how I would provision the nodes in the future.

References

• openSUSE MicroOS portal · transactional-update(8) · rebootmgrctl(1) · kured
• RKE2 high availability · requirements · server config reference · backup & restore · known issues · SELinux
• Ingress NGINX retirement / Traefik migration