k3sm apache-2.0 (DCO)

Upgrades

Moving a k3sm node or cluster to a new release.

Single Node#

For script (gen-1) installs, re-run the one-liner:

curl -fsSL https://k3sm.io/install.sh | sh

An unpinned re-run installs the latest release. Pin K3SM_INSTALL_VERSION=vX.Y.Z to re-install the version you are already on (repair without upgrading), and pin an older version to downgrade. That pin-and-re-run is the script channel’s rollback path.

The re-run replaces the binary and restarts the k3sm LaunchDaemons (the io.k3sm.netd root helper and the _k3sm server/agent jobs) onto the new version, through the sudo k3sm install it performs. The new binary replaces the old one, with no side-by-side window and no flag to switch back, so the node is momentarily unavailable while the daemons restart.

The node daemon’s open-file limit (NumberOfFiles in its plist) binds only when launchd boots the job out and bootstraps it again, which k3sm install does. launchctl kickstart -k alone keeps the old limit, so reinstall rather than kickstart after an upgrade that changes it; the install log says when it did.

The kernel also allocates a process at most kern.maxfilesperproc file descriptors, whatever limit launchd grants it, and that ceiling scales with installed RAM (measured on macOS 26: 245760 on a 64 GB Mac, 10240 on an 8 GB one). When it is below the limit k3sm requests, the install logs a warning and the node’s UDP Service flow capacity drops toward its floor. sudo sysctl -w kern.maxfiles=131072 kern.maxfilesperproc=131072 raises it until the next reboot.

Homebrew is planned; the k3sm-io/tap is not published yet. When it ships, brew upgrade k3sm will do the same job, restarting the daemons via launchctl kickstart.

Multi-Node Rolling Upgrade#

A cluster upgrades one node at a time, not all at once. Restarting each node’s daemon via launchctl kickstart creates a short binary-version-skew window where old and new nodes coexist; k3sm releases are designed so adjacent versions interoperate across that window. Upgrade agents first and the control-plane Mac last unless a release note says otherwise. See Multi-node and HA.

Before You Upgrade#

  • Back up the datastore with sudo k3sm snapshot save --out <somewhere off this node>. It is safe to run while the cluster is serving, and it verifies the copy before it reports success. The file-level fallback (stop the daemon, PRAGMA wal_checkpoint(TRUNCATE), copy, verify the copy) is in Backup & restore. k3sm also takes an automatic pre-migration backup when a release changes the datastore engine (below), but that copy lives on the same disk as the cluster it protects, so it is not a substitute for yours.
  • Know your rollback path. On the script channel, prior releases stay downloadable, so rollback is K3SM_INSTALL_VERSION=<prior-tag> and a re-run. The planned Homebrew channel will retain the prior bottle, so rollback there will not need a rebuild round-trip.
  • Check the version skew. Confirm the target Kubernetes pin with k3sm version; see Versions.

Upgrading Across a Datastore-Engine Change#

Some releases move to a newer kine (the etcd-shim over SQLite). A newer kine re-runs its schema migrations against your existing state.db, and that is one-way. The migrated database is not converted back if you reinstall the older k3sm.

You do not have to do anything for this, but you should know what it does.

  • Before the new version opens the database, the server takes a verified backup at db/state.db.pre-<kine-version>.bak and preserves the old kine binary beside it as kine.pre-<kine-version>. The mechanics (the WAL drain, the integrity check, the write-once rule) are in Backup & restore.
  • It refuses to start if the volume does not have twice the database size free, rather than writing a partial backup. Free space and start it again; nothing was changed.
  • The first boot on the new engine is slower than usual (the checkpoint + the copy). Later boots are not.
  • Keep the .bak until you are satisfied with the new release, then delete it. It is a full copy of the database.

A server started with --cluster-init keeps its state in an embedded etcd member instead, and none of this applies there. A multi-server control plane is not available in v0.1.6; see HA.

Upgrading Into the Reserved-Port Policy#

The release that moved LoadBalancer listeners to the wildcard also provisions a ValidatingAdmissionPolicy that rejects a type: LoadBalancer Service declaring a port k3sm’s own listeners own, either the NodePort range 30000-32767 or the kubelet API port 10250. It matches on CREATE and UPDATE, and it does not ratchet on oldObject.

That means a cluster already carrying such a Service is not grandfathered in. The object stays in the datastore and keeps working, but every subsequent write to it is denied, not just a port change. A kubectl label, an annotation added by an unrelated controller, any kubectl apply of the same manifest: all rejected, with a message naming the port.

Check before you upgrade:

kubectl get svc -A -o json | jq -r '
  .items[] | select(.spec.type=="LoadBalancer")
  | select(.spec.ports[]? | .port==10250 or (.port>=30000 and .port<=32767))
  | "\(.metadata.namespace)/\(.metadata.name)"'

Anything listed has two escape hatches, both a single write before the upgrade (or from a still-permitted path after it):

  • Change the port to one outside the reserved set. This is the intended fix, since the Service could never have had a working listener on a port k3sm already holds.
  • Patch type away from LoadBalancer (e.g. to ClusterIP or NodePort); the policy is scoped to LoadBalancer Services only, so the object becomes writable again immediately.

There is no --force and no exemption. A policy that tolerated a pre-existing offender would leave the collision silently in place, along with the kubectl logs/exec outage it can cause.

Upgrading Into the Denied-Local-Port Policy#

Every pod’s sandbox denies connect() to the node’s datastore (kine) listener port. The deny is by port number, whatever address the pod dials, so a Service published on that port is unreachable from pods no matter its type. The release that added the deny also provisions a ValidatingAdmissionPolicy that rejects such a Service, so you meet the problem at kubectl apply rather than inside a pod.

Two things it does differently from the reserved-port policy above:

  • No type scope. ClusterIP, NodePort, LoadBalancer and headless are all rejected. A headless Service is not an exception: its clients dial a pod IP, which the deny covers too.
  • Only spec.ports[].port is matched. A targetPort on the datastore port is fine (that is the port a pod listens on, not one another pod dials), and so is an allocated nodePort.

Like the reserved-port policy, it matches CREATE and UPDATE and does not ratchet on oldObject, so an existing Service on that port is not grandfathered: it keeps working as an object, and every subsequent write to it is denied.

Check before you upgrade. 2379 is the default datastore port; use your --kine-port value if you moved it:

kubectl get svc -A -o json | jq -r '
  .items[] | select(.spec.ports[]? | .port==2379)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Anything listed has two ways out:

  • Publish it on a different spec.ports[].port. This is the intended fix, since no pod can reach the Service on the current one.
  • Start the server with a different --kine-port, which moves the denied number. That is a server restart, and it changes the port anything else of yours uses to reach the datastore.

One limit: the policy is a single cluster-scoped object carrying the ports of the server that provisioned it, while --kine-port is per server. A control plane whose servers used different datastore ports would have a policy that names one of them, and a Service on another server’s port would be admitted by the API and still unreachable from the pods on that server’s node. A multi-server control plane is not available in v0.1.6 (see HA).

Rollback#

Rollback means reverting to the previous binary, plus the daemon restart that comes with it. There is no runtime flag to flip. On the script channel that is K3SM_INSTALL_VERSION=<prior-tag> and a re-run. On the planned Homebrew channel it will be a brew pin or a switch to the prior bottle.

If the release you are leaving changed the datastore engine, reverting the binary is only half of it, because the database has already been migrated in place. Restore the db/state.db.pre-<kine-version>.bak the upgrade left behind (or your own pre-upgrade snapshot) with the daemon stopped:

sudo launchctl bootout system/io.k3sm.server
sudo k3sm snapshot restore /var/lib/k3sm/server/db/state.db.pre-<kine-version>.bak
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plist

k3sm snapshot restore refuses while the daemon is running, verifies the backup before it touches anything, and prints the verification step you must then run; see Backup & restore. Rolling the binary back without restoring the backup leaves an older k3sm pointed at a database a newer engine has migrated.

Mesh Keys After Rolling Back Past the Key Directory Move#

The release that made /var/lib/k3sm root-owned also moved each node’s mesh keys and pod address record from /var/lib/k3sm/run/keys to /var/lib/k3sm/keys. sudo k3sm install makes that move once and removes the old copy, and the move is one-way: an older binary looks only in the old directory, finds nothing, and gives the node a new mesh identity that no peer knows. Before you downgrade, copy the files back by hand:

sudo mkdir -p -m 0700 /var/lib/k3sm/run/keys
sudo cp -p /var/lib/k3sm/keys/server.key /var/lib/k3sm/keys/node.key /var/lib/k3sm/keys/node-pod-cidr /var/lib/k3sm/run/keys/

A node has only the key for its own role, so cp reports the missing one; that is expected. The older binary’s installer also rebuilds its key copy from the node’s own work-dir copy, so this step is a second safeguard rather than the only one.

Durable State After Rolling Back Past the LoadBalancer Bind Change#

The release that moved LoadBalancer/Ingress listeners to the wildcard also changed what k3sm writes into the cluster, and the older binary has no code to clean either of those up. Reverting the binary does not revert them; you have to.

  1. Stale EXTERNAL-IP entries. The new server advertises the node’s derived InternalIP (e.g. 100.64.0.1). The old server only ever retracted the address it was configured with, the loopback default, so it will never remove a derived entry. A rolled-back cluster keeps advertising an address its listeners are no longer on. Retract them by hand:

    kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="LoadBalancer")]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'
kubectl patch svc <name> -n <ns> --subresource=status --type=merge -p '{"status":{"loadBalancer":{}}}'

    Do the same for any Ingress of the k3sm class (kubectl patch ingress … --subresource=status).

  2. The reserved-port Deny policy. The new server provisions the k3sm-reject-loadbalancer-reserved-port ValidatingAdmissionPolicy, which lives in the datastore and survives the downgrade. The old binary neither knows about it nor deletes it, so a type: LoadBalancer Service on a NodePort-range port or on 10250 stays rejected at kubectl apply. If you want the old (unguarded) behaviour back, delete both objects:

    kubectl delete validatingadmissionpolicybinding k3sm-reject-loadbalancer-reserved-port-binding
kubectl delete validatingadmissionpolicy        k3sm-reject-loadbalancer-reserved-port

    Leaving them in place is also a valid choice, because the policy reflects a real collision on the old binary too.

  3. The denied-local-port Deny policy. The same release provisions k3sm-reject-service-denied-local-port, which also lives in the datastore and survives the downgrade, so a Service published on the datastore port stays rejected at kubectl apply. Delete both objects if you want the old behaviour back:

    kubectl delete validatingadmissionpolicybinding k3sm-reject-service-denied-local-port-binding
kubectl delete validatingadmissionpolicy        k3sm-reject-service-denied-local-port

    Check what you are getting back first. If the binary you rolled back to still denies that port in every pod sandbox, deleting the policy only hides the problem: the Service is created, its endpoints go Ready, and no pod can reach it.

Next#

  • Backup & restore covers backing up before an upgrade, and the automatic pre-migration copy.
  • Versions describes the version you are moving to.
  • Troubleshooting is where to go if the daemon does not restart.