Backup and Restore
k3sm keeps cluster state in an embedded kine datastore over SQLite (WAL), or, on an HA server, in an embedded etcd member. Backing it up and restoring it is how you protect and recover a cluster.
What Holds the State#
The control-plane state of record is the kine/SQLite database under the server work directory (the
db/state.db family, including its WAL). This is distinct from PersistentVolume data, which lives in
local-path directories on each node (see Storage) and must be backed up separately.
Three things back it up. k3sm snapshot save (below) runs on your own schedule. k3sm takes an
automatic pre-upgrade backup when a release changes the datastore engine. The file-level procedure
further down is the fallback for a node where the binary will not run. An HA server also takes
scheduled etcd snapshots on its own.
If the cluster was installed with secrets encryption, back up /var/lib/k3sm/server/cred with
every backup and snapshot: neither contains the key, and without it the Secrets are unreadable. See
Secrets Encryption at Rest.
Taking a Backup with k3sm snapshot save#
k3sm snapshot save # -> <work-dir>/db/snapshots/k3sm-snapshot-<UTC>.db
k3sm snapshot save --out /Volumes/backups/k3sm.db
k3sm snapshot save --compress # -> <work-dir>/db/snapshots/k3sm-snapshot-<UTC>.zipIt is safe to run while the control plane is serving, because SQLite takes the copy inside a read transaction and a concurrent write cannot tear it. What it does, in order:
- Refuses unless the destination volume has twice the database size free, rather than writing a partial snapshot.
- Writes a consistent point-in-time image of the datastore, runs
PRAGMA integrity_checkon that image, and only then renames it into place. A snapshot that exists under its final name is therefore complete and was confirmed readable as a database.
The work directory is owned by the _k3sm service user, so run it under sudo (or pass
--work-dir) when your shell user cannot read it. The snapshot is written 0600.
--compress writes the snapshot as a zip archive holding one file, the same snapshot a plain save
writes, which is the format k3s uses for compressed snapshots. The snapshot is checked before it is
compressed, and the archive is read back before it takes its name. A generated name ends in .zip;
a file named with --out keeps the name you gave it. Compression applies to the file on disk only:
a restore decompresses it, and the datastore it puts in place is byte for byte what a plain snapshot
restores. It works the same way on an etcd server.
Copy it off the node. The default location is the same volume as the cluster it protects, which
does not survive losing that volume. --out onto another disk or host is the better habit.
The snapshot does not contain PersistentVolume data; see Storage.
Putting a Snapshot Back with k3sm snapshot restore#
sudo launchctl bootout system/io.k3sm.server # 1. stop the control plane
sudo k3sm snapshot restore /Volumes/backups/k3sm.db # 2. restore
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plist # 3. start it againThe restore is built to be survivable when it goes wrong:
- It refuses while a control plane is running (the launchd job, a foreground
k3sm server, or ak3sm devcluster) and names what to stop. Swapping the datastore under a live kine is corruption, not a restore, because kine keeps writing to the file it already holds open. - It verifies the snapshot before touching anything. A snapshot that fails
integrity_checkcosts you an error and nothing else; your current datastore is untouched. - It recognizes a compressed snapshot by its content, not its name, so a renamed file restores either way. A compressed snapshot is decompressed into a temporary file beside the datastore and checked there before anything is replaced. An archive that is truncated, fails its checksum, holds more than one file, decompresses to more than its header declares, or would decompress to more than 8 GiB is refused, and nothing is changed. The temporary copy holds every Secret in plaintext, so it is removed when the restore ends, and a copy left by an interrupted restore is removed by the next restore or the next start of the server.
- It preserves what it replaces. The superseded
state.dbis moved tostate.db.restore-<UTC>.bakand never deleted, and its-wal/-shmsidecars and kine pin stamp go with it, because a stale sidecar left beside a restored database is exactly how a “successful” restore comes back with the state you were trying to discard. - It prints the verification step below, and the
.bakto keep if verification fails.
Restoring onto a node with no datastore at all (a rebuilt Mac) is supported, and is what the drill is for.
Scheduled etcd Snapshots#
A server installed with --cluster-init or --server-join snapshots its etcd member on a schedule,
the way k3s does. Four flags control it, on both k3sm install and k3sm server, with the k3s
names and defaults:
| flag | default | meaning |
|---|---|---|
--etcd-snapshot-schedule-cron | 0 */12 * * * | when to take a snapshot: a five-field cron expression (minute, hour, day of month, month, day of week) in the server’s local time; the default is midnight and noon |
--etcd-snapshot-retention | 5 | how many of this server’s scheduled snapshots to keep |
--etcd-disable-snapshots | false | take no scheduled snapshots |
--etcd-snapshot-compress | false | write each scheduled snapshot as a zip archive, named etcd-snapshot-<node>-<unix seconds>.zip |
sudo k3sm install --cluster-init --node-ip 192.0.2.10 \
--etcd-snapshot-schedule-cron '0 */6 * * *' --etcd-snapshot-retention 8k3sm install writes the flags you pass into the server’s recorded arguments, so a later
reinstall keeps them; passing one again replaces the recorded value. The cron fields accept *,
numbers, ranges (1-5), steps (*/6, 0-12/3) and comma lists; names such as MON and shortcuts
such as @daily are not accepted, and an expression that never fires is refused at start. When
both day fields are set, a day matching either one fires, except that a day field starting with
* (*/2 included) counts as unset, as in Vixie cron.
What the schedule does:
- Every server snapshots its own member, into its own
<work-dir>/db/snapshots(/var/lib/k3sm/server/db/snapshotson an installed server, made0700when the schedule starts). A three-server cluster keeps three sets. Nothing is copied between servers or off the node. - A Mac that was asleep at a scheduled time takes one snapshot within a minute of waking, and logs that it was late and by how much. Several missed times still give one snapshot, not one each.
- A scheduled snapshot is named
etcd-snapshot-<node>-<unix seconds>, the k3s name. It goes through the same path ask3sm snapshot save: streamed from the running member, checked against etcd’s SHA-256 hash and opened read-only before it takes its name, and written0600. - Retention keeps the newest
--etcd-snapshot-retentionscheduled snapshots of this server and deletes older ones. It runs only after a new snapshot was written and checked, so a failing schedule never deletes the snapshots it already has. It never deletes a snapshot taken withk3sm snapshot save(those are namedk3sm-etcd-snapshot-<UTC>.db) or one named for another node. Compressed and plain snapshots count alike, so turning--etcd-snapshot-compresson or off still keeps the newest--etcd-snapshot-retentionof this server’s snapshots of either kind. Snapshots are ordered by the earlier of the time in the name and the file’s modification time, and one dated in the future (taken while the clock was wrong) counts as the oldest. If you rename a server, its snapshots under the old name are left for you to remove withk3sm snapshot delete. - When the schedule starts it removes the partial
.tmpfiles an interrupted snapshot of this server left behind, and nothing else. - A snapshot is skipped when the snapshot directory’s volume has less than twice the last snapshot’s size free, so the schedule does not fill the disk the datastore lives on. With compression on, or when the newest snapshot is compressed, the etcd database’s own size is used when it is larger, because the snapshot is written and checked uncompressed before it is compressed.
- Each snapshot taken and each one removed is an
INFOline in the server log; a failed or skipped snapshot is aWARNline.k3sm status -o wideshows, on theetcdrow, the schedule, when the newest scheduled snapshot was taken, how many are kept, and, until the next snapshot succeeds, the last failure and the last skip (no room, or the free space could not be read).
A snapshot holds every Secret in plaintext, including service-account tokens and any credential
stored in the cluster. With the defaults, every server therefore keeps up to five plaintext copies
of every Secret on its own disk, one per retained snapshot; --etcd-disable-snapshots turns the
schedule, and those copies, off. The files are 0600 and owned by the service user; keep any copy
you make just as private, and do not put them on shared storage.
Like a manual snapshot, a scheduled one lives on the same volume as the cluster it protects, and
does not contain PersistentVolume data. Copy the snapshots you want to keep off the node. On a
single-server control plane (no --cluster-init or --server-join) the flags are accepted and
ignored with one log line, as in k3s, which takes no scheduled snapshot of a SQLite datastore; use
k3sm snapshot save there.
Listing, Deleting and Pruning Snapshots#
k3sm snapshot ls # name, size, created, kind, compressed
k3sm snapshot ls -o json
k3sm snapshot delete k3sm-etcd-snapshot-20261001T000000Z.db
k3sm snapshot prune --snapshot-retention 3 --dry-run
k3sm snapshot prune --snapshot-retention 3These work on the snapshots directory under the work directory
(/var/lib/k3sm/server/db/snapshots on an installed server), so run them under sudo there. They
match k3s etcd-snapshot ls, delete and prune.
ls lists every snapshot in the directory with its size, when it was created, and its kind:
scheduled (taken by this server’s schedule), manual (taken with k3sm snapshot save), or
other (another node’s scheduled snapshot, or a file k3sm did not name). The created time is the
one retention orders by: the earlier of the time in the name and the file’s modification time. A
snapshot dated in the future shows as unknown. ls reads file metadata only and never opens a
snapshot. Partial .tmp files, symlinks and directories are not listed. A missing directory is an
empty list.
delete removes the snapshots you name, by the bare file names ls prints. It removes only names
k3sm gives a snapshot (scheduled or manual, for any node), and only from the snapshots directory.
It refuses a path, a name starting with -, any other file name, a symlink, a directory and a
partial .tmp file; remove another file by hand if it must go. Deleting a name that is already gone succeeds. A deleted snapshot cannot be
recovered.
prune removes this server’s scheduled snapshots past the newest --snapshot-retention (default
5), with the same selection the schedule’s own retention makes. It never removes a manual
snapshot or one named for another node; use delete for those. It prints how many scheduled
snapshots it found for the node and how many it removed, so on a single server with a kine/SQLite
datastore, which takes no scheduled snapshots, it reports 0 scheduled snapshots and removes
nothing. --dry-run prints what it would remove. The node is --node-name, else the installed
server’s --node-name, else this host’s default node name.
delete and prune refuse to remove anything from a snapshots directory that is group- or
world-writable, or that is owned by anyone other than root, the _k3sm service user or the user
running the command.
Restoring an HA Server (Embedded etcd)#
On a server installed with --cluster-init or --server-join, the datastore is an embedded etcd
member, and k3sm snapshot save writes an etcd snapshot. k3sm snapshot restore puts one back.
It does what k3s server --cluster-reset --cluster-reset-restore-path does: it turns this server
into the only member of a new etcd cluster holding the snapshot’s data. Every other server then
joins that cluster again as a new member.
Run it on the mint-authority server, the one installed with --cluster-init. If that server is
lost, run it on a joined server, then reinstall that server with --cluster-init in place of
--server-join, as HA describes for --cluster-reset. Stop every server first, and
do not start the others again until step 4.
# 1. On every server: stop it.
sudo launchctl bootout system/io.k3sm.server
# 2. On the server you restore: restore, then start it.
sudo k3sm snapshot restore /Volumes/backups/k3sm-etcd-snapshot.db
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plist
# 3. Verify it (see Verify the Restore below), and check that `k3sm status` shows one voting
# etcd member with a leader present.
# 4. On every OTHER server: move its old etcd data aside, then start it. It joins the restored
# server as a new member and is promoted.
sudo mv /var/lib/k3sm/server/etcd /var/lib/k3sm/server/etcd.pre-restore
sudo k3sm server --clear-crashloop # only if it was started on its old data and parked
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plistThe restore reads the server’s member name, etcd peer address and peer port from the installed
server. Pass --node-name, --etcd-peer-ip and --etcd-peer-port to override them, for example on
a rebuilt Mac with no installed server yet. A server joins through its --server address, so a
server whose --server names a server other than the restored one has to be reinstalled with
--server set to the restored server before step 4. If its join is refused for its token, mint a
new server token on the restored server and reinstall with it.
What the restore does, in order:
- It refuses while the server is running: the work-dir lock, the etcd client, peer and apiserver
ports, the launchd job, and the etcd process the server last recorded. A server stopped with
kill -9can leave its etcd process running, and the restore names that process. - It refuses the wrong kind of file. A kine SQLite snapshot is not restored into an etcd server,
and an etcd snapshot is not restored into a single-server control plane. A copy of an etcd
member’s
member/snap/dbis not a snapshot: it carries no integrity hash, so it cannot be verified, and the restore refuses it by name. Take snapshots withk3sm snapshot save. - It verifies the snapshot before touching anything: the SHA-256 hash etcd appends to every snapshot, then a read-only open of the database with etcd’s own buckets. A corrupt or truncated file costs you an error and nothing else.
- It refuses unless the volume has twice the snapshot’s size free.
- It rebuilds the data dir beside the live one (
etcd.restore-<UTC>.tmp, mode 0700, owned like the data dir it replaces) withetcdutl, built from the same etcd release the server runs. The rebuilt cluster gets a new cluster ID, and its revision is raised by one billion and marked compacted. Every client that watched the old cluster holds a resource version the restored one has not reached, so without that bump a watcher could take old data for new. With it, every watcher and informer lists again from scratch. - It preserves what it replaces. The old data dir becomes
etcd.restore-<UTC>.bakand the etcd status record moves beside it. Nothing is deleted. If moving either into place fails, both are moved back and the work dir is as it was. - If the restore is cut off between those moves (a crash or a power cut), the server has no
etcddirectory and the restore’s.bakor.tmpbeside it. It then refuses to start, rather than form an empty cluster or join as a new member, and says how to finish the restore (run it again) or undo it (move the.bakback toetcd).
When the .bak Directories Can Go#
Keep etcd.restore-<UTC>.bak and etcd-status.json.restore-<UTC>.bak until the restored cluster
has been verified and every other server has joined it again. They are the only copy of the data
the restore replaced. After that they are safe to delete. The etcd.pre-restore directories moved
aside on the other servers can go at the same time.
What a Restore Puts Back, and What It Does Not#
The snapshot is the datastore and nothing else. Restoring it puts the whole datastore back to the moment the snapshot was taken:
- Objects created after the snapshot are gone, and objects deleted since are back. That includes Nodes and Pods, leases, the node-password bindings that tie node names to Macs, and bootstrap tokens. A worker that joined after the snapshot has to join again.
- The PKI is not reverted. The cluster CA, the etcd CAs, the service-account key and every certificate stay exactly as they are; the restore only reads them. A certificate issued after the snapshot stays valid.
- PersistentVolume data is not in the snapshot. Restore it separately (see Storage).
Servers Left on the Old Data#
A server started on its old etcd data after the restore belongs to the cluster the restore
replaced. It asks the other servers about their clusters. When one answers as the only member of a
different cluster, with a revision past the billion a restore adds, that is the restored server,
and this one refuses to start. k3sm status shows the etcd row as this server's etcd member belongs to a cluster that was reset or restored, naming the peer and both cluster IDs. The fix
starts with a check: confirm that peer is the server you restored, then move <work-dir>/etcd
aside and join again, which is step 4 above. The refusal parks the server until the crash-loop
record is cleared.
A peer that answers for a different cluster without that shape (a Mac reinstalled at a server’s
address, for example) is only a warning on the status row: a peer answers for a different etcd cluster; check it. The server keeps running, and nothing tells you to delete its data.
If two servers on old data are started while the restored server is down, they can elect a leader and serve the old data between them until they reach it. Start the restored server first.
A Snapshot Proves Integrity, Not Origin#
The SHA-256 hash at the end of a snapshot proves the file is complete and unchanged since it was written. It does not prove who wrote it: anyone can write a file with a valid hash. Restoring a snapshot makes everything in it the cluster’s state, including its RBAC roles and bindings, its Secrets and its service accounts. Restore only a snapshot you took yourself, from storage only you can write to.
Snapshots Hold Secrets in Plaintext#
An etcd snapshot holds every Secret in the cluster, unencrypted unless the cluster uses
secrets encryption. k3sm snapshot save writes it 0600.
Keep it that way, and keep it off shared storage. k3sm snapshot restore warns when the file it is
given is readable by its group or by others. It does not change the file’s mode, because the file is
yours: run chmod 600 on it yourself.
Automatic Pre-Migration Backup#
A release may move to a newer kine, which re-runs its schema migrations against your existing database. That is one-way, so before the new version opens the database for the first time, the server takes a backup with the control plane stopped and no writer running:
- It refuses to continue unless the volume has twice the database size free. You get a clear error and nothing is written; free space and start the server again.
- It checkpoints the write-ahead log into the main database and verifies the log drained. Without
this, a copy of
state.dbalone would silently omit committed writes. - It copies the database to a temporary name, runs
PRAGMA integrity_checkon the copy, and only then renames it into place, so a backup that exists is complete and verified. - It preserves the kine binary that wrote the database beside the backup, because rolling back needs the version that can read it.
In the server work directory’s db/ you will find:
| File | What it is |
|---|---|
state.db | the live datastore |
state.db.pre-<kine-version>.bak | the verified backup taken before moving to <kine-version> |
kine.pre-<kine-version> | the kine binary that wrote that backup |
state.db.kine-pin | which kine version last opened state.db successfully |
The backup is write-once. Once it exists, later boots leave it alone. A crash-restart loop never overwrites it, and a copy of an already-migrated database never replaces it.
Backing Up by Hand#
The file-level procedure below does what k3sm snapshot save does, with sqlite3(1) and cp. Use it
when the k3sm binary will not run on the node (a broken install, a rescue boot from another machine’s
disk), or when you want to see every step. Otherwise prefer the command, which verifies the copy for
you and refuses rather than writing a partial one.
Because SQLite runs in WAL mode, do not copy state.db out from under a running server. The copy
would be missing whatever is still in the log.
# 1. Stop the control plane. (kickstart RESTARTS; to back up you need it stopped.)
sudo launchctl bootout system/io.k3sm.server
# 2. Fold the WAL into the database and confirm it drained (the -wal must be 0 bytes or gone).
sqlite3 /var/lib/k3sm/server/db/state.db 'PRAGMA wal_checkpoint(TRUNCATE);'
ls -l /var/lib/k3sm/server/db/state.db-wal
# 3. Copy it, and verify the COPY before you trust it.
cp /var/lib/k3sm/server/db/state.db ~/k3sm-backup-$(date +%Y%m%d).db
sqlite3 -readonly "file:$HOME/k3sm-backup-$(date +%Y%m%d).db?immutable=1" 'PRAGMA integrity_check;' # must print: ok
# 4. Start the control plane again.
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plistAdjust the work-dir path if you run unprivileged (~/server under the service user’s home) or passed
--work-dir. Keep backups off the node, on another disk or another host, so losing the machine
does not lose the backup with it.
Restoring by Hand#
The same fallback rule applies. Prefer k3sm snapshot restore, which performs the steps below and
verifies the snapshot before it moves anything. Restoring replaces the datastore with the backup’s
state. The server must be stopped, because an open datastore file swapped underneath a running kine
is corruption, not a restore.
# 1. Stop the control plane.
sudo launchctl bootout system/io.k3sm.server
# 2. Move the current datastore aside — do NOT delete it. Take its sidecars too; a stale
# -wal/-shm beside a restored database is exactly how a "successful" restore comes back
# with the state you were trying to discard.
cd /var/lib/k3sm/server/db
sudo mv state.db state.db.broken
sudo rm -f state.db-wal state.db-shm
# 3. Put the backup in place.
sudo cp state.db.pre-v0.17.1.bak state.db # or your own copy from above
sudo chown _k3sm state.db
# 4. Start the control plane.
sudo launchctl bootstrap system /Library/LaunchDaemons/io.k3sm.server.plistVerify the Restore#
k3sm snapshot restore prints these steps when it finishes; run them either way. A restore that
starts the daemon is not a restore that worked. Check that the API server is serving and that the
objects you expected came back:
k3sm kubectl get --raw='/readyz?verbose' # every check ok
k3sm kubectl get nodes # your node(s), Ready
k3sm kubectl get pods -A # the workloads the backup should contain
k3sm doctor # the datastore row: journal_mode=wal, kine pin reportedIf the objects are missing or the datastore check reports a non-WAL journal, stop, keep
state.db.broken, and do not let workloads reconcile against a half-restored cluster.
Rolling Back to the Previous kine as Well#
If you are restoring a state.db.pre-<version>.bak because a version move went wrong, install the
previous k3sm binary too (see Upgrade § Rollback). The preserved
kine.pre-<version> binary beside the backup is there for that case. The superseded kine pin cannot
be rebuilt from source without a module proxy that still carries it, so those bytes are the copy you
have.
Retention#
- Keep the automatic
.bakuntil you are confident in the new version. A week of real workload is a reasonable bar, and it is the only pre-migration copy that exists. - Once you are confident, delete it. It is a full copy of the database and it does not shrink.
- Keep your own off-node backups on your own schedule (
k3sm snapshot save --out …); the automatic one only appears when a release changes the datastore engine, so it is not a backup policy. k3sm snapshot restoreleaves astate.db.restore-<UTC>.bak(plus its sidecars) behind on every restore. Keep the most recent one until you are confident in the restored cluster; they are full copies and do not shrink.- Deleting a
.bakre-arms nothing. The automatic backup is taken per target version, and that version has already been recorded as having opened the database.
Consistency Notes#
Single-node datastore reads are consistent-LIST. Under heavy churn, watch staleness is possible and its soak validation is still pending. Factor that into recovery expectations; see Limitations.