Troubleshooting
When a k3sm cluster does not behave. Start with k3sm status, then the common failure modes below.
Start with k3sm status#
k3sm status is one screen covering the two LaunchDaemons, the apiserver, the node, the workloads, the
data root, the datastore and your kubeconfig. A one-line verdict sits at the top, and the command that
fixes whatever is down sits at the bottom.
A healthy cluster looks like this:
k3sm running: 1/1 nodes ready v0.2.0 · kube v1.37.1 · macOS 26.1
install ok /Library/k3sm/k3sm · launcher /usr/local/bin/k3sm · 2 LaunchDaemons in /Library/LaunchDaemons
netd running pid 512 · socket /var/lib/k3sm/run/netd.sock
server running pid 840
apiserver ready https://127.0.0.1:6444 readyz ok
node ready 1/1 nodes ready · my-mac kubelet v1.37.1
workloads ok 3 running
data-root ok /var/lib/k3sm (apfs volume k3sm, 29.9G of 100G used, mounted)
datastore ok kine sqlite, wal, user_version 1
kubeconfig ok ~/.kube/config context "k3sm"
Next: k3sm kubectl get pods -AA control plane that will not start looks like this. The verdict names the cause, and the server row quotes the last line of its log:
k3sm stopped: io.k3sm.server is crash-looping (12 runs, last exit 1) v0.2.0 · kube v1.37.1 · macOS 26.1
install ok /Library/k3sm/k3sm · launcher /usr/local/bin/k3sm · 2 LaunchDaemons in /Library/LaunchDaemons
netd running pid 512 · socket /var/lib/k3sm/run/netd.sock
server crash-loop 12 runs, last exit 1
log: level=ERROR msg="control plane exited" err="bind: address already in use" (×12)
apiserver down https://127.0.0.1:6444: connection refused
node unknown apiserver unreachable
workloads unknown apiserver unreachable
data-root ok /var/lib/k3sm (apfs volume k3sm, 29.9G of 100G used, mounted)
datastore ok kine sqlite, wal, user_version 1
kubeconfig ok ~/.kube/config context "k3sm"
Next: sudo launchctl kickstart -k system/io.k3sm.server
k3sm status logs serverThree more views go deeper when the overview is not enough:
k3sm status daemons # per-daemon detail: run count, last exit, plist and log paths
k3sm status cluster # readyz, node, workloads by phase, datastore, runtime daemon
k3sm status logs server # tail a daemon's log (netd or server; both when omitted)And two flags for scripts and for watching a restart:
k3sm status -o json # the machine interface; the text screen is rendered from it
k3sm status daemons -o json # a cheap launchd-only probe, safe to poll
k3sm status --wait # poll until the cluster is running, or --timeout elapses
k3sm status --watch -o json | your-tool # a stream: one JSON report per changeBoth JSON shapes are versioned; Status JSON is their contract.
CPU and memory: k3sm status top#
k3sm status top lists each node’s CPU and memory use against what it can allocate, with its pod
count and the age of the sample, then the busiest pods by CPU and then memory:
k3sm status top # one screen
k3sm status top --watch=2s # re-render every 2 seconds
k3sm status top --sort memory -n default --limit 10
k3sm status top -o json # the same rows, for scriptsThe figures come from metrics-server (the metrics.k8s.io API, the same source as kubectl top).
metrics-server samples every 15 seconds, so --watch reads it at most every 5 seconds whatever the
interval; the age column shows how old each node’s sample is. A node it has not sampled yet says
no metrics yet. When the metrics API is not available, the first line gives the reason and
message from the v1beta1.metrics.k8s.io APIService, and the tables list nodes and pods without
usage. --sort, -n and --limit shape the pod table only; the node table always covers the
whole cluster. The view exits 0 when it read the cluster, with or without metrics, and 6 when it
could not.
Under sudo, the view reads with your own ~/.kube/config and the installed k3sm context when it
can list nodes, because a worker’s node credential may read only its own node; it refuses that file
if you do not own it, if others can write it, or if it runs a credential plugin or reads its
credential from another file, and when no credential can list nodes it says to run the view
without sudo.
The exit code is the verdict, so a script can branch on it. k3sm status --help lists the codes.
Rows say needs-admin when only root can run the check, such as the runtime socket. Nothing has
failed there; run sudo k3sm status for those rows. A row says unknown when the report could not
see the answer from your account another way, for example when the apiserver does not answer.
Logs#
k3sm daemons run as launchd jobs under the io.k3sm.* labels and log to the unified log system:
log show --predicate 'subsystem BEGINSWITH "io.k3sm"' --last 10m
launchctl print system/io.k3sm.server # daemon state (label may differ per role)Not everything lands there. The server process’s own structured logs (the LoadBalancer decider, the
ingress host, the control-plane supervisor) go to stderr, which launchd routes to
/var/log/k3sm/server.log. The unified-log predicate above shows none of them.
The /var/log/k3sm directory and the daemon log files in it (server.log, netd.log,
datavol.log) are readable by root and by members of the admin group only, because they carry
daemon arguments, cluster endpoints and failure detail. Read them with sudo if your account is not
an administrator of the Mac. k3sm status prints “not readable by this account” in place of a log
tail it cannot read.
k3sm: command not found After Install#
Install links /usr/local/bin/k3sm → /Library/k3sm/k3sm. If the shell cannot find it:
ls -l /usr/local/bin/k3sm # should be a symlink into /Library/k3sm
echo $PATH # should contain /usr/local/bin
/Library/k3sm/k3sm version # always works, the binary itselfThe link is there but the shell does not see it. A terminal opened before the install may have cached the old lookup. Run
hash -r, or open a new window./usr/local/binis missing fromPATH. It is the first entry in/etc/paths, so a login shell normally has it; aPATHthat is overwritten (rather than appended to) in a shell profile can drop it. Add it back, or invoke/Library/k3sm/k3smdirectly.A regular file sits at
/usr/local/bin/k3sm. Install refuses to replace anything that is not a symlink, because that file belongs to something else. Move it aside and re-runsudo k3sm install.Install failed with
refusing to link into /usr/local/bin. The directory is not root-owned, or it is group/other-writable, which would let another account swap the launcher out from under you. Fix it and re-run the install:sudo chown root:wheel /usr/local/bin && sudo chmod 755 /usr/local/bin sudo k3sm install
kubeconfig … not found From k3sm kubectl#
k3sm kubectl and k3sm kubeconfig say this when they were pointed at a specific server work
directory that holds no admin kubeconfig, or when they found no credentials at all. Which one it is
depends on how you ran them:
K3SM_WORK_DIRis set. That names one server and is never second-guessed, so if the directory holds no kubeconfig the command stops there. Point it at the right work directory, or unset it.Nothing was found anywhere. The message names both places it looked. If you are root or the
_k3smservice user, that means the control plane has not written its kubeconfig. Check the server is actually up (see below) and read its log.Your own kubeconfig has no
k3smcontext. For an ordinary account that context is the way in, because the installed server’s work directory belongs to the service user and is not readable by you:kubectl config get-contexts # is there a k3sm context? sudo k3sm install # idempotent; re-merges it into ~/.kube/config k3sm kubeconfig --write # with the context in place, refreshes it (--path writes elsewhere)
See kubectl access for the full resolution order.
Control Plane Startup Failure#
- Run
k3sm statusfirst. It says whether the server daemon is running, crash-looping, stopped or not loaded, and it quotes the last line of the server log for the two failure cases. - Confirm install completed:
k3sm versionandsudo k3sm install(idempotent). - Check the datastore is present and not locked. The kine/SQLite DB lives under the server work directory (see Backup & restore).
- Restart the daemon:
launchctl kickstart -k system/io.k3sm.server(label per your role).
The Data Root Is Declared But Not Mounted#
If you keep /var/lib/k3sm on its own APFS volume, declared either in /etc/fstab or by sudo k3sm install --data-volume, that volume must be mounted before the daemons start. When it is not,
writes land on the boot disk in the bare mountpoint and shadow the real volume, which looks exactly
like an empty cluster. k3sm status reports the data root as not-mounted and the verdict as
stopped, because the daemons refuse to run against an unmounted mountpoint rather than build a
second, empty datastore on top of your data.
sudo k3sm datavol mount
sudo launchctl kickstart -k system/io.k3sm.netd
sudo launchctl kickstart -k system/io.k3sm.serverIf the volume was never declared with a k3sm record (an /etc/fstab line you wrote yourself, with
no sudo k3sm install --data-volume run since), datavol mount has nothing to work from and you
mount it by hand instead. Check first: ls /var/lib/k3sm/run must list only the netd socket before
you remove it, because anything else there is data the remove would delete.
diskutil info /var/lib/k3sm # the Device Node line names the volume
sudo rm -r /var/lib/k3sm/run # the shadow holds only the netd socket
sudo diskutil mount -mountPoint /var/lib/k3sm /dev/diskNsM
sudo launchctl kickstart -k system/io.k3sm.netd
sudo launchctl kickstart -k system/io.k3sm.serversudo k3sm install refuses to run until the volume is mounted, so mount it first and then re-run the
install if you were in the middle of one.
The Data Root Has The Wrong Owner#
The data root itself, /var/lib/k3sm, belongs to root (root:wheel, mode 0755), the same way
k3s keeps its data dir. The directories the daemons write under it (run, server, agent,
pods, storage, the image store) belong to the unprivileged _k3sm service user, and the
node’s mesh keys live in /var/lib/k3sm/keys, which only root can read or change. If the root is
owned by anyone else, k3sm status reports the data root as wrong-owner with the uid it found.
A root owned by _k3sm is the layout older releases left behind. Restarting the netd helper
realigns the ownership:
sudo launchctl kickstart -k system/io.k3sm.netd
sudo k3sm install # the authoritative repair, and idempotentStuck or Crash-Looping Pods#
On the default runtime (what every installed cluster runs), restartPolicy is honored. The
container is restarted in place with an upstream-shaped CrashLoopBackOff backoff, and
kubectl get pod shows the restart count climbing. If yours is not restarting, work through these:
- Is it a plain init container? A non-sidecar init container that fails is not re-run in place. That is the one remaining restart gap on the default runtime, and only a controller replacing the Pod unsticks it.
- Did you start the node with
--runtime hostprocess? That explicit rootless-dev opt-out honors norestartPolicyat all. An exited container is reaped once and never respawned. Drop the flag to get the default runtime back. - Is the Pod’s policy
Never, or is it aJobthat has already succeeded? Both are working as specified. - Restart count stuck at 0 on a
vmPod? Runkubectl describe pod. When the node refuses the restart, each attempt records a WarningFailedEvent,Error: restart container <name>: <cause>, and the cause follows theCrashLoopBackOffmessage in the container’s state. If it says restarting a container in place is not supported by the guest this Pod booted, delete the Pod: its replacement boots a guest that supports it. The node keeps retrying under the backoff until then. - Stuck rather than exiting? Check the Pod was adapted to the native image model, because a raw Linux image cannot run as a Darwin process (see Images).
See Limitations for the precise scope.
A Pod’s Image Will Not Pull#
A container whose image cannot be resolved does not fail the rest of the Pod. The Pod is
created, the containers that can start do start, and the one that cannot sits Waiting with the
reason kubectl get pod prints in its STATUS column. The Pod’s phase stays Pending while any
container waits, and the restart count stays at 0, because a container that never started has not
restarted.
Read the reason first, because each one has a different recovery:
| Reason | What it means | How it recovers |
|---|---|---|
ErrImagePull | The attempt that just failed: the registry pull, the imagePullSecret, the platform match, or the signature gate. | Retried automatically, 10s then 20s, doubling to a 300s ceiling. |
ImagePullBackOff | The same container between attempts. | Nothing to do; the next attempt is already scheduled. |
ErrImageNeverPull | imagePullPolicy: Never and the image is not in this node’s store. Nothing is pulled and no registry is contacted. | Load it (k3sm image load), then wait for the node to re-check: up to two resync intervals, about 20s. |
CreateContainerConfigError | The container’s run spec could not be built: a missing ConfigMap or Secret key, an env reference that does not resolve, an entrypoint the image config does not supply. | Fix the referenced object, then wait for the node to re-check: up to two resync intervals, about 20s. |
InvalidImageName | The reference does not parse. | Only a spec change helps: kubectl set image, or delete and recreate. Parsing the same string again cannot give a different answer. |
kubectl describe pod carries the same story as Events. Pulling opens every attempt and names
the image; Pulled closes a successful one, saying either how long the fetch took or that the image
was already on the machine. On the failing side: Failed on each failed attempt, BackOff when a
retry is scheduled, InspectFailed for an unparseable reference, and ErrImageNeverPull each time
the node re-checks for an image that policy forbids it to fetch. A container whose image is a host
binary gets no Pulling or Pulled at all, because nothing is fetched for it: that is the native
sentinel, or an absolute path on a container that sets neither command nor args. Give such a
container a command and the reference is treated as an image again.
One attempt announces Pulling for every container of the Pod, not just the one that is behind.
The node resolves a Pod’s images in order inside a single start attempt, so a later container’s
Pulled reports a wait that includes the earlier containers’ fetches. That is what the
(… including waiting) figure means here, and it is why the two durations in a Pulled message can
be far apart. The message also omits the image-size clause upstream Kubernetes appends: the runtime
reports no size for a resolution, and a fabricated byte count would be worse than a shorter sentence.
Four things worth knowing before you debug further:
- On the
vmRuntimeClass, one bad image holds the whole Pod. AvmPod is a single guest, so an image that will not resolve fails the guest build rather than leaving one container behind. Every container reportsPending: the container the failure names carries the reason and the message, and the others readContainerCreating, because they never started for a reason of their own. The Pod is kept and the create is retried on the same schedule described above, so nothing goesFailedand no controller churns replacements. A host-process Pod behaves as the paragraph at the top of this section says: its other containers start and keep running. - The message is bounded. The waiting message and the
Failedevent carry a truncated copy of the underlying error, where upstream Kubernetes prints it whole. The full text, including the registry response, is in the node log (/var/log/k3sm/server.log, oragent.logon a worker, orlog show --predicate 'subsystem == "io.k3sm.node"'). - The schedule is per Pod and per image. Ten Pods referencing one bad image are ten independent retry schedules, exactly as they would be on a kubelet. Restarting the node daemon resets every schedule to the 10s base, so a restart is the fastest way to retry everything at once after you fix a registry.
- Deleting and recreating the Pod resets its backoff. The schedule lives in the running daemon and is keyed to the Pod, so a fresh Pod starts at 10s again rather than inheriting a 300s wait.
DNS From Inside a Pod Does Not Resolve Cluster Names#
On the default runtime this should work. Service A records, headless Services, StatefulSet per-Pod names, SRV and PTR all resolve. Check these in order:
- Is the lookup coming from
/bin/shor another/usr/bintool? macOS stripsDYLD_INSERT_LIBRARIESfrom SIP platform binaries, so k3sm’sgetaddrinfoshim never loads into them and their lookups go to the host resolver. Do the lookup from your compiled workload instead. macOS imposes that limit; it is not a misconfiguration. - Is the Pod’s
dnsPolicyDefaultorNone? Neither selects cluster DNS, so nothing is injected. UseClusterFirst(the default when the field is unset). - Is it an AAAA lookup? k3sm’s CIDRs are IPv4; AAAA is never answered.
- Are you on
--runtime hostprocess? In-pod cluster DNS is not wired there, so every lookup goes to the host resolver. ThevmRuntimeClass does not share this gap; cluster DNS works there.
See Limitations for the full per-path picture.
UDP Service Failures#
ClusterIP UDP Services work on ports 1024 and above, and cluster DNS answers on :53. A UDP
ClusterIP Service on a port below 1024 fails to bind, and NodePort UDP is not implemented. See
Limitations.
LoadBalancer Stuck <pending>#
Start with the Service’s Events, which name the cause when k3sm knows it:
kubectl describe service <name>NoEligibleNodes. No node may host the Service: itssvclb.k3sm.io/lbpoollabel names a pool no node is in, or allow-list mode is on (some node carriessvclb.k3sm.io/enablelb=true) and no node in the pool is enabled. See Multi-node.HostNotBound. The named node was chosen but could not bind every port of the Service. Read that node’s log for the reason below. A chosen node that has never reported at all, such as a worker still running an older k3sm during an upgrade, raises no event: the server logs that the node has not reported, and leaves it out of the Service’s status until it does.
Do not look in the unified log for the rest. The LoadBalancer decider and the ingress host log
through slog to stderr, which the launchd job routes to a file, so the
log show --predicate 'subsystem BEGINSWITH "io.k3sm"' command at the top of this page shows nothing
for them. Read the file instead: server.log on the server, agent.log on a joined worker, which
is where that node’s listeners log:
grep svclb /var/log/k3sm/server.log | tail -50
grep svclb /var/log/k3sm/agent.log | tail -50
grep ingress /var/log/k3sm/server.log | tail -50The server logs one line at start naming its own node and the address it publishes for it, and every node logs one line when its listeners start:
svclb: loadbalancer decider starting self=node-a self_addr=100.64.0.1 hold_down=30s mesh_peers=true
svclb: loadbalancer hoster starting node=node-a bind=0.0.0.0Then look for one of these:
svclb: loadbalancer port is RESERVED by a k3sm listener (the NodePort range, the kubelet API port, or a port the ingress host's kube-system/k3sm-ingress Service declares) or for a macOS host service (22, 53, 88, 445, 548, 631); no listener bound, this node is not advertised. The Service declares a port k3sm’s own listeners own (the NodePort range30000-32767, the kubelet API port10250, or a port the ingress host’skube-system/k3sm-ingressService declares,80and443by default) or a macOS host-service port (22,53,88,445,548or631). No listener is bound and the status stays empty, because taking10250would breakkubectl logs/exec/topon this node. Pick a differentspec.ports[].port. Normally the API rejects such a Service atkubectl applywith a message naming the port; if it did not, the admission policy failed to provision, and the log carries a matchingprovision reserved-loadbalancer-port DENY policyerror.svclb: listener bind failed. Something else already holds that wildcard port. The log line carries the exact diagnostic command; run it:lsof -nP -iTCP:<port> -sTCP:LISTENThe usual culprits are another process on the Mac, a pod (macOS has no network namespaces, so pods share the node’s port space; see Limitations), or a second LoadBalancer Service declaring the same port. k3sm never picks a different port for you.
loadbalancer/ingress status will not list this node: no advertisable node address could be derived. The listeners are bound and serving, but there is no address that would be correct to publish, so the server’s node is left out rather than advertised at an unreachable address. A worker is listed at the mesh address the server recorded when it joined. Checkkubectl get node -o wideshows a non-loopbackINTERNAL-IP, and that you did not start the server with--network none.ingress bind retries exhausted. The ingress listeners gave up (bounded retry, no port fallback). Free the port and restart the daemon.
Restart the control plane after fixing the conflict:
sudo launchctl kickstart -k system/io.k3sm.serverSome causes are not an Event: a node that has never reported, and the bind errors listed above, appear only in the node’s log file.
kubectl top Returns No Metrics#
k3sm ships metrics-server as an add-on, and every server applies it, single-node included, once the
server’s own node has its kubelet certificate from the cluster CA and a node has registered. Until
then kubectl top has no data. Check which case you are in:
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get nodes -l node-role.kubernetes.io/control-plane=true
kubectl -n kube-system get pods -l k8s-app=metrics-server -o wideNo APIService means the add-on was not applied yet. The server re-checks every two minutes and
applies it once its node has started and registered, so a fresh install or an upgrade gets it within
a few minutes of the server coming up. An APIService that is not Available means the metrics-server Pod in kube-system is not
Ready yet. The Pod runs only on a server node, so a Pod left Pending with no server node listed by
the second command means no server has labelled its Node yet; each server does that at start.
kubectl top node shows host-wide figures for every Ready node. See
kubectl access and Limitations.
kubectl exec -it or attach Into a Linux Pod Is Refused#
Both verbs are served by the Pod’s guest, so the refusal message says which half is missing.
- "…does not advertise the … capability … recreate the pod". The Pod is running a guest image older than the one this node pins, which can only happen under a development override. Delete and recreate the Pod so it boots the pinned image.
FailedPreconditionon attach. The container kept no stdin to write to. Setstdin: true(andtty: trueif you want a terminal) on the container and recreate the Pod; stdio is decided when the container starts, not when you attach.Unimplementedonkubectl attach -i/-t. That Pod is on the default native path, where attach is output-only. Usekubectl exec -it, or thevmRuntimeClass. See Limitations.
A garbled first screen after attaching is not a failure, because attach replays a bounded buffer and
can start mid-escape-sequence. Redraw with Ctrl-L.
A Pod Cannot Pull From the Cluster Registry#
- Connection refused. The registry runs on the server, on port 6450 unless
--registry-portsays otherwise. A server started with--registry-port 0runs none. Read the address back from thelocal-registry-hostingConfigMap inkube-publicrather than remembering it. - A
vmPod cannot reach it. The in-cluster nameregistry-k3sm:6450needs the server to be running guests or to be on a mesh (see Using It From Inside a Pod). - Push denied. Pushes need a token, pulls do not. Run
k3sm registry token(see Pushing With Other Tools), or for a Pod see Pushing From Inside a Pod.
See The cluster registry for the full model.
Certificate Nearing Expiry#
Component certificates are re-issued on every control-plane boot, so a restart renews them.
sudo k3sm certificate rotate reports what a restart would re-issue (and both CA pins, which
never change); --restart performs it. Rotation does not revoke anything and does not cover
worker-node certs, so read Certificates before you rely on it.
Rosetta-Selecting Pods Stuck Pending#
The node advertises a capability label only when its start-time probe said yes.
- Check what the node advertises:
kubectl get nodes -L k3sm.io/virtualization,k3sm.io/rosetta,k3sm.io/rosetta-linux. - If the key is missing, the node’s log carries two lines for it: one naming the
conditionand thereason(NotInstalled,TranslationFailed,NotSupported,QueryFailed,VMBackendUnavailable) explaining why the capability was not advertised, and one naming thelabelkey that was therefore left absent. Grep the log for the key itself (k3sm.io/rosetta-linux) and you land on it. k3sm.io/rosetta-linuxneeds both virtualization and guest Rosetta. A node withk3sm.io/rosettabut nok3sm.io/virtualizationwill never carry it.- Installed Rosetta after the node came up? The probes run once at daemon start, so restart it:
sudo launchctl kickstart -k system/io.k3sm.server. - Your Pod must also keep
kubernetes.io/os: darwinin itsnodeSelector; a Pod with only the capability key is rejected with a422. SeevmRuntimeClass. - Scheduled, but stuck
ProviderFailedwith aProviderCreateFailedevent naming a platform mismatch? (kubectl describe podshows it, and it is neverImagePullBackOff.) The Rosetta labels are advertised but not yet honored at pull. Multi-arch selection itself works, and k3sm reads the manifest list and picks a platform, butlinux/amd64is not among the platforms it will accept on either path today. The guest path itself (runtimeClassName: vm,linux/arm64) is built and works; the node’s VM host attaches no Rosetta directory share to its guests, so nothing inside a guest could translate an amd64 payload yet. An amd64-only image is therefore refused at pull rather than started and left to crash. That is a documented gap, not a broken node; seevmRuntimeClass.
Multi-Node Join Fails#
- Re-mint the token (
sudo k3sm token create); tokens expire. - Confirm the agent can reach the server on
9345(the join) and6444(the apiserver) and the wireguard mesh is up. See Multi-node.
The Thunderbolt Interconnect Is Not Used#
Start with sudo k3sm doctor on each cabled Mac. Its rows starting with ic- check bridge0 and
its Thunderbolt members, the netd socket, the node’s interconnect address, the peers, the attached
Pods, cable loops and stray state, and each row that is not a pass prints the command that fixes
it. Then:
- Run
k3sm statuson the control plane. Adisabledinterconnect row means nobody armed it; runk3sm interconnect enablethere. A node named as out of range has a cluster index above 30 and never gets an address. - Read the Pod’s
k3sm.io/InterconnectAttachedcondition (kubectl get pod <name> -o jsonpath='{.status.conditions}'). AFalsecondition names why the attach was refused, and the Pod’s Warning Events say the same. - A listener has to bind
$K3SM_INTERCONNECT_IPto be reached over the cable. A wildcard bind stays on the Pod IP. - A peer that stops answering is often asleep. Thunderbolt has no Wake-on-LAN;
k3sm doctor’sic-sleeprow prints the setting that keeps it awake.
See Interconnects for how the cable is armed and used.
A Node Went Offline and Pods Are Stuck Terminating#
A Mac that goes away while it is running Pods leaves them stuck: kubectl get pod shows them
Terminating indefinitely, because nothing on that Mac can confirm the container is actually gone.
Work through this in order.
Confirm the Mac is genuinely off, not asleep or off-network. Ping its LAN address, and check how long
kubectl get node <name>has shown itNotReady(the age of that condition, not the node). A Mac that is asleep or briefly disconnected heals itself on wake and needs nothing below; do not tell a node that is coming back that it is out of service.If the Mac is coming back, stop after applying the taint below and remove it once the node rejoins. Deleting the node is only for a Mac that is actually leaving the cluster; see Removing a worker for that flow instead.
Apply the out-of-service taint once you are sure the Mac is not coming back on its own. Check first: the Mac is powered off or disconnected for good, not asleep, because the taint lets its Pods be deleted while their containers may still run there:
kubectl taint node <name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecutek3sm statusprints this exact command in the workloads row once a Pod has been stuck terminating on aNotReadynode for more than two minutes, so you rarely have to type it from memory. The taint tells the pod garbage collector it may finish deleting Pods bound to that node. It rechecks every 20 seconds (the pinned v1.37 podgc interval), so a stuck Pod clears within about that long of the taint landing.Remove the taint once the node is back:
kubectl taint node <name> node.kubernetes.io/out-of-service:NoExecute-Force-delete as a last resort, only if a Pod is still stuck after the taint, or you cannot wait for it. Check first: the Pod has no attached volume another copy could write to at the same time:
kubectl delete pod <name> --grace-period=0 --forceA force-deleted Pod may still be running on the partitioned Mac. The cluster starts a replacement, and two copies can run at once until the Mac returns and is cleaned up. For a Pod with attached storage the stakes are higher: check the volume’s node binding first, because two writers on one volume corrupt data where two stateless copies merely waste cycles.
Prevent one offline node from freezing a rollout. A DaemonSet’s default
RollingUpdatestrategy waits for every existing Pod to be replaced in place, so a node that never reports back holds up the whole rollout. SetmaxSurge: 1andmaxUnavailable: 0on the DaemonSet’supdateStrategy.rollingUpdateso a new Pod comes up beside the old one instead of waiting for a slot an offline node will never free.
Checks That Need sudo#
Some rows read files or sockets that only root may open: the runtime daemon’s socket, the datastore
file, the serving certificate, the etcd and mesh records, the installed daemon’s plist. Run as an
ordinary account, k3sm status reports those rows as needs-admin (or unknown with sudo k3sm status as the remedy) instead of guessing. A needs-admin row never moves the verdict, and it does
not mean anything is wrong. Run the same report as an administrator to get the answer:
sudo k3sm statusIf the row is still unknown under sudo, the file it reads is missing or damaged, and the row’s
detail names it.
A k3sm Daemon Is Not Running#
The netd, server and agent rows report what launchd says about each LaunchDaemon. Read the
state first:
not-loaded: nothing runs the daemon. The install is incomplete or was partly removed;sudo k3sm installputs it back and is safe to re-run.disabled: launchd will not start it, not even at boot. If you rank3sm stop, that is on purpose andsudo k3sm startbrings it back (see Stopping and starting). Otherwise re-enable it withsudo launchctl enable system/<label>, then restart it.stopped: it is loaded but not running. Restart it withsudo launchctl kickstart -k system/<label>.failedorcrash-loop: it exited with an error. The row quotes the last line of its log, andk3sm status logs <row>prints more. Fix what the log names, then restart it.unknownwith a warning: launchd says it is running but its process is gone. Restart it.
The labels are io.k3sm.netd, io.k3sm.server on a control-plane Mac and io.k3sm.agent on a
worker. When netd is down, pods on that Mac lose their network and cluster names stop resolving on
the host.
The Crash-Loop Breaker Tripped#
When a control-plane component crashes again and again, or the control plane fails to come up
several times in a row, the server daemon parks itself: it stays running so launchd does not keep
restarting it, but it serves nothing. k3sm status reports the server row as crash-loop with the
time the breaker tripped.
A server that was killed outright leaves its control-plane processes running. Its next start stops
the ones it left behind before starting its own, so they do not count toward the breaker; a process
holding the datastore port that this server did not start is still refused. A server parked on the
crash-loop breaker keeps its work dir locked, so stop it (sudo k3sm stop or launchctl bootout)
before a restore.
Read what it recorded. The row’s remedy names the crash-loop record; each entry carries the failing component and the tail of its log:
k3sm status logs server sudo tail -n 50 /var/lib/k3sm/server/crashloop.jsonFix the fault it names. A fault that cannot heal on retry (a bad flag, a missing file) is quoted in the row’s remedy with the command that repairs it. A breaker tripped by repeated loss of the leader lease points at etcd quorum or an overloaded apiserver; check the etcd and apiserver rows on every server.
Clear the record. The parked daemon notices and restarts the control plane by itself:
sudo k3sm server --clear-crashloop
The Staged Control Plane Does Not Match This Build#
Each k3sm build carries the Kubernetes control-plane binaries it was tested with, and stages them when it installs. The server row warns when the binaries that are actually serving differ from the ones this build pins:
- Older than this build: k3sm was upgraded but the control-plane binaries were not re-staged,
so the previous Kubernetes version is still serving. Re-run
sudo k3sm install; the next start of the server picks up the new set. - Newer than this build: the daemon refuses to downgrade the control plane and keeps serving the newer set. Install a k3sm release whose control-plane version is at or above the staged one.
A Node Is Not Ready#
The node row counts the cluster’s nodes that report Ready, and the heartbeat row says
whether this Mac’s own node still reports in on time. A node stops being Ready when its daemon
stopped, its Mac is asleep or off the network, or its credential expired.
k3sm kubectl get nodes
k3sm kubectl describe node <name> # the conditions say which check failed and since when- On the Mac that is not ready, run
k3sm status. Its daemon rows and itsapiserverrow usually name the cause. - A stale heartbeat on a running node points at the node daemon itself. Read
k3sm status logs server(oragenton a worker) and restart it if it is wedged. - A worker whose credential expired needs to join again; see Multi-node.
- A Mac that is gone for good leaves its pods stuck terminating; see the section below.
The etcd Member Is Unhealthy#
On a server that runs embedded etcd, the etcd row reads the member’s status record, which the
running server rewrites every minute. What it reports, and what to do:
- NOSPACE alarm, or nearing the quota: the database reached its backend quota and the cluster turned read-only, or is about to. Compact the keyspace to the current revision, defragment every member, then disarm the alarm. Writes resume only after the disarm.
- No leader: the member cannot see a majority of servers, so the cluster accepts no writes.
Start the other servers; each waits for its peers. If a majority is gone for good, stop the
daemon on one survivor, run
sudo k3sm server --cluster-resetthere, and start it again. Check first: take a snapshot of the survivor, and be sure the other servers will not come back, because a reset makes this server a cluster of one and the others must re-join with wiped etcd data. - Peer address drift: the member is registered under an address that is no longer this
server’s, so its peers cannot reach it. Restore the previous address (
--etcd-peer-ip); server addresses must be stable, for example through a DHCP reservation. Otherwise remove the member through another server and re-join this one with a wiped etcd data directory. - Another alarm, such as CORRUPT: read
k3sm status logs serverfor the cause. A corrupt member must be removed and re-joined with a wiped etcd data directory. - The record is stale or does not parse: the server stopped refreshing it. Read the server log for the reason.
See High availability and Backup & restore before you remove or reset a member.
A Service Routes to the Wrong Backends#
The services row compares the Service proxy’s own record of what it routes with the Ready
endpoints the API holds. When they differ, traffic for those Service ports may reach the wrong
pods, or none.
Wait a minute. The proxy re-lists Services and EndpointSlices every 30 seconds and repairs a lost update by itself.
If the row still differs, restart the server:
sudo launchctl kickstart -k system/io.k3sm.serverIf the row says Services could not be listed, check that the API answers for your account:
k3sm kubectl get endpointslices -A.Right after a start the row may say the proxy has not recorded yet. It does so 30 seconds after the server starts.
The Ingest Registry#
Every server runs the cluster registry unless it was started with --registry-port 0, and records
whether it is serving. The ingest-registry row checks that record against the registry itself:
down: the registry was recorded as serving but does not answer on its port, so pushes and pulls through it fail. Restart the server, which restarts the registry:sudo launchctl kickstart -k system/io.k3sm.server.disabled: the server gave up starting it, and the row’s detail says why (for example, the port is already in use). Fix that cause, or choose another--registry-port, then restart the server.- The record does not parse, or names an unknown state: restart the server, which rewrites the record. An unknown state means the record was written by a newer k3sm than the one you ran the report with.
See The cluster registry and A Pod Cannot Pull From the Cluster Registry.
The Runtime Daemon Is Unhealthy#
The runtime daemon starts, stops and inspects the containers on its node. Its socket is
owner-only, so an ordinary account sees the runtimed row as needs-admin; run sudo k3sm status for the answer. When it does not answer, or reports itself unhealthy, pods on that node
may not start. The row names the conditions that are not true. Restart the node daemon, which
restarts the runtime with it:
sudo launchctl kickstart -k system/io.k3sm.server # io.k3sm.agent on a workerIf it stays unhealthy, read k3sm status logs server (or agent). A NetworkReady condition
that is false points at netd: check the netd row.
The Node Resolver Does Not Answer#
Cluster names (*.svc and the cluster domain) resolve through a resolver on each node, which netd
publishes into the host’s resolver configuration. Two rows cover it:
dnsasks the resolver for thekubernetesService. If it does not answer over UDP, or answers with no address, restart the node daemon:sudo launchctl kickstart -k system/io.k3sm.server(io.k3sm.agenton a worker). If it answers over TCP but not UDP, its UDP socket stopped receiving, and the same restart fixes it. A--dns-vipthat is not an IP address serves no resolver at all; re-runsudo k3sm installwith a valid address, or without the flag.node-resolverchecks that the host routes cluster names to that resolver. If the entry is missing, or netd is not running to maintain it, restart netd, which republishes it:sudo launchctl kickstart -k system/io.k3sm.netdscutil --dnsshows the host resolver configuration directly.
For names that fail only inside a pod, see DNS From Inside a Pod Does Not Resolve Cluster Names.
Host Binaries Changed After a macOS Update#
Pods that start through /bin/sh run re-signed copies of a few host binaries, which carry k3sm’s
DNS and path shims. A macOS update replaces the originals, and the copies then lag behind. The
shadow-shells row reports drift with the binaries that changed, missing when no copies exist,
and a warning when the copies come from an older install that lacks some this k3sm makes. Re-run
the install, which re-makes the set from the current host binaries:
sudo k3sm installNext#
- Stopping and starting covers a node that reads as stopped on purpose, and a stop that left something behind.
- FAQ has quick answers.
- Limitations answers whether something is a bug or a documented gap.
- Backup & restore covers recovering the datastore.
- Certificates covers the PKI, rotation, and its limits.