k3sm

Compare

A side-by-side of k3sm against Kubernetes in a Linux VM on a Mac, macOS VMs orchestrated across Macs, and a Linux Kubernetes cluster.


k3sm is Kubernetes whose nodes are Macs and whose Pods are Darwin processes, so it answers a different question than the VM-based tools. This page compares them point by point; the narrative version is why k3sm.

Four things are being compared:

  • Kubernetes in a Linux VM on your Mac: desktop container tools, local-cluster tools, and Linux VM managers that boot Linux Kubernetes in one or more guests on a single Mac, with ports and directories forwarded back to the host.
  • A Linux Kubernetes cluster: anything from a managed cloud offering to k3s on hardware.
  • macOS VMs orchestrated across Macs: tools that boot every job as a macOS guest under Virtualization.framework and place guests on hosts from a controller, so a build farm boots a fresh macOS guest for every job.
  • k3sm: Kubernetes whose nodes are Macs and whose Pods are Darwin processes.

Side by Side#

Verdicts only; every no and partial in the k3sm column is expanded in the two sections below.

Linux VM on a Maca Linux Kubernetes clustermacOS guests across Macsk3sm
runs your existing Linux imagesyes, unmodifiedyes, unmodifiedno, VM images rather than container imagespartial, linux/arm64 unmodified via vm, whose guest cold-boots in a 165 ms median on an M1 Ultra Mac Studio, single node; amd64-only images do not run
CNCF [Conformance] certifiedyes, typicallyyes, typicallythe unit of work is a VM, not a Podno, cannot be
highly available control planeno, one Macyes, typicallyn/apartial, an experimental two-server control plane on embedded etcd that tolerates no server failure (high availability)
runs a macOS-only workloadnonoyes, in a macOS guestyes
GPU and unified memoryno, behind the hypervisoryes, Linux GPUspartial, the guest’s paravirtualized GPUyes, direct
filesystem performance for host filespartial, bind mountsyes, nativepartial, directories shared into the guestyes, native
schedules a fleet of Macsno, single laptopno, different hardwareyes, a controller places guests on hostsyes, wireguard mesh on two Macs today (multi-node); experimental until it graduates in a later release on the roadmap
per-tenant isolation on a shared nodeyes, namespaces and cgroupsyesyes, one VM per jobpartial, shared trust domain
enforced CPU limitsyes, CFS millicoresyes, CFS millicoresyes, vCPUs per guestno, accounting only
enforced memory limitsyes, cgroup guaranteeyes, cgroup guaranteeyes, guest memory sizepartial, sampled then SIGKILL
NetworkPolicy as a security boundaryyes, with a CNIyeseach guest has its own network stackno, a hint each node applies to Service ingress for its own Pods
eviction under node pressureyesyesguests per host is the placement limitpartial, memory only, one Pod at a time
Helm charts as HelmChart objectsyes, when the guest runs k3syes, on k3sn/ayes, k3s’s fields in helm.k3sm.io/v1; no private-repo credentials (Helm)
auto-applied manifest directoryyes, when the guest runs k3syes, on k3sn/apartial, apply-only; RBAC, admission, Secret and ServiceAccount objects refused
Secrets encrypted at restyes, an API server optionyes, an API server optionn/apartial, opt-in on a new cluster; no key rotation
the API surfaceupstreamupstreamthe tool’s own APIupstream
release statusstable linesstable linesreleased tools0.x releases (v0.2.3 is current); the app and the install-script tarball are Developer ID signed and notarized
installone downloadn/aan agent per Macinstall script, or the k3sm app, a Developer ID signed, notarized disk image
resource cost on your Macan always-on VMn/aa VM per jobprocesses

Where VM-Based Tools Are Ahead#

  1. An amd64-only Linux image does not run at all. It is refused at pull on both paths. The default path wants darwin/arm64, the vm path wants linux/arm64. An unmodified linux/arm64 image does run: adding runtimeClassName: vm boots it in a per-Pod micro-VM on a single node. On the default path a Pod is a Mac binary, so a workload that has to run there is rebuilt, not relabelled, though registry, tags, digests and pull secrets all behave as expected (what runs). A Linux VM tool runs any Linux image, amd64 or arm64, on a mature path today.
  2. It is not CNCF-conformant, and there is no path to becoming so. The suite requires Linux containers, cgroups, CNI, and network namespaces; k3sm has none of them by construction and cannot carry a Certified Kubernetes badge.
  3. The release line is younger. k3sm is on 0.x releases, and a supported-versions policy comes with the first stable release planned. There is no support contract on offer.
  4. The isolation model is weaker. Same-node native Pods share a uid and a port space. Seatbelt (macOS’s process sandbox) bounds what each Pod can reach, but it is neither a namespace nor a tenancy boundary. A Pod can name the vm RuntimeClass for a per-Pod micro-VM boundary instead (linux/arm64 only).
  5. The resource model is weaker. CPU limits are accounting and QoS, never a millicore ceiling. Memory limits are sampled proc_pid_rusage followed by SIGKILL, which is best-effort and not a cgroup guarantee. Node-pressure eviction covers memory only, one Pod at a time.
  6. NetworkPolicy is not a security boundary. It applies to Service-VIP ingress only, with no egress and no ipBlock. Each node, workers included, enforces it for its own Pods.
  7. Developer ID signing and notarization do not run inside a native Pod. A Pod reaches the Apple frameworks, not the login session, so a keychain read comes back empty with a zero exit status and a workload that tests for a certificate takes the wrong branch without an error. Signing with a Developer ID identity and notarizing both rest on that keychain. SwiftPM does not finish either: swift build re-signs its product by absolute path as its last step, and a Pod may name a file for signing only by a relative path. Sign and notarize on the host. A vm Pod runs Linux and cannot stand in for it (limitations).

Each of these is a place to contribute; see community. k3sm publishes no head-to-head performance comparison against these tools; its own figures, such as the vm path’s cold boot and idle cost, are on limitations.

Where k3sm Is Ahead#

  1. Your workload can be a Mac. Metal, CoreML inference on the Neural Engine, and full-speed local disk at host paths are reached directly, because there is no guest between the Pod and the OS. MLX model serving runs on the Apple GPU. The Seatbelt profile a Pod runs under bounds this. What it reaches is the frameworks, not the login session. A Pod also builds. clang and swiftc compile C, Objective-C and Swift against an installed SDK and ad-hoc sign the result, with no guest to cross to do it. The limitations page lists the Apple services it holds a Pod away from.
  2. A room full of Macs becomes a cluster. A Mac joins with one token, the upstream scheduler places work on it, a joined worker serves NodePorts across nodes and resolves cluster DNS in-Pod, and Pod traffic between nodes crosses a wireguard mesh. A worker runs as its own LaunchDaemon, reuses its stored node credential after a restart, and deregisters from the cluster on uninstall. Two Macs can also share the control plane on embedded etcd (high availability); two servers tolerate no server failure. Both are experimental. Multi-node has run on two Macs and graduates in a later release on the roadmap.
  3. The API is the upstream one. Your manifests, RBAC, CRDs, and controllers are the same ones you use everywhere, because the control-plane binaries are the same ones. Helm charts install through k3s’s HelmChart and HelmChartConfig kinds, with the same fields under the helm.k3sm.io/v1 group.
  4. No always-on guest. Native Pods are processes. Only a vm Pod adds a micro-VM, one per Pod, created when the Pod starts.
  5. The unit of work is a process, not a macOS guest. A macOS guest is a whole operating system with its own memory and boot, so a host runs only a few at once. A node runs as many native Pods as its CPU and memory carry, and there is no hypervisor between the Pod and Metal or unified memory.

How to Choose#

  • For a full Linux development environment on a Mac, or for amd64: a Linux VM tool. For running a single arm64 Linux image inside a k3sm Pod, the vm RuntimeClass does that directly.
  • For a production Kubernetes cluster: Linux Kubernetes, the platform the ecosystem builds for, certifies, and supports.
  • For a fleet of Macs where every job must be a fresh macOS VM: a macOS VM orchestrator.
  • For orchestrating work that must run on macOS (builds, Apple-platform tests, on-device inference) through the Kubernetes API, as processes on shared nodes rather than as guests: k3sm.

To try it: install.

Full detail: what runs for the workloads the default path takes, limitations for what does not, and the conformance profile for the formal self-assessment.