SSH machines: bare and container runtimes

An SSH machine is a remote host under [machines.<id>] with kind = "ssh". Three runtimes can run on it:

bare — uses an existing Git project directory on the remote machine. Create isolated checkout chooses a session-specific clone under .mj/clones/<session-id> instead of using the selected directory directly.
podman — starts a rootless container on the remote machine (the same model as on this machine, just reached over SSH) and runs the session inside it.
docker — uses the remote host’s Linux Docker daemon; see Docker for Mjolnir.

All three shell out to the local ssh CLI rather than using an SSH library.

Provisioning a session runs 15 to 25 ssh and scp commands against the same host, and every attached session keeps one more ssh process open for its relay. On unix, Mjolnir runs these as sessions on a small number of shared, already authenticated connections (“masters”) instead of letting each command pay for its own handshake.

A stock sshd allows at most 10 sessions on one connection (MaxSessions 10). Mjolnir therefore places at most 8 sessions on each master and opens another master for the same host when all of its masters are full. Set MJ_SSH_SESSIONS_PER_CONNECTION in the daemon’s environment to change the number; an unset or invalid value falls back to 8. The two spare sessions are for mj doctor and Tab completion, which can join a master from another process without being counted.

Mjolnir opens each master explicitly with ssh -f -N -o ControlMaster=yes -o BatchMode=yes -o ControlPersist=60 and confirms it with ssh -O check. Every other command then runs with ControlMaster=no, the master’s ControlPath, and ProxyCommand=false. That combination can only use the master: if the master is gone, the command fails at once with exit status 255 instead of quietly opening its own connection. Mjolnir then checks the master again, reopens it, and retries the command. If a master cannot be opened at all, the command fails with an error that names the host and quotes ssh’s own message. A master exits 60 seconds after its last session closes, so ssh processes can outlive the daemon by that long.

The control sockets live in $XDG_RUNTIME_DIR/mjolnir/<instance>/ when that variable is set, and in <data dir>/ssh/ otherwise. <instance> is default for the default instance, the name for --instance, and a 16-character hash of the data directory when only MJ_DATA_DIR is set. Each instance has its own directory, so two daemons never share a master. A socket is named <hash>-<shard>, where the hash covers the destination and your extra_args, and the shard number counts the masters for that host from 0. A <hash>-<shard>.lock file beside each socket makes processes of the same instance (for example the old and new daemon during a restart) take turns checking and opening that master, so they never open two. Mjolnir creates the directory with mode 0700. If it cannot, or if the socket path would be too long for a unix socket address, every command opens its own connection instead. Commands that share a master also share its fate: if the underlying connection drops, every session on it fails at once.

Two kinds of command only ever reuse a master and never create one: the target validation probes (including the mj doctor connectivity probe) and remote Tab completion. They set a short ConnectTimeout and a one-miss keepalive so they fail fast instead of hanging the interface, and a master holding those settings would drop every later session on it after a stall of a couple of seconds. They carry ControlMaster=no and the path of a master, so they join it when it is up and otherwise open their own direct connection. In the daemon, a validation probe counts as one of the 8 sessions on the master it joins, so probes for new sessions cannot push a master past MaxSessions. mj doctor and Tab completion run in other processes and use the two spare sessions.

If the server refuses a session anyway, the command never started, so Mjolnir retries it, up to three attempts with a growing delay. The dashboard’s own Git reads for the conversation header, the capacity probe, and the relay connection to each session’s worker retry the same way. A refused session that gets in on a retry is logged only at debug level; one still refused on its last attempt is logged as a warning that names “refused another session on a shared connection (MaxSessions)”. A connection closed before authentication (MaxStartups) is always a warning.

If the target’s extra_args already set ControlMaster, ControlPath, or -S, Mjolnir adds no sharing options at all for that target and leaves sharing to you. A -J jump host in extra_args is used when a master is opened. To turn sharing off entirely, set MJ_SSH_CONTROL_MASTER=0 (off, false, and no also work) in the daemon’s environment; every command then opens its own connection. Sharing is unix-only; Windows OpenSSH does not implement it.

Limiting concurrent connections

Mjolnir starts one ssh or scp process per remote operation, so a daemon restart with many sessions opens a burst of connections to the same host at once. To keep that burst from being refused, Mjolnir admits at most 6 concurrent connections per destination and makes the rest wait. Set MJ_SSH_MAX_CONCURRENT in the daemon’s environment to change the limit; an unset or invalid value falls back to 6.

The number to compare it to is the remote sshd’s MaxStartups, which counts unauthenticated connections. Its stock value, 10:30:100, starts randomly dropping connections at the eleventh concurrent pre-auth connection and drops all of them past a hundred. Keep MJ_SSH_MAX_CONCURRENT comfortably below the first number, or raise MaxStartups on the host. A dropped connection exits 255 with Connection closed by <host> port 22 or kex_exchange_identification: read: Connection reset by peer; Mjolnir recognizes those and retries the invocation rather than reporting a failure, because the remote command never ran.

Connection sharing removes most of this pressure, because a master authenticates once for up to 8 sessions. The concurrency limit still applies to opening masters, and to every command when sharing is off.

Prerequisites you set up by hand

Key-based SSH that works non-interactively. Mjolnir runs ssh without a pseudo-terminal and does not prompt for a password or passphrase, so the target user must already accept your key without interaction (an unlocked key, ssh-agent, or a passphrase-free key).
A host-key policy you have chosen. By default Mjolnir passes -o StrictHostKeyChecking=accept-new: ssh trusts a host key it has never seen and records it in known_hosts, and refuses a key that has changed. Mjolnir does not otherwise manage known_hosts. To require that the key already be in known_hosts, add the option to the machine’s extra_args:
```
[machines.builder]
kind = "ssh"
host = "builder"
extra_args = ["-o", "StrictHostKeyChecking=yes"]
```
OpenSSH uses the first value it sees for an option, and Mjolnir puts extra_args before its own defaults, so this setting wins. The same holds for UserKnownHostsFile and any other option. Setting StrictHostKeyChecking yes in ~/.ssh/config is not enough: options on the command line override the config file. Add the host key first, for example with ssh-keyscan <host> >> ~/.ssh/known_hosts after you have checked the key’s fingerprint.
For a bare runtime: an existing remote Git project with a valid HEAD. The SSH user must be able to create .mj/clones/ below the repository when Create isolated checkout is selected. Uncommitted changes in the source are not copied into that clone.
For a Podman runtime: rootless Podman on the remote host, meeting the same postconditions Mjolnir expects locally. See Podman for Mjolnir — the remote host needs Podman 4.3 or newer and the same rootless user-namespace setup as a local Podman host.

Machine and runtime configuration

The SSH host is one [machines.<id>] entry, and every runtime on it names that machine. Sessions run on runtimes, so a machine with no runtime is not offered as a session target.

You do not have to write the runtimes yourself. When you save a machine under Settings › Machines, Mjolnir probes it over SSH and adds every runtime it can run: a bare runtime named after the machine when the host is Linux or macOS on x86_64 or arm64, and <machine>-podman and <machine>-docker for each engine that passes the same check a launch runs. A machine already in the file without a runtime gets the same treatment each time mj starts and when you run mj setup, and Settings › Runtimes › Detect runtimes probes every machine. A machine that answers with nothing usable, or does not answer, keeps no runtime; Settings › Machines, the new-session picker and mj doctor say why, with the fix. Removing one runtime of a machine is kept; to stop using a host altogether, remove the machine.

The machine’s keys are:

Key	Required	Notes
`kind`	yes	`ssh`.
`host`	yes	SSH destination: hostname, IP, or an alias from your SSH config.
`user`	no	SSH login user; omit to use your SSH config / default.
`identity_file`	no	Path to the private key.
`extra_args`	no	Extra arguments for every `ssh` invocation for this machine. They come before Mjolnir’s own options, so they override Mjolnir’s defaults.
`workspace_prefix`	no	Per-session lifecycle path recorded for cleanup as `<prefix>/<session-id>`. It does not select or relocate the Git project. Defaults to `.local/share/hel/workspaces` relative to the login home.

A bare runtime on the machine also takes:

Key	Required	Notes
`permissions`	no	`guardian` (the default) preserves configured harness approvals; `yolo` runs unconstrained.

Guardian approvals are supported by Codex, Claude Code, Grok Build, OpenCode, and Muse Code. Kimi does not provide them.

[machines.builder]
kind = "ssh"
host = "builder"
workspace_prefix = ".local/share/hel/workspaces"

[targets.builder]
kind = "bare"
machine = "builder"
permissions = "guardian"

macOS hosts, including EC2 Macs

A bare SSH runtime can run directly on an Intel or Apple silicon Mac. The controller detects the remote operating system and architecture and uploads the worker from its own release. Linux controllers can manage Mac workers; release archives include both per-architecture macOS workers, and each macOS npm package includes its own architecture’s worker.

Enable Remote Login on the Mac and configure noninteractive SSH key access. Install working Git (including the Xcode Command Line Tools if using Apple’s Git) and the tools your project needs. Codex and Claude also require Node.js 22 or newer and npm. Mjolnir checks these through the account’s login shell, so Homebrew’s PATH setup belongs in that shell’s login configuration, such as .zprofile for zsh. The worker uses the target account’s login environment and then applies any configured profile environment overrides.

[machines.mac]
kind = "ssh"
host = "my-mac"

[targets.mac]
kind = "bare"
machine = "mac"
permissions = "guardian"

Select an existing Git project on the Mac when starting a session. Workspaces and checkpoints follow the same SSH-bare lifecycle described below. Closing a session does not shut down the Mac or stop other sessions.

For EC2, provision the Mac instance and Dedicated Host yourself, then use its SSH address in this configuration. Use the SSH machine kind for this persistent host: the aws-ec2 machine kind owns disposable instances and terminates them on session teardown. This SSH configuration does not allocate or release AWS hosts, install Xcode, or manage signing identities.

When developing from source, provide a worker built from the same source revision as the controller. MJ_WORKER_DIR accepts mj-worker-aarch64-apple-darwin or mj-worker-x86_64-apple-darwin; the older combined mj-worker-universal-apple-darwin is still accepted as a fallback for bundles that predate the per-architecture split. Build an architecture-specific worker on a Mac with cargo build -p brokk-mj-worker --bin mj-worker; copy it under the matching name in that directory. A release controller and a worker from another revision cannot be mixed.

How a remote bare project is prepared

The new-session wizard asks for an existing absolute Git directory on the SSH host. Mjolnir validates that path remotely; it does not clone a configured bundle into it.

Create isolated checkout defaults on for a primary checkout and off for an existing linked worktree; you can change either choice in the final review. When checked, Mjolnir creates an independent clone at <repository>/.mj/clones/<session-id> on the default branch. Changes in the source checkout stay there. When unchecked, the session uses the selected checkout directly.

Because the session works in a checkout that belongs to the host, it stays there: mj move and mj resume refuse a target on another machine (this session's working tree lives on <host>; resume it there). Changing the profile in place works. To continue elsewhere, push the session’s branch and start a new session from it.

workspace_prefix is separate from this project workflow. It derives a Mjolnir-owned lifecycle path that is recorded with the target and removed during teardown. It does not control the selected project, the linked-worktree location, or the fixed worker and staged-profile roots. A leading ~/ is treated as relative to the SSH login home; leave the default unless you need a different cleanup namespace.

A Podman runtime always runs unconstrained and does not accept permissions. It takes the same container keys as a local Podman runtime (image, and optionally platform, cpus, memory, environment, pull_policy, workspace_storage):

[targets.builder-podman]
kind = "podman"
machine = "builder"

Verifying a target

mj doctor includes a dedicated check per runtime on an SSH machine, and a machine.<id> check for a machine that has no runtime, saying what probing it found. It first probes connectivity with ssh -o BatchMode=yes <host> true, and its failure messages include the exact command to fix the common causes (ssh-copy-id for key auth, ssh-keyscan for an untrusted host key). For a Podman runtime it then runs the same Podman probes as on this machine, over SSH. With --smoke, Podman over SSH also gets a disposable run/exec/remove test; a bare runtime does not have a smoke test.

You can also always sanity-check reachability by hand before relying on a target:

ssh <host> true

If that succeeds non-interactively, Mjolnir’s own SSH invocations should too.