herdr 0.9.0 added multi-machine support: herdr machine add <ssh-target> --label <label> saves a machine, and its workspaces and agents show up right alongside your local ones in the same window. I’ve got three Macs, a laptop and two servers, so I tried wiring all three together. Two of the three connections worked immediately. The third one didn’t, and the error message it gave me was almost useless.
Gotcha 1: argument order
Small one first, since it’ll bite you before the real story even starts. --help lists --label before the target, so it’s natural to write:
herdr machine add --label server-b server-bThat prints only the usage line and exits with status 2. The target has to come first:
herdr machine add server-b --label server-bGotcha 2: “lost connection to server”
This is the one that actually took a debugging session. Adding server-a and server-b from the laptop worked fine. Adding server-b from server-a failed, and so did the reverse direction:
error: lost connection to server: server closed connection; machine was not savedA plain ssh server-b from server-a worked without any issue, so it wasn’t a broken SSH config in any obvious sense. Neither server log (~/.config/herdr/herdr-server.log) had anything useful either, the remote side never even saw a connection attempt. A failed machine add isn’t logged anywhere, and the CLI’s own message doesn’t say why.
Watching what herdr actually runs
herdr shells out to ssh for all of this, so the fastest way to see what it’s really doing is to put a fake ssh first in PATH, just for one command, that logs every invocation and forwards to the real binary.
Create a folder for it (mkdir -p /tmp/ssh-shim) and save this as /tmp/ssh-shim/ssh, a script that logs the call, runs the real ssh unchanged, and logs how that went too:
#!/bin/bash
echo "ssh $*" >> /tmp/ssh-calls.log # log the exact arguments herdr called ssh with
/usr/bin/ssh "$@" # run the real ssh, unchanged, same arguments
rc=$? # rc = its exit code: 0 success, nonzero failure
echo " -> rc=$rc" >> /tmp/ssh-calls.log # log that result too
exit $rc # and pass it back to herdr, as if we were never hereThen make it executable, and run the one command you actually want to inspect with that folder at the front of PATH, so herdr finds this fake ssh before the real one, only for this one invocation:
chmod +x /tmp/ssh-shim/ssh
PATH="/tmp/ssh-shim:$PATH" herdr machine add server-b --label server-bThat’s the only change needed, PATH gets the shim folder tacked onto the front for this one command, and reverts on its own right after, nothing to undo, nothing installed permanently.
Every ssh call herdr makes shows up in the log, in order, with its real exit code. Here’s the whole run: six setup calls, the one call that actually failed, and herdr’s own cleanup call at the end ($TMPDIR below is macOS’s per-session temp dir, herdr’s own scratch files live there, not literally in /tmp):
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b '/bin/sh -s'
-> rc=0
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b 'command -v herdr'
-> rc=1
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b '/bin/sh -s'
-> rc=1
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b '/bin/sh -s'
-> rc=0
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b '/bin/sh -s'
-> rc=0
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -T server-b '/bin/sh -s'
-> rc=0
ssh -o BatchMode=yes -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=yes -o ConnectTimeout=10 -o ConnectionAttempts=1 -o ServerAliveInterval=15 -o ServerAliveCountMax=4 -T server-b 'exec $HOME/.local/bin/herdr remote-client-bridge'
-> rc=255
ssh -F $TMPDIR/herdr-ssh-64355-0/config -S $TMPDIR/herdr-ssh-64355-0/ctl -o ControlMaster=auto -o ControlPersist=yes -O exit -o BatchMode=yes server-b
-> rc=0Two of the first six exit 1, and that’s not the bug: command -v herdr failing is exactly the PATH problem in the notes below, herdr isn’t on the non-interactive PATH on the remote, so that check fails and herdr falls back to finding the binary another way, which is also why the call right after it fails too before the rest succeed. Chasing every nonzero exit code here sends you the wrong direction.
The seventh call, line 13 above, is the one that matters, and the only one in the whole log that failed. It’s also the only line without -F/-S at all, which turns out to be exactly why: no -F means it skips herdr’s own temp config entirely and opens a fresh connection using your normal ~/.ssh/config instead. The eighth and last call, -O exit, just tells herdr’s shared connection to shut down once it’s done, unremarkable by itself, but it confirms the first six really were sharing one connection.
What -F and -S actually do
-F <file> picks which config file ssh reads at all. Normally that’s ~/.ssh/config, then /etc/ssh/ssh_config; with -F, ssh reads only the named file and skips both of those entirely. herdr writes its own config into that temp folder fresh for each run (I didn’t inspect its contents, so I won’t guess what’s in it), and every setup call points -F at it. The bridge call doesn’t, so it’s the one call in the whole log that actually reads your real ~/.ssh/config, which is exactly where a host’s UserKnownHostsFile /dev/null line takes effect.
-S <path> names where a shared connection’s control socket lives, and it only does anything paired with two more flags every setup call also carries: ControlMaster=auto means if nothing is already listening at that socket, this ssh logs in normally and opens the socket for others to share; if something is listening, it skips logging in entirely and just runs its command over the existing connection. ControlPersist=yes keeps that shared connection open in the background after the command that opened it exits, instead of closing it the moment that one command finishes.
Put together: the first setup call does the only real login, authentication and host key check included, and opens the socket. Calls two through six just reuse it, no new login, no new host key check, which is also why they’re fast and why none of them cared about the host key at all.
What herdr runs on the remote
The /bin/sh -s calls aren’t a black box either, -s just means the shell reads its script from standard input instead of a filename, and with -T disabling any pseudo-terminal on the ssh side, there’s nothing interactive about it at all, no prompt, nobody typing. Seeing what actually gets piped in took extending the wrapper to also save stdin and stdout, but only for /bin/sh -s calls specifically, the bridge call carries herdr’s own live protocol stream rather than a discrete script, so capturing it the same way wouldn’t have meant anything. Re-running machine add against a healthy host, one where it actually succeeds, filled in the rest of the sequence ($HOME below stands in for the actual absolute home directory herdr sends, it always uses the real path, never the literal string $HOME):
| # | Script sent on stdin | Output | Purpose |
|---|---|---|---|
| 1 | uname -s; uname -m | Darwin / arm64 | Detect the remote’s OS and CPU type, which decides which binary it would need |
| 2 | command -v herdr (sent as a command argument, then again via sh -s) | nothing, rc=1 both times | Is herdr on the PATH? No, ~/.local/bin isn’t on the non-interactive PATH. These are the two rc=1 calls in the log above |
| 3 | a discovery script (below) | $HOME/.local/bin/herdr | Checks the usual install locations in order: ~/.local/bin, Homebrew, mise, Nix, and prints the first one that exists and is executable |
| 4 | herdr status client --json | version 0.9.0, protocol 22, capabilities including surface_interest, health_check | Is the installed binary compatible with this client? |
| 5 | herdr status server --json | running, compatible, restart_needed: false | Is a compatible server already running, or does it need starting or replacing? |
| — | the plain bridge connection (no -F/-S) | rc=0 on a healthy host | First real connection attempt, the same call that exits 255 in the failing run |
| 6 | exec herdr remote-client-bridge </dev/null | nothing, rc=0 | A quick bridge check over the shared connection |
| 7 | herdr status server --json again | same as before | Confirms the server is still fine |
| — | a second plain bridge connection | rc=0 | Connects again right before saving the profile |
| — | -O exit | rc=0 | Closes herdr’s shared connection |
A few things worth pulling out of that table:
- The failing run matches steps 1 through 5 exactly, one
rc=0, tworc=1, three morerc=0, then dies on the first plain bridge connection. It never reaches steps 6, 7, or the second bridge call, because it never gets past the first one. - The two
rc=1calls are expected, not a symptom. They’re just “not on PATH” answers, and step 3 exists specifically to work around that. - Nothing here installs or restarts anything. Step 5 reported a compatible server already running, so herdr skips straight past any install or restart path entirely. A missing or incompatible install would trigger more scripts, downloading, installing, restarting, and those need approval in an interactive terminal, a path that never triggered in this session, so I won’t guess at what it looks like.
- Custom install locations aren’t searched. If herdr lives somewhere not on that list and not on
PATH, step 3 won’t find it, andmachine addfails for a completely different reason than this post.
Step 3’s actual script, for anyone who wants to know exactly what “the usual install locations” means:
home=${HOME:-}
user=${USER:-}
version=0.9.0
emit() {
path=$1
if [ -n "$path" ] && [ -x "$path" ]; then
printf '%s\n' "$path"
fi
}
if [ -n "$home" ]; then
emit "$home/.local/bin/herdr"
fi
emit "/opt/homebrew/bin/herdr"
emit "/usr/local/bin/herdr"
if [ -n "$home" ]; then
emit "$home/.local/share/mise/installs/herdr/$version/bin/herdr"
emit "$home/.local/share/mise/installs/herdr/$version/herdr"
emit "$home/.local/share/mise/installs/github-ogulcancelik-herdr/$version/herdr"
emit "$home/.nix-profile/bin/herdr"
fi
if [ -n "$user" ]; then
emit "/etc/profiles/per-user/$user/bin/herdr"
fi
emit "/nix/var/nix/profiles/default/bin/herdr"
emit "/run/current-system/sw/bin/herdr"It stops at the first path that exists and is executable, checked in exactly that order.
You can’t usefully run that exact bridge command by hand, remote-client-bridge expects to speak herdr’s own protocol on stdin, not print something and exit. Swap the remote command for something harmless and keep the same flags, and you get the real error herdr was swallowing:
ssh -o BatchMode=yes -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=yes -o ConnectTimeout=10 -T server-b "echo bridge-ok" </dev/nullNo ED25519 host key is known for 192.0.2.10 and you have requested strict checking.
Host key verification failed.~/.config/herdr/herdr-client.log confirmed the same thing later, when a running client retried a saved machine: remote platform detection failed: No ED25519 host key is known ... Run herdr --remote <target> interactively ...
Root cause
herdr enforces host key checking itself. Its connection command passes -o StrictHostKeyChecking=yes directly on the command line, and command-line options always win over ~/.ssh/config. So a StrictHostKeyChecking no sitting in your config does nothing for herdr specifically, even though it works fine for a plain ssh to the same host.
The failing host’s SSH config had the usual “don’t bother me about this” pair:
StrictHostKeyChecking no
UserKnownHostsFile /dev/nullherdr overrides the first line but not the second. So ssh has nowhere to actually look up a host key, and strict mode refuses to proceed without one.
The laptop worked by the same ControlMaster mechanism explained above, not by having the right config. Its own Host * block had ControlMaster auto plus a ControlPath, and a connection to that host already happened to be open before herdr ever tried. The bridge call reused it instead of logging in fresh, exactly like setup calls two through six, so it never ran its own host key check at all. That’s fragile: close the master connection (reboot, network drop, anything) and reconnecting fails exactly the same way the direct attempt did.
The reverse direction had its own, unrelated problems on top: no SSH alias configured for the other machine at all, and a stale known_hosts entry throwing WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED. Strict mode refuses a changed key just as readily as a missing one.
Fix
- In the target’s
Hostblock, removeUserKnownHostsFile /dev/null(and the now-pointlessStrictHostKeyChecking noalongside it). If you want to set the file path explicitly instead of removing the line, use~/.ssh/known_hostsand watch out if your home directory contains a space (common on an external volume), an absolute path with an unquoted space breaks, becausesshsplits the value on whitespace. - Make sure
known_hostsactually has the right key:
ssh-keygen -R <host> # drop the stale entry
ssh-keyscan -t ed25519 <host> >> ~/.ssh/known_hosts
ssh-keygen -lF <host> # compare with the next line
# on the target, over a channel you already trust:
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub- Run a pre-flight check with the same flags herdr’s own bridge connection uses. It should print
okwith zero prompts:
ssh -o BatchMode=yes -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=yes -o ConnectTimeout=10 -T <alias> 'echo ok'- Only then run
herdr machine add <alias> --label <label>.
You can test any of this before touching a config file at all, by passing the same options as -o flags on the pre-flight command, or by feeding them through the logging wrapper above. Both directions here were validated that way first, config files only got edited once the pre-flight check already passed.
Smaller notes
- On the remote side, herdr lives in
~/.local/bin, which typically isn’t onPATHfor a non-interactive SSH session.machine addalready accounts for that on its own, nothing to fix there. - Automatic reconnects never prompt for anything. Anything that would need a prompt, a host key, a passphrase, an install step, shows that machine as “Attention” in the TUI instead of hanging.
Talking to agents on other machines, still
Multi-machine in 0.9.0 brings the TUI together: one window, all your machines’ workspaces and agents. What it doesn’t do yet is bring the agent CLI together. The maintainer says as much directly in the 0.9.0 announcement: “The TUI can now bring your machines together, but the agent CLI still works within one server. It doesn’t yet see the agents running on your other machines,” with cross-machine agent collaboration listed as future work.
Today, the documented way to reach an agent on another machine is still to run the CLI over there yourself:
ssh server-a 'HERDR_SESSION=default ~/.local/bin/herdr agent list'
ssh server-a 'HERDR_SESSION=default ~/.local/bin/herdr agent prompt <pane-or-name> "..." --wait --timeout 300000'Pane IDs and agent names are scoped to one server, so always look them up on the host you’re actually sending to, not the one you’re sitting at.
Small fact for the record: the debugging notes behind this whole post were handed to me exactly that way, one agent, on one machine, prompting another over SSH. Same mechanism as getting two agents talking across machines in the first place, just used here to write up a bug instead of to say hello.
Leave a Reply