0051. The node protocol carries streams over the agent's own connection
Status: accepted, 2026-10-30. Refines 0008 and 0015: a
node behind NAT can be a relay target.
Context
The agent keeps one outbound WebSocket to appsd (ADR 0008) for its desired state, its reports and a few requests.
Everything else appsd asks of a node goes the other way, to an HTTPS port on the node with a pinned certificate (ADR
0015): relayed app traffic, uploads for builds, cron runs, database dumps and restores, volume and image moves. So a
node behind NAT can't take relayed traffic, build, run a scheduled job or have a database dumped. The owner decided
terminals in containers, logs that follow and connections to databases, from phones: each would need more of that
inbound path, held open by appsd for as long as a person keeps a screen open, and cut whenever appsd restarts.
Decision
- Protocol 2 multiplexes streams over the agent's own connection. The agent offers the WebSocket subprotocol
xvii-node.2; when appsd accepts it, the connection carries yamux, the agent as client and appsd as server. Without
it the connection is protocol 1, today's JSON messages, so old agents and old control planes keep working with new
ones.
- The first stream is the control stream, opened by the agent: protocol 1's messages, unchanged, one per line.
- Every other stream is opened by appsd and is an HTTP/1.1 connection to the agent's relay handler — the endpoints
and relayed traffic its TLS port serves — upgraded where it has to stay open (a terminal, a TCP connection). appsd
closes any other stream a node opens.
- The TLS port becomes a fast path for relayed traffic and bulk transfers where appsd can reach it; a node it can't
reach gets everything over its own connection, so nodes behind NAT are relay targets too. Edge and co-ingress still
need the node's own 80 and 443.
- Long-lived work lives on the node, by id and offset. Terminals, runs and log follows belong to the node, not to a
connection: a client or appsd that reconnects asks again from where it was, and results the node finishes are sent
again until appsd acknowledges them.
- Agents say what they can in their hello (
capabilities); appsd answers node_outdated for what a node's agent
can't do yet.
Consequences
- One TCP connection carries a node's control, interactive and, behind NAT, bulk bytes: a large transfer can delay a
keystroke. Bulk bytes prefer the TLS port, and appsd caps streams per kind.
- A NAT node's app traffic crosses the control plane, as every relayed node's does.
- appsd holds a yamux session per node; the agent serves its handler on two listeners and enforces limits for its
interactive endpoints itself.
- The messages other designs add (backup jobs, agent updates, build sources) ride the control stream or a stream
unchanged.
- Design:
superpowers/specs/2026-10-30-apps-operations-design.md in this repository.