Skip to content

Commit 54d0de2

Browse files
crimsonsunsetcrimsonsunset
committed
fix(gateway): treat rmcp "transport closed" as reconnectable
Live stdio kill stringifies as MCP call failed: Transport closed, which missed the old matcher and skipped reconnect_fresh. Signed-off-by: crimsonsunset <jsangio1@gmail.com>
1 parent c09e569 commit 54d0de2

3 files changed

Lines changed: 204 additions & 1 deletion

File tree

‎crates/mcpmux-gateway/src/pool/routing.rs‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -956,8 +956,15 @@ impl RoutingService {
956956
}
957957

958958
/// Check if an error string indicates the backend transport died.
959+
///
960+
/// Live rmcp stdio kill (Aug 20) stringified as `MCP call failed: Transport closed`.
959961
fn is_transport_closed_error(error_str: &str) -> bool {
960-
let indicators = ["connection closed", "-32000", "transport channel closed"];
962+
let indicators = [
963+
"connection closed",
964+
"-32000",
965+
"transport closed",
966+
"transport channel closed",
967+
];
961968
indicators.iter().any(|s| error_str.contains(s))
962969
}
963970

@@ -1070,6 +1077,13 @@ mod call_failure_classify_tests {
10701077
assert_eq!(reconnect_path_for_error(err), Some(ReconnectPath::Fresh));
10711078
}
10721079

1080+
#[test]
1081+
fn rmcp_stdio_transport_closed_is_not_unmatched() {
1082+
let err = "mcp call failed: transport closed";
1083+
assert_eq!(reconnect_path_for_error(err), Some(ReconnectPath::Fresh));
1084+
assert!(!RoutingService::is_auth_error(err));
1085+
}
1086+
10731087
#[test]
10741088
fn auth_error_still_routes_to_oauth_reconnect() {
10751089
let err = "401 unauthorized: token expired";

‎docs/planning/backend-connection-resilience-test.md‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
# Backend Connection Resilience — Manual Test Playbook
2+
3+
**For:** any agent (or human) verifying `c09e569` / `root-resolution`
4+
**Last Updated:** Aug 20, 2026
5+
**Implements:** [`backend-connection-resilience.md`](./backend-connection-resilience.md)
6+
**Do not implement code.** Execute the cases below, record pass/fail, stop.
7+
8+
---
9+
10+
## Answer first: new sessions in different roots?
11+
12+
**No. Do not open a new Cursor chat, do not switch workspace roots, do not call `mcpmux_set_workspace_root`.**
13+
14+
That was the old workaround. It opens a fresh *inbound* rmcp session and hides the outbound-pool bug. Using it here makes Case B/C inconclusive.
15+
16+
| Action | Use for this test? |
17+
| --- | --- |
18+
| Same Cursor chat, same workspace (`/Users/joe/Desktop/Repos/Personal/mcp-mux`) | **Yes — required** |
19+
| Reload Cursor MCP tools (once, before starting, if the gateway just rebuilt) | Yes, once |
20+
| New chat / different root / `set_workspace_root` | **No** (confounds reconnect) |
21+
| Bind a real FeatureSet onto another repo | **No** (Case A uses a fake UUID; no write) |
22+
23+
Bindings are keyed by exact `workspace_root` + `machine_id`, not by chat. This repo already has two rows for the same path (Gondor vs Rohan), both on FeatureSet `All`. Stay here.
24+
25+
---
26+
27+
## Snapshot (live DB, Aug 20 2026)
28+
29+
```
30+
DB=~/Library/Application\ Support/com.mcpmux.desktop/mcpmux.db
31+
LOG=~/Library/Application\ Support/com.mcpmux.desktop/logs/mcpmux.2026-08-20.log
32+
```
33+
34+
| Fact | Value |
35+
| --- | --- |
36+
| Space | `00000000-0000-0000-0000-000000000001` |
37+
| This root | `/Users/joe/Desktop/Repos/Personal/mcp-mux` |
38+
| Binding (Gondor `ec211deb…`) | `5a588b93-5ed1-4ada-b7cb-8e32a9f11058` → FeatureSet `All` (`fs_default_00000000-0000-0000-0000-000000000001`) |
39+
| Binding (Rohan `5d581ac9…`) | `8e2b36b6-eeff-4818-9259-948c1b9c3b6b` → `All` |
40+
| HA backend | `home-assistant-new` (HTTP, enabled, 95 features) |
41+
| Binding row count | 38 parents / 74 junction rows (re-count before Case A) |
42+
| Pool stats | **in-memory only** — `consecutive_failures` will not appear in SQLite |
43+
44+
Re-count before Case A (numbers drift):
45+
46+
```bash
47+
sqlite3 "$HOME/Library/Application Support/com.mcpmux.desktop/mcpmux.db" \
48+
"SELECT COUNT(*) FROM workspace_bindings;
49+
SELECT COUNT(*) FROM workspace_binding_feature_sets;"
50+
```
51+
52+
---
53+
54+
## Preconditions
55+
56+
1. Debug gateway is the listener on `:45818` (ancestor = `launchd`, not Cursor Helper). Health: `curl -sf http://127.0.0.1:45818/health` → `{"status":"ok","version":"0.5.0"}`.
57+
2. Cursor MCP `user-mcpmux` points at `http://localhost:45818/mcp`. Reload tools **once** if the binary was just rebuilt, then do not reload again.
58+
3. Only call mux via `user-mcpmux` (`mcpmux_search_tools` → `mcpmux_get_tool_schema` if needed → `mcpmux_invoke_tool`). No direct backend MCP servers.
59+
4. Do not run `pnpm dev:stop` / rebuild mid-test (evicts the pool and invalidates Case B/C).
60+
61+
---
62+
63+
## Case A — bind FK guard (~2 min)
64+
65+
**Goal:** a nonexistent `feature_set_id` returns `invalid_argument`, not `FOREIGN KEY constraint failed`, and the DB does not grow.
66+
67+
1. Snapshot counts (query above). Call them `B0` / `J0`.
68+
2. Call `mcpmux_bind_current_workspace` with `feature_set_id` = `00000000-0000-0000-0000-00000000dead` (or any other unused UUID). Do **not** approve anything — the guard runs before consent.
69+
3. **Pass** if the tool error JSON has `"error":"invalid_argument"` and the message contains `mcpmux_list_feature_sets`.
70+
4. **Fail** if the message contains `FOREIGN KEY`, `internal_error`, or `constraint`.
71+
5. Re-count. `B0` and `J0` must be unchanged.
72+
6. Grep the log:
73+
74+
```bash
75+
rg "bind_current_workspace rejected" \
76+
"$HOME/Library/Application Support/com.mcpmux.desktop/logs/mcpmux.$(date +%Y-%m-%d).log"
77+
```
78+
79+
Expect a `warn` with the fake `feature_set_id` and this Space id.
80+
81+
---
82+
83+
## Case B — reconnect after a killed stdio child (~5 min)
84+
85+
**Goal:** a transport-closed error triggers `reconnect_fresh` (not OAuth `reconnect_instance`) and the retry succeeds. Same session.
86+
87+
Pick a **stdio** server that is already invokable under `All`. Cheap options on this machine: `wakatime`, `markitdown`, `chrome-devtools`. Confirm first:
88+
89+
```
90+
mcpmux_search_tools({ "server_id": "wakatime", "mode": "browse", "limit": 5 })
91+
```
92+
93+
If `server_readiness` is not `ready`, pick another stdio id from that browse, or `mcpmux_list_servers` and take one with `ready`.
94+
95+
1. Invoke a cheap read-only tool once so the instance is live. Example (only if search returned it): `mcpmux_invoke_tool` `server_id=wakatime` `tool=wakatime_wakatime_summaries` with explicit `start`/`end` dates for today. Any successful call is enough.
96+
2. Note the time (`date -u +%H:%M:%S`).
97+
3. Kill **only the child**, not McpMux:
98+
99+
```bash
100+
# find the stdio child (example: wakatime). Do not kill target/debug/mcpmux.
101+
pgrep -lf wakatime
102+
# then: kill <child-pid>
103+
```
104+
105+
If you cannot identify a safe child, stop and report INCONCLUSIVE. Do not `pkill -f mcpmux`.
106+
107+
4. In **this same chat**, invoke the same tool again. Do not call `set_workspace_root`.
108+
5. **Pass** if the invoke succeeds (or returns a normal tool error, not `-32000` / `Connection closed`).
109+
6. Grep from the timestamp in step 2:
110+
111+
```bash
112+
LOG="$HOME/Library/Application Support/com.mcpmux.desktop/logs/mcpmux.$(date +%Y-%m-%d).log"
113+
rg "backend call_tool failed|reconnect attempted after call_tool failure|reconnect_fresh completed" "$LOG"
114+
```
115+
116+
**Pass logs:**
117+
118+
- `backend call_tool failed` with `trigger=transport_closed` (or `auth` only if the error was actually 401)
119+
- `reconnect attempted after call_tool failure` with `ok=true`
120+
- `reconnect_fresh completed` with `ok=true` (this is the new path; `reconnect_instance` / `Reconnecting instance ... after OAuth` must **not** be the line for this failure)
121+
122+
**Fail:** invoke returns `MCP error -32000: Connection closed` to the agent, or the log shows OAuth reconnect for a stdio kill.
123+
124+
---
125+
126+
## Case C — original HA idle (optional, 15–20 min)
127+
128+
**Goal:** reproduce the reported HTTP shape against `home-assistant-new`.
129+
130+
1. Search: `mcpmux_search_tools({ "server_id": "home-assistant-new", "mode": "browse", "limit": 5 })`.
131+
2. Invoke one cheap read-only HA tool. Confirm success.
132+
3. Wait 15–20 minutes. Do not invoke that server. Do not reload MCP. Do not re-pin the workspace. Other mux tools are fine.
133+
4. Invoke the **same** HA tool again in this chat.
134+
5. **Pass / fail / log checks** are identical to Case B, except `server_id=home-assistant-new` and the error string historically was `MCP error -32000: Connection closed`.
135+
136+
If you cannot wait, mark Case C SKIPPED and rely on B.
137+
138+
---
139+
140+
## Case D — unmatched errors still surface (~2 min)
141+
142+
**Goal:** a failure that is neither auth nor transport-closed is not swallowed.
143+
144+
1. `mcpmux_invoke_tool` against a ready server with a tool name that does not exist, e.g. `server_id=wakatime` `tool=definitely_not_a_real_tool`.
145+
2. **Pass** if the raw error comes back to the caller (not a silent success, not a reconnect).
146+
3. If the failure is classified unmatched, the log line is `trigger=unmatched` and there is **no** `reconnect attempted after call_tool failure` for that call.
147+
148+
A permission / not-found error that never hits the backend is also acceptable — note it as "never reached classifier" rather than fail.
149+
150+
---
151+
152+
## Do not
153+
154+
- Call `mcpmux_set_workspace_root` "to fix" a closed connection.
155+
- Open a second agent chat in another repo to "compare sessions."
156+
- `pnpm dev:stop` / rebuild / quit McpMux mid-case.
157+
- `pkill -f mcpmux`.
158+
- Write a real FeatureSet bind as part of Case A.
159+
- Dump `credentials` / OAuth tables from the DB.
160+
161+
---
162+
163+
## Report back (copy this)
164+
165+
```
166+
Case A bind FK: PASS | FAIL | BLOCKED
167+
error code/message:
168+
binding counts before/after:
169+
log line present: yes/no
170+
171+
Case B stdio reconnect: PASS | FAIL | INCONCLUSIVE | SKIPPED
172+
server_id / tool:
173+
first invoke: ok/err
174+
child kill: pid / skipped why
175+
second invoke: ok / raw -32000 / other
176+
trigger= :
177+
reconnect_fresh ok= :
178+
oauth reconnect used: yes/no
179+
180+
Case C HA idle: PASS | FAIL | SKIPPED
181+
(same fields)
182+
183+
Case D unmatched: PASS | FAIL | SKIPPED
184+
error returned:
185+
trigger= :
186+
187+
Confounders (reload MCP / set_workspace_root / new chat / rebuild): none | list
188+
```

‎docs/planning/backend-connection-resilience.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,7 @@ Manual:
242242

243243
## Related Documentation
244244

245+
- [`docs/planning/backend-connection-resilience-test.md`](./backend-connection-resilience-test.md) — agent-followable manual playbook (same session, no re-pin)
245246
- [`docs/planning/aug14-gateway-ops-bugs.md`](./aug14-gateway-ops-bugs.md) — this branch's parent investigation (inbound session keep-alive, log noise); marked stale by this doc (Decision 5)
246247
- [`docs/planning/clone-auth-header-config-editing.md`](./clone-auth-header-config-editing.md) — separate "Connection closed" bug (stale pool reuse after a config edit), same `PoolService::connect_server()` healthy-reuse code path this doc's Decision 1/2 also touches
247248
- [`docs/planning/deny-by-default-bindable-callers.md`](./deny-by-default-bindable-callers.md) — established rmcp's inbound keepalive as expected client-hang behavior (Jun 29), informed PR #221's Decision 3

0 commit comments

Comments
 (0)