Skip to content

Commit 993f378

Browse files
crimsonsunsetcrimsonsunset
committed
docs(planning): add meta-gateway invoke manual QA runbook
Single-session checklist with copy-paste prompts for Phases A–C validation. Signed-off-by: crimsonsunset <jsangio1@gmail.com>
1 parent ed8a5fb commit 993f378

1 file changed

Lines changed: 299 additions & 0 deletions

File tree

‎docs/planning/meta-gateway-invoke-qa.md‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# Meta-Gateway Invoke — Manual QA Runbook
2+
3+
**Last Updated:** May 25, 2026
4+
**Branch:** `feat/meta-gateway-invoke`
5+
**Related:** [`meta-gateway-invoke.md`](./meta-gateway-invoke.md)
6+
7+
One-session checklist for validating Phases A–C (search → schema → invoke, result shaping, FeatureSet ACL + surfaced tools).
8+
9+
---
10+
11+
## Quick prep
12+
13+
- [ ] Rebuild/restart gateway if you haven't since the branch (`pnpm dev` or run the built app)
14+
- [ ] Cursor → MCP → **Reload tools**
15+
- [ ] Confirm McpMux endpoint: `http://localhost:45818/mcp`
16+
- [ ] Have at least one OAuth server (GitHub) **installed and connected** but **inactive** in session (for enable-flow tests)
17+
- [ ] Optional for Phase C tests: create a FeatureSet with 1–2 GitHub tools, bind to workspace; leave surfaced off until test 7
18+
19+
**Tester:** _______________
20+
**Date:** _______________
21+
**McpMux version / commit:** _______________
22+
23+
---
24+
25+
## 0. Sanity — meta-only surface
26+
27+
**Prompt:**
28+
29+
```
30+
You have McpMux meta tools only — no direct backend tools like github_*.
31+
32+
1. Call mcpmux_list_servers and show installed servers and active/inactive status.
33+
2. Tell me how many tools you see in your available tool list total, and list their names.
34+
```
35+
36+
| Check | Pass | Fail | Notes |
37+
| ----- | ---- | ---- | ----- |
38+
| `mcpmux_list_servers` returns installed servers | ☐ | ☐ | |
39+
| Only **10** `mcpmux_*` tools exposed (no backend names) | ☐ | ☐ | Expected: bind, create_feature_set, disable/enable_server, get_tool_schema, invoke_tool, list_all_tools, list_feature_sets, list_servers, search_tools |
40+
| Backend servers show **inactive** until enabled | ☐ | ☐ | |
41+
| Tool list count stable (~10 meta + Cursor/plugin tools) | ☐ | ☐ | |
42+
43+
---
44+
45+
## 1. Happy path — GitHub read (Phase A)
46+
47+
**Prompt** (swap repo if needed):
48+
49+
```
50+
Use ONLY the McpMux meta workflow — do not guess backend tool names or params.
51+
52+
Goal: list open issues in mcpmux/mcp-mux.
53+
54+
Steps you must follow explicitly:
55+
1. mcpmux_list_servers — check if github is active
56+
2. If inactive: mcpmux_enable_server for github
57+
3. mcpmux_search_tools with query "list issues", server_id "github", detail_level "description"
58+
4. mcpmux_get_tool_schema for the best match
59+
5. mcpmux_invoke_tool with exact args from the schema
60+
61+
Show each step briefly, then the first 5 issues.
62+
```
63+
64+
| Check | Pass | Fail | Notes |
65+
| ----- | ---- | ---- | ----- |
66+
| Agent enabled github when inactive | ☐ | ☐ | |
67+
| Search before invoke (no param guessing) | ☐ | ☐ | |
68+
| Schema read before invoke | ☐ | ☐ | |
69+
| Invoke succeeded with correct param names | ☐ | ☐ | |
70+
| `tools/list` still ~10 meta tools after enable | ☐ | ☐ | |
71+
72+
---
73+
74+
## 2. Fail-closed + recovery (Phase A errors)
75+
76+
**Prompt:**
77+
78+
```
79+
Try to invoke a GitHub tool WITHOUT enabling github first (disable it if needed).
80+
81+
1. mcpmux_invoke_tool on github with tool list_issues and dummy args
82+
2. Show the exact error message
83+
3. Follow whatever it tells you to do
84+
4. Retry invoke successfully
85+
```
86+
87+
| Check | Pass | Fail | Notes |
88+
| ----- | ---- | ---- | ----- |
89+
| Invoke denied when server inactive | ☐ | ☐ | |
90+
| Error mentions `mcpmux_enable_server` with server_id | ☐ | ☐ | |
91+
| Recovery via enable → retry works | ☐ | ☐ | |
92+
93+
---
94+
95+
## 3. Search detail levels + compact schema (Phase A)
96+
97+
**Prompt:**
98+
99+
```
100+
On github (enabled):
101+
102+
1. mcpmux_search_tools query "list" detail_level "name" limit 5
103+
2. Same query detail_level "description"
104+
3. Pick one tool — mcpmux_get_tool_schema compact: true
105+
4. Same tool — compact: false
106+
107+
What did compact strip?
108+
```
109+
110+
| Check | Pass | Fail | Notes |
111+
| ----- | ---- | ---- | ----- |
112+
| `name` level omits descriptions | ☐ | ☐ | |
113+
| `description` level includes descriptions | ☐ | ☐ | |
114+
| `compact: true` strips descriptions/examples | ☐ | ☐ | |
115+
| Batch schema (array of tools) works if agent tries it | ☐ | ☐ | |
116+
117+
---
118+
119+
## 4. Session toggle — list size unchanged (Phase A)
120+
121+
**Prompt:**
122+
123+
```
124+
1. Enable github — confirm search finds github tools
125+
2. Disable github via mcpmux_disable_server
126+
3. Search again for github tools
127+
4. Report tools/list count before and after — must stay the same
128+
```
129+
130+
| Check | Pass | Fail | Notes |
131+
| ----- | ---- | ---- | ----- |
132+
| Search empty / no github matches when disabled | ☐ | ☐ | |
133+
| Meta tool count unchanged across enable/disable | ☐ | ☐ | |
134+
135+
---
136+
137+
## 5. Default truncation (Phase B)
138+
139+
**Setup:** Enable a heavy server — `posthog-personal`, `firebase-dev`, or GWorkspace clone.
140+
141+
**Prompt:**
142+
143+
```
144+
Enable [heavy server]. Find a list/analytics tool via search, read schema, invoke WITHOUT filter.
145+
146+
Show whether response includes { returned, total, truncated: true } or similar metadata.
147+
Paste payload size estimate (rough char count is fine).
148+
```
149+
150+
| Check | Pass | Fail | Notes |
151+
| ----- | ---- | ---- | ----- |
152+
| Large array auto-truncated | ☐ | ☐ | Default ~50 rows / 64KB |
153+
| Truncation metadata present | ☐ | ☐ | |
154+
155+
---
156+
157+
## 6. Explicit filter (Phase B)
158+
159+
**Prompt:**
160+
161+
```
162+
Same tool as test 5. Invoke with filter: { "max_rows": 3, "format": "summary" }
163+
164+
Then again with fields projection if the tool returns objects with id/name/title fields.
165+
```
166+
167+
| Check | Pass | Fail | Notes |
168+
| ----- | ---- | ---- | ----- |
169+
| `max_rows: 3` honored | ☐ | ☐ | |
170+
| `format: summary` applied | ☐ | ☐ | |
171+
| `fields` projection limits keys per row (if tested) | ☐ | ☐ | |
172+
173+
---
174+
175+
## 7. Clone disambiguation (server_id filter)
176+
177+
**Setup:** You have GWorkspace ×2 clones — enable **only one**.
178+
179+
**Prompt:**
180+
181+
```
182+
Enable ONLY taylorwilsdon.google-workspace-mcp-uvx (not the s2h clone).
183+
184+
mcpmux_search_tools query "drive" or "list files" with server_id set explicitly.
185+
Confirm results are scoped to that server_id only.
186+
```
187+
188+
| Check | Pass | Fail | Notes |
189+
| ----- | ---- | ---- | ----- |
190+
| `server_id` filter scopes search | ☐ | ☐ | |
191+
| Other clone's tools not in results | ☐ | ☐ | |
192+
193+
---
194+
195+
## 8. FeatureSet ACL — partial tool set (Phase C)
196+
197+
**Setup:** FeatureSet with 1–2 GitHub tools included, bound to workspace, surfaced **off**.
198+
199+
**Prompt:**
200+
201+
```
202+
I bound a FeatureSet that only allows specific GitHub tools.
203+
204+
1. mcpmux_search_tools query "github" detail_level "name"
205+
2. Try mcpmux_invoke_tool on a tool NOT in the FeatureSet
206+
3. Invoke one tool that IS included
207+
```
208+
209+
| Check | Pass | Fail | Notes |
210+
| ----- | ---- | ---- | ----- |
211+
| Search only finds allowed tools | ☐ | ☐ | |
212+
| Invoke denied for disallowed tool | ☐ | ☐ | |
213+
| Invoke succeeds for allowed tool | ☐ | ☐ | |
214+
215+
---
216+
217+
## 9. Surfaced tool promotion (Phase C)
218+
219+
**Setup:** In FeatureSet editor, toggle **Surface in client** on one included tool. Reload MCP tools.
220+
221+
**Prompt:**
222+
223+
```
224+
1. List all tools available — identify mcpmux_* vs surfaced backend
225+
2. Call the surfaced tool directly (one hop)
226+
3. Call a different tool on same server via mcpmux_invoke_tool
227+
```
228+
229+
| Check | Pass | Fail | Notes |
230+
| ----- | ---- | ---- | ----- |
231+
| Surfaced tool appears in client tool list | ☐ | ☐ | |
232+
| Surfaced tool callable without invoke wrapper | ☐ | ☐ | |
233+
| Non-surfaced backend still requires invoke | ☐ | ☐ | |
234+
235+
---
236+
237+
## 10. Diagnostic — list_all_tools vs search
238+
239+
**Prompt:**
240+
241+
```
242+
mcpmux_list_all_tools with server_id "github" (or one enabled server).
243+
Compare count to mcpmux_search_tools with query "" and same server_id.
244+
Explain why agents should prefer search.
245+
```
246+
247+
| Check | Pass | Fail | Notes |
248+
| ----- | ---- | ---- | ----- |
249+
| `server_id` filter on list_all_tools works | ☐ | ☐ | |
250+
| Agent recommends search over full dump | ☐ | ☐ | |
251+
252+
---
253+
254+
## 11. End-to-end agent task (realism)
255+
256+
**Prompt:**
257+
258+
```
259+
Brief status report on mcpmux/mcp-mux repo:
260+
- open issue count
261+
- 3 most recent issue titles
262+
- one paragraph summary
263+
264+
Rules: McpMux meta tools only, read schemas before invoke, note truncation if any.
265+
```
266+
267+
| Check | Pass | Fail | Notes |
268+
| ----- | ---- | ---- | ----- |
269+
| Completed without backend tool name guessing | ☐ | ☐ | |
270+
| Schema-first invoke pattern | ☐ | ☐ | |
271+
| Sensible output despite truncation | ☐ | ☐ | |
272+
273+
---
274+
275+
## Red flags (stop and file a bug)
276+
277+
- [ ] Backend tools (`github_*`, etc.) appear in `tools/list` without surfacing
278+
- [ ] Agent can call backend tools directly (bypassing `mcpmux_invoke_tool`)
279+
- [ ] Enable server expands `tools/list` beyond meta + surfaced
280+
- [ ] Search returns tools from inactive or unbound servers
281+
- [ ] Invoke succeeds for tools outside FeatureSet ACL
282+
- [ ] Large list invoke returns unbounded payload with no truncation metadata
283+
- [ ] Opaque errors (no enable/invoke redirect hints)
284+
285+
---
286+
287+
## Sign-off
288+
289+
| Area | Result |
290+
| ---- | ------ |
291+
| Phase A — meta invoke core | ☐ Pass ☐ Fail |
292+
| Phase B — result shaping | ☐ Pass ☐ Fail |
293+
| Phase C — ACL + surfaced | ☐ Pass ☐ Fail ☐ Skipped |
294+
| Overall | ☐ Ship ☐ Block |
295+
296+
**Blockers / issues filed:**
297+
298+
```
299+
```

0 commit comments

Comments
 (0)