Skip to content

Commit 43155a8

Browse files
mcp-tool-shopclaude
andcommitted
fix(hook): add score floor HOK_MIN_SCORE=0.3 to silence weak matches (HOK-01)
The hook took the top-5 matches by rank, ignoring r.score, so incidental single-keyword matches (~0.1-0.25) leaked irrelevant pointers (observed: claude-guardian/duel-system on a memory-os prompt). Add a calibrated min-score floor: against the live 336-entry index, noise tops out ~0.25 and genuine topical matches begin ~0.33+, so 0.3 delivers the design's 'below-threshold -> emit nothing'. Recall on keyword-rich entries (game canon) is a separate Phase-2 keyword-curation concern. smoke-test.ps1 now does a mirror-vs-live drift check (HOK-05) and runs under a scratch HOME so it never writes the live usage.jsonl. Applied to mirror + live (byte-identical, hash 21879425...); live backup at loadout-hook.mjs.bak-pre-hok01-20260616. settings.json wiring unchanged. Verified: noise/off-topic prompts silent, strong matches preserved, ~51ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a4fa04f commit 43155a8

2 files changed

Lines changed: 82 additions & 9 deletions

File tree

apps/hook/loadout-hook.mjs

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
// • matchLoadout(prompt, index) against it
77
// • Emit AT MOST 5 entries as one-line pointers: "- <id> — <summary> → <path>"
88
// • Total additionalContext kept ≤ ~200 tokens
9-
// • Below-threshold match emit nothing (silent)
9+
// • Below-threshold match (score < HOOK_MIN_SCORE) → emit nothing (silent)
1010
// • Always-record loaded events to ~/.ai-loadout/usage.jsonl
1111
//
1212
// Off-switch: AI_LOADOUT_HOOK=off no-op exit 0.
@@ -25,6 +25,13 @@ const INDEX_PATH = resolve(HOME, '.ai-loadout', 'index.json');
2525
const USAGE_PATH = resolve(HOME, '.ai-loadout', 'usage.jsonl');
2626
const MAX_ENTRIES = 5;
2727
const MAX_LINE_CHARS = 180;
28+
// Minimum match score for a DOMAIN entry to be injected. Core entries (score 1.0)
29+
// always pass. Calibrated 2026-06-16 against the live 336-entry index: observed
30+
// incidental single-keyword noise tops out at ~0.25, genuine topical matches begin
31+
// ~0.33+, so 0.3 is the "confident match" floor that delivers the design's
32+
// "below-threshold → emit nothing". Recall on keyword-rich entries (e.g. game
33+
// canon) is a separate Phase-2 keyword-curation concern, not this floor's job.
34+
const HOOK_MIN_SCORE = 0.3;
2835

2936
function readStdinSync() {
3037
try {
@@ -66,7 +73,9 @@ async function main() {
6673

6774
let results;
6875
try { results = matchLoadout(prompt, index); } catch { safeExit(0); return; }
69-
const top = (results || []).filter(r => r && r.entry && r.entry.priority !== 'manual').slice(0, MAX_ENTRIES);
76+
const top = (results || [])
77+
.filter(r => r && r.entry && r.entry.priority !== 'manual' && r.score >= HOOK_MIN_SCORE)
78+
.slice(0, MAX_ENTRIES);
7079
if (top.length === 0) { safeExit(0); return; }
7180

7281
const lines = top.map(r => {

apps/hook/smoke-test.ps1

Lines changed: 71 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,73 @@
1+
#requires -Version 5
2+
<#
3+
Smoke test for the loadout hook.
4+
5+
- Drift check (HOK-05): mirror (apps/hook) vs live (~/.claude/loadout-hook) must be byte-identical.
6+
- Threshold check (HOK-01): drives the hook with representative prompts and shows what it injects.
7+
A min-score floor means weak/incidental matches and off-topic prompts go silent.
8+
9+
Isolation: runs the hook under a SCRATCH HOME (a copy of the live ~/.ai-loadout/index.json),
10+
so testing never appends to the live usage.jsonl.
11+
12+
Usage:
13+
./smoke-test.ps1 # tests the MIRROR copy (this repo) by default
14+
./smoke-test.ps1 -HookPath live # tests the LIVE copy (~/.claude/loadout-hook)
15+
#>
16+
param(
17+
[string]$HookPath = 'mirror'
18+
)
119
$ErrorActionPreference = 'Continue'
2-
$hook = 'C:/Users/mikey/.claude/loadout-hook/loadout-hook.mjs'
20+
21+
$mirror = Join-Path $PSScriptRoot 'loadout-hook.mjs'
22+
$live = Join-Path $HOME '.claude/loadout-hook/loadout-hook.mjs'
23+
24+
# ── Drift check (HOK-05) ────────────────────────────────────────
25+
Write-Output ('=' * 78)
26+
Write-Output 'DRIFT CHECK — mirror vs live (must be byte-identical)'
27+
Write-Output ('=' * 78)
28+
$mirrorHash = (Get-FileHash $mirror -Algorithm SHA256).Hash
29+
if (Test-Path $live) {
30+
$liveHash = (Get-FileHash $live -Algorithm SHA256).Hash
31+
if ($mirrorHash -eq $liveHash) { Write-Output "OK — identical ($mirrorHash)" }
32+
else { Write-Output "DRIFT — mirror=$mirrorHash live=$liveHash" }
33+
} else {
34+
Write-Output "live copy not found at $live"
35+
}
36+
Write-Output ''
37+
38+
# ── Resolve which hook to drive ─────────────────────────────────
39+
$hook = if ($HookPath -eq 'live') { $live } else { $mirror }
40+
Write-Output "Driving: $hook"
41+
Write-Output ''
42+
43+
# ── Scratch HOME (isolate from live usage.jsonl) ────────────────
44+
$scratch = Join-Path ([System.IO.Path]::GetTempPath()) ("loadout-smoke-" + [guid]::NewGuid().ToString('N').Substring(0,8))
45+
New-Item -ItemType Directory -Force -Path (Join-Path $scratch '.ai-loadout') | Out-Null
46+
$liveIndex = Join-Path $HOME '.ai-loadout/index.json'
47+
if (Test-Path $liveIndex) {
48+
Copy-Item $liveIndex (Join-Path $scratch '.ai-loadout/index.json')
49+
} else {
50+
Write-Output "WARN: no live index at $liveIndex — matching will be empty"
51+
}
52+
53+
# Redirect the hook's homedir() (USERPROFILE on Windows) to the scratch dir so
54+
# usage events land in scratch, never the live ~/.ai-loadout/usage.jsonl.
55+
$origUserProfile = $env:USERPROFILE
56+
$origHome = $env:HOME
57+
$env:USERPROFILE = $scratch
58+
$env:HOME = $scratch
359

460
$prompts = @(
561
@{ label = '1. GAME prompt (Star Freight visual canon)';
662
payload = @{ prompt = 'I want to update the Star Freight visual canon — the Renna identity packet needs a new portrait variant. Open the right canon paths and the workflow profile.' } },
763
@{ label = '2. TOOL prompt (shipcheck a repo before publish)';
864
payload = @{ prompt = 'Run shipcheck audit on the role-os repo before npm publish — full treatment afterwards if it passes.' } },
965
@{ label = '3. GENERIC prompt (mundane filesystem)';
10-
payload = @{ prompt = 'list the files in the current directory' } }
66+
payload = @{ prompt = 'list the files in the current directory' } },
67+
@{ label = '4. NOISE prompt (the known bad case — should be silent or strong-only)';
68+
payload = @{ prompt = "Let's build memory-os out using the dogfood swarm protocol." } },
69+
@{ label = '5. OFF-TOPIC prompt (should be silent)';
70+
payload = @{ prompt = 'what is the weather like today and should I bring an umbrella' } }
1171
)
1272

1373
foreach ($p in $prompts) {
@@ -20,17 +80,21 @@ foreach ($p in $prompts) {
2080
$start.Stop()
2181
$ms = $start.ElapsedMilliseconds
2282
if (-not $out -or $out.Trim().Length -eq 0) {
23-
Write-Output "(silent — no pointer injection)"
83+
Write-Output '(silent — no pointer injection)'
2484
} else {
2585
try {
2686
$parsed = $out | ConvertFrom-Json
27-
$ctx = $parsed.hookSpecificOutput.additionalContext
28-
Write-Output $ctx
87+
Write-Output $parsed.hookSpecificOutput.additionalContext
2988
} catch {
3089
Write-Output "RAW OUTPUT: $out"
3190
}
3291
}
33-
Write-Output ""
92+
Write-Output ''
3493
Write-Output ("latency: {0} ms" -f $ms)
35-
Write-Output ""
94+
Write-Output ''
3695
}
96+
97+
# ── Restore env + cleanup scratch ───────────────────────────────
98+
$env:USERPROFILE = $origUserProfile
99+
$env:HOME = $origHome
100+
Remove-Item -Recurse -Force $scratch -ErrorAction SilentlyContinue

0 commit comments

Comments
 (0)