I run a personal Obsidian vault — a curated wiki of research on Claude Code, agentic AI, and related tooling, built on the idea that you curate sources once and let an agent do the summarizing, cross-referencing, and bookkeeping afterward (the approach is laid out well in Using Claude Code to Set Up a Second Brain, AKA an LLM Wiki, which is where I got the idea). To make the vault actually useful across projects, I built a small Claude Code sub-agent, second-brain-lookup: before a research task on some unrelated project starts, it checks the vault first, so I’m not re-discovering sources I’ve already captured.

The pitch was simple: this should be cheaper and faster than a fresh web search, without being worse. I never actually tested that. So I did — and the result surprised me enough that I retired the agent.

The setup

A matched-pair A/B test, run automatically, no manual steps mid-run:

• Condition A — the sub-agent, invoked headlessly inside the vault: claude -p "<question>" --agent second-brain-lookup --output-format json
• Condition B — a plain session with no vault access at all, restricted to WebSearch/WebFetch only, run from a scratch directory with no CLAUDE.md — a genuine no-reuse baseline.

Both get the identical question. Which one runs first is randomized per topic, to cancel out order effects. I picked topics the vault already covers well, since an uncovered topic gives the lookup agent nothing to reuse — that wouldn’t be a fair test.

For every run, I parse Claude Code’s own session transcripts rather than hand-instrumenting anything — each assistant message in the JSONL log carries a usage object with all four token buckets (input, output, cache-write, cache-read) plus tool-call counts and timestamps. From that I get pricing-normalized cost, wall-clock time, and tool usage, no extra instrumentation needed.

Quality gets scored separately and blind: a third claude -p call scores both answers on a faithfulness/relevancy rubric, with the two responses labeled “Response 1” / “Response 2” in randomized order so the judge never knows which condition produced which — only re-mapped to A/B after scoring. The decision rule: A only counts as a quality win if its score is at least as good as B’s — cheaper-but-worse doesn’t count.

What went wrong first

Testing this surfaced real bugs before it surfaced a real result:

• The transcript parser assumed ~/.claude/projects/..., but my setup runs Claude Code under a custom CLAUDE_CONFIG_DIR — fixed by reading that env var.
• macOS resolves /var to /private/var and folds underscores differently than a naive string replace when escaping project-folder paths — fixed with os.path.realpath() plus a proper regex fold.
• The big one: condition B, even restricted to search tools, frequently didn’t search at all — it just answered from memory when confident, on 3 of 4 topics in the first full run. That silently turned “fresh web search” into “no search, no cost” — not the baseline I was trying to measure. Fixed with a system-prompt instruction forcing at least one real search, verified before re-running everything.
• One transient API failure (exit 1, empty stderr) that succeeded immediately on an identical retry — added basic retry logic rather than treating it as a real bug.

The result

Four topics, run end to end after both fixes:

topic	cost A	cost B	time A	time B	quality A	quality B
context window management	$1.481	$0.189	91.8s	12.9s	5/5	4/4
browser automation for AI agents	$0.867	$0.040	111.1s	6.2s	5/5	5/1
agentic tool & error design	$0.560	$0.290	61.5s	17.0s	5/4	4/5
output validation & hallucination guards	$1.246	$0.065	159.2s	15.8s	4/5	4/5

(quality shown as faithfulness/relevancy, 1–5 each)

The sub-agent was 1.9×–21.6× more expensive and 3.6×–18× slower than a fresh search — on every single topic, consistently across all three runs I did (buggy and fixed alike). Quality came out roughly comparable once the baseline was actually forced to search properly — before that fix, the baseline had swept quality by answering from memory alone, which had inflated its apparent edge.

The verdict

It doesn’t hold. The sub-agent pays a real, consistent cost and time premium for comparable — not better — answer quality. That’s the opposite of what it was built to do. Based on this, I retired it: removed the agent definition from my Claude Code config entirely. The vault still has the pipeline and the write-up, both as the record of why, and as a reusable test for the next lookup-agent idea I have.

Four topics isn’t a large sample — well below the ~30–100 pairs you’d want for a real statistical claim. But the direction was consistent across every topic and every run, which is a different, more useful kind of evidence than a single clean number would have been.

The honest lesson here isn’t really about this one sub-agent. It’s that an agent design that feels like it should save something is worth actually measuring before you keep using it — the intuition and the token bill don’t always agree.

---