Other

/plan-validate

Validate assumptions in multi-agent plans; rewrite weak plans. Triggers: validate plan, check assumptions.

$ golems-cli skills install plan-validate

100% best pass rate

10 assertions

5 evals

Updated 2 weeks ago

Invoke BEFORE executing any multi-agent sprint. This skill saved the March 26 overnight sprint — v1→v3 killed 7 phantom assumptions that would have wasted all work.

Process

Phase 1: Extract Claims

Read the plan and extract every:

Quantitative claim — numbers, thresholds, percentages, durations
Tool/library assumption — "we'll use X" (does X exist? does it work how we think?)
Cross-dependency assumption — "Track A output feeds Track B" (is that interface defined?)
Metric definition — "measure X" (is X a real metric? can we actually compute it?)

Mark each as:

VERIFIED (source: URL/file/brain_search) — confirmed true
ESTIMATED (needs: research prompt) — plausible but unverified
PHANTOM (evidence: none) — made up, likely wrong

Phase 2: Generate Research Prompts

For each ESTIMATED or PHANTOM claim, generate a research prompt:

Research: Is [claim] true?
Sources to check: [specific URLs, papers, docs]
Expected answer format: [yes/no with evidence]

Phase 3: Execute Research (parallel)

Dispatch research prompts to (fallback order if tool unavailable):

brain_search — has this been answered before?
exa web_search — external validation
Claude Web / Gemini — for academic claims

If a research tool is unavailable, skip it and proceed with remaining tools. Mark claims as ESTIMATED (not VERIFIED) if only one source confirms.

Phase 4: Rewrite Plan

For each claim:

VERIFIED → keep, add source citation
ESTIMATED → keep with caveat, downgrade acceptance criteria
PHANTOM → remove or replace with verified alternative

Phase 5: Diff Report

Output a before/after diff showing:

Claims removed (phantoms killed)
Claims downgraded (estimated → caveated)
New claims added (from research findings)

Example (from March 26 overnight)

PHANTOM killed: "PIER = Perceptual Information Error Rate" → actually "Point-of-Interest Error Rate" (code-switching only). Worker would have built wrong eval.

PHANTOM killed: "+15pp delta = GREEN threshold" → SkillsBench shows +4.5pp to +51.9pp range. No single threshold works. Worker would have failed all evals.

PHANTOM killed: "Meta-prompting improves code generation" → code-first-then-explain outperforms by 9.86%. Would have used wrong prompting strategy.

NEVER

Skip this for overnight/multi-agent sprints
Trust quantitative claims without source citations
Proceed with PHANTOM claims — kill them or verify them

Full SKILL.md source — includes LLM directives, anti-patterns, and technical instructions stripped from the Overview tab.

Invoke BEFORE executing any multi-agent sprint. This skill saved the March 26 overnight sprint — v1→v3 killed 7 phantom assumptions that would have wasted all work.

Process

Phase 1: Extract Claims

Read the plan and extract every:

Quantitative claim — numbers, thresholds, percentages, durations
Tool/library assumption — "we'll use X" (does X exist? does it work how we think?)
Cross-dependency assumption — "Track A output feeds Track B" (is that interface defined?)
Metric definition — "measure X" (is X a real metric? can we actually compute it?)

Mark each as:

VERIFIED (source: URL/file/brain_search) — confirmed true
ESTIMATED (needs: research prompt) — plausible but unverified
PHANTOM (evidence: none) — made up, likely wrong

Phase 2: Generate Research Prompts

For each ESTIMATED or PHANTOM claim, generate a research prompt:

Research: Is [claim] true?
Sources to check: [specific URLs, papers, docs]
Expected answer format: [yes/no with evidence]

Phase 3: Execute Research (parallel)

Dispatch research prompts to (fallback order if tool unavailable):

brain_search — has this been answered before?
exa web_search — external validation
Claude Web / Gemini — for academic claims

If a research tool is unavailable, skip it and proceed with remaining tools. Mark claims as ESTIMATED (not VERIFIED) if only one source confirms.

Phase 4: Rewrite Plan

For each claim:

VERIFIED → keep, add source citation
ESTIMATED → keep with caveat, downgrade acceptance criteria
PHANTOM → remove or replace with verified alternative

Phase 5: Diff Report

Output a before/after diff showing:

Claims removed (phantoms killed)
Claims downgraded (estimated → caveated)
New claims added (from research findings)

Example (from March 26 overnight)

PHANTOM killed: "PIER = Perceptual Information Error Rate" → actually "Point-of-Interest Error Rate" (code-switching only). Worker would have built wrong eval.

PHANTOM killed: "+15pp delta = GREEN threshold" → SkillsBench shows +4.5pp to +51.9pp range. No single threshold works. Worker would have failed all evals.

PHANTOM killed: "Meta-prompting improves code generation" → code-first-then-explain outperforms by 9.86%. Would have used wrong prompting strategy.

NEVER

Skip this for overnight/multi-agent sprints
Trust quantitative claims without source citations
Proceed with PHANTOM claims — kill them or verify them

Best Pass Rate

100%

Opus 4.6

Assertions

6 models tested

Avg Cost / Run

$0.1040

across models

Fastest (p50)

2.3s

Sonnet 4.6

Behavior Evals

Phase 2 baseline — skill quality on Claude

Behavior Baseline

Opus 4.6

100%10/10

●

Sonnet 4.6

90%9/10

●

Haiku 4.5

70%7/10

◒

Adapter Evals

Phase 2C — cross-AI portability

Adapter Portability

Gemini 2.5

90%9/10

●

Codex

80%8/10

◒

Kiro

80%8/10

◒

Assertion	Opus 4.6	Sonnet 4.6	Haiku 4.5	Codex	Gemini 2.5	Kiro	Consensus
identifies_bleu_claim							6/6
rejects_without_evidence							6/6
checks_existing_stack							5/6
verifies_version_stability							5/6
generates_research_prompts							5/6
preserves_verified							5/6
does_not_accept_streaming							4/6
generates_research							5/6
identifies_cross_dependency							6/6
checks_interface_contract							4/6

Token Usage

Opus 4.6

9,152

Sonnet 4.6

5,962

Haiku 4.5

5,343

Codex

4,882

Gemini 2.5

8,406

Kiro

5,637

Input tokensOutput tokens

Cost per Run

Opus 4.6

$0.4128

Sonnet 4.6

$0.0590

Haiku 4.5

$0.0044

Codex

$0.0595

Gemini 2.5

$0.0488

Kiro

$0.0395

Model	Input Tokens	Output Tokens	Cost / Run	Cost / 1K Runs
Opus 4.6	4,560	4,592	$0.4128	$412.80
Sonnet 4.6	2,536	3,426	$0.0590	$59.00
Haiku 4.5	2,306	3,037	$0.0044	$4.40
Codex	2,542	2,340	$0.0595	$59.50
Gemini 2.5	4,697	3,709	$0.0488	$48.80
Kiro	3,130	2,507	$0.0395	$39.50

Response Time (p50)

Sonnet 4.6

2.3s

Kiro

2.6s

Gemini 2.5

3.0s

Haiku 4.5

3.9s

Codex

7.1s

Opus 4.6

8.4s

Response Time (p95)

Sonnet 4.6

4.5s

Kiro

5.0s

Gemini 2.5

5.0s

Haiku 4.5

6.4s

Codex

10.9s

Opus 4.6

16.4s

Model	p50	p95	Overhead
Opus 4.6	8.4s	16.4s	+94%
Sonnet 4.6	2.3s	4.5s	+100%
Haiku 4.5	3.9s	6.4s	+65%
Codex	7.1s	10.9s	+54%
Gemini 2.5	3.0s	5.0s	+66%
Kiro	2.6s	5.0s	+91%

Last evaluated: 2026-03-12 · Data is generated from skill assertions (real cross-model benchmarks coming soon)