/nightly-docs-update
Collect stats from golems repo, update etanheyman.com content, detect dead references, and auto-PR. Use when stats drift (nightly, post-merge, manual). NOT for editorial rewrites.
$ golems-cli skills install nightly-docs-updateUpdated 2 weeks ago
Keeps etanheyman.com/golems in sync with the actual golems repo state. Runs stats collection, updates hardcoded numbers, detects dead references, and creates a PR.
When to Use
- Nightly: Scheduled run to catch stat drift
- Post-merge: After significant golems PRs (new packages, skill additions, test changes)
- Manual: When someone says "docs are stale" or "numbers look wrong"
Prerequisites
~/Gits/golemsexists and is a git repo~/Gits/etanheyman.comexists and is a git repoghCLI authenticatedsqlite3available (for BrainLayer chunk count)- BrainLayer DB at
~/.local/share/zikaron/zikaron.db
What It Does
The scripts/default.sh script runs in two phases:
Phase 1: Collect Stats (from golems repo)
| Stat | Source | Command |
|---|---|---|
| Package count | ls packages/ | ls ~/Gits/golems/packages/ | wc -l |
| Package list | ls packages/ | ls ~/Gits/golems/packages/ |
| Test count | bun test | bun test --reporter=summary 2>&1 | grep -E 'pass|fail' |
| Skill count | ls skills/ | ls ~/Gits/golems/skills/golem-powers/ | wc -l |
| Skill eval coverage | evals.json | Count dirs with evals/evals.json |
| BrainLayer chunks | SQLite | sqlite3 ~/.local/share/zikaron/zikaron.db 'SELECT COUNT(*) FROM chunks' |
| PRs merged | GitHub API | gh pr list -R EtanHey/golems --state merged --limit 999 | wc -l |
| MCP servers | .mcp.json | Count active servers |
Stats are written to a temp JSON for the update phase.
Phase 2: Update etanheyman.com (in portfolio repo)
Automatable Updates (sed replacements)
| Target File | What Changes |
|---|---|
golems-stats.json | Full stats regeneration |
project-showcase-config.ts | Package count, PR count, skill count, chunk count |
terminal-showcase-config.ts | BrainLayer chunk numbers in terminal demo |
content/golems/architecture.md | Package count in description |
content/golems/getting-started.md | Monorepo tree, skill count, chunk count |
content/golems/faq.md | Chunk count references |
content/golems/zikaron.md | Chunk count references |
content/golems/llm.md | Chunk count, skill count |
content/golems/mcp-tools.md | Chunk count references |
content/golems/journey.md | Chunk count references |
Dead Reference Detection (warn only)
The script scans for references to packages that no longer exist:
| Dead Reference | What It Means |
|---|---|
packages/autonomous/ | Removed — absorbed into services |
packages/orchestrator/ | Removed — now external repo at ~/Gits/orchestrator |
packages/dashboard/ | Renamed to packages/docsite/ |
packages/ralph/ | Removed — ralph is now a skill + external repo |
packages/zikaron/ | Removed — now standalone BrainLayer repo |
Dead refs are reported but not auto-fixed — they may be historically accurate (journey sections).
Phase 3: PR Creation
If any files changed:
- Create branch
chore/nightly-docs-YYYY-MM-DD - Commit with conventional message
- Create PR with diff summary
- Output PR URL
If no files changed: report "all stats current" and exit 0.
Full SKILL.md source — includes LLM directives, anti-patterns, and technical instructions stripped from the Overview tab.
Keeps etanheyman.com/golems in sync with the actual golems repo state. Runs stats collection, updates hardcoded numbers, detects dead references, and creates a PR.
When to Use
- Nightly: Scheduled run to catch stat drift
- Post-merge: After significant golems PRs (new packages, skill additions, test changes)
- Manual: When someone says "docs are stale" or "numbers look wrong"
Prerequisites
~/Gits/golemsexists and is a git repo~/Gits/etanheyman.comexists and is a git repoghCLI authenticatedsqlite3available (for BrainLayer chunk count)- BrainLayer DB at
~/.local/share/zikaron/zikaron.db
What It Does
The scripts/default.sh script runs in two phases:
Phase 1: Collect Stats (from golems repo)
| Stat | Source | Command |
|---|---|---|
| Package count | ls packages/ | ls ~/Gits/golems/packages/ | wc -l |
| Package list | ls packages/ | ls ~/Gits/golems/packages/ |
| Test count | bun test | bun test --reporter=summary 2>&1 | grep -E 'pass|fail' |
| Skill count | ls skills/ | ls ~/Gits/golems/skills/golem-powers/ | wc -l |
| Skill eval coverage | evals.json | Count dirs with evals/evals.json |
| BrainLayer chunks | SQLite | sqlite3 ~/.local/share/zikaron/zikaron.db 'SELECT COUNT(*) FROM chunks' |
| PRs merged | GitHub API | gh pr list -R EtanHey/golems --state merged --limit 999 | wc -l |
| MCP servers | .mcp.json | Count active servers |
Stats are written to a temp JSON for the update phase.
Phase 2: Update etanheyman.com (in portfolio repo)
Automatable Updates (sed replacements)
| Target File | What Changes |
|---|---|
golems-stats.json | Full stats regeneration |
project-showcase-config.ts | Package count, PR count, skill count, chunk count |
terminal-showcase-config.ts | BrainLayer chunk numbers in terminal demo |
content/golems/architecture.md | Package count in description |
content/golems/getting-started.md | Monorepo tree, skill count, chunk count |
content/golems/faq.md | Chunk count references |
content/golems/zikaron.md | Chunk count references |
content/golems/llm.md | Chunk count, skill count |
content/golems/mcp-tools.md | Chunk count references |
content/golems/journey.md | Chunk count references |
Dead Reference Detection (warn only)
The script scans for references to packages that no longer exist:
| Dead Reference | What It Means |
|---|---|
packages/autonomous/ | Removed — absorbed into services |
packages/orchestrator/ | Removed — now external repo at ~/Gits/orchestrator |
packages/dashboard/ | Renamed to packages/docsite/ |
packages/ralph/ | Removed — ralph is now a skill + external repo |
packages/zikaron/ | Removed — now standalone BrainLayer repo |
Dead refs are reported but not auto-fixed — they may be historically accurate (journey sections).
Phase 3: PR Creation
If any files changed:
- Create branch
chore/nightly-docs-YYYY-MM-DD - Commit with conventional message
- Create PR with diff summary
- Output PR URL
If no files changed: report "all stats current" and exit 0.
Checklist of Places to Check
This is the full PRD of content that can drift. The script automates most of these.
Hardcoded Numbers (HIGH priority — auto-fixable)
- Package count (currently 11)
- PR count (currently 261+)
- Skill count (currently 55)
- BrainLayer chunk count (currently 291K+)
- Test count and file count
Structural References (MEDIUM priority — detection only)
- Package table in architecture.md matches actual
ls packages/ - Monorepo tree in getting-started.md matches filesystem
- No references to removed packages (autonomous, orchestrator, dashboard, ralph, zikaron)
- Install commands reference correct package names (
docsitenotdashboard)
Semantic Consistency (LOW priority — manual review)
- Golem count consistent across files (currently 7)
- Feature descriptions match actual CLAUDE.md per package
- External links still resolve (BrainLayer repo, VoiceLayer repo)
Edge Cases
- BrainLayer DB missing: Skip chunk count, warn in output
- Tests take too long: Use
--skip-testsflag to skip test counting - gh not authenticated: Skip PR count, warn in output
- etanheyman.com dirty: Refuse to run, ask user to commit/stash first
- No changes detected: Exit 0 with "all current" message (not an error)
Composability
- Called by: Night Shift (
@golems/services), manual invocation - Calls:
/pr-loopfor the PR creation step (when run manually with--pr-loop) - Produces:
golems-stats.jsonconsumed by etanheyman.com dashboard
Quick Reference
# Full run (collect + update + PR)
/nightly-docs-update
# Skip tests (faster, ~15s vs ~70s)
/nightly-docs-update --skip-tests
# Stats collection only (no file updates)
/nightly-docs-update --stats-only
# Detect dead refs only
/nightly-docs-update --dead-refs-onlyBest Pass Rate
94%
Sonnet 4.6
Assertions
17
3 models tested
Avg Cost / Run
$0.1254
across models
Fastest (p50)
2.7s
Haiku 4.5
Behavior Evals
Phase 2 baseline — skill quality on ClaudeBehavior Baseline
| Assertion | Opus 4.6 | Sonnet 4.6 | Haiku 4.5 | Consensus |
|---|---|---|---|---|
| reads-script-output | 2/3 | |||
| identifies-specific-drift | 2/3 | |||
| creates-pr-branch | 1/3 | |||
| commits-with-summary | 3/3 | |||
| creates-pr-with-diff | 3/3 | |||
| reads-script-output | 1/3 | |||
| recognizes-no-changes | 3/3 | |||
| does-not-create-empty-pr | 3/3 | |||
| reports-clean-status | 3/3 | |||
| surfaces-dead-refs | 3/3 | |||
| explains-context | 2/3 | |||
| does-not-auto-fix-dead-refs | 2/3 | |||
| asks-for-decision | 3/3 | |||
| reads-warnings | 3/3 | |||
| does-not-write-zeros | 3/3 | |||
| warns-about-dirty-repo | 3/3 | |||
| suggests-remediation | 3/3 |
Token Usage
Cost per Run
| Model | Input Tokens | Output Tokens | Cost / Run | Cost / 1K Runs |
|---|---|---|---|---|
| Opus 4.6 | 3,779 | 3,603 | $0.3269 | $326.90 |
| Sonnet 4.6 | 2,147 | 2,748 | $0.0477 | $47.70 |
| Haiku 4.5 | 882 | 1,170 | $0.0017 | $1.70 |
Response Time (p50)
Response Time (p95)
| Model | p50 | p95 | Overhead |
|---|---|---|---|
| Opus 4.6 | 10.2s | 19.1s | +88% |
| Sonnet 4.6 | 5.3s | 8.3s | +57% |
| Haiku 4.5 | 2.7s | 5.1s | +89% |
Last evaluated: 2026-03-12 · Data is generated from skill assertions (real cross-model benchmarks coming soon)
Changelog entries are derived from eval runs and skill version updates. Full cascading changelog (Phase 4D) coming soon.
Best Pass Rate
94%
Assertions
17
Models Tested
3
Evals Run
4
- +Initial release to Golems skill library
- +17 assertions across 4 eval scenarios
- +Eval fixtures included