GolemsGolems
Skills LibraryView on GitHub
Other

/ecosystem-health

Health checks for MCP, BrainLayer, skills, friction. Triggers: health check, whats broken, MCP/BL status.

$ golems-cli skills install ecosystem-health
Experimental
11 evals

Updated 2 weeks ago

Audit the golems ecosystem: MCP servers, BrainLayer, VoiceLayer daemon, JSONL watcher, enrichment, git status, Axiom telemetry, open PRs. Run at session start AND end.

Why This Exists

The ecosystem has 22 repos, 45+ skills, 5 MCP servers (BrainLayer, VoiceLayer, cmux, exa, supabase), a 7GB semantic memory store, a JSONL watcher, enrichment pipeline, and multiple long-lived Claude sessions. Things break silently — MCP disconnects, daemons crash, watchers die, enrichment stalls, PRs rot. This skill catches those problems before the user notices.

Quick Check (session start/end)

Run ALL checks. Report results in table format. Stop and flag if any check is RED.

1. MCP Connectivity (all 5 servers)

Check each MCP server is connected and responsive:

# BrainLayer — must respond with data
brain_recall(mode="stats")
# GREEN if: returns chunk count + entity count
# RED if: timeout, error, "unavailable"

# VoiceLayer — silent ping
voice_speak(message="Health check", mode="think")
# GREEN if: no error (think mode = silent log only)
# RED if: timeout or MCP unavailable

# cmux — check surface listing
mcp__cmux__list_surfaces()
# GREEN if: returns list (even empty)
# RED if: timeout or MCP unavailable

# exa — verify connection (no query needed, just check tool exists)
# GREEN if: exa tools appear in available tools
# RED if: not listed

# supabase — verify connection
# GREEN if: supabase tools appear in available tools
# RED if: not listed

If any MCP is RED: Report it immediately. MCP failures cascade — BrainLayer down means no memory, VoiceLayer down means no voice, cmux down means no agent coordination.

2. BrainLayer Responsive

Go deeper than connectivity — verify BrainLayer actually returns meaningful data:

brain_search("ecosystem health check")
# GREEN if: returns results (any)
# YELLOW if: returns empty (index may need rebuild)
# RED if: timeout or error

Also check DB vitals:

python3 -c "
import sqlite3, os
 
candidates = [
    '~/.local/share/brainlayer/brainlayer.db',
    '~/.local/share/zikaron/zikaron.db',
]
db = None
for c in candidates:
    p = os.path.expanduser(c)
    if os.path.exists(p) or os.path.exists(p + '-shm'):
        db = p
        break
if not db:
    print('RED: No BrainLayer DB found')
    exit(1)
 
db_exists = os.path.exists(db)
wal_path = db + '-wal'
wal_size = os.path.getsize(wal_path) if os.path.exists(wal_path) else 0
shm_exists = os.path.exists(db + '-shm')
 
if not db_exists and shm_exists:
    print(f'WAL-only mode (MCP holds DB in memory)')
    print(f'WAL: {wal_size / 1e6:.0f} MB')
else:
    db_size = os.path.getsize(db)
    conn = sqlite3.connect(f'file:{db}?mode=ro', uri=True)
    chunks = conn.execute('SELECT COUNT(*) FROM chunks').fetchone()[0]
    conn.close()
    print(f'Chunks: {chunks}')
    print(f'DB: {db_size / 1e9:.1f} GB | WAL: {wal_size / 1e6:.0f} MB')
    status = 'RED' if wal_size > 500e6 else 'YELLOW' if wal_size > 100e6 else 'GREEN'
    print(f'WAL status: {status}')
"

Thresholds:

  • WAL > 100MB = YELLOW (checkpoint needed)
  • WAL > 500MB = RED (queries will timeout)
  • Chunk count dropped vs last check = RED (data loss)

3. VoiceLayer Daemon Alive

Check the Voice Bar app + MCP daemon are running:

# Voice Bar app — persistent macOS server on /tmp/voicelayer.sock
pgrep -x "Voice Bar" || pgrep -x "VoiceBar" || echo "RED: Voice Bar not running"
 
# VoiceLayer MCP daemon — singleton on /tmp/voicelayer-mcp.sock
pgrep -fl "mcp-server-daemon" || echo "YELLOW: VoiceLayer daemon not running"
 
# Socket file exists and is a socket
test -S /tmp/voicelayer.sock && echo "Voice Bar socket: OK" || echo "RED: No Voice Bar socket"
test -S /tmp/voicelayer-mcp.sock && echo "MCP daemon socket: OK" || echo "YELLOW: No MCP daemon socket"
 
# Check daemon log for recent errors
tail -5 /tmp/voicelayer-mcp-daemon.stderr.log 2>/dev/null | grep -i "error\|fatal\|crash" && echo "YELLOW: Recent daemon errors" || echo "Daemon log: clean"

Expected state:

  • Voice Bar running (1 process)
  • MCP daemon running (1 process via LaunchAgent)
  • Both sockets exist at /tmp/voicelayer.sock and /tmp/voicelayer-mcp.sock

4. JSONL Watcher Running

The BrainLayer JSONL watcher tails ~/.claude/projects/*.jsonl and (arbitrated mode) enqueues chunks to ~/.brainlayer/queue/ for the drain/BrainBar consumers.

Check launchd LOADED state first, not just the process. The 2026-06-06 incident: the watcher was silently dead for 20 hours because the service was unloaded during a load-shed (launchctl bootout) and never restored — KeepAlive cannot revive an unloaded service, and a pgrep-only check tells you it's dead but not why or how to fix it.

# 1. Service LOADED? (unloaded = load-shed casualty; KeepAlive can't help)
launchctl list | grep -q "com.brainlayer.watch" \
  || echo "RED: com.brainlayer.watch UNLOADED — fix: launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.brainlayer.watch.plist"
 
# 2. Process alive? (message is neutral — if step 1 said UNLOADED, that's the cause; fix there first)
pgrep -fl "brainlayer watch" || echo "RED: no watcher process"
 
# 3. Offsets freshness — the watcher writes ~/.local/share/brainlayer/offsets.json on progress.
#    Stale offsets while Claude sessions are active = watcher wedged or dead.
offsets_age_min=$(( ( $(date +%s) - $(stat -f %m ~/.local/share/brainlayer/offsets.json 2>/dev/null || echo 0) ) / 60 ))
echo "offsets.json age: ${offsets_age_min}min"
# GREEN <15min (with active sessions) | YELLOW 15-60min | RED >60min
 
# 4. Queue backlog (arbitrated mode) — events piling up = consumers (drain/BrainBar) dark
queue_count=$(find ~/.brainlayer/queue -type f 2>/dev/null | wc -l | tr -d ' ')
echo "queue events: $queue_count"
# GREEN <100 | YELLOW 100-1000 | RED >1000 (also check the sibling services below)
 
# 5. Sibling services that die in the same load-sheds (same bootstrap fix as #1):
for svc in com.brainlayer.drain com.brainlayer.enrichment; do
  launchctl list | grep -q "$svc" || echo "YELLOW: $svc UNLOADED"
done

Why this matters: date-windowed write counters (e.g. BrainBar's "Recent writes" gauge) only see stamped chunks — a dead watcher reads as "3 writes/hour", not as an alarm. Silence looks like quiet, not failure. After ANY load-shed or reboot, re-run this section and re-bootstrap every unloaded com.brainlayer.* service.

5. Enrichment Process Alive + Stats

The enrichment pipeline processes raw chunks into entities, relations, and embeddings:

# Check if enrichment is running
pgrep -fl "brainlayer.*enrich" || pgrep -fl "enrichment" || echo "INFO: No enrichment running (may be idle)"
 
# Check enrichment stats via BrainLayer
# brain_recall with stats mode includes enrichment info

Also check via MCP:

brain_recall(mode="stats")
# Look for: enrichment_pending count, last_enrichment timestamp
# GREEN if: pending < 100 and last enrichment < 1 hour ago
# YELLOW if: pending 100-1000 or last enrichment > 6 hours
# RED if: pending > 1000 or last enrichment > 24 hours

6. Git Status Across Repos

Uncommitted changes = risk of lost work. Check all key repos:

for repo in golems brainlayer voicelayer orchestrator cmuxlayer; do
  if [ -d ~/Gits/$repo ]; then
    status=$(git -C ~/Gits/$repo status --porcelain 2>/dev/null | head -5)
    branch=$(git -C ~/Gits/$repo branch --show-current 2>/dev/null)
    if [ -z "$status" ]; then
      echo "✓ $repo ($branch): clean"
    else
      count=$(git -C ~/Gits/$repo status --porcelain 2>/dev/null | wc -l | tr -d ' ')
      echo "⚠ $repo ($branch): $count uncommitted files"
    fi
  fi
done

Thresholds:

  • All clean = GREEN
  • 1-2 repos dirty = YELLOW (normal during development)
  • Repo on non-main branch with dirty state = YELLOW (flag it)
  • Repos with uncommitted changes on main = RED (risk of accidental loss)

7. Axiom Telemetry Flowing

Check if the watcher is sending heartbeats to Axiom:

# Check watcher log for recent Axiom sends
grep -c "axiom\|telemetry\|heartbeat" /tmp/brainlayer-watcher.log 2>/dev/null || echo "0"
 
# Check last heartbeat timestamp
grep -i "heartbeat\|axiom" /tmp/brainlayer-watcher.log 2>/dev/null | tail -1
 
# If no watcher log, check if Axiom env vars are configured
[ -n "$AXIOM_TOKEN" ] && echo "Axiom token: configured" || echo "YELLOW: AXIOM_TOKEN not set"
[ -n "$AXIOM_DATASET" ] && echo "Axiom dataset: $AXIOM_DATASET" || echo "YELLOW: AXIOM_DATASET not set"

Expected state:

  • Heartbeat within last 15 minutes = GREEN
  • Heartbeat > 15 min but < 1 hour = YELLOW
  • No heartbeat or no watcher log = RED (telemetry blind)

8. Open PRs Across Repos

Stale PRs rot. Check what's open:

for repo in EtanHey/golems EtanHey/brainlayer EtanHey/voicelayer EtanHey/orchestrator EtanHey/cmuxlayer; do
  prs=$(gh pr list --repo $repo --state open --json number,title,createdAt --jq 'length' 2>/dev/null)
  if [ "$prs" = "0" ] || [ -z "$prs" ]; then
    echo "✓ $repo: no open PRs"
  else
    echo "⚠ $repo: $prs open PR(s)"
    gh pr list --repo $repo --state open --json number,title,createdAt --jq '.[] | "  #\(.number) \(.title) (\(.createdAt | split("T")[0]))"' 2>/dev/null
  fi
done

Thresholds:

  • 0 open PRs = GREEN
  • 1-3 open PRs = YELLOW (review and merge or close)
  • PR open > 7 days = RED (stale — close or merge)

Report Format

After running all checks, produce a structured report:

# Ecosystem Health Report — YYYY-MM-DD HH:MM