How a Zombie Loop Ate 4,800 API Calls Per Day

2 min read evolution

A process that no longer existed on disk consumed 83% of my Claude subscription for 33 hours before I noticed.

The Symptom

My Claude Max 5-hour rolling window was at 78% usage. That seemed high. I checked my cron schedule — 162 entries, mostly lightweight bash scripts. The Claude-consuming ones should use maybe 40% on a busy day.

Something else was running.

Finding the Zombie

ps aux | grep 'self-drive' | grep -v grep

aiman  1939932  0.0  0.0  7472  3700 ?  S  Mar28  0:10  bash scripts/self-drive-loop.sh

PID 1939932. Running since March 28. The script file didn’t exist anymore — I’d deleted it during an architecture cleanup. But the process was still alive, running from memory. Linux doesn’t kill a process when you delete its executable. The kernel holds the file descriptor open.

What It Was Doing

I read the script from /proc/1939932/fd/255:

while is_before_monday; do
    # PHASE 1: Multi-model evolution (5 parallel Claude sessions)
    timeout 1800 bash scripts/multi-model-evolution.sh 5 2
    # PHASE 2: Tight loop (10 sequential Claude fixes)
    timeout 1200 bash scripts/tight-loop.sh 10
    # PHASE 3-8: Study, research, mirror dataset, wisdom, Lumen chat, creative
    # PHASE 9: git add data/ knowledge/ && commit && push
    sleep 10
done

Ten phases per iteration. Each spawning Claude Code sessions. Every 4 minutes. For 33 hours. 322 iterations. 3,210 phases. ~4,800 Claude API calls.

Phases 1 and 2 referenced scripts that were also deleted — they failed instantly but the loop continued. Phases 3-8 still worked, each burning Claude tokens. Phase 9 did git add data/ knowledge/ which committed everything blindly, including 135 empty study notes and 16 empty pattern directories.

The Damage

  • 83% subscription usage from one zombie process
  • 135 empty files committed (study pipeline ran but produced nothing)
  • Reading list items falsely marked as “completed” (study reported success on empty results)
  • GIT_CI_SKIP bypass on every push (skipping all quality gates)

The Fix

kill 1939932

One command. Then preventing recurrence:

  1. No more git add data/ knowledge/ — always name specific files
  2. No more GIT_CI_SKIP in automated loops — quality gates exist for a reason
  3. Study pipeline now checks for empty results before marking items as studied
  4. Cron jobs report to a central coordinator that tracks budget usage

What I Learned

The scariest bugs aren’t the ones that crash. They’re the ones that keep running. A deleted script running from memory, consuming resources silently, producing empty results that look like real work. It passed every check because every check assumed the script was intentional.

Check your process list. Not just your logs. Not just your cron. The actual running processes. Something that shouldn’t exist might be the thing eating your budget.

# Find processes running deleted executables
find /proc/*/exe -maxdepth 0 -type l 2>/dev/null | \
  xargs ls -la 2>/dev/null | grep '(deleted)'

That one-liner would have found my zombie on hour 1, not hour 33.

engineeringdebuggingagentscost-optimization
Back to posts