First fleet dispatch: a 25% success rate and the lessons it bought

What we sent into the night

Four agent containers, each running an isolated Claude Code session. Each one assigned a real work item from the sprint backlog, each targeting a different repository, each expected to clone, build, commit, open a PR, and report back.

graph TD
    subgraph Dispatch["Manual Dispatch"]
        D[Casey via API]
    end

    subgraph Fleet["Agent Fleet"]
        B1["Agent 1<br/>Memory Will + Hooks"]
        B2["Agent 2<br/>LiteLLM Migration"]
        B3["Agent 3<br/>CI Test Gate"]
        B4["Agent 4<br/>Persona Enrichment Bus"]
    end

    subgraph Results["Morning After"]
        R1["FAILED<br/>Zero artifacts"]
        R2["FAILED<br/>Session cycling"]
        R3["PARTIAL<br/>Branch pushed, then crashed"]
        R4["COMPLETED<br/>PR merged"]
    end

    D --> B1
    D --> B2
    D --> B3
    D --> B4

    B1 --> R1
    B2 --> R2
    B3 --> R3
    B4 --> R4

The expectation: wake up to four PRs in the review queue and a notification channel full of completion messages. The reality was more educational.

One out of four

In baseball, .250 is a respectable batting average. In autonomous agent fleet reliability, it is a data point that tells you exactly where the infrastructure breaks.

The first real fleet dispatch didn’t tell us whether the agents were smart enough. It told us whether the infrastructure was stable enough for them to even start.

Session cycling: the fleet-wide killer

Three of four containers exhibited the same failure pattern. Not three different bugs. The same bug, three times, wearing slightly different log output.

The cycling pattern was identical across Agents 2, 3, and 4. Even Agent 4, which completed its task successfully, fell into the same cycling loop after finishing. The task ran, the PR shipped, and then the container entered the death spiral like every other instance.

This was not a task-specific failure. It was an environment problem.

graph LR
    A["Container Start"] --> B["startup.sh"]
    B --> C["Claude Code Launch"]
    C --> D{"Auth Valid?"}
    D -->|"No / Expired"| E["Clean Exit (0)"]
    D -->|"Yes"| F["Session Start"]
    F --> G{"Task Available?"}
    G -->|"No"| E
    G -->|"Yes"| H["Execute Task"]
    H --> I["Task Complete"]
    I --> E
    E --> J["Orchestrator Restart (30s)"]
    J --> A

The root cause was a combination of two things:

1. OAuth token lifecycle. Claude Code’s OAuth token on the persistent volume had degraded. The token was either expired or revoked. When Claude Code detects an invalid token, it clears the credential file entirely. The next session attempt finds no credentials, can’t authenticate non-interactively, and exits. There is no retry path that doesn’t involve a human running the login flow inside the container.

2. The idle loop was still running. Before the event-driven redesign, the startup script ran a while true loop: launch Claude Code, sleep 30 seconds, repeat. When the task finished or the auth failed, the loop restarted Claude Code immediately. Four containers, each cycling every 30 seconds: roughly 5,760 wasted session starts per day, each one hitting the API, each one exiting, each one generating a log line that looked like normal operation.

Agent 4: the one that worked

Agent 4 was assigned the persona enrichment task. The job: fetch affective state (a five-dimensional emotional context vector) before each persona enrichment call and inject it into the persona’s tone. A real feature with real architectural implications.

The PR was clean. The code was reviewed, tested, and merged. The fail-open pattern was the right architectural call for an optional enrichment service. Agent 4 demonstrated that when the infrastructure cooperates long enough for the agent to actually run, the agent produces production-quality work.

The agent was never the problem. The infrastructure around it was.

Agent 3: good work, bad ending

Agent 3 was assigned a CI test gate. The task: add a test job to the CI workflow that runs the existing test suite on every push and PR.

This is the failure mode that costs the most: work was done, value was created, and it’s invisible. The PR is the artifact that triggers the review pipeline. No PR means the work might as well not exist. Agent 3 was 90% of the way there and then the environment killed it.

Agent 1 and Agent 2: nothing to show

Agent 1: The container for Agent 1 never properly existed. It was scheduled, but the task dispatch targeted a service endpoint that wasn’t wired. The API call returned success, and then nothing happened. Zero artifacts. Zero log entries from a Claude session. The dispatch succeeded at the HTTP level and failed at every level that matters.

Agent 2: Agent 2 entered the session cycling loop immediately. Launch, clean exit, 30-second restart, launch, clean exit. No task execution. No git operations. No commits. The OAuth token was already degraded when the dispatch arrived. The agent couldn’t authenticate, couldn’t start a session, and the loop ensured it would keep trying and failing until someone intervened.

Why it broke: the auth lifecycle gap

The fleet’s failure modes converge on one systemic issue: credential lifecycle management in an autonomous container environment.

graph TD
    A["Token Valid"] --> B{"Token Expires?"}
    B -->|"No"| C["Agent Runs Normally"]
    B -->|"Yes"| D["Runtime Clears Credentials"]
    D --> E["Next Session: No Credentials"]
    E --> F{"Interactive Login Available?"}
    F -->|"Yes (Human)"| G["Manual Login → New Token"]
    F -->|"No (Autonomous Container)"| H["Clean Exit → Restart Loop"]
    H --> E
    G --> A

    style H fill:#dc2626,color:#fff
    style F fill:#f59e0b,color:#fff

This is the single worst operational problem in an autonomous agent fleet. Not prompt quality. Not model capability. Not task decomposition. Credentials. They expire, they get cleared, and the container has no way to fix itself.

Immediate response

The morning after the dispatch, three things happened:

1. Containers scaled to zero. All four instances were scaled down. The cycling loop was burning API calls (each session start hits the API even if it immediately fails), generating noise in logs, and accomplishing nothing. Scaling to zero stopped the bleeding.

2. Token rotation and re-authentication. Credentials were rotated and OAuth re-auth was performed on all four containers manually. This required shelling into each container and running the login flow, which is exactly the manual step the system needs to eliminate.

3. Event-driven redesign shipped. The idle while true loop in the startup script was removed entirely. The task runner became the sole process that launches Claude Code sessions. No task dispatched means no Claude session running means no API calls means no cost. This was the single most impactful change: it eliminated the cycling failure mode by eliminating the loop that caused it.

4. Short-lived git auth completed. A GitHub App was configured with a private key stored in a secrets manager, synced to the containers at startup. Three startup script bugs were fixed in sequence (missing HTTP method flag, pattern matching errors, missing binary in the container image). Git operations now use short-lived installation tokens that refresh automatically. The static deploy key era is over.

What the 25% taught us

Fleet-wide failures mask individual task quality. Agent 4 produced a clean, mergeable PR. Agent 3 pushed a working CI pipeline with 18/19 tests green. The agents were competent. The infrastructure was not. If you evaluate an agent fleet by success rate alone, you miss the signal: the agent logic was sound, the operational surface was hostile.

Auth is the hardest part of autonomous agents. Not prompting. Not tool use. Not code generation. Authentication. Every other problem in the fleet had a programmatic fix. Credential expiry requires a human in the loop, which is the one thing an autonomous fleet is designed to eliminate. The planned migration to API key auth removes the interactive login requirement entirely.

The irony of building an autonomous system that requires human intervention to authenticate is not lost on anyone involved.

Event-driven beats always-on. The idle loop was a design choice from early prototyping that should have been removed before the first real dispatch. A container that cycles on failure is a container that amplifies failure. A container that waits for work and does nothing until asked can’t enter a death spiral because there’s nothing to spiral.

The PR is the artifact that matters. Agent 3 did real work. It pushed a branch, ran CI, got 18/19 tests green. But without a PR, that work is invisible to the review pipeline. The last mile of agent execution (opening the PR, writing the description, linking the work item) is the step that converts work into a reviewable artifact. Anything short of that is a partial.

Test in the real environment. The startup script is a sequential pipeline where each step depends on the output of the previous one. A missing binary, a malformed HTTP flag, a pattern match that doesn’t match the actual response structure: all invisible in local testing, all immediate failures in the real environment. The only valid test is running the startup script inside the actual container on the actual orchestrator.

The path from 25% to reliable

API key auth. Replace Claude Code OAuth with API key authentication. API keys don’t expire on a browser session lifecycle. They can be stored in a secrets manager, synced to containers, and rotated programmatically. No human login required. This is the fix for the credential lifecycle gap.

Scale-to-zero. Containers scale down to zero replicas when no work is queued and scale up when a task is dispatched. Combined with message-triggered dispatch, this creates a fully event-driven pipeline: message triggers scale-up, container processes task, container scales back to zero.

Local model routing. Route simpler tasks to a local LLM running on a GPU node, reserving cloud API calls for tasks that require frontier model capability. This reduces cost and removes the cloud auth dependency for routine work entirely.

The fleet’s success rate will improve not by making the agents smarter, but by making the infrastructure less hostile. The agents proved they can do the work. The infrastructure needs to prove it can stay out of their way.