An AI agent in production, network_error bursts in chains, and an invisible mechanism: the provider’s prefix cache was dying in 2 minutes.
This is a real production post-mortem, with A/B-measured calibration and the result in settings.yaml. The kind of diagnosis that only exists when the harness is yours.
The symptom: the retry chain
The initial picture: bursts of network_error right after image-reading calls returning large files. The tool took long, the next call failed, the retry failed again, in a chain.
The instinct is to blame the unstable gateway. The instinct misses the target: the next failure was a consequence of the previous interval, not of the instability itself.
The mechanism: a blown 2-minute TTL
The investigation isolated the mechanism: the provider’s prefix cache expires after about 2 minutes without a request. The long tool (large image reading) blew the TTL, the cache died, and each retry resent the whole context: 40,000 text tokens plus 1 MB of accumulated images.
A cold upload of that size over an unstable gateway is the whole burst explained. Naming the mechanism, prefix invalidation, is what turns the rest into common engineering: with the cause named, the fix becomes failure classification plus measured calibration.
The fix, on two fronts
In the code: gateways synthesize finish_reason, and those errors entered as fatal failures. I remapped them as recoverable transport, with a test suite, and the work became a contribution branch to the harness mainstream.
In configuration: A/B-measured calibration, each value with its rationale recorded in settings.yaml:
- A 6 MiB cap per image request: bursts 76% smaller, peak 82% smaller.
- A 60-second stream timeout, because 30 inflated repeats.
- 100 retries with 750ms backoff, capping the worst case under 2 hours.
And the operational rule closing the design: retry ceiling exceeded, abort and compact the context. Result: 62 retries per day absorbed automatically, zero lost shifts, with the model running the whole window.
Why this requires an open harness
The same harness serves the cloud and my own data center (a DGX H200 serving GLM via NIM), and the debug was possible because the harness is mine: getting down to the prefix-invalidation mechanism and fixing it with tests does not fit in a support ticket.
A closed harness would turn this diagnosis into a deadline-less ticket, and probably a solution-less one: the cache provider is not the agent’s owner, and the agent’s owner cannot see the TTL. An open harness turns diagnosis into a tested fix, and that is why the orchestration layer is worth its price.
Conclusion
The same post-mortem teaches three lessons: a prefix cache is a living resource that dies in the silence between calls; transport failures and fatal failures need different classifications; and calibration without measured A/B is guessing with an archive.
Anyone running agents in production will find this pattern (or its cousin) sooner or later. The difference between 2 hours of worst case and a lost shift is the harness you control. The model is a commodity; the operation serving it is the product.