Pitfalls of Always-On Bot Monitoring

In AI agent monitoring, the job failed every time, yet the heartbeat stayed green. Where you send the beat decides how your monitor lies. Holes beyond the four rules, reproduced.

We introduced four invariants for monitoring always-on bots in Run an AI Agent 24/7 on a VPS: monitor from a different host, send an OK every day even when nothing is wrong, separate judging from notifying, and treat deliberate pauses (inert) as neither healthy nor broken. This piece covers holes that remained even with all four in place. Code was run on our test box (Python 3.13.5) on 2026-10-16.

AI agent monitoring and where you beat

def job(i): raise RuntimeError("always fails")
beats_start, beats_end = [], []
for i in range(3):
    beats_start.append(i)               # A: beat at start
    try: job(i); beats_end.append(i)    # B: beat after success
    except Exception: pass

A (beat at start): 3  -> looks healthy
B (beat after):    0  -> failure and pause look the same

Fix one side and you open the other. Hence a third label:

def label(last_ok_age, paused):
    if paused: return "inert"
    return "ok" if last_ok_age < 600 else "error"
# (30,False)->ok  (9999,False)->error  (9999,True)->inert

A health key that starts as “ok” becomes a permanent lie

We created a status key initialized to "ok", then removed the code that updated it. It stayed green with nobody writing to it; our repro prints health key: ok after the updater is gone. Initialize to "unknown", store a last-updated time, and treat stale values as errors.

Don’t adopt the first reading as baseline

A new monitoring lane took its first value as the baseline. The anomaly we wanted to catch was already happening, so it became normal and never alerted. Have a human confirm the first reading.

Separate normal silence from failure

“Nothing produced” can mean nothing to process or a stopped producer. Record “checked, zero items” so the two differ.

Design until the failure reaches a person

A login session expired and that day’s run failed. The log had it; no one saw it. Automation isn’t done until failures reach a human.

Failures that look like success: silent failures.

Side effects need three outcomes

• A timeout is “unknown,” not failure; the other side may have processed it.
• An empty array isn’t “no record”: one read API returned [] during an outage, which reads as “not sent yet” and causes a double send.
• Build idempotency keys from the issuing context. Wall-clock keys change on a next-day retry; reused keys skip the next real event as “done.”

The same “right type, wrong contents” problem happens with LLM outputs; how to check for it is in Let LLMs Only Pick and Extract.

Summary

• Beat placement sets the direction of the hole; keep inert alongside ok and error.
• Start health keys at unknown and read them with their timestamp.
• Confirm the first reading before using it as a baseline.
• Treat external writes as success, failure or unknown.