My Automated Pipeline Has No Alarm If It Silently Stops

Published 2026-10-21

I already wrote about this exact gap once. My content pipeline runs on cron jobs scheduled inside a live Claude Code session, and if that session closes, the jobs die with it. No error, no notification, nothing. I named that tradeoff in a past post and then did nothing about it. The gap sat there for weeks while the pipeline kept running fine, which is exactly the kind of thing that stops looking urgent right up until the day it isn't.

The Tradeoff I Already Named, And Never Acted On

Session-native scheduling replaced Windows Task Scheduler for a real reason: the API key kept running out, and Task Scheduler had no way to tell the difference between a job that ran and a job that exited early on a billing failure. Moving to CronCreate fixed that specific problem.

It introduced a different one in trade. A CronCreate job lives only inside that session. Close the window, lose power, run out of context, and every scheduled job disappears with no error anywhere.

Windows Task Scheduler at least kept running when nobody was watching, even if the job inside it silently failed. CronCreate flips that: the scheduling mechanism itself goes away the moment the session does, which is a sharper failure mode than the one it replaced.

I wrote that limitation down explicitly when I made the switch. I just never built anything that actually watches for it, because the pipeline kept working fine for weeks and there was always something else to build first.

"It Ran Yesterday" Isn't Proof It's Running Today

The thing that let this sit unfixed for so long is that the pipeline's own output looks identical whether it's healthy or quietly dead. Each successful cycle appends a row to a content log, blog posts go out, everything downstream looks normal.

The failure only shows up as an absence, a gap in the log that nobody's looking at until someone asks why nothing's posted in a week. Absence is a bad signal to rely on noticing by accident.

What A Real Check Actually Needs To Verify

A useful check doesn't need to know anything about why a session closed. It only needs one fact: how long has it been since the last successful cycle, and does that gap exceed what the schedule allows.

My posting cadence is every 2 days, so a gap past 2 days means something's already late, and a gap past 3 or 4 days means the pipeline very likely isn't running at all anymore.

That's the whole check. It doesn't need to inspect the session, the cron jobs, or anything about Claude Code itself, just the log the pipeline already writes on every successful cycle.

The Actual Check

The first piece reads the content log and finds the last row tagged as a real posting cycle, not a note.

import re
from datetime import date

LOG_PATH = "socials/content-log.md"
MAX_GAP_DAYS = 3

def last_cycle_date(log_path):
    with open(log_path, encoding="utf-8") as f:
        rows = [line for line in f if line.startswith("| 20") and "| cycle |" in line]
    last_date_str = rows[-1].split("|")[1].strip()
    return date.fromisoformat(last_date_str)

The check itself is a straight date subtraction against the cadence.

def check_pipeline_alive(log_path):
    last = last_cycle_date(log_path)
    gap = (date.today() - last).days
    if gap > MAX_GAP_DAYS:
        raise RuntimeError(f"No cycle logged in {gap} days, last one was {last}")
    return gap

Three days of slack on top of the 2-day cadence instead of alerting the moment a cycle is one day late. A single slow night shouldn't trip the same alarm as the pipeline actually being dead.

What It Does When It Trips

When the gap exceeds the threshold, it reuses the same email alert script the lead-review side of this project already calls, rather than building a second notification path for one script.

import subprocess

if __name__ == "__main__":
    try:
        gap = check_pipeline_alive(LOG_PATH)
        print(f"Pipeline alive, last cycle was {gap} day(s) ago.")
    except RuntimeError as e:
        subprocess.run([
            "python", "send_lead_alert.py",
            "--subject", "[ALERT] Pipeline may have stopped",
            "--body", str(e),
        ])

This runs as its own scheduled check, separate from the content pipeline itself, since a watchdog that depends on the same session it's watching defeats the entire point.

That separation matters more than it sounds. If the watchdog ran as another job inside the same session, it would die exactly when it's needed most, the moment the session itself goes down.

Putting it somewhere else, a different scheduled task entirely, is what makes it an actual check instead of a second thing that silently stops.

The alert itself stays plain on purpose. It doesn't need a dashboard or a dedicated notification service, just an email with the gap and the last known-good date, enough to know something needs attention without building infrastructure around a problem that happens rarely.

What This Doesn't Fix

The check only detects the silence. It doesn't restart a closed session, and it can't, a CronCreate job has no way to relaunch the Claude Code window that held it. The fix after an alert fires is still manually opening the session again.

That's a real, sizeable limitation. But it's a separate problem from having no alarm at all. Knowing the pipeline stopped two days ago instead of finding out a week later when nothing's posted is most of the value here. The rest is still on me to notice and act on.

The honest version of this post is that the check didn't need to be clever. It needed to exist. I'd written the exact failure mode down weeks earlier and treated naming it as close enough to solving it, which it wasn't.