When it breaks at 3am: retries, alerts and idempotency

Workflows that run unattended fail in three ways. Here is a small pattern for each so a bad night costs you one notification, not a weekend.

A workflow you run by hand fails in front of you. A scheduled one fails while you sleep. The difference is not whether it fails, it is whether you find out, and whether it did damage on the way down.

The three failures

  1. Transient. The API was slow, the token expired, the network blinked. Retrying fixes it.
  2. Bad input. An email with no body, a feed item with no date, a row with text where a number should be. Retrying does not fix it; skipping does.
  3. Logic. The model returned the wrong shape, or a step produced something downstream cannot use. Only a human fixes it.
Members read the full version

The step-by-step section, the resources and the troubleshooting notes are part of membership.

Done with this one? Mark it to track your path.Sign in
Save it for laterSign in
Questions about this piece?Ask AI about this

Comments (0)

No comments yet. Start the conversation.

Sign in to join the discussion.Sign in