Alerting on Failed Tile Jobs

Evaluate a few rules against the run manifest after every run and send one message if any fail:

def check(manifest, history):
    alerts = []
    failed = len(manifest["tiles"]["failed"])
    if failed > max(5, 0.005 * manifest["tiles"]["attempted"]):
        alerts.append(f"{failed} tiles failed")
    if manifest["wall_s"] > 2 * median(h["wall_s"] for h in history):
        alerts.append("runtime more than 2x recent median")
    return alerts

An alert is a request for someone’s attention, so every rule should describe a situation where a person needs to act. This page belongs to monitoring and observability for raster pipelines in Cloud Execution & Orchestration.

Four Rules That Cover Most Failures

A small rule set A failed-tile rule fires when failures exceed a small absolute or relative threshold. A missing-run rule fires when no manifest appears by the expected time. A data-quality rule fires when tiles are written with nodata fraction near one or values outside the valid range. A regression rule fires when runtime, bytes read or cost exceed twice the recent median. Rules that deserve a notification failed tiles > 5 or > 0.5% missing run no manifest by 07:00 data quality empty or out-of-range regression > 2× recent median Red: something broke. Amber: something changed. Both need a person, with different urgency.

The missing-run rule is the one most often forgotten and the one that catches the worst failures. When a scheduler stops, credentials expire before start-up, or a deployment breaks the entry point, nothing runs and therefore nothing logs an error. Only an external check that expects a manifest by a given time notices the silence — a “dead man’s switch”.

Environment & Setup

Package Version pin Used for
httpx >=0.26 Posting to a Slack webhook
boto3 >=1.34 Reading manifests from S3
prefect >=2.14 Optional: automations on flow state 
pip install "httpx>=0.26" "boto3>=1.34" "prefect>=2.14"

Complete Working Example

import json
import os
import statistics
from dataclasses import dataclass

import httpx


@dataclass
class Rules:
    max_failed_abs: int = 5
    max_failed_frac: float = 0.005
    max_empty_tiles: int = 0
    regression_factor: float = 2.0


def evaluate(m: dict, history: list[dict], r: Rules = Rules()) -> list[str]:
    out = []
    t = m["tiles"]
    failed = t["failed"]
    if len(failed) > max(r.max_failed_abs, r.max_failed_frac * t["attempted"]):
        out.append(f"{len(failed)}/{t['attempted']} tiles failed, e.g. {', '.join(failed[:5])}")
    empty = m.get("quality", {}).get("empty_tiles", [])
    if len(empty) > r.max_empty_tiles:
        out.append(f"{len(empty)} tiles written as all nodata, e.g. {', '.join(empty[:5])}")
    if len(history) >= 3:
        for key in ("wall_s", "bytes_read", "cost_usd"):
            base = statistics.median(h[key] for h in history if key in h)
            if base and m.get(key, 0) > r.regression_factor * base:
                out.append(f"{key} {m[key]:.3g} vs recent median {base:.3g}")
    return out


def notify(run_id: str, alerts: list[str], webhook: str | None = None) -> None:
    if not alerts:
        return
    text = f":warning: raster pipeline run {run_id}\n" + "\n".join(f"• {a}" for a in alerts)
    httpx.post(webhook or os.environ["SLACK_WEBHOOK_URL"], json={"text": text}, timeout=10)


def after_run(run_id: str, manifest: dict, history: list[dict]) -> list[str]:
    alerts = evaluate(manifest, history)
    notify(run_id, alerts)
    return alerts

The function sends at most one message per run, listing every failing rule with a few example tiles. Grouping matters: a hundred individual messages for a hundred failed tiles are read once and then muted, while one message saying “100 of 8,000 tiles failed, e.g. …” is read every time. The manifest itself comes from the run summary described in monitoring and observability for raster pipelines.

Keeping Alerts Worth Reading

Alert fatigue As the number of alerts per week rises, the fraction that receive a prompt response falls steeply. A pipeline that alerts a few times a month when something is genuinely wrong keeps a high response rate; one that alerts daily on known flaky tiles trains people to ignore it. Response rate vs alert volume a few per month daily Every alert that needs no action lowers the chance the next real one is read.

Every alert that turns out to need no action erodes trust in the channel. Three habits keep the signal clean. First, fix the causes of recurring alerts rather than raising thresholds: tiles that always fail because they lie over open ocean belong in an explicit skip list recorded in the manifest, not in a tolerance. Second, separate urgency: a failed run that blocks a morning product goes to a pager or direct message, a runtime regression goes to a team channel for daytime review. Third, include enough in the message to start work — run identifier, example tiles and a link to the logs or performance report — so the first response is investigation, not searching.

The Dead Man’s Switch

Rules evaluated at the end of a run cannot notice runs that never ended. A separate scheduled check — a small cloud function or a cron job on different infrastructure — should look for the expected manifest and alert if it is missing or stale:

import datetime as dt
import boto3

def check_heartbeat(bucket: str, prefix: str, max_age_h: float = 26) -> list[str]:
    s3 = boto3.client("s3")
    objs = s3.list_objects_v2(Bucket=bucket, Prefix=prefix).get("Contents", [])
    if not objs:
        return ["no manifests found"]
    newest = max(o["LastModified"] for o in objs)
    age = (dt.datetime.now(dt.timezone.utc) - newest).total_seconds() / 3600
    return [f"latest manifest is {age:.0f} h old"] if age > max_age_h else []

Running it somewhere other than the pipeline’s own scheduler is the point: if the scheduler is what broke, a check that depends on it breaks too. Hosted heartbeat services that expect a ping and alert on its absence achieve the same with less code.

Alerts from the Orchestrator

Orchestrators provide their own alerting for task and run state, and it is worth using as a backstop. Prefect automations can notify on any flow run entering a failed or crashed state; AWS Batch job state changes are published to EventBridge, where a rule can forward failed jobs to SNS. These catch crashes that happen before the pipeline’s own code can write a manifest. They do not catch quiet failures — a successful run that wrote empty tiles — so they complement the manifest rules rather than replacing them. The retry policies in retrying failed raster tasks in a Prefect pipeline decide which failures reach this point at all.

Verification

Every rule should have fired at least once An alert drill injects a failed tile, an empty tile, a slow run and a missing manifest in a test environment, and checks that each produces the expected message. Rules that have never fired are untested. Alert drill inject failed tiles fired inject empty tile fired slow run fired skip a run fired Repeat after any change to the rules or the manifest format. 
m = {"tiles": {"attempted": 1000, "failed": [f"t{i}" for i in range(12)]},
     "quality": {"empty_tiles": ["t99"]}, "wall_s": 9000}
hist = [{"wall_s": 3000}] * 5
alerts = evaluate(m, hist)
assert any("tiles failed" in a for a in alerts)
assert any("nodata" in a for a in alerts)
assert any("wall_s" in a for a in alerts)

Common Errors

Nobody noticed the pipeline stopped for a week

There was no missing-run check. Add a heartbeat evaluated on separate infrastructure.

The channel is muted by everyone

Too many alerts need no action. Fix recurring causes and move low-urgency rules to a digest.

Alerts contain presigned URLs

Messages were built from raw log lines. Include identifiers and dashboard links, never credentials.

Regression rule fires after every intentional change

The history includes runs with a different configuration. Reset the baseline when the pipeline version changes.

Frequently Asked Questions

Q: What should trigger an alert in a raster pipeline? Failed tiles above a small threshold, a run that never produced a manifest, tiles written empty or with impossible values, and runtime or cost more than about twice the recent median.

Q: How do I detect a pipeline that did not run at all? With a heartbeat check on separate infrastructure that expects a fresh manifest by a set time and alerts when it is missing.

Q: Should every failed tile send a notification? No. Group failures into one message per run with counts and examples. Individual messages per tile lead to muted channels.

Q: Do orchestrator alerts replace custom rules? No. They catch crashed runs, but not runs that succeed while producing bad data. Use both.