Monitoring and Observability for Raster Pipelines

A raster pipeline that processes thousands of tiles overnight will fail in ways nobody sees unless it reports on itself. The minimum useful setup is one structured event per tile and stage, a per-run summary, and an alert when anything in that summary is wrong:

import json, logging, time

log = logging.getLogger("pipeline")

def event(stage, tile, status, **fields):
    log.info(json.dumps({"ts": time.time(), "run": RUN_ID, "stage": stage,
                          "tile": tile, "status": status, **fields}))

event("ndvi", "33UVP_0412", "ok", seconds=41.2, bytes_read=512_000_000, nodata_frac=0.03)

Observability here means being able to answer, from the records alone, three questions: what happened in this run, why did this tile fail, and is this run behaving differently from the last one. This topic belongs to Cloud Execution & Orchestration and builds on the scaling and orchestration practices elsewhere in that section.

---

Prerequisites

Observability is added to a working pipeline rather than built first, so the assumptions are modest:

• A pipeline that processes rasters in independent units — tiles, scenes or chunks — whether on Dask, Prefect, AWS Batch or a plain loop.
• Python 3.10+ with structlog or the standard logging module, and dask.distributed if the pipeline uses Dask.
• Somewhere to send logs and metrics: CloudWatch, a log aggregator, or even a bucket of JSON lines files. The practices below do not depend on a particular vendor.
• A notification channel — email, Slack or a pager — for alerts.

pip install "structlog>=24.1" "dask[distributed]>=2023.1" "prefect>=2.14" "memray>=1.11"

---

What Makes Raster Pipelines Hard to Observe

Generic application monitoring is built around requests and error rates. Raster pipelines have a different shape: long-running batch jobs, thousands of near-identical units of work, heavy reads from object storage, and outputs whose correctness is a property of the pixels rather than of a response code. A job can exit with status zero and still have written a tile of nodata because a cloud mask was inverted, or a STAC search came back empty for one date and the composite silently used one scene instead of ten. The observability that matters is the one that makes those quiet failures visible.

That is why this topic leans on data-aware metrics — nodata fraction, scene count, bytes read, value ranges — logged alongside the usual status and duration. Each is cheap to compute at the moment a tile is written and nearly impossible to reconstruct afterwards.

---

Step-by-Step Workflow

Step 1 — Emit one structured event per tile and stage

Replace free-text log lines with JSON events that share a fixed set of keys: run identifier, stage, tile, scene or date, status, duration and a handful of data metrics. Structured events can be filtered, counted and joined; free text can only be searched. Logging structured events from raster tasks shows how to set this up with structlog, including binding context once per task so every line carries it automatically.

import structlog

log = structlog.get_logger().bind(run=RUN_ID, stage="composite")
log = log.bind(tile=tile_id)
log.info("done", seconds=round(elapsed, 1), scenes=len(items), nodata_frac=round(nd, 4))

Step 2 — Capture task and resource metrics

For Dask pipelines, the scheduler already knows every task’s duration, the bytes transferred between workers and each worker’s memory over time. Saving a performance report for every run, and extracting a few headline numbers from it, turns that knowledge into a history. Tracking Dask task metrics during a run covers performance reports, the scheduler plugin interface and exporting to Prometheus.

from dask.distributed import performance_report

with performance_report(filename=f"reports/{RUN_ID}.html"):
    result = cube.median("time").compute()

Step 3 — Write a run manifest

At the end of every run, write a small JSON document listing tiles attempted, succeeded, failed and skipped, with totals for duration, bytes read and estimated cost. The manifest is the single artefact that answers “did last night work?”, and comparing manifests between runs is how regressions surface.

manifest = {
    "run": RUN_ID, "started": t0, "finished": t1,
    "tiles": {"attempted": n, "ok": n_ok, "failed": failed_ids, "skipped": skipped_ids},
    "bytes_read": total_bytes, "cpu_hours": cpu_h,
}

Step 4 — Alert on the right conditions

Alerts should fire on things that need a human: failed tiles above a small threshold, a run that did not finish, a manifest missing entirely, or metrics far outside their recent range. Alerting on failed tile jobs builds these rules and discusses how to keep them quiet enough that people keep reading them.

Step 5 — Profile memory when workers die

Out-of-memory kills are the most common loud failure in raster work and the least informative, because the process dies before it can log anything useful. Profiling one representative tile in isolation shows where the memory goes — usually an unexpected full-resolution read, an accidental dtype promotion to float64, or an intermediate held longer than needed. Profiling memory in a raster worker walks through memray and tracemalloc on a real task.

---

Choosing What to Measure

The temptation is to log everything. Resist it: logs that nobody reads cost money and bury the lines that matter. The six metrics above cover the failure modes seen most often in practice, and each maps to a specific question. If a proposed metric does not answer a question you would ask during an incident, it can wait.

Data-quality metrics are particularly worth the few lines they cost. A nodata fraction of 1.0 on a tile that should be over land is an immediate signal that something upstream is wrong. A maximum reflectance of 10,000 in a product that should be scaled to 0–1 means the scale factor was skipped. Neither shows up in a status code, and both corrupt every product built downstream.

---

Logs, Metrics and Traces

Three kinds of telemetry complement one another. Logs are discrete events with context — what happened to this tile. Metrics are numeric time series aggregated across many events — how many tiles failed per hour, what the ninety-fifth percentile duration was. Traces follow one unit of work through every stage and service it touches. For most raster pipelines, structured logs plus a small set of derived metrics give most of the value; distributed tracing becomes worthwhile when a tile passes through several services — a queue, a Batch job, a tiler — and latency questions span them.

A practical architecture keeps logs as the source of truth and derives metrics from them. Every event carries the fields needed to count and aggregate it, so dashboards and alerts are queries over the log store rather than a second instrumentation path to maintain. CloudWatch Logs Insights, Loki, or DuckDB over a bucket of JSON lines files all support this pattern.

---

Observability Across Orchestrators

Keeping the per-tile events in application code, rather than relying on each orchestrator’s native logging, makes the pipeline portable. The same function can run in a notebook, on a local Dask cluster, inside a Prefect flow or as a Batch array job, and the events it emits look identical in every case. The orchestrator adds run-level context — which flow run, which job, which attempt — and that context should be bound into the logger at the start of each task.

Prefect deserves a specific note. Its task and flow run states already give a queryable history of success and failure, and its artifacts are a natural home for the run manifest, rendered as a table in the UI. Automations can then send notifications on failed flow runs without any extra infrastructure. The retry logic described in retrying failed raster tasks in a Prefect pipeline should log each attempt as its own event so that tiles which succeed only on the third try are visible, since those are usually the next to fail outright.

---

Parameter Reference

---

Cost of Observability

Telemetry is not free. CloudWatch Logs ingestion is charged per gigabyte, and a pipeline that logs every chunk read of every tile at DEBUG level can spend more on logs than on compute. The per-tile event design keeps volume proportional to the number of tiles rather than the number of operations: ten thousand tiles times five stages is fifty thousand small events per run, a few tens of megabytes. Sampling is rarely needed at that volume. Performance reports are a few megabytes each and belong in cheap object storage rather than in the log system.

Metrics derived from logs cost query time rather than ingestion, so it pays to keep the event schema flat and the field names stable. A dashboard built on seconds breaks silently when someone renames it to duration_s; treat the event schema as an interface and change it deliberately.

---

Verification & Testing

A monitoring setup is only trustworthy once it has been seen to catch a failure. Test it deliberately:

# 1. Inject a failure and confirm it is logged and alerted
os.environ["FORCE_FAIL_TILE"] = "33UVP_0412"
run_pipeline(tiles)
events = [json.loads(l) for l in open(f"logs/{RUN_ID}.jsonl")]
assert any(e["tile"] == "33UVP_0412" and e["status"] == "failed" for e in events)

# 2. Confirm the manifest matches the events
m = json.load(open(f"manifests/{RUN_ID}.json"))
ok = {e["tile"] for e in events if e["stage"] == "write" and e["status"] == "ok"}
assert len(ok) == m["tiles"]["ok"]

# 3. Confirm a nodata-only tile is flagged
assert any(e.get("nodata_frac", 0) > 0.99 for e in events if e["tile"] == EMPTY_TILE)

Run this kind of drill whenever the pipeline or the alerting changes. An alert rule that has never fired is an untested assumption.

---

Troubleshooting

Logs from Dask workers never reach the log store

Worker processes configure logging independently of the client. Configure logging in a worker plugin or via the Dask config so that every worker emits JSON to stdout, where the platform collects it.

Events cannot be joined across stages

A key such as the tile identifier is formatted differently in different stages. Define identifiers in one place and bind them into the logger rather than formatting them inline.

Alerts fire every night for the same handful of tiles

Those tiles are permanently failing — often over the ocean or outside the data footprint. Fix the tiling or add them to an explicit skip list recorded in the manifest, rather than muting the alert.

The log bill is larger than the compute bill

Logging is at DEBUG or per-chunk. Return to one event per tile and stage at INFO and move detailed diagnostics to on-demand profiling.

Out-of-memory kills leave no trace

The process dies before logging. Record the peak memory of successful tiles, alert when it approaches the limit, and profile a failing tile locally.

---

Frequently Asked Questions

Q: What should a raster pipeline log? One structured event per tile and stage with the run, stage, tile, status and duration, plus data metrics such as scene count, bytes read, nodata fraction and value range. Those cover most loud and quiet failures.

Q: Do I need a dedicated monitoring platform? No. Structured JSON logs in CloudWatch or even a bucket, a run manifest and a few alert rules are enough for most pipelines. Dedicated platforms help once several pipelines and teams share the infrastructure.

Q: How do I detect silent failures? Log data-aware metrics — nodata fraction, scene count, value range — and compare each run with recent runs. A tile that is all nodata or a run that read far more bytes than usual is a silent failure made visible.

Q: Should I use distributed tracing? Only when a tile passes through several services and latency questions span them. For single-cluster batch pipelines, structured logs and metrics are sufficient.

Q: How long should telemetry be kept? Keep detailed logs for 30 to 90 days and run manifests indefinitely. Manifests are tiny and give the long-term history of success rates, runtimes and costs.

---

Related

• Logging Structured Events from Raster Tasks — the event design in code.
• Tracking Dask Task Metrics during a Run — performance reports and scheduler plugins.
• Alerting on Failed Tile Jobs — rules that people keep reading.
• Profiling Memory in a Raster Worker — explaining out-of-memory kills.
• Diagnosing Dask Memory Spills in Raster Workflows — the Dask side of memory trouble.