Building a Prefect Deployment for Nightly Composites

Deploy a flow with a cron schedule onto a work pool:

from prefect import flow

@flow(log_prints=True)
def nightly_composite(days_back: int = 30, tiles: list[str] | None = None, out_prefix: str = "s3://composites/"):
    ...

if __name__ == "__main__":
    nightly_composite.from_source(
        source="https://github.com/org/raster-pipelines.git",
        entrypoint="flows/composite.py:nightly_composite",
    ).deploy(name="nightly", work_pool_name="raster-ecs", cron="0 2 * * *",
             parameters={"days_back": 30}, image="ghcr.io/org/raster-flows:2026.09")

A flow that runs in a notebook becomes a service once it has a schedule, somewhere to run, and guarantees about what happens when two runs overlap or one fails. This page belongs to orchestrating pipelines with Prefect in Cloud Execution & Orchestration.

The Pieces of a Deployment

From flow code to a nightly run Flow code lives in a repository or image. A deployment registers the flow with its entry point, default parameters and cron schedule. At the scheduled time, Prefect creates a flow run in a work pool. A worker polling that pool starts the run on its infrastructure — a Docker container, an ECS task or a Kubernetes job — which executes the flow and reports state back. Flow → deployment → work pool → worker → infrastructure flow code repo or image deployment params + cron work pool queue of runs worker polls the pool infrastructure ECS / K8s / Docker The worker is small and always on; the heavy infrastructure exists only while a run is active.

Separating the always-on worker from the per-run infrastructure is what keeps a nightly pipeline cheap. The worker is a lightweight process that polls for scheduled runs; when one is due, it launches an ECS task or Kubernetes job sized for compositing, which exits when the flow finishes. Nothing large sits idle for the other twenty-two hours of the day.

Environment & Setup

Package Version pin Used for
prefect >=3.0 Flows, deployments, work pools, artifacts
prefect-aws >=0.5 ECS work pools and S3 blocks
odc-stac >=0.3.9 Loading scenes inside the flow
rioxarray >=0.15 Writing composite COGs 
pip install "prefect>=3.0" "prefect-aws>=0.5" "odc-stac>=0.3.9" "rioxarray>=0.15"
prefect work-pool create raster-ecs --type ecs
prefect worker start --pool raster-ecs

Complete Working Example

# flows/composite.py
import datetime as dt

from prefect import flow, task, get_run_logger
from prefect.artifacts import create_table_artifact
from prefect.concurrency.sync import concurrency


@task(retries=2, retry_delay_seconds=[60, 300], tags=["stac-read"])
def composite_tile(tile: str, start: dt.date, end: dt.date, out_prefix: str) -> dict:
    key = f"{out_prefix}{end:%Y/%m/%d}/{tile}.tif"
    if exists(key):                                    # idempotent: skip finished tiles
        return {"tile": tile, "status": "skipped", "key": key}
    with concurrency("stac-api", occupy=1):            # global cap on catalogue load
        items = search(tile, start, end)
    if not items:
        return {"tile": tile, "status": "no_data", "key": None}
    cube = load(items, tile)
    write_cog(median_composite(cube), key)
    return {"tile": tile, "status": "ok", "key": key, "scenes": len(items)}


@flow(log_prints=True)
def nightly_composite(days_back: int = 30, tiles: list[str] | None = None,
                      out_prefix: str = "s3://composites/", run_date: dt.date | None = None):
    log = get_run_logger()
    end = run_date or dt.date.today()
    start = end - dt.timedelta(days=days_back)
    tiles = tiles or default_tiles()
    futures = composite_tile.map(tiles, start, end, out_prefix)
    results = [f.result(raise_on_failure=False) for f in futures]
    rows = [r if isinstance(r, dict) else {"tile": t, "status": "failed", "key": None}
            for t, r in zip(tiles, results)]
    create_table_artifact(key="composite-manifest", table=rows,
                          description=f"Composite {start} to {end}")
    failed = [r["tile"] for r in rows if r["status"] == "failed"]
    log.info(f"{len(rows) - len(failed)} ok/skipped, {len(failed)} failed")
    if len(failed) > max(5, 0.01 * len(rows)):
        raise RuntimeError(f"{len(failed)} tiles failed")

Three design choices make this safe to run unattended. Outputs are written to keys derived from the run date and tile, and existing outputs are skipped, so a rerun after a failure only does the missing work. A global concurrency limit caps how many tasks hit the STAC API at once, however many tiles map in parallel. And the flow fails only when failures exceed a threshold, so a handful of problem tiles produce a successful run with a manifest listing them rather than a red run that hides the 99% that worked. The retry settings follow retrying failed raster tasks in a Prefect pipeline.

Schedules, Time Zones and Backfills

Nightly runs and a backfill Scheduled runs occur at two in the morning each night. When two nights are missed due to an outage, a backfill submits runs for those dates with the run_date parameter set explicitly, producing the same outputs as if the scheduled runs had succeeded. Scheduled runs and a backfill missed during outage backfill with run_date Because run_date is a parameter, a backfill is just two ordinary runs.

Cron schedules are evaluated in UTC unless a time zone is given, which matters twice a year when daylight saving shifts a 02:00 local run by an hour. Setting the time zone explicitly on the schedule keeps runs at the intended local time. More important is making the run date a parameter rather than reading the clock inside the flow: a backfill for missed nights is then just a run with run_date set, and reprocessing last month after a bug fix is a loop over dates. A flow that calls date.today() deep inside a task can never reproduce yesterday.

Choosing the Work Pool

Process work pools run flows on the worker’s own machine and suit small pipelines or development. Docker pools isolate each run in a container on the worker’s host. ECS and Kubernetes pools launch each run as a separate task or job with its own CPU and memory, which is the right choice for compositing: each night’s run gets a machine sized for it, and the worker host stays small. The image should contain GDAL and the flow’s dependencies, built as in building a slim GDAL Docker image, with the tag pinned in the deployment so a new image cannot change last night’s behaviour without a new deployment.

Preventing Overlapping Runs

If a run takes longer than a day — because of a backlog, a slow catalogue or a large backfill — the next scheduled run starts while the previous one is still writing. With idempotent outputs this is wasteful rather than dangerous, but it doubles load on the catalogue and the cluster. A deployment-level concurrency limit of one queues the second run until the first finishes; alternatively, a tag-based limit shared by several deployments caps the combined load on a shared resource. Alerting on runs that exceed their usual duration, as in alerting on failed tile jobs, catches the backlog before it compounds.

Verification

Before trusting the schedule Confirm the deployment's next scheduled time is the intended local time. Trigger one manual run with a fixed run date and confirm its manifest artifact. Trigger the same run again and confirm every tile is skipped, proving idempotency. Three checks after deploying next run time correct local hour manual run manifest artifact present rerun every tile skipped The rerun check is what makes backfills and retries safe. 
prefect deployment inspect 'nightly-composite/nightly'
prefect deployment run 'nightly-composite/nightly' --param run_date=2026-09-01 --param tiles='["33UVP"]'
# run the same command again: the manifest should show status "skipped" for 33UVP

Common Errors

Scheduled runs never start

No worker is polling the work pool. Start a worker for that pool and keep it running as a service.

Runs start an hour off after a clock change

The schedule has no time zone. Set it explicitly.

A rerun reprocesses everything

Outputs are not deterministic or not checked. Derive keys from run date and tile and skip existing ones.

Two runs overlap and double the load

No concurrency limit on the deployment. Set a limit of one.

Frequently Asked Questions

Q: What is a Prefect deployment? A server-side registration of a flow with its entry point, default parameters, schedule and the work pool its runs should use. It turns a flow into something that runs on a schedule or on demand.

Q: Which work pool suits raster compositing? An ECS or Kubernetes pool, so each run gets its own appropriately sized container and nothing large sits idle between runs.

Q: How do I backfill missed nights? Make the run date a flow parameter, then trigger runs for the missed dates. With idempotent outputs, a backfill is just ordinary runs.

Q: Should a few failed tiles fail the whole run? Usually not. Fail only above a threshold and list the failed tiles in a manifest artifact, so partial success is visible and recoverable.