Cloud Execution & Orchestration

A raster pipeline that runs cleanly on your laptop rarely survives contact with a continental archive. The moment the working set outgrows local RAM, or a nightly job must process thousands of scenes inside a fixed window, the bottleneck shifts from how you compute to where and how you run. This section is about that gap: taking correct single-machine code — the Core Raster Fundamentals & STAC Mapping primitives and the Satellite Processing Workflows & Index Pipelines stages — and executing it across many machines reliably, on schedule, and without a surprise cloud bill. It is written for data engineers and remote sensing analysts who already have working rasterio, xarray, and dask code and now need it to run at scale in production.

Scaling out is not one decision but four interlocking ones: how to parallelise the numerical work (Dask), how to schedule and supervise the workflow (Prefect), where to rent the compute (Coiled, AWS Batch, Kubernetes), and how to keep the whole thing cheap and fast (cost and performance tuning). Each has a dedicated topic below; this page gives you the architecture that ties them together and the failure modes that only appear once real infrastructure is involved.

From One Script to a Distributed System

On a single machine, a raster pipeline is a Python process that reads pixels, computes, and writes results. Distributing it introduces three new layers that did not exist before: a cluster of workers that hold and process array chunks, an orchestrator that decides when runs happen and what to do when they fail, and object storage that replaces the local filesystem as the source of truth. Understanding how a request flows through these layers is the mental model everything else hangs on.

The path starts with your driver script — a laptop, a CI runner, or an orchestrator worker — which builds a lazy task graph rather than computing eagerly. That graph is submitted to a Dask scheduler, which distributes chunks across workers. Workers stream bytes directly from object storage (Cloud-Optimized GeoTIFFs over HTTP range requests, or Zarr chunks), compute in parallel, and write results back to the same object store. An orchestrator wraps the whole run so it can be scheduled, retried, and observed.

The critical insight is that the compute goes to the data, not the other way around. Workers run in the same cloud region as the imagery, so they read Cloud-Optimized GeoTIFFs over the internal network at near-zero egress cost, and only small results — a composite, a statistics table — leave the region. Getting this topology wrong is the single largest source of both slow pipelines and inflated bills, which is why cost awareness runs through every topic in this section.

Key Components

The table below lists the objects, decorators, and services that recur across every workflow here, with a one-line note on the role each plays once you leave a single machine.

Production Patterns

The four patterns below are the load-bearing ones. Each links to the topic that develops it fully with tuning, verification, and troubleshooting.

Pattern 1 — Provision a Dask cluster inside a context manager

Never create a Dask cluster you cannot guarantee will be destroyed. A with block ties the Dask cluster lifetime to the code that uses it, so workers shut down even if the computation raises. This single habit prevents most runaway-cost incidents.

from dask.distributed import Client, LocalCluster

# Development: workers on the local machine. Swap LocalCluster for a
# coiled.Cluster in production without changing the code below it.
with LocalCluster(n_workers=4, threads_per_worker=2, memory_limit="4GB") as cluster:
    with Client(cluster) as client:
        print(client.dashboard_link)   # inspect the graph while it runs
        result = expensive_raster_graph().compute()
# Cluster and client are both torn down here, even on exception.

Scaling Raster Processing with Dask covers cluster types, worker sizing, and how chunk structure maps onto the task graph.

Pattern 2 — Lazy STAC-to-cube computation on the Dask cluster

The pipeline should stay lazy from the STAC query all the way to the terminal write. Building an xarray cube backed by Dask means the graph is defined on the driver but every chunk read and arithmetic operation runs on the workers, close to the data.

import stackstac
import pystac_client

catalog = pystac_client.Client.open("https://earth-search.aws.element84.com/v1")
items = catalog.search(
    collections=["sentinel-2-l2a"],
    bbox=[-3.2, 51.4, -2.9, 51.6],
    datetime="2024-05-01/2024-09-30",
    query={"eo:cloud_cover": {"lt": 20}},
).item_collection()

# Lazy, Dask-backed: nothing is read until .compute() runs on the cluster.
cube = stackstac.stack(items, assets=["B04", "B08"], resolution=10,
                       chunksize={"x": 2048, "y": 2048})
red, nir = cube.sel(band="B04"), cube.sel(band="B08")
ndvi = ((nir - red) / (nir + red + 1e-10)).median(dim="time")
ndvi_result = ndvi.compute()   # graph submitted to the connected Client

The NDVI arithmetic itself follows the spectral index calculation pipelines conventions; here it simply runs distributed. See Computing NDVI with xarray on a Dask Cluster for the end-to-end version.

Pattern 3 — Wrap stages as a retrying Prefect flow

Transient S3 timeouts and rate limits are not bugs; they are the normal weather of cloud I/O. Prefect turns each stage into a supervised task that retries with backoff and records state, so a single flaky read does not sink a multi-hour run.

from prefect import flow, task
from prefect.tasks import exponential_backoff

@task(retries=4, retry_delay_seconds=exponential_backoff(backoff_factor=2))
def process_scene(item_id: str) -> str:
    # per-scene work: read COG, mask, index, write result to S3
    return write_result(item_id)

@flow(name="ndvi-monthly", log_prints=True)
def ndvi_monthly(item_ids: list[str]) -> list[str]:
    futures = [process_scene.submit(i) for i in item_ids]
    return [f.result() for f in futures]

if __name__ == "__main__":
    ndvi_monthly(discover_scene_ids())

Orchestrating Pipelines with Prefect covers deployments, schedules, concurrency limits, and mapping tasks over dynamic work.

Pattern 4 — Submit heavy per-scene jobs to AWS Batch

When a stage is embarrassingly parallel across scenes and each scene is a self-contained container run — for example reprojecting and re-tiling thousands of COGs — AWS Batch is often cheaper and simpler than holding a large Dask cluster open. You submit one job per scene and let Batch manage the compute fleet.

import boto3

batch = boto3.client("batch", region_name="us-west-2")

def submit_scene(scene_key: str) -> str:
    resp = batch.submit_job(
        jobName=f"cog-reproject-{scene_key.replace('/', '-')}",
        jobQueue="raster-queue",
        jobDefinition="cog-reproject:3",
        parameters={"input": f"s3://scenes/{scene_key}"},
        retryStrategy={"attempts": 3},   # retry transient container failures
    )
    return resp["jobId"]

Distributed Processing on Coiled & AWS Batch contrasts the Batch container model with Coiled’s Dask-native model and shows when each wins.

Choosing and Composing the Execution Stages

The four topics in this section are not alternatives to pick one of — they are layers you combine. Deciding how to run a pipeline means answering four questions in order.

How do I parallelise the computation? Start with Dask. Scaling Raster Processing with Dask is the foundation: it explains how chunked dask.array and xarray operations turn into a task graph, how to size workers and memory, and how chunk geometry decides whether your graph is efficient or catastrophic. The companion guide on tuning Dask chunk sizes for raster cubes is where most real performance is won or lost, because chunking interacts directly with COG internal tiling and with the pixel resolution and scaling of the source imagery.

When and how often does it run, and what happens on failure? That is orchestration. Orchestrating Pipelines with Prefect covers turning a script into a scheduled, observable deployment — including scheduling Sentinel-2 downloads with Prefect flows on a cron cadence and retrying failed raster tasks so transient S3 and network errors self-heal. Orchestration is the layer that makes a pipeline a product rather than a script you babysit.

Where does the compute physically run? Distributed Processing on Coiled & AWS Batch is the provisioning layer. Coiled launches ephemeral Dask clusters in your own cloud account — see launching a Coiled cluster for STAC processing — while AWS Batch runs containerised per-scene jobs, covered in processing COGs on AWS Batch with Docker. The choice hinges on whether your work is one large shared computation (Dask/Coiled) or many independent container runs (Batch).

How do I keep it fast and affordable? Cost is not a post-hoc audit; it is a design constraint. Optimizing Pipeline Cost and Performance covers instance selection, spot capacity, and the profiling that tells you dollars per result. The largest single lever for Earth observation workloads is data transfer, addressed head-on in reducing S3 egress costs in raster pipelines — keep compute in the imagery’s region, read only the COG windows you need, and never pull whole scenes when a range read suffices.

Common Pitfalls and Failure Modes

Oversized Dask graphs exhaust worker memory

Symptom: Workers climb to their memory limit, spill to disk, then get killed and restarted, causing the computation to thrash or stall. Cause: Chunks are too large, or the graph holds many wide intermediate arrays alive at once because reductions were expressed so that every input chunk must materialise before any output is produced. Fix: Reduce spatial chunk dimensions to 1024×1024 or 2048×2048, target 256–512 MB per chunk after decompression, and insert .persist() only at genuine checkpoints. Watch the distributed dashboard’s memory pane to find the widest point in the graph before scaling hardware.

Unbounded task concurrency hammers S3 and triggers throttling

Symptom: A burst of thousands of simultaneous reads returns SlowDown / HTTP 503 errors from S3, and throughput collapses instead of scaling. Cause: Mapping a task over ten thousand scenes with no concurrency ceiling opens ten thousand concurrent connections against the same prefix, exceeding S3’s per-prefix request rate. Fix: Cap fan-out with a Prefect concurrency limit or a bounded Dask worker count, spread objects across key prefixes to raise the aggregate request ceiling, and always configure client-side retries with exponential backoff so throttled requests recover rather than fail.

Serializing GDAL and rasterio objects across workers

Symptom: TypeError: cannot pickle errors, segfaults, or silent corruption when passing an open dataset between the driver and workers. Cause: A rasterio.DatasetReader (and the underlying GDAL handle) wraps native resources and file descriptors that cannot be pickled and shipped to another process. Fix: Never send open datasets across the wire. Pass the path or URL and have each task open it inside its own with rasterio.open(url) as src: block. For repeated reads of the same object, open within the worker and reuse locally rather than scattering the handle.

Forgetting to scale down clusters, burning money while idle

Symptom: The cloud bill shows hours of running instances long after the job finished; nobody remembers leaving a Dask cluster up. Cause: A Dask cluster was created outside a context manager, or a notebook kernel kept the Client alive, so workers ran idle indefinitely. Fix: Create every cluster inside a with block or with an idle timeout (coiled.Cluster(..., idle_timeout="20 minutes", shutdown_on_close=True)). Add adaptive scaling so idle workers release automatically, and set a cloud billing alarm as a backstop for the case the code path is skipped.

Missing retries on transient S3 and network errors

Symptom: A multi-hour run fails at 90% because one read timed out or hit a momentary rate limit. Cause: Neither the storage client nor the task layer was configured to retry, so a single transient error propagates as a fatal exception. Fix: Configure boto3 with a retry policy (Config(retries={"max_attempts": 5, "mode": "adaptive"})) and wrap each task with Prefect retries plus exponential backoff. Idempotent tasks — writing to a deterministic output key — make retries safe to apply liberally.

Container and driver environment drift

Symptom: Code that runs locally fails on workers with ModuleNotFoundError or subtly different results, or GDAL cannot open a format it read fine on the driver. Cause: The worker image, the AWS Batch container, and the driver have mismatched versions of GDAL, rasterio, or the reader plugins. Fix: Pin the entire stack and build a single container image used by both driver and workers. With Coiled, use package_sync or an explicit image; with Batch, reference an immutable image digest, not a floating latest tag.

Performance and Scale Considerations

Keep compute in the data’s region

For Earth observation the dominant cost and latency factor is data movement, not CPU. Public archives like Sentinel-2 on AWS live in specific regions; running workers in that same region means COG reads travel the internal network for free and fast, while cross-region or egress-to-internet reads are billed per gigabyte and add latency to every chunk. Confirm your cluster’s region matches the imagery bucket before optimising anything else.

Read only what you need

Distributed execution amplifies wasteful reads. A pipeline that pulls entire scenes to use a small window multiplies that waste by every worker. Exploit COG structure so each chunk fetch is an HTTP range request for exactly the overview level and window in play, and align Dask chunks to the COG’s internal tile size so one chunk maps cleanly to a small set of range reads rather than straddling tile boundaries.

Match cluster shape to the workload

I/O-bound COG reading tolerates many workers with a few threads each, because the GIL is released during reads and network bandwidth is the ceiling. CPU-bound work — resampling, regression, classification — wants one or two threads per worker to avoid GIL contention and more, smaller workers. Size memory_limit to roughly 80% of RAM per core, leaving headroom for OS buffers and inter-process transfer, and enable adaptive scaling so the Dask cluster grows for the heavy stage and shrinks for the light ones.

Prefer object-store-native formats

Writing results as COG (single spatial products) or Zarr (multi-temporal cubes) lets distributed workers write concurrently without file locking and lets downstream consumers read partial extents cheaply. Zarr in particular stores each chunk as an independent object, so a hundred workers can write a hundred chunks in parallel — impossible with a single monolithic GeoTIFF that requires serialized writes.

Profile dollars per result, not just seconds

Wall-clock time and cost diverge on the cloud: a bigger cluster finishes sooner but may cost the same or more, while spot instances cut cost at the risk of interruption. Instrument runs to record instance-hours and egress alongside runtime, then optimise the ratio of dollars to delivered product. This is the discipline the cost and performance topic formalises.

Frequently Asked Questions

When should I move a raster pipeline from one machine to a Dask cluster?

Move to a Dask cluster when the working set exceeds single-machine RAM, when wall-clock time per run blocks iteration, or when a scheduled workflow must process more scenes than one host can finish in its window. Below that threshold a single large instance with a LocalCluster is cheaper and simpler than a distributed cluster, because you avoid network serialization and scheduler overhead. Scale out only when a single fat node genuinely cannot keep up.

What is the difference between Dask and Prefect in a raster pipeline?

Dask parallelises the numerical computation inside a job, splitting arrays into chunks that workers process concurrently. Prefect orchestrates the workflow around that computation: scheduling runs, sequencing tasks, retrying transient failures, and recording state. They compose cleanly — a Prefect task submits work to a Dask cluster (via DaskTaskRunner or a direct Client), and Prefect owns the run-level lifecycle that Dask has no concept of.

How do I stop a forgotten Dask or Coiled cluster from wasting money?

Always create clusters inside a context manager or with an idle timeout so idle workers shut down automatically. For Coiled set shutdown_on_close=True and an idle_timeout; for LocalCluster and distributed clusters wrap them in a with Client(...) block that tears the Dask cluster down when the flow exits, even on exception. Enable adaptive scaling so idle workers release, and add a cloud billing alarm as a backstop.

Can I run Dask inside AWS Batch, or should I pick one?

They solve different shapes of problem. Use a Dask cluster (often via Coiled) when the work is one large shared computation over a cube — a mosaic, a temporal composite — where workers must exchange chunks. Use AWS Batch when the work is many independent per-scene container runs that share nothing. You can even nest them: a Batch array job where each element runs a small Dask computation, though most pipelines are clearer picking the model that matches the dominant workload.

Do I need Kubernetes to run Dask in the cloud?

No. Kubernetes via dask-kubernetes is one option and is worth it if your organisation already runs a Dask cluster, but Coiled provisions Dask clusters directly in your cloud account with far less operational overhead, and a single large VM running a LocalCluster covers a surprising amount of production work. Reach for Kubernetes when you need tight integration with existing k8s infrastructure, not as a default.

---

Related

• Core Raster Fundamentals & STAC Mapping — the raster data model, COG byte layout, CRS handling, and STAC querying that every distributed worker relies on.
• Satellite Processing Workflows & Index Pipelines — the single-machine masking, mosaicking, index, and time-series stages that this section scales out.
• Scaling Raster Processing with Dask — chunked dask.array and xarray, cluster types, worker sizing, and task-graph tuning.
• Orchestrating Pipelines with Prefect — flows, tasks, schedules, deployments, and retry policies for supervised runs.
• Distributed Processing on Coiled & AWS Batch — provisioning ephemeral Dask clusters versus containerised per-scene batch jobs.
• Optimizing Pipeline Cost and Performance — instance selection, spot capacity, egress reduction, and dollars-per-result profiling.