Launching a Coiled Cluster for STAC Processing

Create an ephemeral Coiled Dask cluster from Python, attach a Client, and every downstream .compute() runs on cloud workers sitting next to the data — then close it so nothing bills while idle:

import coiled
from distributed import Client

with coiled.Cluster(n_workers=6, worker_memory="16GiB", region="us-west-2") as cluster:
    client = Client(cluster)
    # ... stackstac a STAC search into xarray, compute a composite ...
    print(client.dashboard_link)
# cluster is torn down automatically on exit — no idle VMs

This is the interactive, shared-memory counterpart to the containerized approach in the parent guide, Distributed Processing on Coiled & AWS Batch. Here the whole job is one live Dask graph you iterate on, not a fixed script fanned out per scene.

Why This Arises in Remote Sensing Workflows

Building a cloud-free composite over a season means reading dozens to hundreds of scenes, aligning them on a common grid, and reducing across time. That is a single connected Dask computation over an xarray cube — exactly the shape a live cluster handles best. Coiled provisions that cluster from a couple of lines of Python, so you stay in a notebook instead of managing container images and queues.

The inputs come from a STAC search, and stackstac turns the matching items into a lazy cube whose reads execute in parallel on the workers when you finally call .compute(). Because the Dask cluster holds intermediate results in distributed memory, refining the composite — swapping a median for a percentile, tightening the cloud filter — does not re-read every COG from cold storage. When the exploration is done and you want to run the finalized recipe over a full archive, the sibling guide Processing COGs on AWS Batch with Docker is the throughput-oriented path. For the wider picture, see Cloud Execution & Orchestration.

The ephemeral cluster lifecycle

The Dask cluster exists only for the duration of the computation. The diagram shows the four phases and where cost accrues.

Environment & Setup

You need Coiled and the STAC-to-array stack locally; Coiled replicates a matching software environment onto the workers so the client and cluster agree on versions.

pip install "coiled>=1.0" "stackstac>=0.5" "pystac-client>=0.7" \
            "xarray>=2024.1" "dask>=2024.1"

Authenticate once with coiled login (or a token) before creating clusters. Build a named software environment from these pins so worker packages match your session:

import coiled

coiled.create_software_environment(
    name="rs-stac-2026",
    pip=["stackstac>=0.5", "pystac-client>=0.7", "xarray>=2024.1",
         "rioxarray>=0.15", "dask>=2024.1"],
)

Complete Working Example

The function below provisions a Dask cluster in the data’s region, searches a STAC API for Sentinel-2 L2A items, stacks the red and NIR bands into a lazy cube, reduces the time axis to a median composite, and returns the computed array. The Dask cluster is created in a with block so it is always torn down — even if the computation raises.

import coiled
from distributed import Client
import stackstac
import xarray as xr
from pystac_client import Client as StacClient

STAC_API = "https://earth-search.aws.element84.com/v1"

def sentinel2_median_composite(
    bbox: tuple[float, float, float, float],
    date_range: str,
    region: str = "us-west-2",
    max_cloud: float = 20.0,
) -> xr.DataArray:
    """
    Build a cloud-filtered Sentinel-2 median composite on an ephemeral Coiled cluster.

    bbox        : (west, south, east, north) in EPSG:4326
    date_range  : STAC datetime interval, e.g. "2023-06-01/2023-08-31"
    region      : cloud region — MUST match the COG bucket to avoid egress
    max_cloud   : discard scenes cloudier than this percentage
    """
    # 1. Find items first — cheap metadata call, no pixels yet
    catalog = StacClient.open(STAC_API)
    search = catalog.search(
        collections=["sentinel-2-l2a"],
        bbox=bbox,
        datetime=date_range,
        query={"eo:cloud_cover": {"lt": max_cloud}},
    )
    items = search.item_collection()
    if len(items) == 0:
        raise ValueError("STAC search returned no items for the given filters")

    # 2. Provision workers in the data's region; context manager guarantees teardown
    with coiled.Cluster(
        name="s2-median",
        n_workers=6,
        worker_memory="16GiB",
        region=region,               # keep compute next to the assets
        software="rs-stac-2026",     # pinned env → client/worker parity
        idle_timeout="20 minutes",   # safety net if the session is abandoned
    ) as cluster:
        client = Client(cluster)     # every .compute() below runs on the cluster
        print("dashboard:", client.dashboard_link)

        # 3. Lazy cube — stackstac builds a Dask graph, reads nothing yet
        cube = stackstac.stack(
            items,
            assets=["red", "nir"],
            epsg=32610,              # common projected grid for all items
            resolution=10,
            chunksize=2048,          # Dask tile edge per worker task
        )

        # 4. Reduce the time dimension → median composite, then trigger reads
        composite = cube.median(dim="time", skipna=True)
        result = composite.compute()   # parallel COG reads happen here, on workers

    # 5. Cluster closed on with-exit; result is a plain in-memory DataArray
    return result


if __name__ == "__main__":
    arr = sentinel2_median_composite(
        bbox=(-122.6, 37.6, -122.3, 37.9),   # San Francisco
        date_range="2023-06-01/2023-08-31",
    )
    print(arr.dims, arr.shape)

The .compute() call is the only line that moves pixels: stackstac and the .median() reduction just extend the Dask graph. All the COG range reads then run in parallel across the six workers, and the reduced composite — far smaller than the raw stack — is pulled back to your session.

Variant Patterns

Adaptive scaling for uneven workloads

If the graph’s width varies during the session, let Coiled grow and shrink the worker pool automatically instead of pinning n_workers. This adds workers while a large .compute() runs and releases them afterward.

with coiled.Cluster(name="s2-adaptive", worker_memory="16GiB", region="us-west-2") as cluster:
    cluster.adapt(minimum=2, maximum=20)   # scale between 2 and 20 workers on demand
    client = Client(cluster)
    # ... build and compute the cube; workers scale to the graph ...

Persisting the cube for iterative analysis

When you plan several reductions over the same stack, persist it into distributed memory once so later computes reuse the cached reads rather than re-hitting S3. This is where a live cluster earns its keep: the expensive COG reads happen once, and each subsequent experiment reduces an already-materialized cube.

cube = stackstac.stack(items, assets=["red", "nir"], epsg=32610, resolution=10)
cube = cube.persist()            # materialize reads in worker memory, keep it lazy
ndvi = ((cube.sel(band="nir") - cube.sel(band="red")) /
        (cube.sel(band="nir") + cube.sel(band="red"))).median("time").compute()

Writing the composite back to S3

Once computed, the composite is a plain in-memory DataArray. Attach the CRS and transform stackstac recorded and write it as a Cloud-Optimized GeoTIFF to an output bucket so downstream consumers can range-read it — the same COG access pattern the inputs used.

import rioxarray  # registers the .rio accessor

# result carries CRS + transform from stackstac; write a tiled, compressed COG
result.rio.to_raster(
    "s3://my-raster-archive/composites/sf_2023_summer.tif",
    driver="COG", compress="zstd", blocksize=512,
)

Run the write from the session after .compute(), or submit it as a Dask task so the bytes go straight from a worker to S3 without transiting your laptop.

Common Errors

Slow reads and unexpected egress charges

The Dask cluster is in a different region than the STAC assets. Set coiled.Cluster(region=...) to the collection’s storage region so range reads stay on the internal network instead of crossing regions.

KilledWorker during .compute()

A worker ran out of memory holding a chunk. Lower chunksize in stackstac.stack, raise worker_memory, or reduce the number of assets stacked at once so each task’s resident array is smaller.

Cluster keeps billing after the notebook finishes

The Dask cluster was created without a with block and never closed. Wrap it in a context manager or call cluster.close() in a finally, and set idle_timeout so an abandoned session self-terminates.

Related

• Distributed Processing on Coiled & AWS Batch — parent guide comparing live Coiled clusters with containerized AWS Batch jobs
• Querying STAC Catalogs Programmatically — build the STAC search that feeds stackstac
• Scaling Raster Processing with Dask — the distributed engine underneath every Coiled cluster
• Band Math Operations with xarray — reduce and combine the cube once it is in memory