Distributed Processing on Coiled & AWS Batch

When a raster workload outgrows a single machine, the question is no longer how to compute an index but where the compute should live and how it reaches the data. This guide contrasts two production deployment models for running Python raster pipelines in the cloud: Coiled, which spins up managed Dask clusters directly from Python, and AWS Batch, which runs containerized jobs against a managed compute queue. Both keep compute next to the data and write results back to object storage; they differ in execution shape, ergonomics, and cost profile. For the broader orchestration context, see the parent section on Cloud Execution & Orchestration.

The audience here is remote sensing analysts and data engineers who already have a working local pipeline and now need it to run over hundreds or thousands of scenes without saturating a laptop’s memory or bandwidth. The two hands-on guides in this section walk through each model end to end; this page gives you the decision framework and the shared infrastructure concerns that apply to both.

---

Two deployment models at a glance

Coiled and AWS Batch solve the same problem from opposite directions. Coiled gives you a live, elastic Dask scheduler you talk to from a notebook or script — ideal when the computation is one large distributed graph over an xarray cube. AWS Batch gives you a queue that runs an image to completion per job — ideal when the work is naturally embarrassingly parallel, one container invocation per scene. The diagram below places both against the same data plane.

---

Prerequisites

Both models sit on the same rasterio/GDAL core; you additionally install the Coiled or AWS SDK depending on which you drive. Pin the geospatial stack so it matches whatever runs on the workers or inside the container — GDAL and PROJ drift is the top cause of silent coordinate errors on remote compute.

# Shared core
pip install "rasterio>=1.3" "numpy>=1.24"
# Coiled path
pip install "coiled>=1.0" "dask>=2024.1" "distributed>=2024.1" "stackstac>=0.5"
# AWS Batch path (local tooling to submit jobs)
pip install "boto3>=1.34"

Conceptual prerequisites: you should already have a working single-machine pipeline and be comfortable with Dask chunking and with reading assets from a STAC catalog. An AWS account with S3, IAM, and (for Batch) ECR access is assumed; a Coiled account is assumed for the managed-cluster path.

---

Key components

---

Head-to-head

Neither model is strictly better. A mature deployment often uses both: Coiled for the interactive research loop where you design a composite over a STAC collection, then AWS Batch to run the finalized script over the full archive.

---

When to choose Coiled

Choose Coiled when the computation is a single connected Dask graph and you want to iterate on it. Streaming a STAC search into an xarray cube, computing a cloud-free composite, and inspecting the result interactively is the canonical fit. The Dask cluster holds intermediate results in distributed memory, so re-running a slightly changed reduction does not re-read every COG from scratch.

import coiled
from distributed import Client

# Provision a right-sized Dask cluster in the data's region
cluster = coiled.Cluster(
    name="stac-composite",
    n_workers=8,
    worker_memory="16GiB",
    region="us-west-2",              # same region as the COG bucket
    software="rs-stac-2026",         # pinned software environment
)
client = Client(cluster)             # route all .compute() calls here
print(client.dashboard_link)

From here you would stackstac a STAC item collection into a lazy xarray cube and reduce it — the full walkthrough lives in Launching a Coiled Cluster for STAC Processing. Because the Dask cluster is elastic, adaptive scaling can grow workers while a large .compute() runs and release them afterward. Always close the Dask cluster when the session ends so idle VMs stop billing.

---

When to choose AWS Batch

Choose AWS Batch when the work is naturally one job per scene and each job runs a fixed script to completion. There is no shared state to exploit, so a live scheduler adds nothing; a container image and a queue fit better. The array job primitive lets a single submission fan out over thousands of scenes, and the queue scales the underlying EC2 fleet to match depth.

import boto3

batch = boto3.client("batch", region_name="us-west-2")

# One array job → N child tasks, each processes a different scene by index
response = batch.submit_job(
    jobName="cog-index-run",
    jobQueue="raster-queue",
    jobDefinition="cog-processor:3",   # versioned job definition
    arrayProperties={"size": 500},     # 500 scenes, 500 child tasks
)
print(response["jobId"])

Each child task reads its scene’s COG from S3, runs the transform in-container, and writes the result back — details in Processing COGs on AWS Batch with Docker. Batch pairs naturally with EC2 Spot compute environments, which cut cost substantially for interruption-tolerant, idempotent per-scene work.

---

Shared infrastructure concerns

Both models succeed or fail on the same three fundamentals.

Same-region compute

Placing workers in the same region as the bucket is the single highest-leverage decision. A Cloud-Optimized GeoTIFF pipeline issues many small HTTP range reads per file; cross-region, each pays egress and tens of milliseconds of latency, and it compounds across thousands of files. Set region on the Coiled cluster and the jobQueue compute environment to match the data. Verify the bucket’s region first:

import boto3

s3 = boto3.client("s3")
loc = s3.get_bucket_location(Bucket="sentinel-cogs")["LocationConstraint"]
print(loc or "us-east-1")   # None → us-east-1; pin compute to this region

VM sizing

Size VMs to the per-task memory footprint, not to a round number of cores. For raster work the binding constraint is memory per worker: a single Sentinel-2 scene read into float32 across several bands can hold a few gigabytes resident. On Coiled set worker_memory above the largest chunk a worker holds at once; on Batch set the job definition’s memory above the peak resident set of one scene, with headroom for GDAL’s block cache.

Software / package environment parity

Workers must run the same GDAL, rasterio, and PROJ versions you tested locally. GDAL or PROJ version drift silently changes coordinate transforms and resampling output. On Coiled, build a named software environment from pinned requirements; on Batch, pin exact versions in the Docker image. Never rely on latest tags for either — an image rebuilt months later can pull a newer GDAL and change your results.

---

Verification & testing

Before committing a full run, prove three things on a single scene: the worker environment matches, the data is local, and results write back correctly.

import rasterio

# Run this ON a worker (Coiled) or inside the container (Batch) to confirm parity
def assert_geo_stack(expected_gdal: str) -> None:
    got = rasterio.__gdal_version__
    assert got == expected_gdal, f"GDAL drift: worker {got} != pinned {expected_gdal}"
    # Confirm a header-only remote open succeeds from the compute environment
    url = "https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/N/YF/2023/6/S2A_36NYF_20230615_0_L2A/B04.tif"
    with rasterio.open(url) as src:
        assert src.crs is not None, "remote COG opened but CRS is missing"
    print("environment parity + remote read OK")

On Coiled, submit this with client.submit(assert_geo_stack, "3.9.2").result() so it executes on a worker, not your laptop. On Batch, run it as the container entrypoint of a one-task test job and read the result from CloudWatch logs. Confirm a written output object appears in S3 with the expected size and CRS before scaling to the full archive.

---

Troubleshooting

ModuleNotFoundError on a Coiled worker but not locally — the worker software environment does not match your session. Rebuild the environment from the same pinned requirements, or enable package sync, and confirm the Dask cluster picked up the new software= name. Client/worker mismatch also surfaces as pickle/deserialization errors.

AWS Batch task stuck in RUNNABLE — the compute environment cannot place the job. The requested vCPU/memory exceeds any instance type in the environment, the environment is at its maxvCpus ceiling, or the subnet has no capacity. Lower the job definition’s resource request or raise the compute environment limits.

AccessDenied reading or writing S3 from a job — the job role lacks the right IAM permissions. The Batch job role (not the execution role) needs s3:GetObject on the input prefix and s3:PutObject on the output prefix. On Coiled, attach the equivalent instance profile or pass credentials to workers.

High cost with low utilization — an idle Coiled cluster left open bills for its VMs, and oversized Batch jobs waste reserved memory. Close clusters at session end, enable adaptive scaling, and right-size Batch memory to the measured peak. See Optimizing Pipeline Cost and Performance for the full cost model.

Cross-region slowness — reads are slow and egress charges appear even though the pipeline “works”. Compute is in a different region than the bucket. Move the Dask cluster/queue to the data’s region rather than moving the data.

---

FAQ

When should I use Coiled instead of AWS Batch for raster processing?

Use Coiled when the work is an interactive or exploratory Dask computation — streaming a STAC stack into an xarray cube, iterating on a composite, and inspecting results in a notebook. Reach for AWS Batch when you have thousands of independent per-scene jobs that each run a fixed script to completion, where an array job and a container image fit the shape of the work better than a live scheduler.

Why does same-region compute matter so much for cloud raster jobs?

Cloud providers charge egress when bytes leave a region and add latency on every cross-region range read. A COG pipeline issues many small HTTP range requests per file, so placing workers in the same region as the bucket keeps reads on the internal network — near-zero egress cost and single-digit-millisecond latency instead of tens of milliseconds per request.

How do I keep the worker environment identical to my local environment?

On Coiled, build a named software environment from the same pinned requirements you run locally, or let package sync mirror your active environment. On AWS Batch, bake dependencies into the Docker image and pin exact versions so the GDAL, rasterio, and PROJ stack on the worker matches what you tested. Version drift in GDAL or PROJ is the most common source of silent coordinate errors on remote workers.

---

Related

• Cloud Execution & Orchestration — parent section covering scheduling, scaling, and cost across the full cloud pipeline
• Scaling Raster Processing with Dask — the distributed compute engine that Coiled provisions and manages for you
• Processing COGs on AWS Batch with Docker — package a rasterio job in a container and fan it out with an array job
• Launching a Coiled Cluster for STAC Processing — spin up an ephemeral cluster, stream a STAC stack, and tear it down
• Understanding Cloud-Optimized GeoTIFF Structure — why range reads make same-region COG access cheap
• Optimizing Pipeline Cost and Performance — control egress, right-size compute, and use spot capacity