Optimizing Pipeline Cost and Performance

A cloud raster pipeline that works on a laptop can become surprisingly expensive at scale: idle CPUs waiting on network round-trips, workers thrashing on oversized chunks, and a monthly bill dominated by S3 egress and per-request charges you never see in a profiler. Making it cheap and fast is a distinct engineering task from making it correct — it means aligning compute placement, worker shape, and read strategy with how object storage and GDAL actually behave. For the full architectural context of where this tuning sits, see the parent section Cloud Execution & Orchestration.

This guide is for remote sensing analysts and data engineers already running distributed jobs who want to drive down cost per scene without rewriting their science code. Every technique below is measurable: establish a baseline, change one variable, and confirm the improvement in the Dask dashboard or the bill.

---

Where the money and time actually go

Before tuning anything, understand the three cost centres of a cloud raster job. Compute (worker vCPU-hours) is the one everyone watches, but data-transfer egress and per-request charges frequently dominate for I/O-heavy index and mosaic workloads. The diagram contrasts a naive full-resolution, cross-region pipeline against a tuned one that reads overviews in-region.

The ordering matters: chasing a 10% compute saving while paying cross-region egress on every full-resolution byte is optimizing the wrong term. Fix data placement and read volume first, then tune the workers.

---

Prerequisites

Install the profiling and distributed stack. Pin versions so scheduler and worker images stay in lockstep — a Dask version skew between client and cluster silently degrades scheduling:

pip install "dask[distributed]>=2024.1" "distributed>=2024.1" \
            "rasterio>=1.3" "rioxarray>=0.15" "bokeh>=3.1" \
            "s3fs>=2024.2"

Conceptual prerequisites: You should understand how Cloud-Optimized GeoTIFF structure places overviews and tile offsets at the head of the file, and how to read COG headers without downloading full files — both are the mechanical foundation for reading less. Familiarity with scaling raster processing with Dask is assumed.

---

Step-by-step workflow

Step 1 — Establish a cost and time baseline

You cannot optimize what you have not measured. Wrap a representative run in performance_report and capture wall-clock, bytes read, and request count so every later change has a reference point:

from dask.distributed import Client, performance_report
import time

client = Client("tcp://scheduler:8786")  # or LocalCluster() for a dry run

def run_pipeline():
    # your real workload: build a lazy graph, then .compute()
    ...

start = time.perf_counter()
with performance_report(filename="baseline.html"):
    run_pipeline()
elapsed = time.perf_counter() - start

print(f"wall-clock: {elapsed:.1f}s")
# The HTML report records task durations, transfer time, and worker memory.

Record the S3 request count and bytes transferred from CloudWatch or the storage console for the same window. Egress and GET charges never appear in the Dask report, so the bill is the second half of your baseline.

Step 2 — Right-size Dask workers: threads, processes, and memory_limit

Raster reads are I/O- and decompression-bound, and GDAL releases the GIL during both, so threads inside a worker parallelize reads while sharing one GDAL cache and one HTTP connection pool. Prefer a few processes each with a handful of threads rather than many single-threaded processes, which duplicate the cache and multiply connections:

from dask.distributed import LocalCluster, Client

# On an 8-vCPU node: 4 worker processes x 2 threads each.
cluster = LocalCluster(
    n_workers=4,           # separate processes → memory isolation
    threads_per_worker=2,  # GIL-releasing GDAL reads run concurrently
    memory_limit="6GB",    # per-worker cap; total must fit the node RAM
    processes=True,
)
client = Client(cluster)

Set memory_limit deliberately: too high and a spike triggers the OS OOM killer, losing the worker and its in-flight tasks; too low and Dask pauses or spills to disk, wrecking throughput. Target roughly 70–80% of node_RAM / n_workers, leaving headroom for the GDAL block cache configured in Step 4.

Step 3 — Read overviews instead of full-resolution pixels

Most analyses do not need native resolution. A COG stores decimated overviews; reading the level that matches your output resolution can cut transferred bytes by 16x or 64x. Select an overview explicitly with rioxarray, or let GDAL pick via OVERVIEW_LEVEL:

import rioxarray

# Read the second overview (typically 1/4 resolution) instead of full-res.
da = rioxarray.open_rasterio(
    "s3://sentinel-cogs/.../B04.tif",
    overview_level=1,          # 0 = first overview, -1 = full resolution
    chunks={"x": 2048, "y": 2048},
    lock=False,                # allow threaded reads within a worker
)
print(da.rio.resolution())     # coarser pixel size, far fewer bytes

If you only need a spatial subset, combine overviews with windowed reads so you fetch neither unnecessary resolution nor unnecessary extent — see optimizing rasterio window reads for memory efficiency. Confirm overviews exist before you depend on them; absent overviews are a COG validation failure you can catch from the header alone.

Step 4 — Tune the GDAL VSI cache and merge consecutive ranges

Every worker runs its own GDAL. Two settings dominate remote-read efficiency: the in-memory block cache (GDAL_CACHEMAX) and range merging (GDAL_HTTP_MERGE_CONSECUTIVE_RANGES), which coalesces adjacent tile requests into one HTTP round-trip. Enable the /vsicurl/ chunk cache (VSI_CACHE) so re-reads of the same header or tile do not hit the network twice:

import rasterio
from rasterio.env import Env

gdal_opts = {
    "GDAL_CACHEMAX": "512",                     # MB of decoded-block cache per process
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES", # coalesce adjacent tile ranges
    "VSI_CACHE": "TRUE",                        # cache /vsicurl/ byte ranges
    "VSI_CACHE_SIZE": "26214400",               # 25 MB per-file range cache
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR", # skip directory listing requests
    "GDAL_HTTP_MULTIPLEX": "YES",               # HTTP/2 multiplexed range requests
}

with Env(**gdal_opts):
    with rasterio.open("s3://sentinel-cogs/.../B04.tif") as src:
        window_data = src.read(1, window=((0, 1024), (0, 1024)))

Size GDAL_CACHEMAX against the per-worker memory_limit from Step 2 — it is allocated per process, so 4 workers x 512 MB is 2 GB of RAM you must budget. On a Dask cluster, apply these globally with dask.config or a worker plugin so every process inherits them, not just the client.

Step 5 — Colocate compute in the bucket region and add spot workers

The single largest lever is placement. Data transferred from S3 to compute in the same AWS region incurs no egress charge; cross-region or internet egress is billed per GB and often exceeds compute cost. Launch workers in the bucket’s region, then let the cheapest capacity do the bulk work:

# Cluster config sketch (Coiled / dask-cloudprovider style)
cluster_spec = {
    "region": "us-west-2",        # MUST match the bucket region → egress free
    "worker_options": {
        "nthreads": 2,
        "memory_limit": "6GB",
    },
    "scheduler_instance": "on-demand",   # keep the brain stable
    "worker_instance": "spot",           # 60-90% cheaper, interruptible
    "worker_spot_fraction": 0.8,         # cap so a reclaim can't drain the cluster
}

Spot and preemptible workers are safe for idempotent, chunked raster work: when a worker is reclaimed, Dask reschedules only its in-flight tasks onto survivors. Keep the scheduler on on-demand capacity, write results atomically, and colocate outputs in the same region. The egress mechanics deserve their own treatment — see reducing S3 egress costs in raster pipelines.

Step 6 — Profile the run with the Dask dashboard and performance_report

Re-run the tuned pipeline under performance_report and compare against baseline.html. In the live dashboard, three panels tell you what is left to fix: the task-stream (are workers saturated or gapped by I/O?), the memory panel (spilling to disk?), and the workers table (skewed load?):

from dask.distributed import performance_report

with performance_report(filename="tuned.html"):
    run_pipeline()

# Open http://<scheduler>:8787 during the run:
#  - Task Stream: red gaps = idle workers waiting on network → read less / merge ranges
#  - Bytes stored: climbing toward memory_limit → shrink chunks or overview_level
#  - Workers: one worker hot, others idle → chunk sizes uneven, re-tile the read

Long red gaps in the task stream mean workers idle on network round-trips: revisit Steps 3 and 4. A memory curve pressing the limit means chunks are too large: coarsen overview_level or shrink chunk edges, as covered in tuning Dask chunk sizes for raster cubes.

---

Parameter reference

---

Verification & testing

Prove each change paid off with numbers, not intuition. Assert that the tuned run beats the baseline on both wall-clock and request count:

import json
from pathlib import Path

def assert_improved(baseline_s: float, tuned_s: float,
                    baseline_reqs: int, tuned_reqs: int) -> None:
    """Fail CI if a 'tuning' change regressed the pipeline."""
    assert tuned_s < baseline_s, (
        f"wall-clock regressed: {tuned_s:.1f}s vs baseline {baseline_s:.1f}s"
    )
    assert tuned_reqs <= baseline_reqs, (
        f"request count rose: {tuned_reqs} vs baseline {baseline_reqs}"
    )
    saving = 1.0 - tuned_s / baseline_s
    print(f"wall-clock improved {saving:.0%}; "
          f"requests {baseline_reqs} → {tuned_reqs}")

# Example: pulled from performance reports + CloudWatch request metrics
assert_improved(baseline_s=812.0, tuned_s=143.0,
                baseline_reqs=41000, tuned_reqs=6200)

For a physical sanity check, confirm the overview read returned the resolution you expected (da.rio.resolution()), and confirm egress dropped to near zero in the billing console after moving compute into the bucket region.

---

Troubleshooting

Workers idle but the job is slow — the task stream shows red gaps between compute blocks. Workers are blocked on network round-trips. Enable GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES, raise GDAL_CACHEMAX, drop to a coarser overview_level, and confirm compute is in the bucket region. Low CPU with a long wall-clock is the signature of an I/O-bound, not compute-bound, pipeline.

KilledWorker / repeated OOM kills — memory_limit is set above what the node can back, or GDAL_CACHEMAX x n_workers overran RAM. Lower memory_limit, shrink chunk edges or coarsen the overview, and remember the GDAL cache is per process. Watch the dashboard memory panel for spill-to-disk before the kill.

Egress bill unchanged after tuning reads — compute is still in a different region (or on-premises) from the bucket. Reading fewer bytes helps, but any cross-region byte is billed. Move the Dask cluster into the bucket’s region; in-region transfer is free.

Spot reclaims stall the whole run — the scheduler or a non-idempotent write step ran on spot capacity, or worker_spot_fraction was 1.0 and a coordinated reclaim drained every worker. Pin the scheduler to on-demand, cap the spot fraction, and make result writes atomic so a retried task cannot corrupt output.

GDAL_CACHEMAX seems ignored — the env vars were set on the client but not on the workers. Distribute them with a dask.distributed worker plugin or dask.config.set so every worker process inherits the GDAL environment, then verify with rasterio.Env().options inside a task.

---

FAQ

Should Dask workers for raster processing use threads or processes?

Use a few processes each with several threads. GDAL and rasterio release the GIL during I/O and decompression, so threads give real parallelism for reads while sharing one GDAL cache and one HTTP connection pool. Processes add memory isolation but duplicate that cache. A common shape is 4 workers x 2 threads on an 8-vCPU node; profile before assuming more threads help.

Why is my pipeline slow even though CPU usage is low?

Low CPU with a slow wall-clock almost always means the workers are blocked on network I/O. Each un-merged HTTP range request to S3 costs a round-trip of tens of milliseconds. Read overviews instead of full resolution, set GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES, raise GDAL_CACHEMAX, and run compute in the same region as the bucket to remove that latency.

Are spot or preemptible workers safe for a raster pipeline?

Yes, when the work is idempotent and chunked. Dask reschedules tasks from an interrupted worker onto survivors, so a spot reclaim costs only the in-flight tasks, not the whole run. Keep the scheduler and any result-writing step on on-demand capacity, write outputs atomically, and cap the fraction of spot workers so a coordinated reclaim cannot drain the Dask cluster.

---

Related

• Cloud Execution & Orchestration — parent section covering distributed execution, orchestration, and deployment of raster pipelines end to end
• Scaling Raster Processing with Dask — worker and cluster fundamentals that this cost tuning builds on
• Tuning Dask Chunk Sizes for Raster Cubes — pick chunk edges that keep memory bounded and scheduling efficient
• Reducing S3 Egress Costs in Raster Pipelines — the egress and request half of the bill, in depth
• Understanding Cloud-Optimized GeoTIFF Structure — why overviews and internal tiling make partial reads possible
• How to Read COG Headers Without Downloading Full Files — fetch metadata cheaply to plan reads before committing bandwidth