Choosing between Threads and Processes for GDAL Workloads

For most raster pipelines, a few processes each running a few threads beats either extreme:

from dask.distributed import LocalCluster, Client

# 16 cores: 4 processes x 4 threads, memory split evenly
cluster = LocalCluster(n_workers=4, threads_per_worker=4, memory_limit="14GiB",
                       env={"GDAL_CACHEMAX": "512", "GDAL_NUM_THREADS": "1"})
client = Client(cluster)

Threads share memory and are cheap, but only help when the work releases Python’s global interpreter lock. Processes avoid the lock entirely but copy data between them. This page belongs to scaling raster processing with Dask in Cloud Execution & Orchestration.

Who Holds the GIL

What runs in parallel under threads GDAL reads and decompression through rasterio release the GIL, as do most NumPy array operations on large arrays, so they parallelise well with threads. Pure-Python loops, shapely operations on many small geometries, and object-heavy pandas code hold the GIL and serialise under threads, needing processes to scale. GIL behaviour of common raster steps releases the GIL — threads fine rasterio / GDAL reads and writes decompression (DEFLATE, ZSTD) NumPy arithmetic on large arrays reprojection with GDAL warp holds the GIL — needs processes pure-Python loops over pixels per-geometry shapely calls object-dtype pandas work many tiny tasks (scheduler overhead) Most raster pipelines are dominated by the left column — which is why threads work as well as they do.

Rasterio releases the GIL around GDAL calls, and NumPy releases it inside most vectorised operations on arrays larger than a few thousand elements. A pipeline that reads COGs, does band arithmetic and writes results therefore spends most of its time outside the lock, and four threads in one process genuinely run four reads at once. The cost of processes — serialising arrays between them — buys nothing for that work.

Environment & Setup

Package Version pin Used for
dask[distributed] >=2023.1 Local and remote clusters
rasterio >=1.3.0 GDAL-backed reads
xarray >=2023.1 The workload under test 
pip install "dask[distributed]>=2023.1" "rasterio>=1.3.0" "xarray>=2023.1"

Complete Working Example

import os
import time

from dask.distributed import Client, LocalCluster

LAYOUTS = [(1, 16), (4, 4), (16, 1)]        # (processes, threads) on a 16-core machine


def run_layout(n_proc: int, n_thr: int, workload, total_mem_gb: int = 60) -> float:
    env = {
        "GDAL_CACHEMAX": str(max(128, 2048 // n_proc)),     # MB per process
        "GDAL_NUM_THREADS": "1",                           # Dask provides the parallelism
        "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
        "OMP_NUM_THREADS": "1", "MKL_NUM_THREADS": "1", "OPENBLAS_NUM_THREADS": "1",
    }
    with LocalCluster(n_workers=n_proc, threads_per_worker=n_thr,
                      memory_limit=f"{total_mem_gb // n_proc}GiB", env=env) as cluster, \
            Client(cluster) as client:
        client.run(lambda: None)                           # warm up workers
        t0 = time.perf_counter()
        workload(client)
        return time.perf_counter() - t0


def benchmark(workload):
    results = {f"{p}x{t}": run_layout(p, t, workload) for p, t in LAYOUTS}
    for k, v in sorted(results.items(), key=lambda kv: kv[1]):
        print(f"{k:>6}  {v:7.1f} s")
    return results

Pinning GDAL_NUM_THREADS and the BLAS thread variables to one is important. Otherwise each Dask thread may spawn its own pool of GDAL or BLAS threads, and a 4×4 layout silently becomes 4×4×16 threads fighting over sixteen cores. Dask should own the parallelism.

How the Layouts Compare

Hybrid layouts are rarely the worst On a read-heavy COG workload, one process with sixteen threads and four processes with four threads are similarly fast, while sixteen single-threaded processes are slower due to serialisation. On a Python-heavy workload, sixteen processes are fastest and one process is slowest. Four by four is close to best on both, which makes it a sensible default. Runtime by layout (shorter is better) read-heavy Python-heavy 1×16 4×4 16×1

The hybrid layout wins by never being badly wrong. It keeps most of the thread advantage for GDAL work and gives Python-heavy steps four independent interpreters. It also limits the damage of a memory problem: when one process is killed, only a quarter of the in-flight work is lost. On machines with many cores, keep threads per process between two and eight and scale the number of processes.

Memory and the Process Count

Every process carries its own Python interpreter, imported libraries and GDAL block cache — typically 300 to 600 MB before any data. Sixteen processes spend several gigabytes on overhead alone, and each gets one-sixteenth of the memory limit, so a single large task can exceed its worker’s share while the machine as a whole has plenty free. Fewer, larger processes pool memory more flexibly. This is the other reason the hybrid layout is a good default, and it interacts directly with the spill behaviour described in diagnosing Dask memory spills in raster workflows.

When to Move to Processes

Switch the balance towards processes when profiling shows workers mostly busy in Python rather than in GDAL or NumPy — a dashboard profile dominated by Python functions, CPU on each worker capped near 100% of one core while other threads wait. Typical culprits are per-feature zonal statistics, polygonisation followed by shapely operations, or custom per-pixel functions not vectorised with NumPy. Often the better fix is to vectorise the step, for example with apply_ufunc over NumPy code or by moving polygon work to a vectorised library, after which threads work again.

Thread Safety of Open Datasets

Threads share memory, which includes open rasterio datasets — and GDAL dataset handles are not safe to use from several threads at once. Reading the same open file object concurrently from two threads can return corrupted blocks or crash. Libraries such as rioxarray and odc-stac handle this by opening a separate handle per thread or by guarding reads with a lock, but custom code that opens a dataset once at module level and reads it from many Dask tasks will misbehave under threads while working perfectly under processes. The fix is to open datasets inside the task, or keep a per-thread handle with threading.local(). Opening a COG is cheap when GDAL_DISABLE_READDIR_ON_OPEN is set and the header is cached, so the per-task open rarely costs anything measurable.

Free-Threaded Python

Python 3.13 introduced an optional build without the global interpreter lock. For raster pipelines the practical effect is small today — the heavy work already runs outside the lock — but it removes the penalty for the Python-heavy steps in the right-hand column above, and may eventually make single-process, many-thread layouts the simple default. Until the scientific stack’s free-threaded wheels are mature, benchmark before relying on it.

Verification

CPU tells you whether threads are working With four threads doing GDAL reads, a worker process shows close to 400% CPU. With four threads running pure-Python code, it shows close to 100% CPU regardless of thread count, because the GIL lets only one run at a time. Worker CPU with 4 threads GDAL reads — ~380% pure Python — ~100% The Workers tab of the dashboard shows this per process. 
import psutil
from dask.distributed import Client

client = Client(cluster)
fut = client.compute(job)
cpu = client.run(lambda: psutil.Process().cpu_percent(interval=5))
print({w.split(":")[-1]: f"{c:.0f}%" for w, c in cpu.items()})
fut.result()

If each process reports near 100% while configured with several threads, the workload is GIL-bound and more processes will help; near the thread count times 100%, threads are already doing their job.

Common Errors

Adding threads makes the job slower

GDAL or BLAS spawn their own threads per Dask thread. Set GDAL_NUM_THREADS and BLAS thread variables to one.

Workers die although the machine has free memory

Too many processes split the memory limit into small shares. Use fewer processes with more threads.

Reads fail with too many open files

Many processes each open many datasets. Raise the file descriptor limit or reuse opened datasets.

Process layout makes no difference

The job is bound by network throughput, not CPU. Tune read concurrency and chunk size instead.

Frequently Asked Questions

Q: Should Dask use threads or processes for rasterio work? Mostly threads, because GDAL and NumPy release the GIL. A hybrid of a few processes each with several threads is the safest default.

Q: Why set GDAL_NUM_THREADS to 1? So that Dask controls parallelism. Otherwise each Dask thread can start its own GDAL thread pool, oversubscribing the CPU.

Q: How many threads per worker process? Two to eight. Fewer wastes memory on per-process overhead; more concentrates failure and can hit GIL contention on Python-heavy steps.

Q: Does this apply to cloud clusters too? Yes. The same trade-offs hold for Coiled, Kubernetes or Batch workers; choose instance size and layout together.