Caching PROJ Data and GDAL Config in Containers

A handful of environment variables decide how many requests a remote read costs and whether coordinate transformations are reproducible. Set them once, in the image:

ENV GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
    GDAL_INGESTED_BYTES_AT_OPEN=65536 \
    GDAL_HTTP_MULTIRANGE=YES \
    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \
    GDAL_CACHEMAX=512 \
    VSI_CACHE=TRUE \
    VSI_CACHE_SIZE=50000000 \
    PROJ_NETWORK=OFF

These are the runtime half of Containerizing Geospatial Python Environments — the build decides what is installed, this decides how it behaves.

Why This Arises in Remote Sensing Workflows

GDAL’s defaults were chosen for local files. On a local disk, listing a directory is free, reading a few extra kilobytes is free, and caching is a minor optimisation. Against object storage every one of those assumptions is wrong: a listing is a request, extra bytes are billed egress, and a cache miss is a network round trip.

The result is that an unconfigured container reads remote data several times slower and more expensively than a configured one, with identical code. The difference is not subtle — a sibling listing on every open can double the request count for a job that touches ten thousand files, and an over-sized block cache in a many-worker container looks exactly like a memory leak.

Because these are environment variables, they can be set anywhere: in the shell, in the job definition, in a rasterio.Env block. Setting them in the image is what makes them consistent, so a pipeline behaves the same whether it is launched by a scheduler, a notebook or a developer.

What the configuration removes from every open With defaults, opening a remote COG costs a directory listing, a small header read and a second read for the rest of the header. With sibling listing disabled and the ingest sized to the header, the same open costs a single request, and the saving multiplies by the number of files the job touches. Cost of one rasterio.open against object storage defaults LIST prefix GET 16 KB GET rest of IFD 3 requests · ~180 ms · × 10,000 files = 30,000 requests configured GET 64 KB 1 request · ~55 ms · × 10,000 = 10,000 requests GDAL_DISABLE_READDIR_ON_OPEN removes the listing. GDAL_INGESTED_BYTES_AT_OPEN sized to the header merges the two GETs into one. Neither changes a single pixel — only how the bytes are fetched.

Environment & Setup

Setting Typical What it changes
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR Removes the sibling listing on every open
GDAL_INGESTED_BYTES_AT_OPEN 32768–65536 Header arrives in the first response
GDAL_HTTP_MULTIRANGE YES Allows several byte ranges per request
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES YES Combines adjacent tile ranges
GDAL_CACHEMAX 256–512 (MB) Per-process block cache
VSI_CACHE / VSI_CACHE_SIZE TRUE / 50 MB Per-file read cache for repeated access
PROJ_NETWORK OFF Prevents runtime grid downloads
PROJ_DATA bundled path Where proj.db and grids are found
CPL_TMPDIR /tmp Where GDAL writes temporary files 
pip install "rasterio>=1.3.0" "pyproj>=3.4"

Complete Working Example

Rather than scattering these across a codebase, define them once and apply them from a single place — the image for defaults, and a helper for anything that depends on the worker.

Which settings belong in the image and which per job Read-path settings and PROJ placement are properties of the pipeline and belong in the image. Cache sizes and thread counts depend on the container the job runs in, so they belong in the job definition or in code that reads the worker size. Image-level versus job-level settings setting set where why GDAL_DISABLE_READDIR_ON_OPEN image always true for remote reads GDAL_INGESTED_BYTES_AT_OPEN image a property of the file layout PROJ_NETWORK / PROJ_DATA image reproducibility of transformations GDAL_CACHEMAX job or code depends on worker memory and count GDAL_NUM_THREADS job or code depends on the container CPU allocation The bottom two vary between deployments of the same image — hard-coding them is the usual mistake. 
"""gdal_env.py — one definition of how this pipeline talks to object storage."""
import os

import rasterio

# Always true for this pipeline, regardless of where it runs
STATIC = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "GDAL_INGESTED_BYTES_AT_OPEN": "65536",
    "GDAL_HTTP_MULTIRANGE": "YES",
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
    "GDAL_HTTP_MAX_RETRY": "5",
    "GDAL_HTTP_RETRY_DELAY": "1",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.jp2",
    "PROJ_NETWORK": "OFF",
}


def worker_env(memory_mb: int, threads: int = 1) -> dict:
    """Settings that depend on the container's size, not on the pipeline."""
    # The block cache is PER PROCESS: divide the container's memory by the worker count
    per_worker = max(64, int(memory_mb * 0.15 / max(threads, 1)))
    return STATIC | {
        "GDAL_CACHEMAX": str(per_worker),
        "VSI_CACHE": "TRUE",
        "VSI_CACHE_SIZE": str(min(100_000_000, per_worker * 100_000)),
        "GDAL_NUM_THREADS": str(threads),
    }


def open_remote(uri: str, **overrides):
    """Open a remote raster with the pipeline's standard configuration."""
    cfg = worker_env(
        memory_mb=int(os.environ.get("WORKER_MEMORY_MB", "2048")),
        threads=int(os.environ.get("WORKER_THREADS", "1")),
    ) | overrides
    env = rasterio.Env(**cfg)
    env.__enter__()          # caller closes via the returned context manager
    return env, rasterio.open(uri)


if __name__ == "__main__":
    import time

    env, src = open_remote("https://example-bucket.s3.amazonaws.com/scene_cog.tif")
    try:
        start = time.perf_counter()
        print("tiled:", src.profile.get("tiled"), "blocks:", src.block_shapes[0])
        print("overviews:", src.overviews(1))
        print(f"metadata read in {time.perf_counter() - start:.3f}s")
    finally:
        src.close()
        env.__exit__(None, None, None)

The worker_env function is the part worth copying. GDAL_CACHEMAX being per-process is the single most misunderstood setting in the list: a container with eight worker processes and GDAL_CACHEMAX=1024 is asking for eight gigabytes of cache, and the resulting pressure surfaces as workers being killed rather than as anything mentioning GDAL — the profile described in Diagnosing Dask Memory Spills in Raster Workflows.

Variant Patterns

1. PROJ data placement

Where PROJ looks for proj.db PROJ searches the PROJ_DATA environment variable, then the pyproj package's bundled directory, then system locations. A container that sets PROJ_DATA to a path it does not ship fails at the first transformation; one that leaves it unset and relies on the wheel's bundled copy is usually correct. Search order, and what a container should do 1. $PROJ_DATA explicit, wins if set 2. pyproj bundled data ships with the wheel 3. system paths /usr/share/proj Wheel-based image: leave PROJ_DATA unset and let step 2 win — it is pinned by the wheel. System-GDAL image: set PROJ_DATA to the system path so both stacks agree on one database. Setting PROJ_DATA to a path the image does not contain is the most common cause of "Cannot find proj.db". 
import pyproj
print(pyproj.datadir.get_data_dir())      # what PROJ will actually use

Printing this in the build smoke test converts a whole class of runtime failure into a build failure.

2. Retries and timeouts for flaky storage

ENV GDAL_HTTP_MAX_RETRY=5 \
    GDAL_HTTP_RETRY_DELAY=1 \
    GDAL_HTTP_TIMEOUT=60 \
    GDAL_HTTP_CONNECTTIMEOUT=15

GDAL’s own retries handle transient 5xx responses below the Python layer, which means a task-level retry policy only sees failures that survived several attempts. That composition is deliberate: keep GDAL’s retries short and few, and let the orchestration layer handle the rest with backoff, as set out in Retrying Failed Raster Tasks in a Prefect Pipeline.

3. Restricting what GDAL will probe

CPL_VSIL_CURL_ALLOWED_EXTENSIONS tells GDAL not to probe for sidecar files it will never find. On archives with many auxiliary files this removes several requests per open.

ENV CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif,.TIF,.tiff,.jp2,.vrt"

Set it to the extensions your pipeline actually reads. Too narrow a list makes a legitimate sidecar invisible, which surfaces as a missing mask rather than as an error.

Verifying the Configuration

Two checks, both runnable inside the image.

Count the requests on an open. Enable CPL_CURL_VERBOSE and read the log: a configured container should show one or two requests to open a COG and read its profile. Three or more means the listing is still happening or the ingest size is too small.

Time a windowed read on a cold container. Compare against the same read with the defaults; a factor of two to three is normal for a small window, and no difference at all means the environment variables are not reaching GDAL — usually because they were set after the process started or in a subshell that the worker does not inherit.

It is worth asserting the important ones at startup, because a missing variable is silent:

import os

REQUIRED = {"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR", "PROJ_NETWORK": "OFF"}
missing = {k: v for k, v in REQUIRED.items() if os.environ.get(k) != v}
if missing:
    raise SystemExit(f"container is misconfigured: {missing}")

Common Errors

Workers are killed and the pipeline looks like it leaks memory

GDAL_CACHEMAX is being applied per process while sized for the whole container. Divide by the worker count, and remember that threads inside one process share a cache while processes do not.

PROJ: Cannot find proj.db

PROJ_DATA points somewhere the image does not contain — often left over from a multi-stage build that copied site-packages without the data directory. Unset it and let pyproj’s bundled copy be found.

Reads are still slow after setting the variables

They are set in the wrong place: variables exported in a shell that the worker does not inherit, or set after GDAL has initialised. Put them in the image’s ENV, or pass them through rasterio.Env at the call site.

Frequently Asked Questions

Q: Which single setting matters most? GDAL_DISABLE_READDIR_ON_OPEN set to EMPTY_DIR. Without it GDAL lists the containing prefix on every open, which on object storage adds a request — and sometimes a very slow one — to every file the job touches.

Q: How large should GDAL_CACHEMAX be? A few hundred megabytes per process, not per machine. The cache is per-process, so eight workers with a 1 GB cache each will try to use 8 GB, and the resulting pressure looks like a memory leak in the pipeline.

Q: Should these be set in the image or per job? In the image for anything that is always true — the read-path settings and PROJ placement. Per job for anything that depends on the worker’s memory or the storage backend, since those vary between deployments of the same image.