Diagnosing Slow COG Reads with GDAL VSI Logging

Turn on GDAL’s HTTP logging for one read and count what it actually does:

import rasterio

with rasterio.Env(CPL_DEBUG="ON", CPL_CURL_VERBOSE="YES"):
    with rasterio.open("https://example-bucket.s3.amazonaws.com/scene.tif") as src:
        src.read(1, window=((0, 512), (0, 512)))
# stderr now lists every request: listings, header reads, tile ranges

Nearly every slow remote read is explained by the log in under a minute: too many requests, requests of the wrong kind, or requests that should not have happened at all. This page belongs to understanding Cloud-Optimized GeoTIFF structure in Core Raster Fundamentals & STAC Mapping.

What a Healthy Read Looks Like

The healthy pattern is the one the COG format was designed to allow: the header lives at the start of the file and fits in one fetch, and the tiles a window needs are near each other and can be fetched in one multi-range request. Anything beyond that is either configuration or file layout.

Environment & Setup

pip install "rasterio>=1.3.0" "rio-cogeo>=5.0"

Complete Working Example

import contextlib
import io
import os
import re
import sys
import time

import rasterio


@contextlib.contextmanager
def capture_stderr_fd():
    """Capture C-level stderr, where GDAL writes its debug output."""
    r, w = os.pipe()
    saved = os.dup(2)
    os.dup2(w, 2)
    buf = io.BytesIO()
    try:
        yield buf
    finally:
        os.dup2(saved, 2)
        os.close(w)
        with os.fdopen(r, "rb") as fh:
            buf.write(fh.read())


def diagnose(url: str, window=((0, 512), (0, 512)), **gdal_opts) -> dict:
    opts = {"CPL_DEBUG": "ON", "CPL_CURL_VERBOSE": "YES"} | gdal_opts
    with capture_stderr_fd() as buf:
        t0 = time.perf_counter()
        with rasterio.Env(**opts):
            with rasterio.open(url) as src:
                src.read(1, window=window)
        elapsed = time.perf_counter() - t0
    log = buf.getvalue().decode(errors="ignore")

    ranges = re.findall(r"Range: bytes=([\d\-,]+)", log)
    return {
        "seconds": round(elapsed, 3),
        "http_requests": len(re.findall(r"^> (GET|HEAD) ", log, re.M)),
        "range_requests": len(ranges),
        "listings": len(re.findall(r"list-type=2|ListBucket|delimiter=", log)),
        "sidecar_probes": len(re.findall(r"\.(ovr|aux\.xml|msk)\b", log)),
    }


if __name__ == "__main__":
    url = "https://example-bucket.s3.amazonaws.com/products/ndvi.tif"
    print("default :", diagnose(url))
    print("tuned   :", diagnose(url, GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
                                CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif",
                                GDAL_INGESTED_BYTES_AT_OPEN="32768",
                                GDAL_HTTP_MERGE_CONSECUTIVE_RANGES="YES"))

Capturing the C-level file descriptor rather than Python’s sys.stderr is necessary because GDAL writes its debug output directly from C and bypasses Python’s stream objects entirely. Running the same read with and without the tuning options side by side turns a vague “it is slow” into a table of which requests disappeared.

Mapping Log Symptoms to Fixes

The last row in that table is the case configuration cannot fix. If the tile requests for a small window are scattered across the whole file, the internal layout is not cloud-optimized — the tiles are not in the order the format expects, or the file is striped rather than tiled. rio cogeo validate will say so, and the fix is to rewrite the file with the tooling in converting a GeoTIFF to a COG with rio-cogeo.

Setting the Fixes Once

Once diagnosed, the fixes belong in the environment of every process that reads the archive, not scattered through code. For a batch job that means the container image; for a tile server, its deployment environment; for a notebook, a rasterio.Env block at the top. The set shown above — disable directory listing, restrict extensions, ingest a larger header, merge consecutive ranges — is safe for essentially every COG archive and worth making the default everywhere.

Two more settings help specific workloads. VSI_CACHE=TRUE with a generous VSI_CACHE_SIZE keeps recently read ranges in memory, which matters when the same file is read repeatedly in small windows. GDAL_HTTP_MULTIPLEX=YES lets several range requests share one HTTP/2 connection, which matters for concurrent reads against one host. The container-level configuration pattern is set out in caching PROJ data and GDAL config in containers.

Verification

before = diagnose(url)
after = diagnose(url, GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
                 CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif",
                 GDAL_HTTP_MERGE_CONSECUTIVE_RANGES="YES")
print(before, after, sep="\n")
assert after["listings"] == 0
assert after["http_requests"] <= 4, "still too many requests — check the file layout"

Measuring the same read before and after is the only way to know the change helped; anecdotal “it feels faster” is unreliable against the natural variance of network latency. Run each measurement a few times and compare medians when the difference is small.

Common Errors

No debug output appears

It went to C-level stderr, which Python redirection does not capture. Capture the file descriptor as in the example, or run the script from a shell and redirect 2>.

Requests are fine but reads are still slow

Latency, not request count, dominates: the compute is far from the bucket. Run the read in the storage region.

Every tile is a separate request that cannot merge

The file is not a proper COG. Validate it and rewrite if needed.

Logging makes the job much slower

Verbose output is expensive. Enable it for one diagnostic read only.

The first read is slow and later ones are fast

That is expected: the first open fetches the header and the later ones hit GDAL’s cache. Diagnose the first read specifically, since it is what every new worker and every cold tile request pays, and raise GDAL_INGESTED_BYTES_AT_OPEN if the header needs several round trips.

Reads slow down when many run concurrently

Each process opens its own connections and repeats its own listings. Share the fixes through the environment rather than per-script, and enable HTTP multiplexing so concurrent reads from one process reuse connections.

Frequently Asked Questions

Q: How many requests should a small COG window read take? One or two for the header on first open, then roughly one per internal tile the window touches — often merged into a single multi-range request. A small window read costing dozens of requests is a configuration or file-layout problem, not a network problem.

Q: What is the most common cause of slow opens? GDAL listing the directory to look for sidecar files. On object storage a listing can take seconds and runs on every open. Setting GDAL_DISABLE_READDIR_ON_OPEN to EMPTY_DIR removes it.

Q: Should verbose logging stay on in production? No. It prints every HTTP exchange and slows reads measurably. Turn it on for a single diagnostic read, capture the output, and turn it off again.

Q: Does the same approach work for Azure and GCS? Yes. The VSI drivers for Azure Blob and Google Cloud Storage use the same curl layer and emit the same debug output, and the listing and extension settings apply identically.

Related

• Understanding Cloud-Optimized GeoTIFF Structure — the parent topic.
• Reading a COG over S3 without Downloading — the read this page diagnoses.
• Choosing Block Size and Overview Levels for COGs — the layout decisions the log reveals.
• Deploying TiTiler for a COG Archive — the same settings in a tile service.