Reading COGs from Azure Blob and GCS

The read code is identical across providers; only the URL prefix and the credentials change:

import rasterio

# Azure Blob Storage
with rasterio.Env(AZURE_STORAGE_ACCOUNT="myaccount", AZURE_STORAGE_SAS_TOKEN=sas):
    with rasterio.open("/vsiaz/mycontainer/scenes/T36MYF_B04.tif") as src:
        print(src.profile["tiled"], src.block_shapes[0], src.overviews(1))

# Google Cloud Storage
with rasterio.Env(GS_NO_SIGN_REQUEST="YES"):          # public bucket
    with rasterio.open("/vsigs/mybucket/scenes/T36MYF_B04.tif") as src:
        window = rasterio.windows.Window(2048, 2048, 512, 512)
        arr = src.read(1, window=window)

Everything about the byte layout described in Understanding Cloud-Optimized GeoTIFF Structure applies unchanged — only the transport differs.

Why This Arises in Remote Sensing Workflows

Public satellite archives are not on one cloud. Sentinel and Landsat collections are mirrored across providers, national agencies publish to whichever platform they procured, and a project that starts on one provider frequently acquires a second when a collaborator brings their own data. A pipeline that only knows how to read s3:// paths becomes the reason a dataset cannot be used.

The good news is that GDAL’s virtual file system abstracts all of this: /vsis3/, /vsiaz/, /vsigs/ and /vsicurl/ all present the same interface, and rasterio sees a file. Windowed reads, overview selection and header-only opens work identically, so the analysis code needs no provider awareness at all.

What does differ is credentials and, more importantly, cost geometry. Reading an Azure-hosted archive from a Google region pays egress on one side and ingress latency on the other, and no amount of COG structure compensates for a cross-cloud read path.

One API, three transports Application code calls rasterio.open with a path. GDAL's virtual file system layer routes /vsis3/, /vsiaz/ and /vsigs/ prefixes to the matching provider, handling authentication and HTTP range requests. The window read, the overview selection and the profile access are identical in all three cases. The provider is a URL prefix, not a code path your code rasterio.open(path) GDAL VSI layer auth + range requests + block cache /vsis3/ — Amazon S3 AWS_* config, IAM roles, requester pays /vsiaz/ — Azure Blob account + key, SAS token, or managed identity /vsigs/ — Google Cloud Storage service account JSON, or no-sign for public Anything above the VSI layer — windows, overviews, masks — is written once and runs everywhere.

Environment & Setup

Package Version Why
rasterio ≥1.3.0 Env configuration and the VSI-backed reads
GDAL ≥3.4.0 /vsiaz/ and /vsigs/ drivers
planetary-computer ≥1.0 Only if you need SAS signing for that catalogue
pystac-client ≥0.7 Resolving asset URLs from a catalogue 
pip install "rasterio>=1.3.0" "pystac-client>=0.7"

Complete Working Example

This helper resolves a provider-agnostic URI into a VSI path, applies the right GDAL configuration, and reads a window — the shape a pipeline needs when its inputs come from several archives.

Credential mechanisms per provider Each provider offers an ambient credential mechanism that avoids putting secrets in code or images: instance roles on AWS, managed identity on Azure, and application default credentials on Google. Anonymous access is available for public data on all three. How to authenticate without secrets in the image provider ambient mechanism anonymous option S3 (/vsis3/) instance role or AWS_PROFILE AWS_NO_SIGN_REQUEST=YES Azure (/vsiaz/) managed identity or CLI session AZURE_NO_SIGN_REQUEST=YES GCS (/vsigs/) application default credentials GS_NO_SIGN_REQUEST=YES HTTPS (/vsicurl/) a signed URL, resolved at read time public URL, no config Prefer the middle column: a credential in an image layer is a credential in every copy of it. 
from urllib.parse import urlparse

import rasterio
from rasterio.windows import Window

# Settings that matter on every provider: no sibling listing, header in one request
COMMON = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "GDAL_INGESTED_BYTES_AT_OPEN": "65536",
    "GDAL_HTTP_MULTIRANGE": "YES",
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": "50000000",
}


def vsi_path(uri: str) -> str:
    """Map an s3://, az:// or gs:// URI onto the matching GDAL VSI path."""
    parsed = urlparse(uri)
    scheme, container, key = parsed.scheme, parsed.netloc, parsed.path.lstrip("/")
    prefixes = {"s3": "/vsis3/", "az": "/vsiaz/", "abfs": "/vsiaz/",
                "gs": "/vsigs/", "https": "/vsicurl/"}
    if scheme == "https":
        return f"/vsicurl/{uri}"
    if scheme not in prefixes:
        return uri                      # a local path
    return f"{prefixes[scheme]}{container}/{key}"


def provider_config(uri: str, *, sas_token: str | None = None,
                    azure_account: str | None = None) -> dict:
    """GDAL options for the provider behind `uri`."""
    scheme = urlparse(uri).scheme
    cfg = dict(COMMON)
    if scheme in ("az", "abfs"):
        cfg["AZURE_STORAGE_ACCOUNT"] = azure_account or ""
        if sas_token:
            cfg["AZURE_STORAGE_SAS_TOKEN"] = sas_token.lstrip("?")
    elif scheme == "gs":
        cfg["GS_NO_SIGN_REQUEST"] = "YES" if sas_token is None else "NO"
    elif scheme == "s3":
        cfg.setdefault("AWS_NO_SIGN_REQUEST", "YES")
    return cfg


def read_window(uri: str, window: Window, *, band: int = 1, **creds):
    """Read one window from any supported provider with sane HTTP settings."""
    with rasterio.Env(**provider_config(uri, **creds)):
        with rasterio.open(vsi_path(uri)) as src:
            if not src.profile.get("tiled"):
                raise ValueError(f"{uri} is not tiled — windowed reads will be expensive")
            return src.read(band, window=window, masked=True)


if __name__ == "__main__":
    arr = read_window("gs://gcp-public-data-landsat/…/LC08_B4.TIF", Window(2048, 2048, 512, 512))
    print(arr.shape, arr.dtype, arr.mask.mean())

Two of the common settings do most of the work. GDAL_DISABLE_READDIR_ON_OPEN stops GDAL listing the containing prefix on every open — the single largest source of surprise requests on all three providers. GDAL_INGESTED_BYTES_AT_OPEN sized to the header turns a two-request open into a one-request open, and both are explained further in How to Read COG Headers Without Downloading Full Files.

Variant Patterns

1. Short-lived signed URLs

Several catalogues hand out assets that must be signed, with tokens valid for minutes. The rule that follows is not about signing but about when you sign.

import planetary_computer as pc
import pystac_client
import rasterio

catalog = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
item = next(catalog.search(collections=["sentinel-2-l2a"], max_items=1).items())

# Sign at read time, not at search time: a URL captured into a manifest will have expired
href = pc.sign(item.assets["B04"].href)
with rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open(href) as src:
        print(src.profile["compress"], src.block_shapes[0])

A pipeline that stores item identifiers and re-resolves hrefs at read time survives token expiry, storage reorganisation and asset renaming; one that stores signed URLs breaks on all three.

2. Credentials without secrets in code

Every provider supports ambient credentials, and using them is both safer and simpler than threading secrets through function arguments.

import rasterio

# Azure: managed identity or the az CLI session, no key in the process
with rasterio.Env(AZURE_STORAGE_ACCOUNT="myaccount", AZURE_NO_SIGN_REQUEST="NO"):
    with rasterio.open("/vsiaz/container/scene.tif") as src:
        profile = src.profile

# GCS: application default credentials from the environment
with rasterio.Env(CPL_MACHINE_IS_GCE="YES"):
    with rasterio.open("/vsigs/bucket/scene.tif") as src:
        profile = src.profile

Keep the configuration inside a rasterio.Env block rather than setting process-wide environment variables: it scopes the credentials to the read, and it makes concurrent access to two providers straightforward.

3. Cross-cloud reads, and why to avoid them

Where the compute sits relative to the bucket A tile read from compute in the same region as the bucket takes tens of milliseconds and no egress charge. The same read from another region of the same cloud adds latency and per-gigabyte egress. From another cloud entirely it adds both plus a second provider's ingress path, and the COG layout changes none of it. One 512 px tile, three placements same region ~25 ms · no egress charge same cloud, other region ~90 ms · inter-region egress per GB cross-cloud ~180 ms · internet egress, and every retry pays again Structure cannot fix placement: a perfect COG read across clouds is still the slowest row here. If a dataset must be used repeatedly from another cloud, copy it once rather than reading it a thousand times.

Common Errors

RasterioIOError: … does not exist in the file system, and is not recognized as a supported dataset name

Usually a missing VSI prefix — an az:// or gs:// URI passed straight to rasterio.open. Map it to /vsiaz/ or /vsigs/ first, as in vsi_path above.

403 from a URL that worked minutes ago

A SAS or signed URL expired. Store item identifiers, not signed URLs, and sign at read time.

Reads are correct but far slower than on S3

Sibling listing is enabled, or the object is in a different region from the compute. Set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR first, then check the region — the cost of getting that wrong is quantified in Reducing S3 Egress Costs in Raster Pipelines.

Frequently Asked Questions

Q: Do I need different code for each cloud provider? No. Only the URL prefix and the credential configuration differ; every read, window and overview call is identical because GDAL abstracts the storage behind its virtual file system.

Q: Can I read a Planetary Computer asset without a token? Some collections are public, but most assets need a short-lived SAS token appended to the URL. Because the token expires in minutes, resolve it at read time rather than storing signed URLs in a manifest.

Q: Why is the first read much slower than later ones? The first open resolves credentials and may list the containing prefix. Setting GDAL_DISABLE_READDIR_ON_OPEN to EMPTY_DIR removes the listing, and reusing one rasterio.Env keeps the credential resolution warm.