Reading Zarr Datacubes from S3 with fsspec

Open the store with an s3:// URL, select lazily, and load only what you need:

import xarray as xr

cube = xr.open_zarr("s3://example-bucket/cubes/ndvi.zarr",
                    consolidated=True,
                    storage_options={"anon": False, "profile": "research"})
summer = cube.ndvi.sel(time=slice("2026-06-01", "2026-08-31")).median("time").compute()

Nothing is downloaded until .compute(), and then only the chunks the selection touches. This page belongs to working with Zarr and cloud-native datacubes in Core Raster Fundamentals & STAC Mapping.

The Layers Between Python and the Bucket

Who does what on a remote read User code asks xarray for a selection. xarray and Dask work out which Zarr chunks are needed. Zarr asks its store for those keys. fsspec's S3 implementation turns each key into an HTTP request against the bucket, applying credentials and optional caching. Each layer can be configured independently. From a selection to HTTP requests your code: cube.sel(...).median().compute() xarray + Dask: which chunks does this need? zarr: fetch keys ndvi/12.3.4, ndvi/12.3.5, … fsspec / s3fs: credentials, retries, optional cache object storage: one GET per chunk storage_options configure this layer

storage_options is passed through to the fsspec layer unchanged, which is why the same argument handles credentials, anonymous access, custom endpoints for S3-compatible stores and request timeouts. Understanding that separation makes most connection problems easy to place: an authentication error lives in the bottom two layers, a shape or dtype error in the middle ones.

Environment & Setup

Package Version pin Used for
xarray >=2024.1 Opening and selecting from the cube
zarr >=2.17 The store format
fsspec >=2024.2 The filesystem abstraction and caching wrappers
s3fs >=2024.2 The S3 implementation behind s3:// URLs
dask[array] >=2024.1 Lazy, parallel chunk reads 
pip install "xarray>=2024.1" "zarr>=2.17" "fsspec>=2024.2" "s3fs>=2024.2" "dask[array]>=2024.1"

Complete Working Example

import fsspec
import xarray as xr


def open_cube(url: str, *, anon: bool = False, profile: str | None = None,
              endpoint_url: str | None = None, cache_dir: str | None = None) -> xr.Dataset:
    """Open a remote Zarr cube with optional local chunk caching."""
    so: dict = {"anon": anon}
    if profile:
        so["profile"] = profile
    if endpoint_url:                                  # S3-compatible stores
        so["client_kwargs"] = {"endpoint_url": endpoint_url}

    if cache_dir:
        # 'filecache' keeps fetched objects on local disk across sessions
        mapper = fsspec.get_mapper(
            f"filecache::{url}",
            s3=so,
            filecache={"cache_storage": cache_dir},
        )
        return xr.open_zarr(mapper, consolidated=True)

    return xr.open_zarr(url, consolidated=True, storage_options=so)


if __name__ == "__main__":
    cube = open_cube("s3://example-bucket/cubes/ndvi.zarr", profile="research",
                     cache_dir="/tmp/zarr-cache")
    print(cube)                                        # metadata only, no chunk reads

    point = cube.ndvi.sel(x=780_050, y=9_950_050, method="nearest")
    series = point.load()                              # fetches only the needed chunks
    print(series.to_series().describe())

The filecache:: prefix is fsspec’s chaining syntax: it wraps the S3 filesystem in a local disk cache, so a chunk fetched once is served from disk thereafter. For interactive work that revisits the same area repeatedly — tuning a model on one region, say — it turns the second and later runs from network-bound into disk-bound, which is typically ten times faster.

Where the Time Goes

Consolidated metadata removes the slowest step Opening an unconsolidated store with a dozen variables issues a request per metadata file before any data is read, which can take several seconds cross-region. A consolidated store reads one metadata object. The chunk reads that follow are the same in both cases and are parallel, so after the open the two behave identically. Open plus one small selection unconsolidated 40 metadata requests chunks consolidated chunks one metadata request The chunk reads are identical in both; only the open changes. For a notebook that reopens the store often, the unconsolidated open dominates everything.

After the open, remote reads are governed by two things: how many chunks a selection touches, and how far the compute is from the bucket. The first is a property of the chunk shape, covered in choosing chunk shapes for Zarr raster cubes. The second is a property of where the code runs, and it is the one people most often ignore.

Keep the Compute Next to the Data

A laptop reading a terabyte-scale cube across the internet will work — every chunk arrives eventually — but each request pays tens of milliseconds of latency and every byte is billed as egress. The same computation on a worker in the bucket’s region pays a few milliseconds per request and nothing for transfer.

The pattern that scales is to push the computation to the data and pull back only the reduced result: run the seasonal median or the trend fit on a cluster in the storage region, write the small output, and download that. The execution side of this is the subject of scaling raster processing with Dask, and the cost side of reducing S3 egress costs in raster pipelines.

For interactive exploration from a laptop, the local file cache above is the pragmatic compromise: the first look at an area is slow, every later look is fast, and the cache directory can be deleted when the work moves elsewhere.

Verification

Confirm the open was lazy and complete After opening, the dataset should be backed by Dask arrays so no chunk data has been read, its CRS should be present, and its time coordinate should span the expected dates. A NumPy-backed variable means the whole array was read eagerly, which is the most expensive possible mistake on a remote store. Three checks right after opening Dask-backed nothing read yet CRS present georeferencing survived time span as expected the right store, up to date A NumPy-backed variable after open means the whole array was just downloaded. 
import dask.array as da

cube = open_cube("s3://example-bucket/cubes/ndvi.zarr", profile="research")
assert isinstance(cube.ndvi.data, da.Array), "variable was loaded eagerly"
assert cube.rio.crs is not None, "no CRS on the remote store"
print(cube.time.values.min(), "to", cube.time.values.max(), f"({cube.sizes['time']} dates)")

It is also worth printing the store’s own chunk encoding at this point, since a remote store built by someone else may be shaped for a different access pattern from yours, and knowing that before the first heavy computation saves a slow surprise. The Dask check is the one worth keeping in any shared notebook. xr.open_dataset(engine="zarr") without chunks can load eagerly, and on a remote cube that is a silent multi-gigabyte download that looks, from the outside, like the notebook hanging.

Common Errors

NoCredentialsError on a public bucket

The filesystem is looking for credentials the bucket does not need. Pass storage_options={"anon": True}.

KeyError: '.zmetadata'

The store is not consolidated. Open with consolidated=False, and consolidate it if you can write to it.

Reads hang with no error

The request is being retried against an unreachable endpoint, often a wrong region or custom endpoint. Set the endpoint explicitly and a timeout in client_kwargs.

The cache directory grows without bound

filecache never evicts. Point it at a scratch location and clear it between projects.

Reads work locally but fail on cluster workers

The credentials exist only in the notebook’s environment. Workers need them too — through an instance role, environment variables set on the cluster, or explicit keys in storage_options that travel with the task graph.

Every read re-downloads the same chunks

The cache is wrapped around the wrong layer or keyed by a URL that changes. Use a stable URL and the filecache:: chaining shown above, and confirm the cache directory is actually filling.

Frequently Asked Questions

Q: How do I read a public Zarr store without credentials? Pass storage_options={‘anon’: True}. Without it, the S3 filesystem looks for credentials, fails to find any and raises, even though the bucket needs none.

Q: Why does opening a remote store take so long? Usually because metadata is not consolidated, so the reader fetches one small file per array and coordinate. With consolidated metadata the open is a single request; ask the store’s owner to consolidate or do it yourself if you control it.

Q: Is it expensive to read a large cube from another region? It is slower, because every chunk request pays cross-region latency, and often billed as inter-region transfer. Run the computation in the store’s region and bring back only the reduced result.

Q: Does the same approach work on Azure or Google Cloud Storage? Yes. fsspec provides equivalent implementations for both, addressed with their own URL schemes, and the same storage_options pattern carries their credentials. Only the scheme and the keys inside the options change.