Choosing Chunk Shapes for Zarr Raster Cubes

Shape the chunks for the read you do most, and size them to tens of megabytes:

bytes_per_px = 2                                # scaled int16

# Maps dominate: one date per chunk, big spatial tile
map_chunks = {"time": 1, "y": 2048, "x": 2048}             # 8 MB

# Time series dominate: whole time axis, small spatial tile
series_chunks = {"time": 365, "y": 128, "x": 128}          # 12 MB

The same cube can be ten times faster or ten times slower depending on this one decision. This page belongs to working with Zarr and cloud-native datacubes in Core Raster Fundamentals & STAC Mapping.

Two Reads, Two Shapes

The same reads against two chunkings A cube chunked as thin time slices serves a single-date map by reading a handful of chunks, but serves a pixel's full time series by reading one chunk per date. A cube chunked as tall columns serves the time series with one chunk but a map with many. Neither shape is universally better; the access pattern decides. map-shaped: time chunk 1 series-shaped: whole time axis map of one date: 1 chunk row pixel series: 7 chunks, one per date pixel series: 1 chunk map of one date: 5 chunks, most of each unused Dark blue: the chunk the highlighted read is centred on. Red outline: what the other read has to touch.

The asymmetry is worst at the extremes. A pixel time series over 300 dates against map-shaped chunks is 300 requests; a single map read against series-shaped chunks fetches the full time depth of every spatial chunk it touches and throws away all but one date. Both are correct and both are slow.

Environment & Setup

Package Version pin Used for
xarray >=2024.1 Inspecting and rechunking the cube
zarr >=2.17 Reading the stored chunk encoding
dask[array] >=2024.1 Executing the rechunk lazily
rechunker >=0.5 Memory-bounded rechunking of large stores 
pip install "xarray>=2024.1" "zarr>=2.17" "dask[array]>=2024.1" "rechunker>=0.5"

Complete Working Example

import math


def chunk_report(shape: dict[str, int], chunks: dict[str, int],
                 bytes_per_px: int = 2) -> dict[str, float]:
    """Size, object count and requests-per-read for a candidate chunk shape."""
    c = {d: (shape[d] if chunks[d] == -1 else chunks[d]) for d in shape}
    chunk_mb = math.prod(c.values()) * bytes_per_px / 1e6
    n_objects = math.prod(math.ceil(shape[d] / c[d]) for d in shape)

    # A map read: one date, a 2048x2048 window
    map_reads = math.ceil(2048 / c["y"]) * math.ceil(2048 / c["x"])
    # A series read: one pixel, all dates
    series_reads = math.ceil(shape["time"] / c["time"])
    return {
        "chunk_mb": round(chunk_mb, 1),
        "objects": n_objects,
        "map_requests": map_reads,
        "series_requests": series_reads,
    }


if __name__ == "__main__":
    shape = {"time": 365, "y": 10_980, "x": 10_980}
    for name, chunks in {
        "map-shaped":    {"time": 1,   "y": 2048, "x": 2048},
        "series-shaped": {"time": -1,  "y": 128,  "x": 128},
        "balanced":      {"time": 32,  "y": 512,  "x": 512},
    }.items():
        print(f"{name:<14}", chunk_report(shape, chunks))

Running a report like this before creating a store takes seconds and replaces a debate with numbers. For a year of daily Sentinel-2 over one tile, the map-shaped option gives an 8 MB chunk, about 13,000 objects, 1 request per map read and 365 per series; the series-shaped option gives a 12 MB chunk, about 7,400 objects, 256 per map and 1 per series. The balanced option lands in between on both.

Rechunking When Both Patterns Matter

Ingest store and analysis store New dates are appended to an ingest store chunked one date at a time, which keeps appends cheap and suits map reads. On a schedule the ingest store is rechunked into an analysis store with the full time axis per chunk, which suits time-series work. Readers pick the store that matches their access pattern. Two stores, each shaped for its readers new acquisitions daily, from the catalog ingest store time chunk 1 cheap appends, map reads analysis store whole time axis per chunk rebuilt weekly rechunk Storage doubles; every read becomes a fast read. For most teams that is a good trade. 
import xarray as xr
from rechunker import rechunk

source = xr.open_zarr("s3://example-bucket/cubes/ndvi_ingest.zarr", consolidated=True)

plan = rechunk(
    source,
    target_chunks={"ndvi": {"time": -1, "y": 128, "x": 128},
                   "time": None, "x": None, "y": None},
    max_mem="2GB",
    target_store="s3://example-bucket/cubes/ndvi_analysis.zarr",
    temp_store="s3://example-bucket/cubes/_rechunk_tmp.zarr",
)
plan.execute()

The max_mem bound is the reason to use a dedicated rechunker rather than a plain .chunk().to_zarr(). Converting map-shaped chunks to series-shaped ones requires, in the naive approach, holding every date of a spatial tile in memory at once — for a large cube that exceeds any single worker. The rechunker stages the transpose through a temporary store so peak memory stays bounded.

Why Not Just Pick the Balanced Shape

The balanced compromise is tempting because it avoids a decision, and for a small cube or an exploratory project it is fine. For anything that will be read heavily, it has a hidden cost: both access patterns pay a moderate penalty on every read, forever. A time series from balanced chunks of 32 dates is twelve requests instead of one; a map from them fetches 32 dates per chunk and discards 31.

Over thousands of reads by a team, that overhead exceeds the storage cost of a second, correctly shaped copy many times over. The general principle — pay once in storage to save repeatedly in reads — is the same one behind overviews in a COG, and the cost model in benchmarking COG read throughput from object storage applies with little modification.

Verification

Measure the reads you care about After creating the store, time one representative map read and one representative time-series read. The dominant pattern should complete in a small number of requests and well under a second from the same region; if not, the chunk shape does not match the access pattern. Two timings decide whether the shape is right series read (dominant) 0.2 s · 1 request map read (occasional) 2.6 s · 256 requests Acceptable if maps really are occasional; if not, keep a second, map-shaped copy. 
import time

import xarray as xr

cube = xr.open_zarr(store, consolidated=True)["ndvi"]
print("stored chunks:", cube.encoding["chunks"])

t0 = time.perf_counter()
cube.isel(y=5000, x=5000).load()
print(f"series read: {time.perf_counter() - t0:.2f} s")

t0 = time.perf_counter()
cube.isel(time=-1, y=slice(4000, 6048), x=slice(4000, 6048)).load()
print(f"map read:    {time.perf_counter() - t0:.2f} s")

The dominant read should be fast from the same region as the storage — well under a second. If it is not, the shape does not match the workload, and it is far cheaper to find out now than after the store has grown to terabytes.

Common Errors

A small read fetches hundreds of megabytes

The chunks are far larger than the reads. Shrink the spatial chunk or split the time axis. Check the stored shape with the encoding’s chunks entry before assuming the reads are at fault.

Opening or listing the store is very slow

There are millions of tiny chunks. Increase chunk size toward the 10–100 MB range.

Rechunking runs out of memory

A naive .chunk().to_zarr() is transposing the whole cube in memory. Use a staged rechunker with a memory bound.

Workers are killed during computation

Chunks are too large for per-task memory once decompressed and processed. Aim nearer 10–30 MB per chunk for heavy computations. Remember that intermediate arrays in a computation are often several times the chunk size.

Frequently Asked Questions

Q: What chunk size should I aim for? Roughly 10 to 100 megabytes uncompressed. Below that, request overhead dominates and stores accumulate millions of objects; above it, reads fetch far more than they need and workers need large amounts of memory per task.

Q: Can one chunk shape serve both maps and time series? A balanced shape serves both moderately and neither well. When both access patterns are frequent, two copies of the cube with different chunking usually cost less in storage than the extra reads a compromise shape imposes.

Q: How expensive is rechunking later? It reads and rewrites the entire store, and naive approaches can need memory proportional to the whole cube. Dedicated rechunking tools stage the operation through intermediate storage; either way, it is far cheaper to choose correctly before the store grows.

Q: Should coordinate arrays be chunked too? Keep coordinates in a single chunk each. They are tiny, every read needs them, and splitting them turns every open into several extra requests for no benefit.