Appending New Time Steps to a Zarr Store

Drop dates already present, match the store’s grid and spatial chunks, then append along time:

import xarray as xr

existing = xr.open_zarr(store, consolidated=True).time.values
new = new.sel(time=~new.time.isin(existing))              # idempotent reruns
if new.sizes["time"]:
    new.chunk({"time": 1, "y": 1024, "x": 1024}) \
       .to_dataset(name="ndvi") \
       .to_zarr(store, append_dim="time", consolidated=True)

The deduplication line is what makes a scheduled job safe to rerun. This page belongs to working with Zarr and cloud-native datacubes in Core Raster Fundamentals & STAC Mapping.

What an Append Touches

Time chunk size decides what an append rewrites With a time chunk of one, appending a date writes only new chunk objects and every existing object is untouched. With a time chunk of eight, the last chunk along time is partly filled, so appending a date rewrites that partial chunk before any new chunk is started, which costs more and briefly exposes readers to a changing object. time chunk = 1 time chunk = 8 new existing objects untouched 8 dates 5 dates + 1 rewritten the partial chunk is read, extended and rewritten For a store updated daily, time chunks of one keep every append cheap and every reader safe.

That choice interacts with how the cube is read. A time chunk of one is ideal for appending and for map reads, and poor for long time series; a larger time chunk is the opposite. Stores that are both appended frequently and read as series often keep two copies — an append-friendly ingest store and a periodically rechunked analysis store — as described in choosing chunk shapes for Zarr raster cubes.

Environment & Setup

Package Version pin Used for
xarray >=2024.1 Reading the time axis and appending
zarr >=2.17 The store
rioxarray >=0.15 Reprojecting new frames onto the store grid
pandas >=2.0 Date handling and deduplication 
pip install "xarray>=2024.1" "zarr>=2.17" "rioxarray>=0.15" "pandas>=2.0"

Complete Working Example

import numpy as np
import pandas as pd
import xarray as xr


def append_dates(store: str, new: xr.DataArray, *, var: str = "ndvi",
                 storage_options: dict | None = None) -> list[str]:
    """Append only the dates not already in the store, aligned and chunked to match."""
    so = storage_options or {}
    current = xr.open_zarr(store, consolidated=True, storage_options=so)

    # 1. Idempotency: drop dates already present
    have = pd.DatetimeIndex(current.time.values)
    new = new.sel(time=~pd.DatetimeIndex(new.time.values).isin(have))
    if new.sizes["time"] == 0:
        return []

    # 2. Ordering: refuse to append anything earlier than the last stored date
    if pd.Timestamp(new.time.values.min()) <= have.max():
        raise ValueError("incoming dates precede the store's last date; rebuild instead")
    new = new.sortby("time")

    # 3. Alignment: exactly the store's grid
    template = current[var].isel(time=0)
    if not (np.array_equal(new.x, template.x) and np.array_equal(new.y, template.y)):
        new = new.rio.reproject_match(template.rio.write_crs(current.rio.crs))

    # 4. Chunking: spatial chunks must match the store exactly
    enc_chunks = current[var].encoding["chunks"]
    chunk_map = dict(zip(current[var].dims, enc_chunks))
    ds = new.to_dataset(name=var).chunk(chunk_map)

    # Encoding is inherited from the store on append; do not pass it again
    ds.drop_vars("spatial_ref", errors="ignore").to_zarr(
        store, append_dim="time", consolidated=True, storage_options=so)

    return [str(pd.Timestamp(t).date()) for t in new.time.values]

Reading the chunk shape from the store’s own encoding rather than hard-coding it is what keeps this function correct if the store was created with different chunking. Not passing an encoding on append is equally important: xarray reuses the store’s scale factor, dtype and compressor, and passing a conflicting encoding raises — which is the right outcome, but a confusing one.

Scheduling Appends Safely

The typical deployment is a scheduled job that searches a catalog for acquisitions since the last run, builds frames for them, and appends. Three properties make that job robust.

It must be idempotent, which the deduplication step provides: a rerun after a partial failure appends only what was not written the first time. It must be serial per store, because two concurrent appends race on the metadata that records the time axis length, and the loser’s data is either lost or duplicated. And it should fail loudly rather than silently when a date arrives out of order, since quietly appending it produces an unsorted axis that breaks every later sel(time=slice(...)).

A workflow orchestrator is the natural place to enforce the second property, with a concurrency limit of one on the append task; the setup is covered in scheduling Sentinel-2 downloads with Prefect flows. Late-arriving acquisitions — reprocessed scenes, delayed deliveries — are better handled by a periodic rebuild of the affected period than by trying to insert into the middle of the axis.

When to Rebuild Instead

Append, or rebuild the period New dates later than the last stored date on the same grid are appended. Dates earlier than the last stored date, a change of grid, or a change of processing baseline for existing dates all call for rebuilding the affected period into a new store and switching readers to it. Most updates append; a few must rebuild append new dates after the last stored date same grid, same processing cheap, incremental, safe to rerun the daily case rebuild the period late or out-of-order dates reprocessed existing dates any change of grid or encoding write new, then switch readers Trying to patch the middle of an axis in place is where Zarr stores get corrupted.

Reprocessing is the case that catches people. When the provider reissues old scenes under a new processing baseline, the existing time steps are not wrong in the sense of being corrupt, but they are inconsistent with anything appended afterwards. The honest answer is to rebuild the affected period with the new baseline, write it to a new store, and switch readers — the same versioning discipline applied to published products elsewhere on this site.

Verification

Three properties of a healthy time axis After every append the time coordinate must be strictly increasing, contain no duplicates, and have grown by exactly the number of dates the job reported appending. Any one failing indicates a concurrency or ordering problem that will break downstream slicing. Check the axis, every time strictly increasing no out-of-order dates no duplicates reruns were harmless length grew as reported no lost or doubled writes These three catch every concurrency and ordering bug an append job can have. 
import numpy as np
import xarray as xr

before = xr.open_zarr(store, consolidated=True).sizes["time"]
added = append_dates(store, new_frames)
after_ds = xr.open_zarr(store, consolidated=True)
t = after_ds.time.values

assert after_ds.sizes["time"] == before + len(added), "axis length does not match"
assert np.all(np.diff(t) > np.timedelta64(0, "s")), "time axis not strictly increasing"
assert len(np.unique(t)) == len(t), "duplicate dates"

Running these after every append in the scheduled job costs one metadata read and catches every concurrency and ordering problem the job can have. It is far cheaper than discovering a duplicated month when a monthly composite comes out twice as bright as its neighbours.

Common Errors

ValueError about encoding conflicts on append

An encoding was passed to the append call. Omit it; the store’s encoding is reused automatically.

Duplicate dates in the store

The job was rerun without deduplication. Filter incoming dates against the existing axis first.

Appended data is misaligned by a pixel

The new frames were on a slightly different grid. Reproject onto the store’s grid before appending.

Readers see fill values for the newest date

They opened the store mid-append. Consolidate metadata only after the chunk writes complete, which to_zarr does by default.

Frequently Asked Questions

Q: What happens if I append a date that is already in the store? It is appended again, creating a duplicate time step. Zarr does not deduplicate. Always read the existing time coordinate first and drop incoming dates already present, so a rerun of the job is harmless.

Q: Can I append a date that is earlier than the last one stored? Mechanically yes, and the time axis then stops being sorted, which breaks every slice and resample that assumes order. Either insert late arrivals into a rebuilt store or sort on read, and prefer to prevent it by appending in date order.

Q: Does appending rewrite existing chunks? Only the chunks that straddle the old end of the time axis, if the time chunk size is larger than one. With a time chunk of one, an append writes only new chunk objects and updates the metadata.

Q: Can I append a new variable rather than new dates? Yes — write the new variable with mode="a" and no append_dim, on exactly the same coordinates. That adds an array alongside the existing ones without touching them, which is how derived layers such as a cloud flag are added to an established cube.