Paginating Large STAC Searches with pystac-client

A STAC search that matches thousands of scenes does not hand them all back in one response — the server returns one page at a time, and pystac-client stops early unless you tell it not to. Set max_items=None and iterate .items(), then check .matched() to prove you got everything:

from pystac_client import Client

catalog = Client.open("https://earth-search.aws.element84.com/v1")

search = catalog.search(
    collections=["sentinel-2-l2a"],
    bbox=[5.0, 45.0, 6.0, 46.0],
    datetime="2023-01-01/2023-12-31",
    max_items=None,        # follow every "next" link, no total cap
)

items = list(search.items())
print(len(items), "of", search.matched(), "matched")

This page belongs to Querying STAC Catalogs Programmatically, and focuses on completeness — getting all the results — rather than narrowing them, which is covered in Using pystac-client to Filter Sentinel-2 Imagery by Date.

Why This Arises in Remote Sensing Workflows

STAC APIs follow the OGC API - Features pagination model: every response carries a links array, and when more results exist, one link has "rel": "next". A client must follow that link to reach the next page. Servers cap page size (commonly 100, 250, or 1000 items) to bound response payloads, so a query over a full year of Sentinel-2 tiles across a region can span dozens of pages.

The failure mode is silent. pystac-client defaults max_items to 100, so a naive list(search.items()) returns exactly 100 items even when 4,000 match — no error, no warning. Downstream, a monthly composite is built from a fraction of the archive, and the gap is invisible until someone audits scene counts. Because the truncation is quiet, it survives code review and reaches production.

Understanding this is a prerequisite for anything that consumes a full catalog query: temporal compositing, gap analysis, or per-tile inventories. Once you have the complete item list, you typically hand asset URLs off to metadata validation — for example How to Read COG Headers Without Downloading Full Files — before committing to any pixel reads. For the broader architectural picture of catalog-driven ingestion, see Core Raster Fundamentals & STAC Mapping.

How pystac-client Walks the Pages

The diagram traces one .items() iteration: the client requests a page, yields its items, follows the next link, and repeats until either the server omits next or max_items is reached.

The key distinction: limit is the page size the server returns per request, while max_items is the total pystac-client will yield before it stops walking next links. Raising limit cuts the number of HTTP round-trips; setting max_items=None removes the ceiling entirely.

Environment & Setup

pip install "pystac-client>=0.7.0"

Confirm connectivity and that the server reports a total count before you rely on it:

from pystac_client import Client

catalog = Client.open("https://earth-search.aws.element84.com/v1")
search = catalog.search(collections=["sentinel-2-l2a"], max_items=5)
print("matched:", search.matched())  # None means the server omits the count

Complete Working Example

The function below collects every matching item, honours a rate limit between page requests, and asserts the collected count against matched() so truncation cannot pass unnoticed.

import time
import logging
from typing import Any

from pystac_client import Client
from pystac import Item

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("stac_paginate")


def fetch_all_items(
    api_url: str,
    collections: list[str],
    bbox: list[float],
    datetime_range: str,
    *,
    page_size: int = 500,
    request_pause_s: float = 0.2,
) -> list[Item]:
    """
    Retrieve every STAC item matching a query, page by page.

    Parameters
    ----------
    api_url : str
        STAC API root (the endpoint that exposes /search).
    collections, bbox, datetime_range :
        Standard STAC search filters.
    page_size : int
        Items per page (the `limit` parameter). Larger pages mean fewer
        HTTP round-trips; most servers cap this near 1000.
    request_pause_s : float
        Seconds to sleep between page fetches to stay under rate limits.

    Returns
    -------
    list[Item]
        Every matching item. Raises if the count disagrees with matched().
    """
    catalog = Client.open(api_url)

    search = catalog.search(
        collections=collections,
        bbox=bbox,
        datetime=datetime_range,
        limit=page_size,     # server-side page size
        max_items=None,      # follow every "next" link — no total cap
    )

    expected = search.matched()  # server's total, or None if unsupported
    log.info("Server reports %s matching items", expected)

    items: list[Item] = []
    # pages() yields one ItemCollection per HTTP response, which lets us
    # pause between requests and log progress page-by-page.
    for page_index, page in enumerate(search.pages(), start=1):
        page_items = list(page)
        items.extend(page_items)
        log.info("page %d: +%d items (running total %d)",
                 page_index, len(page_items), len(items))
        time.sleep(request_pause_s)  # simple client-side rate limiting

    # Completeness gate: never let a dropped page pass silently.
    if expected is not None and len(items) != expected:
        raise RuntimeError(
            f"Incomplete result set: collected {len(items)} "
            f"but server matched {expected}"
        )

    return items


if __name__ == "__main__":
    all_items = fetch_all_items(
        api_url="https://earth-search.aws.element84.com/v1",
        collections=["sentinel-2-l2a"],
        bbox=[5.0, 45.0, 6.0, 46.0],
        datetime_range="2023-01-01/2023-12-31",
        page_size=500,
    )
    print(f"Retrieved {len(all_items)} items")
    # Asset URLs are now ready for header-level validation before any read.
    first = all_items[0]
    print(first.id, first.assets["red"].href)

Why pages() over items() here

search.items() gives a flat generator of Item objects — convenient, but it hides page boundaries. search.pages() yields one ItemCollection per HTTP response, which is what you want when you need to pause between requests, log progress, checkpoint to disk, or release each page’s memory before fetching the next. For very large queries, process and discard each page instead of accumulating a list of tens of thousands of items.

The three knobs that control completeness

The rule of thumb: raise limit toward the server’s ceiling to minimise requests, set max_items=None for any production pull that must be complete, and always cross-check against matched(). Treat the three as a unit — tuning one without the others is how truncation slips through.

Variant Patterns

1. Stream and discard — bounded memory for huge result sets

When a query matches 50,000 items, materialising them all is wasteful. Process each page and keep only what you extract:

from pystac_client import Client

catalog = Client.open("https://earth-search.aws.element84.com/v1")
search = catalog.search(
    collections=["sentinel-2-l2a"],
    datetime="2020-01-01/2024-01-01",
    bbox=[-124.0, 32.0, -114.0, 42.0],
    limit=1000,
    max_items=None,
)

red_hrefs: list[str] = []
for page in search.pages():           # one HTTP response at a time
    for item in page:
        red_hrefs.append(item.assets["red"].href)
    # page goes out of scope here — memory is reclaimed before the next fetch
print(len(red_hrefs), "red-band URLs")

2. Deliberately capping results for a preview

During development you often want a fast, bounded sample rather than the whole archive. Set max_items to a small number — pagination stops as soon as the cap is hit:

search = catalog.search(
    collections=["sentinel-2-l2a"],
    bbox=[5.0, 45.0, 6.0, 46.0],
    datetime="2023-06-01/2023-06-30",
    max_items=20,   # stop after 20 items regardless of how many match
)
preview = list(search.items())
assert len(preview) <= 20

3. Retry with backoff on transient page failures

Long paginated pulls occasionally hit a 429 Too Many Requests or a 502. Wrap page iteration so a single flaky response does not lose the whole run:

import time
from pystac_client import Client
from pystac_client.exceptions import APIError

def pages_with_retry(search, retries: int = 4, base_delay: float = 1.0):
    page_iter = search.pages()
    while True:
        for attempt in range(retries):
            try:
                yield next(page_iter)
                break
            except StopIteration:
                return
            except APIError:
                if attempt == retries - 1:
                    raise
                time.sleep(base_delay * 2 ** attempt)  # exponential backoff

Common Errors

Search returns exactly 100 items when thousands match

max_items defaults to 100. pystac-client stops following next links once it reaches that total. Pass max_items=None to remove the cap, then validate with matched().

search.matched() returns None

Not every STAC server advertises context / numberMatched. When matched() is None, you cannot assert an exact count — instead log the number of pages and items per page, and treat a zero-length final page plus an absent next link as your completeness signal.

APIError: 429 Too Many Requests during a long pull

You are paginating faster than the server’s rate limit allows. Add a short time.sleep() between page fetches (as in the working example) and wrap pages() in exponential-backoff retry logic. Increasing limit also helps by reducing the number of requests needed.

Related

• Querying STAC Catalogs Programmatically — the parent section covering Client.open(), search construction, and CQL2 filtering in full.
• Using pystac-client to Filter Sentinel-2 Imagery by Date — the sibling guide on narrowing a search by temporal and cloud-cover filters before you paginate it.
• How to Read COG Headers Without Downloading Full Files — validate the asset URLs a full search returns before committing to pixel reads.
• Core Raster Fundamentals & STAC Mapping — how catalog queries feed the wider ingestion architecture.