Reducing S3 Egress Costs in Raster Pipelines

Run your compute in the same AWS region as the bucket: S3-to-EC2 transfer within a region carries no egress charge, so the per-gigabyte fee that dominates a raster bill for cross-region or internet reads simply disappears. Confirm the bucket’s region from its metadata, then launch workers there before tuning anything else:

import boto3

def bucket_region(bucket: str) -> str:
    """Return the AWS region a bucket lives in (us-east-1 reports as None)."""
    loc = boto3.client("s3").get_bucket_location(Bucket=bucket)["LocationConstraint"]
    return loc or "us-east-1"

print(bucket_region("sentinel-cogs"))  # → 'us-west-2' : launch workers here

This page narrows the broader Optimizing Pipeline Cost and Performance guide down to one line item: the S3 egress and request charges incurred while reading and writing Cloud-Optimized GeoTIFFs.

Why This Arises in Remote Sensing Workflows

S3 pricing has three components analysts routinely conflate: storage (per GB-month), requests (per thousand GET/PUT), and data transfer, or egress (per GB leaving the region). For a COG pipeline the storage is often someone else’s — public archives like Sentinel-2 on AWS Open Data — so your bill is dominated by requests and egress. Egress is billed only when bytes cross a boundary: out to the internet, or from one region to another. Bytes moving from S3 to compute in the same region are free of transfer charge.

The failure mode is subtle because the pipeline works regardless of where it runs. A Dask cluster in eu-west-1 reading Sentinel-2 from us-west-2 produces identical NDVI to one running in-region, but every full-resolution byte is billed as cross-region transfer, and a full scene is hundreds of megabytes per band across thousands of tiles. Multiply by an archive-scale job and egress alone can exceed the compute cost several times over.

For the parent section context see Cloud Execution & Orchestration; the tactics here assume you already read only the bytes you need, using Cloud-Optimized GeoTIFF structure to fetch tiles and overviews rather than whole files.

In-region versus cross-region data flow

The diagram contrasts the two topologies. In-region reads cost only per-request GET charges; cross-region reads add a per-gigabyte transfer charge on every byte.

The lesson is topological, not algorithmic: the cheapest read strategy in the wrong region still pays egress. Fix placement first, then minimize bytes and requests.

Environment & Setup

pip install "rasterio>=1.3.0" "rioxarray>=0.15" "boto3>=1.34"

Complete Working Example

The function below reads a spatial window from a public COG at a chosen overview level, with every GDAL setting tuned to minimize the number and size of range requests. Run it on a worker in the bucket’s region and no egress is billed at all.

import rasterio
from rasterio.env import Env
from rasterio.windows import from_bounds
import numpy as np

# Env tuned to cut request count (and bytes on any cross-region read).
LOW_EGRESS_GDAL = {
    "AWS_NO_SIGN_REQUEST": "YES",                # public bucket: skip SigV4
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR", # no directory-listing GETs
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES", # coalesce adjacent tile ranges
    "VSI_CACHE": "TRUE",                         # reuse fetched header/tile ranges
    "VSI_CACHE_SIZE": "26214400",                # 25 MB per-file range cache
    "GDAL_HTTP_MULTIPLEX": "YES",                # HTTP/2 multiplexed ranges
}

def read_window_cheaply(
    url: str,
    bounds: tuple[float, float, float, float],  # (left, bottom, right, top) in the raster CRS
    overview_level: int = 1,                    # 0=first overview; -1=full resolution
) -> np.ndarray:
    """
    Read only a spatial window, and only at the resolution you need.
    Transfers a fraction of the bytes a whole-file read would, and — run
    in the bucket region — incurs zero egress charge.
    """
    with Env(**LOW_EGRESS_GDAL):
        with rasterio.open(url, OVERVIEW_LEVEL=overview_level) as src:
            # Convert geographic bounds to a pixel window at the open resolution.
            window = from_bounds(*bounds, transform=src.transform)
            data = src.read(1, window=window)   # only these tiles are fetched
    return data


if __name__ == "__main__":
    s2_red = (
        "https://sentinel-cogs.s3.us-west-2.amazonaws.com"
        "/sentinel-s2-l2a-cogs/36/N/YF/2023/6"
        "/S2A_36NYF_20230615_0_L2A/B04.tif"
    )
    # A small AOI, read at 1/4 resolution: a few tiles instead of the whole band.
    subset = read_window_cheaply(
        s2_red,
        bounds=(600000, 10000, 605000, 15000),
        overview_level=1,
    )
    print(subset.shape)  # far smaller than the full 10980 x 10980 band

Two multiplications drive the saving: the window restricts extent to the tiles overlapping your area of interest, and OVERVIEW_LEVEL restricts resolution. Reading a 5 km window at overview level 1 from a full Sentinel-2 tile can transfer under 1% of the bytes a naive src.read(1) would. For the range-request mechanics behind this, see how to read COG headers without downloading full files.

Variant Patterns

1. Write results to a colocated bucket

Reads are only half the bill. Writing outputs to a bucket in a different region re-incurs egress on every byte you produce. Resolve the compute region and target a same-region result bucket:

import boto3

def assert_colocated(input_bucket: str, output_bucket: str) -> None:
    """Fail fast if results would be written across a region boundary."""
    s3 = boto3.client("s3")
    def region(b: str) -> str:
        return s3.get_bucket_location(Bucket=b)["LocationConstraint"] or "us-east-1"
    in_r, out_r = region(input_bucket), region(output_bucket)
    if in_r != out_r:
        raise ValueError(
            f"Egress risk: reading from {in_r} but writing to {out_r}. "
            "Use an output bucket in the same region as compute."
        )
    print(f"Colocated in {in_r}: writes incur no transfer charge.")

assert_colocated("sentinel-cogs", "my-results-usw2")

2. Batch range reads across bands with a single environment

When a task reads several bands of the same scene, open them under one Env block so the /vsicurl/ range cache and HTTP/2 connection are reused across files, collapsing many small GETs:

import rasterio
from rasterio.env import Env

BANDS = {"red": ".../B04.tif", "nir": ".../B08.tif", "green": ".../B03.tif"}

def read_scene_bands(urls: dict[str, str], overview_level: int = 1) -> dict:
    arrays = {}
    with Env(AWS_NO_SIGN_REQUEST="YES",
             GDAL_HTTP_MERGE_CONSECUTIVE_RANGES="YES",
             GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
             GDAL_HTTP_MULTIPLEX="YES"):
        for name, url in urls.items():
            with rasterio.open(url, OVERVIEW_LEVEL=overview_level) as src:
                arrays[name] = src.read(1)
    return arrays

Sharing one environment lets GDAL multiplex the three bands’ range requests over a single HTTP/2 connection rather than negotiating a fresh TLS handshake per file.

3. Public buckets: skip signing to avoid failed, retried requests

Public Open Data buckets reject SigV4-signed requests, and a rejected request is often retried — doubling your GET count before the read even succeeds. Set AWS_NO_SIGN_REQUEST=YES so the first attempt lands:

import rasterio
from rasterio.env import Env

with Env(AWS_NO_SIGN_REQUEST="YES", GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open(
        "https://sentinel-cogs.s3.us-west-2.amazonaws.com/.../B04.tif"
    ) as src:
        print(src.overviews(1))  # header-only: one small range request

Common Errors

Egress charges appear despite reading overviews Fewer bytes still cross the region boundary if compute is not colocated. Reading overviews cuts the volume but not the rate. Move workers into the bucket’s region (get_bucket_location) so in-region transfer is free, then keep overview reads to also shrink the per-request count.

AccessDenied on a public bucket The request was signed against credentials the public bucket rejects. Set AWS_NO_SIGN_REQUEST=YES; without it, GDAL attaches SigV4 headers that Open Data buckets refuse, and the retry inflates your request count.

Request bill high even in-region Un-merged range requests: every tile becomes its own GET. Enable GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES and VSI_CACHE=TRUE, and read windows rather than scattered pixels so adjacent tiles coalesce into one round-trip.

Related

• Optimizing Pipeline Cost and Performance — the parent guide covering worker sizing, caching, and profiling alongside egress
• Understanding Cloud-Optimized GeoTIFF Structure — how overviews and internal tiling let you transfer a fraction of a file
• How to Read COG Headers Without Downloading Full Files — validate assets with tiny range requests before committing bandwidth
• Optimizing Rasterio Window Reads for Memory Efficiency — restrict extent with Window reads so you fetch only the tiles you need