Processing COGs on AWS Batch with Docker

To run a rasterio job over many Cloud-Optimized GeoTIFFs at scale, bake the job into a Docker image, register it as an AWS Batch job definition, and submit an array job so each child task processes one scene by its AWS_BATCH_JOB_ARRAY_INDEX. The container reads each COG straight from S3 by range and writes its result back:

import os, boto3

batch = boto3.client("batch", region_name="us-west-2")
resp = batch.submit_job(
    jobName="cog-batch-run",
    jobQueue="raster-queue",
    jobDefinition="cog-processor:1",
    arrayProperties={"size": 500},   # 500 scenes → 500 child tasks
)
print(resp["jobId"])

This is the containerized batch pattern from the parent guide, Distributed Processing on Coiled & AWS Batch — one image, one command, fanned out across the archive.

Why This Arises in Remote Sensing Workflows

Per-scene raster transforms are embarrassingly parallel: reprojecting, computing an index, or generating a thumbnail for one scene has no dependency on any other scene. That shape maps cleanly onto AWS Batch, which runs a container to completion per job and scales the underlying EC2 fleet to the queue depth. You get horizontal throughput without standing up a scheduler or holding shared cluster state.

The design hinges on reading Cloud-Optimized GeoTIFFs in place. Because a COG’s header and tiles are range-addressable, the container never copies whole files to local disk; GDAL fetches only the bytes a windowed read touches. For interactive, shared-memory Dask work the sibling guide Launching a Coiled Cluster for STAC Processing is the better fit; Batch wins when the work is a fixed script run once per scene. For the wider orchestration picture, see Cloud Execution & Orchestration.

How an array job fans out over scenes

The diagram traces one submit_job call through the queue to N isolated containers, each pinned to a different scene by its array index.

AWS Batch array job: one submission, N per-scene containers over a shared S3 bucket A boto3 submit_job call with arrayProperties enters the Batch job queue, which launches N child containers built from the same ECR image. Each container reads AWS_BATCH_JOB_ARRAY_INDEX, maps it to a scene key in a manifest, range-reads that COG from S3, runs the rasterio transform, and writes the result to the output prefix. boto3 submit_job arrayProperties size=N Batch job queue + compute environment container idx=0 → scene 0 container idx=1 → scene 1 container idx=N → scene N same ECR image rasterio + job script S3 (same region) in: range-read scene COG · manifest of keys out: write index result to output prefix

Environment & Setup

Only three moving parts run locally: the Docker CLI to build and push the image, the AWS CLI/boto3 to register the job definition and submit jobs, and rasterio inside the image. Pin versions so the in-container GDAL matches what you tested.

Package / tool Version Why required
rasterio 1.3.9 COG range reads via GDAL /vsis3/ inside the container
boto3 1.34+ Submit array jobs and register job definitions from Python
Docker 24+ Build the immutable job image
base image ghcr.io/osgeo/gdal:ubuntu-small-3.9.2 Prebuilt GDAL 3.9 with libcurl for range reads 
pip install "rasterio==1.3.9" "boto3>=1.34"

Complete Working Example

1. Dockerfile on a GDAL/rasterio base

Start from an official GDAL image so /vsis3/ and range reads work out of the box, then add pinned Python deps and the job script. The image is immutable — rebuild and push a new tag when dependencies change.

# Prebuilt GDAL 3.9 with libcurl → COG range reads work immediately
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.9.2

# Pin the Python stack so worker GDAL/rasterio matches local testing
RUN pip install --no-cache-dir "rasterio==1.3.9" "numpy==1.26.4" "boto3==1.34.84"

WORKDIR /app
COPY process_scene.py /app/process_scene.py

# GDAL tuning for cloud reads, baked into the image so every task inherits it
ENV GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
    GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES \
    CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif

ENTRYPOINT ["python", "/app/process_scene.py"]

Build and push to ECR:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <acct>.dkr.ecr.us-west-2.amazonaws.com
docker build -t cog-processor:1 .
docker tag cog-processor:1 <acct>.dkr.ecr.us-west-2.amazonaws.com/cog-processor:1
docker push <acct>.dkr.ecr.us-west-2.amazonaws.com/cog-processor:1

2. The in-container job script

The script reads its array index, maps it to a scene key from a manifest in S3, range-reads the COG, applies a transform (here a simple NDVI-style normalized difference on two bands), and writes a Cloud-Optimized GeoTIFF back to the output prefix. No full file is ever copied to disk.

import os
import json
import numpy as np
import rasterio
from rasterio.io import MemoryFile
import boto3

BUCKET = os.environ["SCENE_BUCKET"]
MANIFEST_KEY = os.environ["MANIFEST_KEY"]      # JSON list of scene records
OUT_PREFIX = os.environ["OUTPUT_PREFIX"]       # e.g. "ndvi/2023/"

def load_manifest() -> list[dict]:
    """Fetch the JSON manifest of scenes this run should process."""
    s3 = boto3.client("s3")
    body = s3.get_object(Bucket=BUCKET, Key=MANIFEST_KEY)["Body"].read()
    return json.loads(body)

def process_one(scene: dict) -> None:
    """Range-read NIR + Red COGs, compute a normalized difference, write result to S3."""
    # GDAL reads directly from s3://; only the header + touched blocks transfer
    with rasterio.open(scene["nir"]) as nir_src, rasterio.open(scene["red"]) as red_src:
        nir = nir_src.read(1).astype("float32")
        red = red_src.read(1).astype("float32")
        profile = nir_src.profile

    denom = nir + red
    with np.errstate(divide="ignore", invalid="ignore"):
        ndvi = np.where(denom != 0, (nir - red) / denom, np.nan).astype("float32")

    profile.update(dtype="float32", count=1, compress="zstd",
                   tiled=True, blockxsize=512, blockysize=512, nodata=np.nan)

    # Serialize the COG in memory, then upload — avoids a local temp file
    with MemoryFile() as mem:
        with mem.open(**profile) as dst:
            dst.write(ndvi, 1)
        out_key = f"{OUT_PREFIX}{scene['id']}_ndvi.tif"
        boto3.client("s3").put_object(Bucket=BUCKET, Key=out_key, Body=mem.read())
    print(f"wrote s3://{BUCKET}/{out_key}")

def main() -> None:
    idx = int(os.environ["AWS_BATCH_JOB_ARRAY_INDEX"])   # injected by AWS Batch
    scenes = load_manifest()
    process_one(scenes[idx])                             # this task owns one scene

if __name__ == "__main__":
    main()

3. The job definition JSON

The job definition declares the image, resources, environment, and — critically — the job role that grants S3 access. Register it with aws batch register-job-definition --cli-input-json file://job-def.json.

{
  "jobDefinitionName": "cog-processor",
  "type": "container",
  "containerProperties": {
    "image": "<acct>.dkr.ecr.us-west-2.amazonaws.com/cog-processor:1",
    "vcpus": 2,
    "memory": 8192,
    "jobRoleArn": "arn:aws:iam::<acct>:role/cog-batch-job-role",
    "executionRoleArn": "arn:aws:iam::<acct>:role/cog-batch-exec-role",
    "environment": [
      { "name": "SCENE_BUCKET", "value": "my-raster-archive" },
      { "name": "MANIFEST_KEY", "value": "runs/2023/manifest.json" },
      { "name": "OUTPUT_PREFIX", "value": "ndvi/2023/" }
    ]
  }
}

The jobRoleArn needs an IAM policy granting s3:GetObject on the input prefix and s3:PutObject on the output prefix. The separate executionRoleArn only lets Batch pull the ECR image and ship logs to CloudWatch — do not conflate the two.

4. Submit the array job

import boto3

batch = boto3.client("batch", region_name="us-west-2")

def submit_run(n_scenes: int, jobdef: str = "cog-processor:1") -> str:
    """One array job → n_scenes child tasks, each keyed by its array index."""
    resp = batch.submit_job(
        jobName="cog-ndvi-2023",
        jobQueue="raster-queue",
        jobDefinition=jobdef,
        arrayProperties={"size": n_scenes},    # must match manifest length
        # Retry transient spot interruptions; the job is idempotent per scene
        retryStrategy={"attempts": 3},
    )
    return resp["jobId"]

if __name__ == "__main__":
    print(submit_run(500))

Set arrayProperties.size to the manifest length so every scene gets exactly one child task. Because each task overwrites a deterministic output key, retries are safe — a re-run of scene 42 produces the same object.

Variant Patterns

Passing the scene list without a manifest

For small runs you can skip the manifest and pass scene keys through job parameters, resolving the index against a comma-separated env var. This avoids an S3 round-trip per task but does not scale past a few hundred entries because of environment-size limits.

import os
scenes = os.environ["SCENE_KEYS"].split(",")     # "s3://.../a.tif,s3://.../b.tif"
key = scenes[int(os.environ["AWS_BATCH_JOB_ARRAY_INDEX"])]

Windowed reads for very large scenes

When a single scene is too large to hold in memory, read and write by block instead of read(1). Pair this with the header-first checks from How to Read COG Headers Without Downloading Full Files to size memory before allocating.

import rasterio
from rasterio.windows import Window

with rasterio.open(scene_url) as src:
    for _, window in src.block_windows(1):     # iterate native COG tiles
        block = src.read(1, window=window)     # only this tile transfers
        # ... transform block, accumulate or stream-write ...

Common Errors

AccessDenied when the container reads or writes S3

The job role, not the execution role, governs application S3 access. Attach an IAM policy to jobRoleArn granting s3:GetObject on the input prefix and s3:PutObject on the output prefix, then resubmit.

RasterioIOError: ... not recognized as a supported file format inside the container

GDAL in the image lacks curl/range support or the URL returned an error page. Confirm the base image is a GDAL build with libcurl and that CPL_VSIL_CURL_ALLOWED_EXTENSIONS includes .tif; test one URL with rasterio.open locally in the same image.

Array task fails with KeyError: 'AWS_BATCH_JOB_ARRAY_INDEX'

You submitted a plain job, not an array job. arrayProperties.size must be set (and at least 2) for Batch to inject the index variable; a single-task run does not get one.