Sizing AWS Batch Jobs for Tile Workloads

Pick tiles per job so each job runs a few minutes, and request resources that pack onto instances:

import boto3

batch = boto3.client("batch")
batch.submit_job(
    jobName="ndvi-2026-09-18", jobQueue="raster-spot", jobDefinition="ndvi:7",
    arrayProperties={"size": 250},                   # 250 jobs x 20 tiles = 5,000 tiles
    containerOverrides={"resourceRequirements": [
        {"type": "VCPU", "value": "2"}, {"type": "MEMORY", "value": "7500"}],
        "environment": [{"name": "TILES_PER_JOB", "value": "20"}]})

Batch schedules containers onto instances; how you slice the tile list into jobs and what each job asks for decide how full those instances are and how much a failure costs. This page belongs to distributed processing on Coiled and AWS Batch in Cloud Execution & Orchestration.

Tiles per Job: Overhead against Retry Cost

The sweet spot in job size With one tile per job, container start-up and image pull dominate and overhead is high. With hundreds of tiles per job, a single failure or spot interruption loses a lot of work and the run's tail is long. Total cost is lowest when each job runs roughly five to fifteen minutes. Cost against tiles per job 5–15 min per job start-up overhead retry and tail cost 1 tile 500 tiles Solid line: total. Dashed: its two components.

A Batch job carries fixed overhead: scheduling, pulling the image if the instance has not cached it, starting Python and importing GDAL — often 20 to 60 seconds. With one tile per job taking 30 seconds, half the bill is overhead. At the other extreme, a job processing 500 tiles for two hours loses all its work to a single spot interruption and leaves the run waiting on its slowest job. Grouping tiles so each job runs five to fifteen minutes keeps overhead under a few percent and makes each retry cheap.

Environment & Setup

Package Version pin Used for
boto3 >=1.34 Submitting and monitoring jobs
rasterio >=1.3.0 The per-tile work inside the container
psutil >=5.9 Measuring peak memory during profiling 
pip install "boto3>=1.34" "rasterio>=1.3.0" "psutil>=5.9"

Complete Working Example

import json
import math
import os

import boto3


def plan(tiles: list[str], seconds_per_tile: float, peak_mb_per_tile: float,
         target_minutes: float = 10, vcpu: int = 2, threads_per_job: int = 2) -> dict:
    per_job = max(1, round(target_minutes * 60 / seconds_per_tile * threads_per_job))
    n_jobs = math.ceil(len(tiles) / per_job)
    # memory: concurrent tiles x peak, plus runtime overhead, rounded to pack onto instances
    mem = int(threads_per_job * peak_mb_per_tile * 1.3 + 600)
    mem = int(math.ceil(mem / 500) * 500)
    return {"tiles_per_job": per_job, "jobs": n_jobs, "vcpu": vcpu, "memory_mb": mem}


def write_manifest(tiles, bucket, key):
    boto3.client("s3").put_object(Bucket=bucket, Key=key, Body=json.dumps(tiles).encode())


def submit(p: dict, run_id: str, manifest_uri: str, queue: str, definition: str):
    return boto3.client("batch").submit_job(
        jobName=f"tiles-{run_id}", jobQueue=queue, jobDefinition=definition,
        arrayProperties={"size": p["jobs"]},
        retryStrategy={"attempts": 3, "evaluateOnExit": [
            {"onStatusReason": "Host EC2*", "action": "RETRY"},   # spot reclaim
            {"onReason": "*", "action": "EXIT"}]},
        containerOverrides={
            "resourceRequirements": [{"type": "VCPU", "value": str(p["vcpu"])},
                                     {"type": "MEMORY", "value": str(p["memory_mb"])}],
            "environment": [{"name": "RUN_ID", "value": run_id},
                            {"name": "MANIFEST", "value": manifest_uri},
                            {"name": "TILES_PER_JOB", "value": str(p["tiles_per_job"])}]})


# inside the container
def my_tiles(all_tiles: list[str]) -> list[str]:
    i = int(os.environ["AWS_BATCH_JOB_ARRAY_INDEX"])
    n = int(os.environ["TILES_PER_JOB"])
    return all_tiles[i * n:(i + 1) * n]

An array job submits one definition and lets Batch create the child jobs, each with its own AWS_BATCH_JOB_ARRAY_INDEX. The container reads the tile manifest from S3 and slices its share. This is much faster to submit than thousands of individual jobs, and array children are retried individually. The retry strategy retries only on spot reclamation; genuine errors exit so they surface quickly instead of burning three attempts each. The container itself is the one built in processing COGs on AWS Batch with Docker.

Packing Jobs onto Instances

Requests that divide the instance evenly An instance with 16 vCPU and 64 GB fits eight jobs requesting 2 vCPU and 7.5 GB each, using nearly all resources. Jobs requesting 2 vCPU and 9 GB fit only seven by memory, stranding 2 vCPU. Requests should divide the instance's usable memory, which is slightly less than its nominal memory, evenly. 16 vCPU / 64 GB instance 2 vCPU, 7.5 GB 8 jobs, instance full 2 vCPU, 9 GB 7 jobs, one slot stranded Usable memory is a little under nominal, because the agent and OS reserve some.

Batch places jobs by their requests, not their actual use. A request of 8 GB on an instance with 64 GB nominal will not fit eight times, because the ECS agent and operating system reserve a few hundred megabytes; eight jobs at 7.5 GB will. Deriving requests from a profiled tile, as the planner does, and rounding them to values that divide common instance sizes keeps instances full. Choosing which instance sizes the compute environment may use is covered in choosing instance types for raster workloads.

Idempotent Jobs

Retries only help if running a job twice is harmless. Write each tile’s output to a deterministic key, skip tiles whose output already exists with a matching run identifier, and write to a temporary key before an atomic copy to the final one. A spot interruption then costs only the tiles in progress, and a re-submitted array can pick up exactly where the last one stopped. Record each tile’s outcome as a structured event so the run manifest can be built afterwards, following logging structured events from raster tasks.

Array Size Limits and Very Large Runs

An array job can have up to 10,000 children, and a queue processes many arrays concurrently. For runs with more jobs than that, submit several arrays over disjoint slices of the manifest, or raise tiles per job. Very large runs also meet account quotas — vCPU limits for on-demand and spot capacity per region — which appear as jobs stuck in the RUNNABLE state. Check the quotas before a large run rather than discovering them at two in the morning, and set the compute environment’s maximum vCPUs to a value the account can actually supply.

Verification

Was the sizing right? After a run, job durations should cluster around the target with a short tail. Peak memory per job should sit at sixty to eighty percent of the request. Instance CPU utilisation should be high, showing jobs packed well. Post-run checks job duration median near target memory headroom peak 60–80% of request instance CPU high, few idle hosts Adjust tiles per job and requests from these numbers, not from guesses. 
import statistics
import boto3

batch = boto3.client("batch")
kids = batch.list_jobs(arrayJobId=array_job_id, jobStatus="SUCCEEDED")["jobSummaryList"]
mins = [(j["stoppedAt"] - j["startedAt"]) / 60000 for j in kids]
print(f"median {statistics.median(mins):.1f} min, max {max(mins):.1f} min, n={len(mins)}")

Common Errors

Jobs stay RUNNABLE forever

Requests exceed any instance the compute environment allows, or the vCPU quota is exhausted. Check both.

Jobs are killed with exit code 137

The container exceeded its memory request. Profile a tile and raise the request, or reduce threads per job.

Instances run half empty

Requests do not divide instance resources evenly. Round requests to fit.

Retries repeat the same failure three times

The retry strategy retries every error. Retry only on host or spot termination.

Frequently Asked Questions

Q: How many tiles should one Batch job process? Enough that each job runs roughly five to fifteen minutes. That keeps start-up overhead small while keeping the cost of a retry or spot interruption low.

Q: Should I use array jobs? Yes. One array submission creates up to 10,000 child jobs, each with an index to select its tiles, and children retry individually.

Q: How do I choose the memory request? Profile a tile, multiply its peak by the number of tiles processed concurrently in the job, add overhead, and round to a value that divides the instance’s usable memory.

Q: Why are my jobs stuck in RUNNABLE? Usually because no allowed instance type can satisfy the request, or the account’s vCPU quota for the region is exhausted.