Passing Credentials to Remote Raster Workers

The best credential is one the worker gets from its own environment — an IAM role:

import rasterio
from rasterio.session import AWSSession
import boto3

# On a worker with an instance profile or task role: no keys anywhere in code
with rasterio.Env(AWSSession(boto3.Session(), requester_pays=True)):
    with rasterio.open("s3://private-bucket/scenes/T33UVP/B04.tif") as src:
        block = src.read(1, window=((0, 512), (0, 512)))

Code that reads private COGs on a laptop often fails on a cluster, because the laptop’s credentials live in files and environment variables that remote workers do not have. This page belongs to distributed processing on Coiled and AWS Batch in Cloud Execution & Orchestration.

Three Ways Workers Get Credentials

Roles, forwarded tokens, signed URLs With IAM roles, each worker obtains and refreshes its own short-lived credentials from the platform; nothing is transmitted. Forwarded temporary credentials are created by the client and sent to workers, expiring after an hour or so. Signed URLs grant access to specific objects for a limited time and need no credentials on the worker at all. Roles are preferred; signed URLs suit sharing a few objects with external compute. In order of preference 1. IAM role worker fetches its own auto-refreshed nothing transmitted 2. forwarded STS client creates, sends once expires in ~1 h must be refreshed 3. signed URLs per-object access no credentials on worker limited lifetime Long-lived access keys belong in none of the three columns.

On AWS Batch, attach a job role to the job definition; on Coiled, Kubernetes or EC2-based Dask clusters, attach an instance profile or service-account role. The AWS SDK and GDAL both discover these automatically through the instance metadata service or the container credentials endpoint, and refresh them before they expire. No secret ever appears in code, configuration or the task graph.

Environment & Setup

Package Version pin Used for
boto3 >=1.34 Sessions and STS
rasterio >=1.3.0 AWSSession for GDAL reads
dask[distributed] >=2023.1 Worker plugins
coiled >=1.0 Optional: managed clusters with credential forwarding 
pip install "boto3>=1.34" "rasterio>=1.3.0" "dask[distributed]>=2023.1" "coiled>=1.0"

Complete Working Example

import os

import boto3
from dask.distributed import WorkerPlugin

GDAL_OPTS = {
    "AWS_REGION": "eu-central-1",
    "AWS_REQUEST_PAYER": "requester",
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.tiff,.TIF",
}


class GDALEnv(WorkerPlugin):
    """Apply GDAL options on every worker. No secrets here."""
    def setup(self, worker):
        os.environ.update(GDAL_OPTS)


class TemporaryAWSCredentials(WorkerPlugin):
    """Fallback when workers cannot have a role: forward short-lived STS credentials."""
    def __init__(self, duration_s: int = 3600):
        creds = boto3.client("sts").get_session_token(DurationSeconds=duration_s)["Credentials"]
        self._env = {
            "AWS_ACCESS_KEY_ID": creds["AccessKeyId"],
            "AWS_SECRET_ACCESS_KEY": creds["SecretAccessKey"],
            "AWS_SESSION_TOKEN": creds["SessionToken"],
        }
        self.expires = creds["Expiration"]

    def setup(self, worker):
        os.environ.update(self._env)


def configure(client, forward_credentials: bool = False):
    client.register_plugin(GDALEnv())
    if forward_credentials:
        plugin = TemporaryAWSCredentials()
        client.register_plugin(plugin)
        return plugin.expires

The plugin approach keeps credentials out of the task graph. Passing keys as function arguments embeds them in every serialised task, where they can end up in the scheduler’s memory, the dashboard, performance reports and error tracebacks. A plugin sends them once per worker and sets the environment that GDAL and boto3 read. Coiled offers built-in forwarding of the client’s temporary credentials, which does the same thing with automatic refresh.

GDAL Needs to Know More than Keys

Settings that look like credential problems Many access failures are not missing credentials. Requester-pays buckets need AWS_REQUEST_PAYER set to requester. Buckets in another region may need AWS_REGION. Public buckets read without credentials need AWS_NO_SIGN_REQUEST. Expired session tokens produce the same 403 error as missing permissions. A 403 has several possible causes requester pays AWS_REQUEST_PAYER wrong region AWS_REGION public bucket AWS_NO_SIGN_REQUEST expired token refresh or use a role Set CPL_CURL_VERBOSE=YES on one worker to see the exact request and response.

A 403 from S3 looks like a permissions problem but frequently is not. Requester-pays buckets reject requests that do not declare the requester pays; public datasets read without credentials need AWS_NO_SIGN_REQUEST=YES, and mixing public and private buckets in one job requires setting that per read with rasterio.Env rather than globally. When a long job starts failing after about an hour, the cause is almost always an expired forwarded session token — the strongest argument for roles, which refresh themselves.

Least Privilege for Workers

Workers need to read inputs and write outputs, nothing more. Scope the role to the specific buckets and prefixes, grant s3:GetObject on inputs and s3:PutObject on the output prefix, and avoid wildcard actions. Separate roles for development and production clusters keep an experiment from overwriting a published product. For cross-account data, prefer a bucket policy granting the worker role access over copying keys between accounts. These controls are also what make it safe to run third-party or experimental code on the cluster.

Other Clouds and Signed URLs

The same principles apply on Google Cloud and Azure: workload identity or managed identities play the role of IAM roles, and GDAL reads GOOGLE_APPLICATION_CREDENTIALS or Azure environment variables when a role is not available. Signed URLs are the right tool when sharing a small number of objects with compute you do not control, such as a partner’s cluster or a tile server; GDAL reads them through /vsicurl/ like any HTTPS file. Never log them — a signed URL is a working credential for its lifetime.

Credentials in Container Images and Notebooks

Two leaks are common enough to name. The first is baking credentials into a container image through an ENV line or a copied ~/.aws directory: every layer of an image is readable by anyone who can pull it, and deleting the file in a later layer does not remove it from the earlier one. Images should contain configuration, never secrets, and get credentials at run time from the platform. The second is notebooks: a key pasted into a cell survives in the saved output and the version history long after the cell is edited. Load credentials from the environment or a profile instead, and add a pre-commit hook that scans for key patterns so a mistake is caught before it leaves the laptop. If a key does leak, rotate it at once — removing it from a repository does not remove it from clones and caches.

Verification

Access works, secrets stay put Run a small read on every worker and confirm success. Inspect the identity each worker uses. Search logs and serialized task graphs for access key patterns and confirm none appear. Two things to prove every worker can read and uses the expected identity no keys in graphs or logs search for AKIA / ASIA patterns Run both checks whenever the cluster or its configuration changes. 
import boto3, rasterio

def probe(path="s3://private-bucket/scenes/T33UVP/B04.tif"):
    ident = boto3.client("sts").get_caller_identity()["Arn"]
    with rasterio.open(path) as src:
        src.read(1, window=((0, 16), (0, 16)))
    return ident

print(client.run(probe))   # every worker should report the expected role ARN

Common Errors

Works locally, 403 on the cluster

Workers lack the laptop’s credentials. Attach a role, or forward temporary credentials with a plugin.

Fails after about an hour

Forwarded session tokens expired. Use roles, or refresh the credentials with a new plugin registration.

Requester-pays reads fail with 403

GDAL was not told the requester pays. Set AWS_REQUEST_PAYER=requester.

Keys appear in the dashboard or tracebacks

They were passed as task arguments. Move them into a worker plugin or, better, a role.

Frequently Asked Questions

Q: How should Dask workers authenticate to S3? With an IAM role or instance profile attached to the workers. The SDK and GDAL obtain and refresh credentials automatically, and no secret is transmitted.

Q: Is it safe to pass AWS keys as task arguments? No. They are serialised into every task and can appear in the dashboard, logs and tracebacks. Use a worker plugin or a role.

Q: Why do my reads fail after an hour? Forwarded session credentials expire, typically after an hour. Roles refresh automatically; forwarded credentials must be renewed.

Q: How do I read public and private buckets in one job? Set AWS_NO_SIGN_REQUEST only for the public reads, using rasterio.Env around those reads, and let the role sign the private ones.