Validating a Cloud Mask against Reference Labels

Compare the mask with labelled pixels and report both kinds of error, broken down by surface:

import numpy as np

omission = np.mean(~mask[ref == "cloud"])          # cloud the mask missed
commission = np.mean(mask[ref == "clear"])          # clear ground wrongly masked
print(f"cloud omission {omission:.1%}, commission {commission:.1%}")

A mask’s reputation is not a measurement of how it behaves on your scenes. This page belongs to cloud and shadow masking strategies in Satellite Processing Workflows & Index Pipelines.

Two Errors, Unequal Costs

What each error does downstream Omission — cloud left unmasked — injects bright, spectrally wrong pixels into composites, indices and training data. Commission — clear ground masked as cloud — only reduces the number of usable observations. For most land applications omission is far more harmful, which should decide how a mask is tuned. Missed cloud versus over-masking omission — cloud kept bright values enter composites indices drop, looking like stress models learn cloud as a class corrupts results commission — clear lost fewer observations per pixel gaps in time series bright surfaces under-sampled reduces results Tune for low omission, then accept whatever commission that costs — unless bright surfaces are the subject.

The exception proves the rule: if the analysis is about snow, deserts or salt flats, commission over those surfaces removes exactly the data you need, and the balance shifts. That is why the breakdown by surface type matters as much as the overall figures.

Environment & Setup

Package Version pin Used for
numpy >=1.23 Error rates
pandas >=2.0 Per-surface breakdowns
geopandas >=0.14 Reference points with labels
rasterio >=1.3.0 Sampling the mask at reference locations 
pip install "numpy>=1.23" "pandas>=2.0" "geopandas>=0.14" "rasterio>=1.3.0"

Complete Working Example

import geopandas as gpd
import numpy as np
import pandas as pd
import rasterio


def sample_mask(mask_path: str, refs: gpd.GeoDataFrame) -> np.ndarray:
    with rasterio.open(mask_path) as src:
        pts = refs.to_crs(src.crs)
        vals = np.array([v[0] for v in src.sample(zip(pts.geometry.x, pts.geometry.y))])
    return vals.astype(bool)


def score(refs: gpd.GeoDataFrame, masked: np.ndarray, *, edge_px: np.ndarray | None = None) -> pd.DataFrame:
    df = refs.assign(masked=masked)
    if edge_px is not None:
        df = df.assign(near_edge=edge_px)
    rows = []
    for surface, g in df.groupby("surface"):
        cloud, clear = g[g.label == "cloud"], g[g.label == "clear"]
        rows.append({
            "surface": surface,
            "n_cloud": len(cloud), "n_clear": len(clear),
            "omission": float((~cloud.masked).mean()) if len(cloud) else np.nan,
            "commission": float(clear.masked.mean()) if len(clear) else np.nan,
        })
    out = pd.DataFrame(rows).set_index("surface")
    total_cloud, total_clear = df[df.label == "cloud"], df[df.label == "clear"]
    out.loc["ALL"] = [len(total_cloud), len(total_clear),
                      float((~total_cloud.masked).mean()), float(total_clear.masked.mean())]
    return out

The per-surface table is the output worth studying. A mask with 5% overall omission may have 1% over forest and 25% over bright urban areas, and whether that is acceptable depends entirely on where your analysis lives. Reference points are best collected with the same spatial and scene-level diversity as any validation sample, following the design principles in validating raster predictions and accuracy assessment.

Edges Deserve Their Own Score

Where masks actually miss cloud Omission is near zero for thick cloud cores, substantial at cloud edges within about a hundred metres of the boundary, and highest for thin cirrus. An overall omission figure averages these very different regimes and hides the edges, which is where most contamination originates. Cloud omission by regime 2% 21% 33% thick core edge, <100 m thin cirrus Score these separately; the overall number is dominated by easy cores.

Stratifying reference samples by distance to the nearest cloud boundary, and reporting omission separately for edges and cores, shows exactly what a dilation buffer should fix. The buffer sizing described in dilating cloud masks to catch thin cirrus can then be tuned against the edge omission rate directly rather than by eye.

Building Reference Labels Efficiently

Labelling pixels is slow, and the reference set is only useful if it covers the situations that break masks. A sampling design stratified by predicted class, by distance to cloud boundaries and by surface type concentrates effort where masks disagree with reality. Interpreting each point on a true-colour and a false-colour composite — plus the cirrus band where available — makes thin cloud visible that neither view shows alone.

Keep the reference set versioned and reuse it: once built, it scores every future mask variant, threshold and buffer in seconds, which turns mask tuning from argument into measurement.

Choosing Thresholds from the Scores

Most masks expose at least one tunable number — a cloud probability threshold for s2cloudless, a buffer distance, a cirrus reflectance cut-off. With a reference set in hand, each setting becomes a point on an omission-versus-commission curve. Sweep the parameter, score every value, and pick the setting where omission reaches the level your application tolerates, then read off what commission that costs. Plotting the curve for each surface type separately often shows that a single global threshold is a poor compromise and that a surface-dependent rule — a stricter threshold over bright ground, a looser one over vegetation — gives better results for both errors.

Guard against tuning to the reference set itself. If the same points are used to choose the threshold and to report its performance, the reported numbers are optimistic. Split the references by scene, tune on one half, and report the score on the other, exactly as with any model validation.

Reporting the Result

A useful cloud-mask report is short: the reference set’s size and composition, overall omission and commission with confidence intervals, the per-surface table, the edge-versus-core breakdown, and the chosen parameters. Keep it next to the pipeline configuration, in version control, so anyone reading a composite months later can find out exactly how clean its inputs were. When the mask algorithm, the processing baseline of the input product, or the region changes, re-run the scoring rather than assuming the old numbers still hold — cloud-mask behaviour is notoriously regional, and a mask validated in temperate Europe can behave very differently over the Sahel or the Andes.

Verification

Is the reference set fit to judge? A reference set should contain enough cloud and clear points per surface type to estimate each rate, should span many scenes and dates, and should include bright surfaces and cloud edges explicitly. A set drawn from a few easy scenes produces flattering and misleading scores. Check the references before trusting the score per-cell counts ≥ 100 per surface × label many scenes seasons and cloud types hard cases edges, cirrus, bright ground A score from easy scenes is a score of nothing in particular. 
counts = refs.groupby(["surface", "label"]).size().unstack(fill_value=0)
print(counts)
assert (counts.min(axis=1) >= 100).all(), "some surface/label cells too small to score"
assert refs["scene_id"].nunique() >= 10, "references come from too few scenes"

Common Errors

Scores look excellent but composites are contaminated

The references under-sample cloud edges and cirrus. Stratify by distance to cloud boundary.

Commission is very high in one region

That region has bright surfaces the mask mistakes for cloud. Check the per-surface table and add a surface-specific rule.

Sampling the mask returns wrong values

The reference points were not reprojected to the mask CRS. Reproject before sampling.

Results change between reviewers

Label definitions differ — especially for thin haze. Write a short labelling guide and double-label a subset.

Frequently Asked Questions

Q: Which cloud-mask error is worse, omission or commission? For most land analysis, omission — missed cloud contaminates composites, indices and models. Commission only removes some clear observations. Tune masks to low omission and accept a modest commission rate.

Q: Why do masks fail over bright surfaces? Snow, salt pans, bright sand and some urban roofs are as bright as cloud in visible bands. Masks commonly over-flag them, so validation must include those surfaces explicitly or it will overstate performance.

Q: How many reference pixels are needed? A few thousand pixels drawn from many scenes and surface types, rather than many pixels from few scenes. Diversity of conditions matters far more than raw count, because cloud mask errors cluster by situation.

Q: Should shadows be scored separately from cloud? Yes. Shadow masks fail differently — mainly through misplacement relative to their clouds and confusion with dark water or terrain shadow — and folding them into the cloud score hides both problems.