Measuring Boundary Accuracy with IoU and F1

Intersection over union scores the mask; matching connected components scores the objects:

import numpy as np

def iou(pred: np.ndarray, ref: np.ndarray) -> float:
    inter = np.logical_and(pred, ref).sum()
    union = np.logical_or(pred, ref).sum()
    return float(inter / union) if union else float("nan")

For anything delineated — fields, buildings, water bodies — that number says far more than overall accuracy does. This page belongs to validating raster predictions and accuracy assessment in Raster Machine Learning & Model Inference.

What Per-Pixel Accuracy Cannot See

Two segmentation results can have identical per-pixel accuracy and be worth completely different amounts. The difference is in the topology: whether objects are separated correctly, not whether their interiors are labelled correctly.

Same pixels, different objects The reference shows two adjacent fields separated by a thin boundary. One prediction reproduces both fields correctly. Another merges them into a single object, losing only the boundary pixels — under one per cent of the area — while halving the field count. A third splits one field in two, inflating the count instead. reference merged split 2 objects · pixel accuracy 100% 1 object · pixel accuracy 99.2% 3 objects · pixel accuracy 99.4% A field-boundary product with 99.2% pixel accuracy and the wrong field count is not usable.

Object-level metrics exist to make that failure visible. Instance IoU asks whether each reference object has a well-matched prediction; boundary F1 asks whether the edges are in the right place; and the count of merges and splits asks whether the topology survived.

Environment & Setup

Package Version pin Used for
numpy >=1.23 Mask arithmetic and the overlap matrix
scipy >=1.11 ndimage.label for connected components, distance transforms
rasterio >=1.3.0 Reading the aligned prediction and reference 
pip install "numpy>=1.23" "scipy>=1.11" "rasterio>=1.3.0"

Complete Working Example

import numpy as np
from scipy import ndimage


def instance_scores(pred_mask: np.ndarray, ref_mask: np.ndarray,
                    *, iou_threshold: float = 0.5) -> dict[str, float]:
    """Match predicted objects to reference objects by overlap."""
    pred_lbl, n_pred = ndimage.label(pred_mask)
    ref_lbl, n_ref = ndimage.label(ref_mask)
    if n_pred == 0 or n_ref == 0:
        return {"precision": 0.0, "recall": 0.0, "f1": 0.0,
                "n_pred": n_pred, "n_ref": n_ref}

    # Pairwise intersection counts via a single 2D histogram
    both = (pred_lbl > 0) & (ref_lbl > 0)
    pairs = np.bincount(pred_lbl[both] * (n_ref + 1) + ref_lbl[both],
                        minlength=(n_pred + 1) * (n_ref + 1))
    inter = pairs.reshape(n_pred + 1, n_ref + 1)[1:, 1:]

    pred_area = np.bincount(pred_lbl.ravel(), minlength=n_pred + 1)[1:]
    ref_area = np.bincount(ref_lbl.ravel(), minlength=n_ref + 1)[1:]
    union = pred_area[:, None] + ref_area[None, :] - inter
    iou_matrix = np.divide(inter, union, out=np.zeros_like(inter, "float64"),
                           where=union > 0)

    matched = (iou_matrix >= iou_threshold)
    tp = int(matched.any(axis=1).sum())          # predictions with a match
    precision = tp / n_pred
    recall = int(matched.any(axis=0).sum()) / n_ref
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return {"precision": precision, "recall": recall, "f1": f1,
            "n_pred": n_pred, "n_ref": n_ref}


def boundary_f1(pred_mask: np.ndarray, ref_mask: np.ndarray,
                *, tolerance_px: int = 2) -> float:
    """F1 over boundary pixels, allowing a tolerance in pixels."""
    def edges(m: np.ndarray) -> np.ndarray:
        return m ^ ndimage.binary_erosion(m, iterations=1)

    pe, re = edges(pred_mask), edges(ref_mask)
    # Distance from every pixel to the nearest reference / predicted edge
    d_ref = ndimage.distance_transform_edt(~re)
    d_pred = ndimage.distance_transform_edt(~pe)

    precision = (d_ref[pe] <= tolerance_px).mean() if pe.any() else 0.0
    recall = (d_pred[re] <= tolerance_px).mean() if re.any() else 0.0
    return (2 * precision * recall / (precision + recall)
            if precision + recall else 0.0)

The overlap matrix built with a single bincount is what keeps this usable on a full scene. The obvious double loop over predicted and reference objects is fine for a chip and hopeless for a tile containing forty thousand fields.

Reading the IoU Curve

Two models that agree at 0.5 and diverge above it Plotting object F1 against the IoU matching threshold, both models score about 0.84 at a threshold of 0.5. As the threshold rises one model holds up to 0.8 while the other collapses, showing that its objects overlap the references loosely rather than sharply. The single score at 0.5 hid that entirely. Object F1 against matching threshold 1.0 0.0 both 0.84 here sharp loose 0.5 0.7 0.9 Report the curve, or at least three thresholds — a single point at 0.5 is not a delineation metric.

The shape of the curve diagnoses the failure mode. A curve that falls steeply between 0.5 and 0.7 means objects are found but their extents are loose, which usually points at a boundary that is systematically dilated or eroded — often the result of all_touched rasterization during training, as discussed in rasterizing labels into segmentation masks. A curve that is already low at 0.5 means objects are being missed or merged outright, which is a topology problem rather than a precision one.

It is worth reporting merge and split counts alongside the curve, because they name the failure directly: a reference object matched by two or more predictions was split, and a prediction matching two or more references merged them.

Verification

import numpy as np

scores = instance_scores(pred_mask, ref_mask, iou_threshold=0.5)
assert 0.0 <= scores["f1"] <= 1.0
print(f"objects: {scores['n_pred']} predicted vs {scores['n_ref']} reference")
print(f"count ratio: {scores['n_pred'] / max(scores['n_ref'], 1):.2f}")
print("boundary F1 @2 px:", round(boundary_f1(pred_mask, ref_mask, tolerance_px=2), 3))
What a boundary tolerance allows A reference boundary is drawn with a tolerance band two pixels wide on each side. A predicted boundary that stays inside the band counts as correct; one that wanders outside it is scored as an error. Setting the band narrower than the imagery's registration accuracy measures georeferencing rather than delineation. Tolerance band around the reference edge reference boundary predicted edge leaves the band here inside the band — counted correct Band half-width = tolerance_px. Choose it from the imagery registration, not from the model.

The count ratio is the fastest possible smoke test. A ratio near 0.5 means objects are being merged in pairs; a ratio near 2 means they are being split. Either is visible before any metric is computed, and either invalidates a delineation product no matter how good the pixel accuracy looks. Compute it per tile rather than only for the scene, because merging and splitting are often concentrated where fields are smallest, and a scene-wide ratio close to one can hide a region where the product is unusable.

Finally, always compute these on the post-processed output rather than the raw prediction, because the minimum mapping unit and majority filter from smoothing and post-processing classification rasters change the object count substantially — usually for the better, and always in a way the metric should reflect.

Common Errors

IoU is high but the object count is wrong

Semantic IoU was computed on the merged mask rather than per instance. Label connected components first; the two metrics answer different questions.

MemoryError building the overlap matrix

The scene contains tens of thousands of objects and the dense IoU matrix is enormous. Compute it per tile, or use a sparse formulation keyed only on pairs that actually intersect.

Boundary F1 is near zero despite a good-looking map

The tolerance is smaller than the registration error between prediction and reference. Raise it to two or three pixels, or fix the alignment first. A quick way to tell the two causes apart is to compute the score at tolerances of one, two and four pixels: a registration problem produces a steep rise across that range, while a genuine delineation problem stays low at all three.

Frequently Asked Questions

Q: Why is per-pixel accuracy misleading for segmentation? Because the pixels that define an object’s shape are a tiny fraction of its area. A model can merge two adjacent fields into one and lose under one per cent of pixels, while destroying the field count that the product exists to produce.

Q: What IoU threshold counts as a match? Fifty per cent is the common convention and it is generous. For delineation work where boundary position matters, report a curve across thresholds from 0.5 to 0.9 rather than a single number, because that curve is where merging and splitting show up.

Q: What tolerance should a boundary F1 use? One to three pixels, chosen from the registration accuracy of the imagery rather than from the model. A tolerance smaller than the georegistration error measures the georeferencing, not the delineation.