vision3d.metrics#

3D object detection evaluation metrics.

Classes

`APInterpolation`(*values)	AP interpolation mode.
`MeanAveragePrecision3D`(class_ids[, ...])	3D detection mAP metric.
`MeanAveragePrecision3DResult`	Structured result returned by `MeanAveragePrecision3D.compute()`.
`NuScenesDetectionScore`(class_ids[, ...])	nuScenes Detection Score (NDS) metric.
`NuScenesDetectionScoreResult`	Structured result returned by `NuScenesDetectionScore.compute()`.
`Prediction3D`	Per-frame detection output.
`Target3D`	Per-frame ground-truth annotations.

class vision3d.metrics.APInterpolation(*values)[source]#

Bases: Enum

AP interpolation mode.

R40 = 'r40'#: 40-point interpolation (modern KITTI default).

R11 = 'r11'#: 11-point interpolation (legacy KITTI, PASCAL VOC07).

R101 = 'r101'#: 101-point interpolation (COCO).

ALL_POINTS = 'all_points'#: VOC07 area-under-curve at every recall change.

class vision3d.metrics.MeanAveragePrecision3D(class_ids, iou_thresholds=(0.5, 0.7), ap_interpolation=APInterpolation.R40, range_bins=None)[source]#

Bases: object

3D detection mAP metric.

Matching is greedy by descending score, one prediction to one ground truth, with precision/recall accumulated globally across frames (KITTI convention).

Parameters:

class_ids (list[int]) – Integer class IDs to score. Predictions and GTs with labels outside this set are ignored.
iou_thresholds (tuple[float, ...]) – IoU thresholds to report AP at. Default (0.5, 0.7).
ap_interpolation (APInterpolation) – Interpolation mode. Default APInterpolation.R40.
range_bins (tuple[tuple[float, float], ...] | None) – Optional distance bins [low, high) in meters from the sensor origin. When set, AP is also broken down per bin; boxes are bucketed by their center’s distance.

update(preds, targets)[source]#

Accumulate one or more frames of predictions vs ground truth.

Parameters:

preds (list[Prediction3D]) – List of per-frame Prediction3D dicts.
targets (list[Target3D]) – List of per-frame Target3D dicts.

Raises:

ValueError – If preds and targets differ in length.

Return type:

None

compute()[source]#

Compute the aggregated metric.

Returns:: Populated MeanAveragePrecision3DResult.
Return type:: MeanAveragePrecision3DResult

reset()[source]#

Clear all accumulated state.

Return type:: None

class vision3d.metrics.MeanAveragePrecision3DResult[source]#

Bases: TypedDict

Structured result returned by MeanAveragePrecision3D.compute().

Undefined slots (buckets with no ground-truth boxes accumulated) are reported as -1.0 and callers can filter them with x >= 0.

Variables:

mAP (float) – Overall mean AP, taken over every defined (class, iou, bin) bucket.
mAP_per_class (dict[int, float]) – AP per class, averaged over the other axes.
AP_per_iou (dict[float, float]) – AP per IoU threshold, averaged over the other axes.
AP_per_class_per_iou (dict[tuple[int, float], float]) – AP per (class, iou) pair, averaged over range bins (or a single value when range bucketing is disabled).
AP_per_range (dict[tuple[float, float], float]) – AP per range bin, averaged over the other axes. Only present when range_bins was set on the metric.
AP_per_class_per_range (dict[tuple[int, tuple[float, float]], float]) – AP per (class, range_bin) pair, averaged over IoU thresholds. Only present when range_bins was set on the metric.

class vision3d.metrics.NuScenesDetectionScore(class_ids, dist_thresholds=(0.5, 1.0, 2.0, 4.0), tp_threshold=2.0, min_recall=0.1, min_precision=0.1, mean_ap_weight=5.0, tp_metrics=TP_METRICS, orientation_periods=None, skip_tp_metrics=None)[source]#

Bases: object

nuScenes Detection Score (NDS) metric.

Predictions are matched to ground truth greedily by ascending 2D center distance (within each frame, one ground truth per prediction), pooled across all frames and sorted by descending score. Average precision is integrated per distance threshold with the nuScenes recall/precision clipping, and the five TP error metrics are measured at tp_threshold.

Parameters:

class_ids (list[int]) – Integer class IDs to score. Predictions and ground truths with labels outside this set are ignored.
dist_thresholds (tuple[float, ...]) – BEV center-distance match thresholds in meters over which AP is averaged. Default (0.5, 1.0, 2.0, 4.0).
tp_threshold (float) – The single distance threshold at which the TP error metrics are computed. Must be one of dist_thresholds. Default 2.0.
min_recall (float) – Recall below which precision is discarded before integrating AP and TP errors. Default 0.1.
min_precision (float) – Precision floor subtracted before integrating AP. Default 0.1.
mean_ap_weight (float) – Weight of mAP relative to the TP scores in the NDS. Default 5.0.
tp_metrics (tuple[str, ...]) – The TP error metrics that participate in the score, a subset of ("trans_err", "scale_err", "orient_err", "vel_err", "attr_err"). The NDS is normalized over mean_ap_weight plus the number of active TP metrics, so dropping a metric cleanly removes it rather than scoring it as a perfect (or worst-case) constant. Default: all five (the nuScenes setting). Drop "vel_err"/"attr_err" for datasets without velocity or attribute annotations. "vel_err" requires per-box velocities and "attr_err" requires attributes on every frame passed to update().
orientation_periods (dict[int, float] | None) – Optional {class_id: period} overriding the orientation periodicity (radians) used for the orientation error. Defaults to \(2\pi\) for every class.
skip_tp_metrics (dict[int, set[str]] | None) – Optional {class_id: set_of_metric_names} forcing the listed TP metrics to nan for that class (and thus excluding it from those metrics’ class averages).

Raises:

ValueError – If class_ids or dist_thresholds is empty, if tp_threshold is not in dist_thresholds, or if tp_metrics is empty or names an unknown metric.

classmethod from_class_names(class_names, dist_thresholds=(0.5, 1.0, 2.0, 4.0), tp_threshold=2.0, min_recall=0.1, min_precision=0.1, mean_ap_weight=5.0, tp_metrics=TP_METRICS)[source]#

Build the metric from class names using the nuScenes presets.

Class IDs are assigned by position (class_names[i] -> i). For the official nuScenes classes this wires up the barrier half-period orientation and the barrier/traffic_cone TP-metric skips.

The orientation_periods and skip_tp_metrics constructor arguments are derived from the names and therefore cannot be set here; the remaining arguments are forwarded to NuScenesDetectionScore unchanged.

Parameters:

class_names (list[str]) – Ordered class names; their indices become the integer class IDs scored by the metric.
dist_thresholds (tuple[float, ...]) – BEV center-distance match thresholds in meters over which AP is averaged.
tp_threshold (float) – The single distance threshold at which the TP error metrics are computed; must be one of dist_thresholds.
min_recall (float) – Recall below which precision is discarded before integrating AP and TP errors.
min_precision (float) – Precision floor subtracted before integrating AP.
mean_ap_weight (float) – Weight of mAP relative to the TP scores in the NDS.
tp_metrics (tuple[str, ...]) – The TP error metrics that participate in the score.

Returns:

A configured NuScenesDetectionScore.

Return type:

NuScenesDetectionScore

update(preds, targets)[source]#

Accumulate one or more frames of predictions vs ground truth.

Parameters:

preds (list[Prediction3D]) – List of per-frame Prediction3D dicts.
targets (list[Target3D]) – List of per-frame Target3D dicts.

Raises:

ValueError – If preds and targets differ in length, or if an active TP metric needs annotations (velocities for vel_err, attributes for attr_err) that a frame does not provide.

Return type:

None

compute()[source]#

Compute the aggregated NDS and its constituent metrics.

Returns:: A populated NuScenesDetectionScoreResult.
Return type:: NuScenesDetectionScoreResult

reset()[source]#

Clear all accumulated frames.

Return type:: None

class vision3d.metrics.NuScenesDetectionScoreResult[source]#

Bases: TypedDict

Structured result returned by NuScenesDetectionScore.compute().

Variables:

nd_score (float) – The nuScenes Detection Score, a weighted combination of mean_ap and the TP scores in [0, 1].
mean_ap (float) – Mean Average Precision, averaged over distance thresholds and classes.
mean_dist_aps (dict[int, float]) – Per-class AP, averaged over distance thresholds. Keyed by class ID.
label_aps (dict[tuple[int, float], float]) – AP per (class_id, distance_threshold) pair.
tp_errors (dict[str, float]) – Mean TP error per active metric name (a subset of trans_err, scale_err, orient_err, vel_err, attr_err), averaged over classes (ignoring skipped classes).
tp_scores (dict[str, float]) – Per-metric score max(0, 1 - error), for the active metrics only.
label_tp_errors (dict[tuple[int, str], float]) – TP error per (class_id, metric_name) pair over the active metrics; nan for skipped class/metric combinations.

class vision3d.metrics.Prediction3D[source]#

Bases: TypedDict

Per-frame detection output.

Variables:

boxes (vision3d.tensors._bounding_boxes_3d.BoundingBoxes3D) – [N, K] predicted 3D bounding boxes; K depends on the box format.
scores (torch.Tensor) – [N] confidence scores.
labels (torch.Tensor) – [N] integer class labels.
velocities (torch.Tensor) – Optional [N, 2] ground-plane (xy) velocities in m/s. Required by NuScenesDetectionScore to compute the mean Average Velocity Error (AVE). Defaults to zeros when omitted.
attributes (torch.Tensor) – Optional [N] integer attribute labels. Used by NuScenesDetectionScore to compute the mean Average Attribute Error (AAE). A negative value marks “no attribute”. Defaults to -1 when omitted.

class vision3d.metrics.Target3D[source]#

Bases: TypedDict

Per-frame ground-truth annotations.

Variables:

boxes (vision3d.tensors._bounding_boxes_3d.BoundingBoxes3D) – [M, K] ground-truth 3D bounding boxes.
labels (torch.Tensor) – [M] integer class labels.
velocities (torch.Tensor) – Optional [M, 2] ground-plane (xy) velocities in m/s. See Prediction3D.
attributes (torch.Tensor) – Optional [M] integer attribute labels. A negative value marks “no attribute”, in which case the box is ignored when computing the attribute error. See Prediction3D.