Atlas
Sign in
Benchmarks / nucleus-instance-segmentation / leaderboard

Nucleus instance segmentation (BBBC039)

imaging
cell-imaging/nucleus-instance-segmentation Task: instance-segmentation Visibility: PUBLIC
OverviewLeaderboard

Leaderboard

Version
Metric

Mean F1, IoU 0.50-0.90 (trained from scratch)

↑ higher is better · supervised · atlas-full-v1

Caicedo et al.'s average F1: per image, the F1 of one-to-one matched nuclei at IoU above each of 0.50, 0.55, ..., 0.90, averaged over the 9 thresholds, then the mean over the 50 test images (a network trained on the 100 training images only); the BBBC039 paper's headline.

Best in the catalogue: 0.894 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (50). Floors are trivial baselines scored on the same items, shown unranked.

# Model Value 95% CI n Source Overlap
1 cell-imaging/monai 0.894 best [0.868, 0.911] 50 reproduced unknown
Secondary metrics (6) · supervised · context, not ranked

AP@0.75 (trained from scratch)

↑ higher is better · supervised

The same Cellpose / StarDist AP as AP@0.5, TP / (TP + FP + FN) per image after one-to-one matching at IoU >= 0.75, then the mean over the 50 test images.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.849 best [0.816, 0.875] 50 reproduced unknown

AP@0.90 (trained from scratch)

↑ higher is better · supervised

The same Cellpose / StarDist AP as AP@0.5, TP / (TP + FP + FN) per image after one-to-one matching at IoU >= 0.90, then the mean over the 50 test images.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.575 best [0.539, 0.612] 50 reproduced unknown

Split rate (trained from scratch)

↓ lower is better · supervised

Caicedo et al.'s split errors per annotated nucleus: the number of annotated nuclei covered by more than one predicted object (matches at IoU > 0.1), summed over the 50 test images, divided by their annotated nuclei.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.011 best [0.008, 0.015] 50 reproduced unknown

Merge rate (trained from scratch)

↓ lower is better · supervised

Caicedo et al.'s merge errors per annotated nucleus: the number of predicted objects covering more than one annotated nucleus (matches at IoU > 0.1), counted once per predicted object, summed over the 50 test images, divided by their annotated nuclei.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.012 best [0.008, 0.017] 50 reproduced unknown

Missed rate (trained from scratch)

↓ lower is better · supervised

Caicedo et al.'s missed errors per annotated nucleus: the number of annotated nuclei no predicted object matches at IoU > 0.7, summed over the 50 test images, divided by their annotated nuclei.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.097 best [0.081, 0.117] 50 reproduced unknown

Extra rate (trained from scratch)

↓ lower is better · supervised

Caicedo et al.'s extra errors per annotated nucleus: the number of predicted objects matching no annotated nucleus at IoU > 0.7, summed over the 50 test images, divided by their annotated nuclei.

Model Value 95% CI n Source Overlap
cell-imaging/monai 0.044 best [0.035, 0.054] 50 reproduced unknown

Protocol

What is measured. Finding every nucleus in a fluorescence image as a separate object (instance segmentation), including touching and dividing nuclei, on BBBC039 v1 (Caicedo et al., Cytometry A 2019): one Hoechst (DNA) field of view per well from 200 compounds of the BBBC022 Cell Painting screen in U2OS cells, 16-bit TIFF, 520 x 696 pixels, with about 23,000 nuclei outlined by hand.

Data and split. The official partition of the metadata archive is used as is: the 50 images of test.txt are the test set and the 100 of training.txt the training set; the 50 validation images are not used. (exc) The test set is the whole official test split (50 images, 5,720 nuclei, 7 to 202 per image) rather than 100 items, and the training set the whole training split (100 images): the dataset has no more, and the official split keeps results comparable with published numbers. Ground-truth masks encode touching nuclei with different colour indices; every 8-connected component of one index is one nucleus (the decoding of the dataset's reference code).

Zero-shot (ap50_zero_shot, count_mae_zero_shot). A generalist segmenter (Cellpose, StarDist, ...) receives the raw 16-bit image and the hint "nucleus" and returns one instance label map of the same size; nothing is trained here. The count error uses the same maps.

Supervised (ap50_supervised). Frameworks with no pretrained weights (a U-Net with an instance head, ...) are trained on the 100 training images and their instance masks only, then predict the 50 test images. Kept apart from zero-shot so that "trained on this benchmark" is never ranked against "used as is".

Metrics. AP@0.5 is the Cellpose / StarDist definition (per image TP / (TP + FP + FN) after an optimal one-to-one matching at IoU >= 0.5, then the mean over images), not COCO's score-ranked AP; no confidence is needed. 95 % bootstrap interval over the test images, 1,000 resamples.

Caveats. Train and test images come from the same 18 BBBC022 plates (the official split is by image, not by plate); every model faces the same split. Cellpose's and StarDist's published generalist models were trained on collections that include parts of the Data Science Bowl 2018 (BBBC038), which overlaps a small fraction of BBBC039: their zero-shot numbers may include images seen in training (a known overlap, reported with the results).

Further metrics (metrics review). mean_f1_iou_zero_shot, mean_f1_iou_supervised are the BBBC039 paper's headline (Caicedo et al. 2019): per image the F1 of nuclei matched one-to-one at IoU above 0.50, 0.55, ..., 0.90, averaged over the 9 thresholds, then the mean over images. ap75_zero_shot, ap75_supervised and ap90_zero_shot, ap90_supervised are the Cellpose AP at stricter IoU (>= 0.75, >= 0.9). split_rate_zero_shot, split_rate_supervised, merge_rate_zero_shot, merge_rate_supervised, missed_rate_zero_shot, missed_rate_supervised and extra_rate_zero_shot, extra_rate_supervised count Caicedo's error types per annotated nucleus, pooled over images: split (a true nucleus covered by more than one prediction) and merge (a prediction covering more than one true nucleus) at IoU > 0.1, missed (a true nucleus without a prediction) and extra (a prediction without a true nucleus) at IoU > 0.7, as in their code; lower is better. count_mae_supervised completes the count error of the zero-shot protocol. Caicedo's F1 and errors use IoU > t, the Cellpose AP IoU >= t (each as its reference code).

Data

  • bbbc039_images: url https://data.broadinstitute.org/bbbc/BBBC039/images.zip, licence CC0-1.0
  • bbbc039_masks: url https://data.broadinstitute.org/bbbc/BBBC039/masks.zip, licence CC0-1.0
  • bbbc039_metadata: url https://data.broadinstitute.org/bbbc/BBBC039/metadata.zip, licence CC0-1.0

Subset

  • seed 20261006
  • test: all item(s)
  • train: all item(s)
  • results at atlas-full-v1 use subset 71d22c95debb3938

Metrics

  • ap50_zero_shot — AP@0.5 (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • count_mae_zero_shot — Nucleus count MAE (zero-shot); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • ap50_supervised — AP@0.5 (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • mean_f1_iou_zero_shot — Mean F1, IoU 0.50-0.90 (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • ap75_zero_shot — AP@0.75 (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • ap90_zero_shot — AP@0.90 (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • split_rate_zero_shot — Split rate (zero-shot); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • merge_rate_zero_shot — Merge rate (zero-shot); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • missed_rate_zero_shot — Missed rate (zero-shot); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • extra_rate_zero_shot — Extra rate (zero-shot); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • count_mae_supervised — Nucleus count MAE (trained from scratch); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • mean_f1_iou_supervised — Mean F1, IoU 0.50-0.90 (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • ap75_supervised — AP@0.75 (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • ap90_supervised — AP@0.90 (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • split_rate_supervised — Split rate (trained from scratch); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • merge_rate_supervised — Merge rate (trained from scratch); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • missed_rate_supervised — Missed rate (trained from scratch); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • extra_rate_supervised — Extra rate (trained from scratch); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples

Sample

Two test images (input: a 16-bit Hoechst image, 520 x 696; expected output: an instance label map of the same size, 0 = background):

item_idBBBC022 plateannotated nuclei
bbbc039-IXMtest_A09_s1_w1CE70AD49-290D-4312-82E6-CDC717F3263720630156
bbbc039-IXMtest_A12_s7_w1EAEEA614-51ED-43B3-A4FF-088730911E4C2064169

A perfect prediction outlines every nucleus once (AP@0.5 = 1, count error 0); merging two touching nuclei costs one false positive and two false negatives at IoU 0.5.