Pediatric pneumonia detection (chest X-ray)

What is measured. Whether a pediatric chest radiograph shows pneumonia (bacterial or viral) or is normal. Data: Kermany et al., Cell 2018 (Mendeley Data doi:10.17632/rscbjbr9sj.2), anterior-posterior radiographs of children aged one to five from Guangzhou Women and Children's Medical Center, graded by two expert physicians (the test set checked by a third), as mirrored by hf-vision/chest-xray-pneumonia (JPEG, sizes vary around 1,000-2,000 pixels).

Split. The official test split (624 images: 234 normal, 390 pneumonia) is the test pool and the official training split (5,216: 1,341 / 3,875) the train pool; the 16-image validation split is not used. atlas-full-v1 tests on the whole official test split (624 images); atlas-n100-v1 drew 100 test images stratified by label. Both draw 500 train images stratified by label (largest-remainder quotas, seed-hash rank). Training images whose patient number (person123 in person123_bacteria_456.jpeg, IM-0115 in IM-0115-0001.jpeg) also appears among the test images are excluded from training, so the two versions' train sets differ slightly (461 of 500 shared; 359 pneumonia / 141 normal in atlas-full-v1). The source numbers patients per folder, so a shared number may not be the same child; dropping it is the conservative choice.

Test size. The leaderboard version is atlas-full-v1 on every profile: the whole official test split, 624 images, the set the literature reports on (an image model scores 624 radiographs in about a minute). The earlier atlas-n100-v1 stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Zero-shot (roc_auc_zero_shot). Either a native pneumonia classifier's probability, or, for image-text models, the similarity of the image to "a chest x-ray showing pneumonia" minus its similarity to "a normal chest x-ray". Nothing is trained here.

Probe (roc_auc_probe). Frozen pooled image embeddings; a logistic head is fit on the 500 training images and scores every test image.

Uncertainty. 95 % bootstrap interval over test images, 1,000 resamples stratified by label.

Caveats. Pneumonia images outnumber normal ones about 3 to 1; ROC-AUC is insensitive to that. The dataset is widely used to fine-tune public chest X-ray classifiers (several keremberke/YOLOv8 classification checkpoints on the hub were trained on a derivative of it), so their zero-shot numbers may include training images: a known overlap, reported with the results. Adult chest X-ray models face a domain shift (children, AP views).

Accuracy, sensitivity, specificity, average precision (accuracy_zero_shot, accuracy_probe, sensitivity_zero_shot, sensitivity_probe, specificity_zero_shot, specificity_probe, average_precision_zero_shot, average_precision_probe; metrics review). Kermany et al. report accuracy, sensitivity and specificity next to ROC-AUC. They read each model's own decision: a classifier's or the probe's probability above 0.5, or a positive prompt margin for an image-text model, pneumonia as the positive class; no threshold is tuned here, so a model whose scores are not calibrated to that cut can score a low sensitivity or specificity with a high ROC-AUC. Always answering pneumonia gives accuracy 0.625 (390 / 624) and specificity 0. Average precision is threshold-free, like ROC-AUC.

Data

• pneumonia: hf_dataset hf-vision/chest-xray-pneumonia at revision c1a67c18df1a52ec332ffc00418f4eaa61a9f2bc, licence cc-by-4.0, catalogued as medical-imaging/chest-xray-pneumonia

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by patient_number
• leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)
• results at atlas-full-v1 use subset 3ae07456bf0152dc
• results at atlas-n100-v1 use subset 09b1f320caf781e4

Metrics

• roc_auc_zero_shot — ROC-AUC (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_probe — ROC-AUC (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_zero_shot — Accuracy (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• sensitivity_zero_shot — Sensitivity, pneumonia (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• specificity_zero_shot — Specificity, pneumonia (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_zero_shot — Average precision (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_probe — Accuracy (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• sensitivity_probe — Sensitivity, pneumonia (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• specificity_probe — Specificity, pneumonia (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_probe — Average precision (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

The leaderboard (atlas-full-v1) scores the 624 official test radiographs (100 in atlas-n100-v1). Three test radiographs (input: the JPEG; expected output: a score that ranks pneumonia above normal):

item_id	label
pneumonia-test-person136_bacteria_648	1 (PNEUMONIA, bacterial)
pneumonia-test-person51_virus_105	1 (PNEUMONIA, viral)
pneumonia-test-NORMAL2-IM-0246-0001	0 (NORMAL)