Chest X-ray findings (NIH ChestX-ray14)

What is measured. Five findings on frontal chest radiographs, the five of the CheXpert competition: atelectasis, cardiomegaly, consolidation, edema and pleural effusion, each present or absent (multi-label). Data: NIH ChestX-ray14 (Wang et al., CVPR 2017; NIH Clinical Center), 112,120 PA/AP radiographs of 30,805 patients, 1024 x 1024 PNG, labels text-mined from the radiology reports. NIH's "Effusion" is pleural effusion.

Split. The official patient-disjoint split (test_list.txt, 25,596 images; train_val_list.txt, 86,524) is re-imposed on the BahaaEldin0/NIH-Chest-Xray-14 mirror, whose own 80/10/10 split shares 5,359 patients between its parts and is not used. Only 4 of the mirror's 93 parquet shards are fetched (1.8 GB of 45 GB); the test pool is the official-test images found in them (1,013 images of 595 patients) and the train pool the official train_val images (3,473), with any training image of a test patient dropped (none in the official split; re-checked).

Test: the whole test pool (atlas-full-v2). Every official-test image of the fetched shards is a test item: 1,013 radiographs (601 AP, 412 PA) with 135 atelectasis, 34 cardiomegaly, 79 consolidation, 31 edema and 183 pleural effusion positives (natural prevalences 3.1 % to 18.1 %); 658 have none of the five. Version 1 (atlas-n100-v1, no result was ever computed on it) chose 100 of them by iterative stratification with at least 8 positives per finding; its frozen subset is kept as history and is contained in this test.

Train: iterative stratification (Sechidis et al. 2011). 500 training images are chosen so that each finding keeps its share of positives, enriched to at least 8 positives per finding (48, 10, 15, 8 and 51 in the order above). The rarest finding is served first and its images, in seed-hash rank (seed 20261006), go where that finding is most needed; images with none of the five findings fill the rest in rank order. The loader makes this selection (the same 500 as in version 1) and the subset builder takes what it returns, re-checking patient disjointness (disjoint_by: patient).

Zero-shot (mean_auroc_5_zero_shot). A native chest X-ray classifier's probability for each of the five findings (models trained on other label sets map their own heads onto these names in their adapter; a model that lacks one of the five is not scored), or, for image-text models, the similarity margin between a positive and a negative prompt per finding (the prompts are in the definition).

Probe (mean_auroc_5_probe). Frozen pooled image embeddings; one binary logistic head per finding is fit on the 500 training images (C by cross-validation per finding) and scores the 1,013 test images.

Metric. ROC-AUC per finding over the 1,013 test images, then the unweighted mean of the five. 95 % bootstrap interval over test images, 1,000 resamples.

Caveats. NIH labels are mined from reports by NLP and are noisy (estimated accuracy above 90 % for most findings); every model faces the same noise. Several hub chest X-ray models were trained on ChestX-ray14 (chest-x-ray-basic's card names NIH ChestX-ray14 among its training data), so their zero-shot numbers can include training images unless their training kept to the official train_val list: a known overlap reported with the results. With 31 (edema) to 183 positives per finding the rarest per-finding AUROCs are still the noisiest; the mean is steadier. The test is the official-test part of 4 of the 93 shards, not the whole official test list (25,596 images): fetching more shards is a new definition_version.

Average precision (mean_average_precision_5_zero_shot, mean_average_precision_5_probe; metrics review). The same scores ranked per finding, the area under each finding's precision-recall curve (scikit-learn's average precision), then the mean of the five; unlike ROC-AUC it depends on each finding's prevalence, so it shows how many of the top-ranked radiographs are positive. The per-finding values are stored with each result.

Data

• nih: hf_dataset BahaaEldin0/NIH-Chest-Xray-14 at revision 932bcdba9d7d9590704d4f20bc70fc2c3a1bbad7, licence NIH Clinical Center: no restriction on use; attribution required, catalogued as medical-imaging/nih-chest-xray-14
• nih_lists: hf_dataset alkzar90/NIH-Chest-X-ray-dataset at revision 36778e3b0e4f4b4fad31d1728d6190f3eda5b543, licence NIH Clinical Center: no restriction on use; attribution required

Subset

• seed 20261006
• test: all item(s)
• train: 500 item(s), disjoint by patient
• results at atlas-full-v2 use subset 53236fd1abc3f902

Metrics

• mean_auroc_5_zero_shot — Mean ROC-AUC, 5 findings (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_auroc_5_probe — Mean ROC-AUC, 5 findings (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_5_zero_shot — Mean average precision, 5 findings (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_5_probe — Mean average precision, 5 findings (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

The leaderboard (atlas-full-v2) scores all 1,013 official-test radiographs of the fetched shards. Three test radiographs (input: a 1024 x 1024 PNG; expected output: one score per finding, in the order atelectasis, cardiomegaly, consolidation, edema, pleural effusion):

item_id	view	label
nih-00009218_032	AP	[1, 0, 0, 0, 1] (atelectasis, effusion)
nih-00012141_022	AP	[0, 0, 0, 1, 1] (edema, effusion)
nih-00009551_000	PA	[0, 0, 0, 0, 0] (none of the five)