CheXpert AUROC

Licence to confirm before publication on the hub. Source(s) chexpert_test_labels declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. The five CheXpert competition findings on frontal chest radiographs: atelectasis, cardiomegaly, consolidation, edema and pleural effusion, each present or absent (multi-label). Data: CheXpert (Irvin et al., AAAI 2019; Stanford Hospital), here the 512-pixel release StanfordAIMI/CheXpert-v1.0-512.

Test labels are radiologists', not NLP. The test pool is the frontal views of the radiologist-labelled studies: the validation studies (consensus of 3 board-certified radiologists) and the 500 test studies (majority of 5, groundtruth.csv of rajpurkarlab/cheXpert-test-set-labels), one frontal image per study (the lowest view number). Training images come from the training split, whose labels are extracted from reports; their uncertain mentions are counted positive (U-Ones, Irvin et al.) and blanks negative. CheXpert's train, validation and test patients are disjoint (re-checked when loading, and disjoint_by: patient).

Subset: iterative stratification (Sechidis et al. 2011), 100 test and 500 training images, each finding kept at its share of positives and at least 8 (seed 20261006; the loader makes the selection, the subset builder takes it). Only a few training shards are fetched; the train pool is the frontal training images they hold.

Zero-shot (mean_auroc_5_zero_shot). A native classifier's probability per finding (adapters map their own heads onto these five names; a model lacking one is not scored), or an image-text model's margin between a positive and a negative prompt per finding.

Probe (mean_auroc_5_probe). One binary logistic head per finding on frozen pooled image embeddings of the 500 training images.

Metric. ROC-AUC per finding, then the unweighted mean of the five, as the CheXpert competition (which scored studies; here one frontal image per study). 95 % bootstrap interval over test studies, 1,000 resamples.

Caveats. Many public chest X-ray models were trained on CheXpert (vit-chest-xray's card says it was fine-tuned on CheXpert): their training data excludes the validation and test studies by construction, but they have seen the institution and labeller, so they are not comparable with models that have not. Blocked until the data can be fetched as pinned.

Average precision (mean_average_precision_5_zero_shot, mean_average_precision_5_probe; metrics review). The same scores ranked per finding, the area under each finding's precision-recall curve, then the mean of the five.

Data

• chexpert: hf_dataset StanfordAIMI/CheXpert-v1.0-512 at revision 03dcca33e1614adc98c184e15540d4f7ca2e08bc, licence Stanford University School of Medicine CheXpert Dataset Research Use Agreement
• chexpert_labels: url https://stanfordaimi.azurewebsites.net/datasets/8cbd9ed4-2eb9-4565-affc-111cf4f7ebe2, licence Stanford University School of Medicine CheXpert Dataset Research Use Agreement
• chexpert_test_labels: url https://raw.githubusercontent.com/rajpurkarlab/cheXpert-test-set-labels/a03eb3438ca55c28eb4379dd0dd09e3b38c0c414/groundtruth.csv, licence none declared (repository without a licence; CheXpert Research Use Agreement applies)

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s), disjoint by patient

Metrics

• mean_auroc_5_zero_shot — Mean ROC-AUC, 5 findings (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_auroc_5_probe — Mean ROC-AUC, 5 findings (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_5_zero_shot — Mean average precision, 5 findings (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mean_average_precision_5_probe — Mean average precision, 5 findings (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Not shown: CheXpert's Research Use Agreement does not allow redistributing its images or labels. An item is one frontal radiograph (512 pixels on the short side) of a validation or test study; the expected output is one score per finding, in the order atelectasis, cardiomegaly, consolidation, edema, pleural effusion, e.g. a study the radiologists read as cardiomegaly + pleural effusion has label [0, 1, 0, 0, 1].