BBBP ROC-AUC

Licence to confirm before publication on the hub. Source(s) bbbp declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. Whether a small molecule crosses the blood-brain barrier (1) or not (0), from its SMILES, on MoleculeNet BBBP (Martins et al., J. Chem. Inf. Model. 2012; Wu et al., Chem. Sci. 2018) as packaged by scikit-fingerprints: 2,039 molecules.

Split. The published OGB scaffold split (ogb_splits_bbbp.json): molecules grouped by Bemis-Murcko scaffold, the largest scaffold groups to train, the rarest to test, so test molecules have scaffolds the train set never shows. atlas-full-v1 takes the whole test part (204 molecules) and the whole train part (1,631); atlas-n100-v1 drew 100 test molecules from the test part and 500 train molecules from the train part, both stratified by label with largest-remainder quotas and seed-hash rank. The valid part is not used. Scaffold disjointness of the two subsets is re-checked when they are built, not taken from the file.

Test size. The leaderboard version is atlas-full-v1 on every profile: the whole OGB test part (204 molecules) and the whole train part (1,631), because a SMILES probe or a small graph network costs seconds per thousand molecules. The earlier atlas-n100-v1 (100 test / 500 train, drawn as described above) stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Probe (roc_auc_probe, average_precision_probe). Frozen embeddings of the SMILES (last layer, mean over tokens); a logistic head is fit on the train molecules and scores the test molecules.

Supervised (roc_auc_supervised, average_precision_supervised). Frameworks with no pretrained weights (graph networks, fingerprints with a learned head) are trained from scratch on the same train molecules and score the same test molecules. Kept apart from the probe so that "trained on this benchmark" is never ranked against "frozen features".

Uncertainty. 95 % bootstrap interval over test molecules, 1,000 resamples stratified by label.

Caveats. About three quarters of BBBP is labelled 1, but the scaffold test part is nearly balanced (108 / 96, the atlas-full-v1 test), and so is the atlas-n100-v1 test (53 / 47); the full train part is 84 % positive (1,369 / 262), and the logistic head is not re-weighted. The labels are literature curation and contain known oddities (ibuprofen is labelled 0, ethylene 1); every model faces the same ones.

Data

• bbbp: hf_dataset scikit-fingerprints/MoleculeNet_BBBP at revision 86db3f465c006b230b1e5af9032a1e05be4db618, licence unknown, catalogued as chem-ml/moleculenet-bbbp

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by scaffold
• leaderboard version atlas-full-v1 on every profile: the whole eligible test and train pool (the sizes above define the n version)

Metrics

• roc_auc_probe — ROC-AUC (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_probe — Average precision (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_supervised — ROC-AUC (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_supervised — Average precision (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

Three training molecules (input: the SMILES; expected output: a score that ranks penetrating molecules above the others):

item_id	molecule	SMILES	label
bbbp-0035	etomidate	CCOC(=O)c1cncn1C(C)c2ccccc2	1
bbbp-0112	pemoline	NC1=NC(=O)C(O1)c2ccccc2	1
bbbp-0262	ibuprofen	CC(C)Cc1ccc(cc1)C(C)C(O)=O	0

The leaderboard (atlas-full-v1) scores every molecule of the scaffold test part, 204 of them (100 in atlas-n100-v1).