Clinical toxicity (ClinTox mean ROC-AUC)

Licence to confirm before publication on the hub. Source(s) clintox declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. Two binary labels per drug or drug candidate, from its SMILES (MoleculeNet ClinTox, Gayvert et al. 2016; Wu et al. 2018, as packaged by scikit-fingerprints, 1,477 molecules): FDA_APPROVED (1 = approved) and CT_TOX (1 = failed a clinical trial for toxicity). The score is the unweighted mean of the two ROC-AUCs.

Split. The published OGB scaffold split (ogb_splits_clintox.json: 1,181 / 148 / 148), whole Bemis-Murcko scaffolds per part. (exc) The test pool is the OGB valid and test parts: the OGB test part alone holds 10 toxicity failures, valid and test together 16. Both parts are scaffold disjoint from train (re-checked when the subsets are built).

Enrichment. As for MoleculeNet HIV, the rare class is enriched: in each pool every CT_TOX = 1 molecule is kept and the others are thinned by seed-hash rank (seed 20261006) to four per positive, topped up so a split can be filled. The test pool has only 16 positives, so the 100 test molecules are 16 toxicity failures / 84 others (16 %, against 7 % in the source); the 500 train molecules are 96 / 404, every train-part failure. Every FDA_APPROVED = 0 molecule is also CT_TOX = 1 in this dataset, so the enrichment covers the rare class of both tasks (14 test and 80 train non-approved molecules). The draws are stratified by CT_TOX.

Test size. The leaderboard version is atlas-full-v1 on every profile: the whole enriched test pool and the whole enriched train pool. Here they are exactly 100 and 500 molecules, so atlas-full-v1 holds the same molecules as the earlier atlas-n100-v1 (same subset_hash); that version stays frozen and its cells stay current, as a separate ranking.

Probe (mean_roc_auc_probe, mean_average_precision_probe). Frozen embeddings of the SMILES (last layer, mean over tokens); one logistic head per task (C chosen by stratified CV on log-loss inside train).

Supervised (mean_roc_auc_supervised, mean_average_precision_supervised). Two-task frameworks with no pretrained weights are trained from scratch on the same 500 train molecules and score the same 100 test molecules.

Uncertainty. 95 % bootstrap interval over test molecules, 1,000 resamples stratified by CT_TOX.

Caveats. The two tasks are nearly complementary (an approved drug rarely failed for toxicity), so the mean is close to one task counted twice; with 14 to 16 minority molecules the interval is wide. The source mixes salts and charged forms as they were registered. Approved drugs are in PubChem and DrugBank, which several chemical language models were pretrained on (structures, not these labels).

Data

• clintox: hf_dataset scikit-fingerprints/MoleculeNet_ClinTox at revision 7d8bc6f06ad01d15a7411ac3f0695df1ca63c89b, licence unknown, catalogued as chem-ml/moleculenet-clintox

Subset

• seed 20261006
• test: 100 item(s), stratified by label_ct_tox
• train: 500 item(s), stratified by label_ct_tox, disjoint by scaffold
• leaderboard version atlas-full-v1 on every profile: the whole eligible test and train pool (the sizes above define the n version)

Metrics

• mean_roc_auc_probe — Mean ROC-AUC (logistic probe per task); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_ct_tox
• mean_average_precision_probe — Mean average precision (logistic probe per task); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_ct_tox
• mean_roc_auc_supervised — Mean ROC-AUC (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_ct_tox
• mean_average_precision_supervised — Mean average precision (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_ct_tox

Three training molecules of the frozen subset (input: the SMILES; expected output: two scores, FDA_APPROVED then CT_TOX):

item_id	molecule	SMILES	FDA_APPROVED	CT_TOX
clintox-00014	ethinamate	C#CC1(CCCCC1)OC(=O)N	1	0
clintox-00209	thalidomide	C1CC(=O)NC(=O)C1N2C(=O)C3=CC=CC=C3C2=O	0	1
clintox-00010	elemental selenium	[Se]	0	1

Labels are the source's as registered (thalidomide is listed as a toxicity failure, not under its later approved indications); a salt or a single atom is a valid input. The leaderboard (atlas-full-v1) scores the whole enriched test pool, 100 molecules (the same 100 as atlas-n100-v1).