Benchmarks

Similarity of 100 biomedical sentence pairs (BIOSSES): Pearson and Spearman correlation of embedding cosine with expert 0-4 ratings, zero-shot.

Chemical mention recognition in PubMed abstracts (BioCreative V CDR official test set): strict entity F1 of a token probe and of native NER heads.

Problems, treatments and tests in clinical case-report sentences (MACCROBAT): strict entity F1, token probe and native NER heads, report-disjoint.

CAFA Fmax (MF, BP, CC) on 100 Swiss-Prot proteins created after 2025-07-01 with experimental GO evidence; probe and native head.

Is a clinical concept negated in its sentence? NegEx test kit: macro F1 of native assertion heads or prompted LMs, and of an embedding probe.

USMLE-style 4-option medical exam questions (MedQA official test split): accuracy, zero-shot letter likelihood and a pairwise embedding probe.

Yes/no/maybe answers to research questions from their PubMed abstract (PubMedQA PQA-L official test set): accuracy and macro F1, zero-shot and probe.

Professions and employment statuses in Spanish COVID-19 tweets (ProfNER validation set): strict entity F1 of a token probe on frozen embeddings.

Zero-shot Spearman, AUC and MCC on in-domain single mutants of the 4 human ProteinGym assays the Clans-0.9 human-PLM repo covers; same spans for every model.

CRISPR-knockout retrieval from Cell Painting wells (RxRx3-core, 20 genes, held-out plates): replicate mAP zero-shot and probe accuracy.

Nucleus instance segmentation on the BBBC039 test split (50 images): AP@0.5 zero-shot and trained from scratch, plus count MAE.

Aqueous solubility (log mol/L) on the OGB scaffold split of MoleculeNet ESOL: RMSE of a ridge probe and of from-scratch models; lower is better.

Blood-brain barrier penetration of small molecules, ROC-AUC on the OGB scaffold split of MoleculeNet BBBP (probe and from-scratch).

Zero-shot Spearman, AUC and MCC on in-domain single mutants of the 4 human ProteinGym assays the 0.5 human-PLM repo covers; same spans for every model.

ClinTox clinical-trial toxicity and FDA approval on scaffold-held-out drugs, mean ROC-AUC over both tasks: logistic probe and from-scratch models.

HIV replication inhibition of small molecules on the OGB scaffold split of MoleculeNet HIV: ROC-AUC (probe, from scratch) and AP on a 20 % positive test set.

Twelve Tox21 toxicity assays on the OGB scaffold split, mean ROC-AUC over measured labels: logistic probe per assay and from-scratch models.

Accuracy of a linear probe on frozen DNA embeddings, mean over three Genomic Benchmarks sets: enhancers, non-TATA promoters, coding vs intergenic.

Zero-shot Spearman, AUC and MCC on in-domain single mutants of the 4 human ProteinGym assays the 0.9 human-PLM repo covers; same spans for every model.

ClinVar pathogenic vs benign SNVs from 512 bp hg38 windows: ROC-AUC of the zero-shot alt/ref log-likelihood ratio and of a linear probe.

Matthews correlation of a linear probe on frozen DNA embeddings, mean over the 18 revised Nucleotide Transformer downstream tasks.

Five CheXpert-competition findings on NIH ChestX-ray14 (official test split): mean ROC-AUC zero-shot and of linear probes.

Five CheXpert competition findings on radiologist-labelled CheXpert validation + test studies: mean ROC-AUC zero-shot and of linear probes.

Spleen segmentation on 2-D CT slices (MSD Task09, 8 held-out patients): Dice and HD95 (mm), zero-shot and trained from scratch.

Pneumonia vs normal on pediatric chest X-rays (Kermany 2018 test split): ROC-AUC zero-shot and of a linear probe.

Share of predicted ligand poses within 2 Å RMSD that also pass the PoseBusters physical checks, on 214 PoseBusters complexes, for docking and for co-folding.

Designability and median self-consistency RMSD of 100 unconditionally generated backbones (lengths 70/100/150), via ProteinMPNN + ESMFold.

Zero-shot Spearman, AUC and MCC on in-domain single mutants of the 4 human ProteinGym assays the Clans-0.5 human-PLM repo covers; same spans for every model.

Native sequence recovery of sequences designed for 100 single-chain 2024 X-ray backbones (backbone coordinates only as input).

Membrane-bound vs soluble from sequence (DeepLoc 1.0 test set, 1,728 proteins): probe Q2 accuracy and MCC.

ΔΔG of single mutants on 100 Mega-scale variants from held-out wild-type clusters: zero-shot Spearman, probe Spearman and RMSE (kcal/mol).

CA lDDT, TM-score and CA RMSD of predicted structures against 100 single-chain X-ray entries released in 2024.

Base-pair F1 on 100 bpRNA TS0 RNAs (≤ 200 nt) of a 2-D convolutional pair head on frozen nucleotide embeddings of 2,000 TR0 RNAs.

Ten-class subcellular localization from sequence (DeepLoc 1.0 test set): probe accuracy and macro F1, plus native classifiers on a 5-class grouping.

Spearman ρ between model scores and measured fitness of single mutants, mean over 10 ProteinGym DMS assays (2 per selection type).

Per-residue secondary structure accuracy on 100 CB513 proteins: 3 states (H/E/C) by probe or DSSP of a predicted structure, 8 DSSP states by probe.

Protein abundance of 3,333 E. coli genes from their coding sequence (CaLM/PaxDb): Spearman ρ of zero-shot pseudo-log-likelihood and of a ridge probe, and RMSE.

Per-nucleotide 2A3 SHAPE reactivity of Ribonanza RNAs (SN-filtered) from a ridge probe on frozen per-nucleotide embeddings: MAE and Pearson r.

SpliceAI top-k accuracy of donor and acceptor sites over 100 whole pre-mRNAs of held-out chromosomes (chr1/3/5/7/9): native and per-nucleotide probe.

Ten liver cell types of a held-out Tabula Sapiens donor (CELLxGENE): macro F1 of probes, clustering (with ARI) and from-scratch annotators.

Norman 2019 K562 CRISPRa, GEARS simulation split (seed 1): Pearson on expression change and DE-gene error for 107 held-out perturbations.

Norman 2019 CRISPRa, GEARS split: cell-eval's DE overlap of predicted single cells (Wilcoxon vs control) with the top-20 DE genes, 107 perturbations.

100 CRISPRi knockdowns in held-out HepG2 cells, trained on K562/RPE1/Jurkat (Replogle 2022, State zero-shot split): Pearson Δ and DE-gene MSE.

100 CRISPRi knockdowns in held-out HepG2 (Replogle, State split): cell-eval's DE overlap of predicted cells (Wilcoxon) with the top-20 DE genes.

Four-donor Tabula Sapiens liver cells (donor = batch): scIB cell-type ASW, batch ASW and graph connectivity of frozen and from-scratch embeddings.