ProteinGym DMS zero-shot (Spearman)

protein

protein-ml/proteingym-dms-zero-shot-spearman Task: regression Visibility: PUBLIC

OverviewLeaderboard

Leaderboard

Version

atlas-n100-v1 140 · default

Protocol

Metric

Spearman ρ (zero-shot)

↑ higher is better · zero-shot · atlas-n100-v1

Spearman correlation between the model's variant-effect score and the measured DMS score on 100 held-out single mutants per assay, averaged over the 10 assays. No training on assay data.

Best in the catalogue: 0.434 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (1000). Floors are trivial baselines scored on the same items, shown unranked.

Model

Value

95% CI

Source

Overlap

protein-ml/amplify-120m

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.469 [0.310, 0.624]	100
CASP3_HUMAN_Roychowdhury_2020	0.553 [0.399, 0.679]	100
CCR5_HUMAN_Gill_2023	0.284 [0.099, 0.463]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.567 [0.382, 0.702]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.300 [0.106, 0.479]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.090 [−0.123, 0.279]	100
MLAC_ECOLI_MacRae_2023	0.402 [0.223, 0.556]	100
MTH3_HAEAE_RockahShmuel_2015	0.696 [0.582, 0.781]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.794 [0.673, 0.872]	100
SPG1_STRSG_Olson_2014	0.184 [−0.018, 0.384]	100

0.434 best

[0.376, 0.483]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)

protein-ml/esm2-t33-650m-ur50d

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.573 [0.428, 0.717]	100
CASP3_HUMAN_Roychowdhury_2020	0.681 [0.573, 0.762]	100
CCR5_HUMAN_Gill_2023	0.311 [0.115, 0.511]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.639 [0.489, 0.740]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.263 [0.078, 0.450]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.096 [−0.099, 0.283]	100
MLAC_ECOLI_MacRae_2023	0.338 [0.173, 0.485]	100
MTH3_HAEAE_RockahShmuel_2015	0.555 [0.390, 0.683]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.663 [0.519, 0.778]	100
SPG1_STRSG_Olson_2014	0.130 [−0.083, 0.342]	100

0.425

[0.366, 0.472]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

protein-ml/progen2-base

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.570 [0.399, 0.717]	100
CASP3_HUMAN_Roychowdhury_2020	0.528 [0.378, 0.663]	100
CCR5_HUMAN_Gill_2023	0.312 [0.133, 0.497]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.596 [0.447, 0.706]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.345 [0.152, 0.509]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.059 [−0.139, 0.261]	100
MLAC_ECOLI_MacRae_2023	0.270 [0.089, 0.433]	100
MTH3_HAEAE_RockahShmuel_2015	0.770 [0.674, 0.837]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.704 [0.583, 0.789]	100
SPG1_STRSG_Olson_2014	0.083 [−0.135, 0.285]	100

0.424

[0.366, 0.471]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90

protein-ml/esm2-t30-150m-ur50d

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.544 [0.394, 0.689]	100
CASP3_HUMAN_Roychowdhury_2020	0.625 [0.508, 0.725]	100
CCR5_HUMAN_Gill_2023	0.296 [0.115, 0.470]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.656 [0.517, 0.761]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.444 [0.248, 0.606]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.050 [−0.160, 0.241]	100
MLAC_ECOLI_MacRae_2023	0.247 [0.064, 0.413]	100
MTH3_HAEAE_RockahShmuel_2015	0.484 [0.307, 0.631]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.740 [0.610, 0.829]	100
SPG1_STRSG_Olson_2014	0.140 [−0.060, 0.344]	100

0.423

[0.361, 0.473]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

protein-ml/amplify-350m

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.521 [0.378, 0.654]	100
CASP3_HUMAN_Roychowdhury_2020	0.554 [0.402, 0.677]	100
CCR5_HUMAN_Gill_2023	0.280 [0.079, 0.479]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.596 [0.439, 0.712]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.288 [0.085, 0.466]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.035 [−0.169, 0.222]	100
MLAC_ECOLI_MacRae_2023	0.321 [0.136, 0.485]	100
MTH3_HAEAE_RockahShmuel_2015	0.773 [0.681, 0.838]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.710 [0.564, 0.811]	100
SPG1_STRSG_Olson_2014	0.079 [−0.134, 0.288]	100

0.416

[0.355, 0.464]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)

protein-ml/prot-bert

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.345 [0.135, 0.541]	100
CASP3_HUMAN_Roychowdhury_2020	0.585 [0.453, 0.699]	100
CCR5_HUMAN_Gill_2023	0.234 [0.044, 0.434]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.470 [0.301, 0.605]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.419 [0.242, 0.576]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.109 [−0.092, 0.290]	100
MLAC_ECOLI_MacRae_2023	0.298 [0.129, 0.450]	100
MTH3_HAEAE_RockahShmuel_2015	0.738 [0.644, 0.806]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.698 [0.551, 0.812]	100
SPG1_STRSG_Olson_2014	0.125 [−0.086, 0.337]	100

0.402

[0.341, 0.452]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: UniRef100

protein-ml/esm2-t36-3b-ur50d

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.603 [0.473, 0.725]	100
CASP3_HUMAN_Roychowdhury_2020	0.605 [0.470, 0.717]	100
CCR5_HUMAN_Gill_2023	0.307 [0.114, 0.496]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.630 [0.475, 0.731]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.235 [0.054, 0.412]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.077 [−0.125, 0.278]	100
MLAC_ECOLI_MacRae_2023	0.283 [0.097, 0.459]	100
MTH3_HAEAE_RockahShmuel_2015	0.584 [0.427, 0.713]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.601 [0.448, 0.725]	100
SPG1_STRSG_Olson_2014	0.092 [−0.127, 0.316]	100

0.402

[0.345, 0.447]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

protein-ml/progen2-small

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.524 [0.356, 0.670]	100
CASP3_HUMAN_Roychowdhury_2020	0.510 [0.356, 0.636]	100
CCR5_HUMAN_Gill_2023	0.326 [0.149, 0.505]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.586 [0.436, 0.699]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.315 [0.132, 0.483]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.142 [−0.064, 0.348]	100
MLAC_ECOLI_MacRae_2023	0.310 [0.145, 0.459]	100
MTH3_HAEAE_RockahShmuel_2015	0.516 [0.345, 0.647]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.728 [0.591, 0.828]	100
SPG1_STRSG_Olson_2014	0.038 [−0.186, 0.240]	100

0.399

[0.340, 0.448]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90

protein-ml/esm2-t12-35m-ur50d

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.495 [0.328, 0.647]	100
CASP3_HUMAN_Roychowdhury_2020	0.547 [0.394, 0.671]	100
CCR5_HUMAN_Gill_2023	0.293 [0.106, 0.494]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.645 [0.496, 0.758]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.378 [0.179, 0.565]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.048 [−0.157, 0.244]	100
MLAC_ECOLI_MacRae_2023	0.202 [0.012, 0.378]	100
MTH3_HAEAE_RockahShmuel_2015	0.396 [0.210, 0.566]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.731 [0.597, 0.825]	100
SPG1_STRSG_Olson_2014	0.139 [−0.068, 0.347]	100

0.388

[0.326, 0.443]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

protein-ml/esm2-t6-8m-ur50d

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.494 [0.316, 0.645]	100
CASP3_HUMAN_Roychowdhury_2020	0.341 [0.155, 0.503]	100
CCR5_HUMAN_Gill_2023	0.262 [0.078, 0.454]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.537 [0.356, 0.670]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.352 [0.148, 0.524]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.043 [−0.160, 0.240]	100
MLAC_ECOLI_MacRae_2023	−0.009 [−0.207, 0.188]	100
MTH3_HAEAE_RockahShmuel_2015	0.206 [0.010, 0.388]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.734 [0.605, 0.825]	100
SPG1_STRSG_Olson_2014	0.196 [−0.020, 0.397]	100

0.316

[0.254, 0.369]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

protein-ml/protgpt2

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.271 [0.079, 0.453]	100
CASP3_HUMAN_Roychowdhury_2020	0.197 [0.010, 0.379]	100
CCR5_HUMAN_Gill_2023	0.228 [0.027, 0.404]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.284 [0.092, 0.461]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	−0.089 [−0.282, 0.091]	100
KCNE1_HUMAN_Muhammad_2023_expression	−0.092 [−0.270, 0.119]	100
MLAC_ECOLI_MacRae_2023	0.122 [−0.097, 0.316]	100
MTH3_HAEAE_RockahShmuel_2015	0.331 [0.120, 0.504]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.526 [0.353, 0.665]	100
SPG1_STRSG_Olson_2014	0.045 [−0.168, 0.261]	100

0.182

[0.121, 0.242]

1000

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Protocol

What is measured. How well a model ranks single amino-acid substitutions of a protein by their experimentally measured effect, on ten deep mutational scanning (DMS) assays of ProteinGym v1 (Notin et al., NeurIPS 2023). DMS_score is oriented by ProteinGym so that higher means fitter.

Assays. Two per ProteinGym coarse_selection_type, drawn by seed-hash rank among the substitution assays whose wild type has at most 1,022 residues and that measure at least 600 single mutants. The choice is a function of the seed and the pinned reference file, not a hand-picked list:

selection type	assays
Activity	ANCSZ_Hobbs_2022, CASP3_HUMAN_Roychowdhury_2020
Binding	CCR5_HUMAN_Gill_2023, SPG1_STRSG_Olson_2014
Expression	CP2C9_HUMAN_Amorosi_2021_abundance, KCNE1_HUMAN_Muhammad_2023_expression
OrganismalFitness	MLAC_ECOLI_MacRae_2023, MTH3_HAEAE_RockahShmuel_2015
Stability	ILF3_HUMAN_Tsuboyama_2023_2L33, RPC1_BP434_Tsuboyama_2023_1R69

Items. Single substitutions only; multi-mutants are dropped. Per assay, 100 test and 500 train mutants, disjoint, both drawn by seed-hash rank.

Zero-shot (spearman_zero_shot). The model scores each test mutant from the wild-type sequence and the mutation alone (for a masked language model, typically the masked-marginal log-ratio), higher meaning fitter, and never sees assay data. Spearman ρ against DMS_score per assay, then the mean over the ten assays.

Probe (spearman_probe). The model embeds each mutated sequence (last layer, mean over residues). Per assay, a ridge head is fit on the 500 train mutants and predicts the 100 test mutants; Spearman ρ per assay, mean over assays. Labels stay in the runner: the model only ever sees sequences.

ProteinGym's other metrics (both protocols, per assay, then the mean over the ten, as ProteinGym's performance_DMS_benchmarks.py computes them). auc_*: ROC AUC of the score against DMS_score_bin, the assay's binarised measurement (ProteinGym's per-assay DMS_binarization_cutoff), joined from the item table's label_bin. mcc_*: Matthews correlation between the score split at its median and DMS_score_bin. ndcg_top10_*: NDCG of the 10 % highest-scored mutants (10 of 100), min-max scaled DMS scores as gains. top10_recall_*: share of the true top 10 % that the score also puts in its top 10 %. A top decile of 10 mutants per assay makes the NDCG and recall intervals wide.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test mutants drawn within each assay.

Caveats. Train and test are disjoint by mutant, not by position, so a probe can learn from train mutants at a test mutant's position that the position is intolerant; the zero-shot metric is unaffected. Ten assays and 100 mutants each are a subset of ProteinGym's own protocol (every mutant of 217 assays): numbers compare models here, not with the official leaderboard.

Data

pg: hf_dataset OATML-Markslab/ProteinGym_v1 at revision 1ea2aa10a328cc085b64328e790c54755fe32b10, licence MIT, catalogued as protein-ml/proteingym-v1
pg_ref: url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/reference_files/DMS_substitutions.csv, licence MIT, catalogued as protein-ml/proteingym

Subset

seed 20261006
test: 100 item(s) per group
train: 500 item(s) per group
results at atlas-n100-v1 use subset 09197ac2b2f80dc3

Metrics

spearman_zero_shot — Spearman ρ (zero-shot); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
auc_zero_shot — AUC (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
mcc_zero_shot — MCC (zero-shot); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
ndcg_top10_zero_shot — NDCG, top 10 % (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
top10_recall_zero_shot — Top-10 % recall (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
spearman_probe — Spearman ρ (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
auc_probe — AUC (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
mcc_probe — MCC (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
ndcg_top10_probe — NDCG, top 10 % (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
top10_recall_probe — Top-10 % recall (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

Sample

Assay SPG1_STRSG_Olson_2014: binding of the protein G B1 domain to IgG-Fc (448-residue wild type MEKEKKVKYF…GALAVASKRKED, 1,045 single mutants). Three of its training mutants:

item_id	mutant	DMS_score
`SPG1_STRSG_Olson_2014:A252K`	A252K	−4.61
`SPG1_STRSG_Olson_2014:G264S`	G264S	0.03
`SPG1_STRSG_Olson_2014:A250Y`	A250Y	2.16

Input: the wild-type sequence, the mutant string and the mutated sequence. Expected output: one score per mutant whose order matches the measured one, here A252K < G264S < A250Y.