Benchmarks · protein-ml/esm-protein-localization

ProteinGym protein-ml/proteingym

ProteinGym DMS zero-shot (Spearman) protein-ml/proteingym-dms-zero-shot-spearman

Drawn from: ProteinGym, ProteinGym_v1

atlas-n100-v1

Metric

Value

95% CI

n

Rank

Best in catalogue

Source

Overlap

Spearman ρ (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.550 [0.368, 0.703]	100
CASP3_HUMAN_Roychowdhury_2020	0.388 [0.226, 0.535]	100
CCR5_HUMAN_Gill_2023	0.253 [0.037, 0.443]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.416 [0.227, 0.579]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.678 [0.515, 0.804]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.486 [0.311, 0.637]	100
MLAC_ECOLI_MacRae_2023	0.364 [0.163, 0.541]	100
MTH3_HAEAE_RockahShmuel_2015	0.609 [0.466, 0.732]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.900 [0.848, 0.930]	100
SPG1_STRSG_Olson_2014	0.803 [0.709, 0.865]	100

0.545

[0.491, 0.588]

1000

10 / 17

0.661

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.777 [0.675, 0.864]	100
CASP3_HUMAN_Roychowdhury_2020	0.722 [0.627, 0.814]	100
CCR5_HUMAN_Gill_2023	0.661 [0.550, 0.760]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.710 [0.603, 0.805]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.914 [0.834, 0.977]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.842 [0.746, 0.922]	100
MLAC_ECOLI_MacRae_2023	0.696 [0.585, 0.795]	100
MTH3_HAEAE_RockahShmuel_2015	0.768 [0.651, 0.877]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.967 [0.924, 0.994]	100
SPG1_STRSG_Olson_2014	0.918 [0.838, 0.979]	100

0.797

[0.765, 0.824]

1000

11 / 17

0.848

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.460 [0.221, 0.624]	100
CASP3_HUMAN_Roychowdhury_2020	0.220 [0.042, 0.421]	100
CCR5_HUMAN_Gill_2023	0.240 [0.020, 0.405]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.361 [0.160, 0.559]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.524 [0.364, 0.684]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.401 [0.242, 0.571]	100
MLAC_ECOLI_MacRae_2023	0.280 [0.081, 0.480]	100
MTH3_HAEAE_RockahShmuel_2015	0.281 [0.096, 0.473]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.776 [0.642, 0.883]	100
SPG1_STRSG_Olson_2014	0.403 [0.288, 0.516]	100

0.395

[0.334, 0.446]

1000

11 / 17

0.472

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.817 [0.714, 0.942]	100
CASP3_HUMAN_Roychowdhury_2020	0.756 [0.681, 0.847]	100
CCR5_HUMAN_Gill_2023	0.704 [0.496, 0.755]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.598 [0.452, 0.731]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.916 [0.830, 0.968]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.658 [0.495, 0.810]	100
MLAC_ECOLI_MacRae_2023	0.829 [0.753, 0.898]	100
MTH3_HAEAE_RockahShmuel_2015	0.846 [0.723, 0.920]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.924 [0.879, 0.960]	100
SPG1_STRSG_Olson_2014	0.960 [0.937, 0.984]	100

0.801

[0.763, 0.827]

1000

11 / 17

0.860

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.200 [0.000, 0.500]	100
CASP3_HUMAN_Roychowdhury_2020	0.000 [0.000, 0.300]	100
CCR5_HUMAN_Gill_2023	0.100 [0.000, 0.357]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.000 [0.000, 0.100]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.400 [0.091, 0.600]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.100 [0.000, 0.455]	100
MLAC_ECOLI_MacRae_2023	0.100 [0.000, 0.364]	100
MTH3_HAEAE_RockahShmuel_2015	0.182 [0.000, 0.501]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.600 [0.100, 0.818]	100
SPG1_STRSG_Olson_2014	0.300 [0.100, 0.700]	100

0.198

[0.133, 0.293]

1000

16 / 17

0.360

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

ProteinGym_v1 protein-ml/proteingym-v1

ProteinGym DMS zero-shot (Spearman) protein-ml/proteingym-dms-zero-shot-spearman

Drawn from: ProteinGym, ProteinGym_v1

atlas-n100-v1

Metric

Value

95% CI

n

Rank

Best in catalogue

Source

Overlap

Spearman ρ (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.550 [0.368, 0.703]	100
CASP3_HUMAN_Roychowdhury_2020	0.388 [0.226, 0.535]	100
CCR5_HUMAN_Gill_2023	0.253 [0.037, 0.443]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.416 [0.227, 0.579]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.678 [0.515, 0.804]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.486 [0.311, 0.637]	100
MLAC_ECOLI_MacRae_2023	0.364 [0.163, 0.541]	100
MTH3_HAEAE_RockahShmuel_2015	0.609 [0.466, 0.732]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.900 [0.848, 0.930]	100
SPG1_STRSG_Olson_2014	0.803 [0.709, 0.865]	100

0.545

[0.491, 0.588]

1000

10 / 17

0.661

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.777 [0.675, 0.864]	100
CASP3_HUMAN_Roychowdhury_2020	0.722 [0.627, 0.814]	100
CCR5_HUMAN_Gill_2023	0.661 [0.550, 0.760]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.710 [0.603, 0.805]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.914 [0.834, 0.977]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.842 [0.746, 0.922]	100
MLAC_ECOLI_MacRae_2023	0.696 [0.585, 0.795]	100
MTH3_HAEAE_RockahShmuel_2015	0.768 [0.651, 0.877]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.967 [0.924, 0.994]	100
SPG1_STRSG_Olson_2014	0.918 [0.838, 0.979]	100

0.797

[0.765, 0.824]

1000

11 / 17

0.848

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.460 [0.221, 0.624]	100
CASP3_HUMAN_Roychowdhury_2020	0.220 [0.042, 0.421]	100
CCR5_HUMAN_Gill_2023	0.240 [0.020, 0.405]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.361 [0.160, 0.559]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.524 [0.364, 0.684]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.401 [0.242, 0.571]	100
MLAC_ECOLI_MacRae_2023	0.280 [0.081, 0.480]	100
MTH3_HAEAE_RockahShmuel_2015	0.281 [0.096, 0.473]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.776 [0.642, 0.883]	100
SPG1_STRSG_Olson_2014	0.403 [0.288, 0.516]	100

0.395

[0.334, 0.446]

1000

11 / 17

0.472

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.817 [0.714, 0.942]	100
CASP3_HUMAN_Roychowdhury_2020	0.756 [0.681, 0.847]	100
CCR5_HUMAN_Gill_2023	0.704 [0.496, 0.755]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.598 [0.452, 0.731]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.916 [0.830, 0.968]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.658 [0.495, 0.810]	100
MLAC_ECOLI_MacRae_2023	0.829 [0.753, 0.898]	100
MTH3_HAEAE_RockahShmuel_2015	0.846 [0.723, 0.920]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.924 [0.879, 0.960]	100
SPG1_STRSG_Olson_2014	0.960 [0.937, 0.984]	100

0.801

[0.763, 0.827]

1000

11 / 17

0.860

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (ridge probe) ↑ higher is better · probe

By assay: 10 levels

assay	Value [95% CI]	n
ANCSZ_Hobbs_2022	0.200 [0.000, 0.500]	100
CASP3_HUMAN_Roychowdhury_2020	0.000 [0.000, 0.300]	100
CCR5_HUMAN_Gill_2023	0.100 [0.000, 0.357]	100
CP2C9_HUMAN_Amorosi_2021_abundance	0.000 [0.000, 0.100]	100
ILF3_HUMAN_Tsuboyama_2023_2L33	0.400 [0.091, 0.600]	100
KCNE1_HUMAN_Muhammad_2023_expression	0.100 [0.000, 0.455]	100
MLAC_ECOLI_MacRae_2023	0.100 [0.000, 0.364]	100
MTH3_HAEAE_RockahShmuel_2015	0.182 [0.000, 0.501]	100
RPC1_BP434_Tsuboyama_2023_1R69	0.600 [0.100, 0.818]	100
SPG1_STRSG_Olson_2014	0.300 [0.100, 0.700]	100

0.198

[0.133, 0.293]

1000

16 / 17

0.360

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

tape protein-ml/tape

Secondary structure (Q3) protein-ml/secondary-structure-q3

Drawn from: tape

atlas-n100-v2

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Q3 accuracy (per-residue probe) ↑ higher is better · probe	0.739	[0.724, 0.755]	100	11 / 17	0.813	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
Q8 accuracy (per-residue probe) ↑ higher is better · probe	0.572	[0.551, 0.595]	100	11 / 17	0.663	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board

UniProt Knowledgebase protein-ml/uniprot-knowledgebase

GO function prediction (Swiss-Prot 2025H2) protein-ml/protein-function-go-prediction

Drawn from: UniProt Knowledgebase

atlas-n100-v1

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Fmax (molecular function, probe) ↑ higher is better · probe	0.691	[0.631, 0.782]	100	6 / 17	0.844	reproduced	unknown	Board
Fmax (biological process, probe) ↑ higher is better · probe	0.458	[0.388, 0.583]	100	8 / 17	0.538	reproduced	unknown	Board
Fmax (cellular component, probe) ↑ higher is better · probe	0.735	[0.680, 0.797]	100	7 / 17	0.756	reproduced	unknown	Board
Micro AUPR (molecular function, probe) ↑ higher is better · probe · secondary	0.718	[0.629, 0.815]	100	—	0.875	reproduced	unknown	Board
Smin (molecular function, probe) ↓ lower is better · bits · probe · secondary	3.192	[2.277, 4.080]	100	—	1.727	reproduced	unknown	Board
Micro AUPR (biological process, probe) ↑ higher is better · probe · secondary	0.467	[0.320, 0.630]	100	—	0.595	reproduced	unknown	Board
Smin (biological process, probe) ↓ lower is better · bits · probe · secondary	5.918	[4.162, 7.379]	100	—	4.842	reproduced	unknown	Board
Micro AUPR (cellular component, probe) ↑ higher is better · probe · secondary	0.714	[0.645, 0.784]	100	—	0.758	reproduced	unknown	Board
Smin (cellular component, probe) ↓ lower is better · bits · probe · secondary	3.362	[2.547, 4.073]	100	—	3.092	reproduced	unknown	Board

Other Benchmarks drawn from no hub dataset you can see

Protein membrane association (DeepLoc) protein-ml/protein-membrane-localization

atlas-full-v1

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Q2 accuracy (logistic probe) ↑ higher is better · probe	0.860	[0.844, 0.877]	1728	4 / 17	0.888	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
MCC (logistic probe) ↑ higher is better · probe · secondary	0.714	[0.681, 0.748]	1728	—	0.771	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board

Protein stability ΔΔG (Mega-scale) protein-ml/protein-stability-ddg

atlas-n100-v1

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Spearman ρ (ridge probe) ↑ higher is better · probe	0.224	[0.017, 0.406]	100	4 / 17	0.297	reproduced	unknown	Board
RMSE, kcal/mol (ridge probe) ↓ lower is better · kcal/mol · probe	1.101	[0.861, 1.313]	100	2 / 17	1.095	reproduced	unknown	Board
Pearson r (ridge probe) ↑ higher is better · probe · secondary	0.189	[−0.021, 0.399]	100	—	0.349	reproduced	unknown	Board
MAE, kcal/mol (ridge probe) ↓ lower is better · kcal/mol · probe · secondary	0.819 best	[0.671, 0.976]	100	—	0.819	reproduced	unknown	Board

Protein subcellular localization (DeepLoc) protein-ml/protein-subcellular-localization

atlas-full-v1

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Accuracy (logistic probe) ↑ higher is better · probe	0.767	[0.754, 0.779]	2742	2 / 17	0.801	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
Macro F1 (logistic probe) ↑ higher is better · probe	0.511	[0.493, 0.527]	2742	6 / 17	0.640	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
Macro F1, 5 classes (native classifier) ↑ higher is better · zero-shot	0.638 best	[0.630, 0.647]	2742	1 / 1	0.638	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
MCC (logistic probe) ↑ higher is better · probe · secondary	0.718	[0.703, 0.733]	2742	—	0.758	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board
Accuracy, 5 classes (native classifier) ↑ higher is better · zero-shot · secondary	0.678 best	[0.670, 0.687]	2742	—	0.678	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50	Board