Models / protein-ml / esm2-t48-15b-ur50d / benchmarks

protein-ml/esm2-t48-15b-ur50d

Visibility: PUBLIC

Model card FilesBenchmarks

Group by

Domain variant fitness, human PLM 0.5 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-pfam-05

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.352 [−0.066, 0.727]	25
MET_HUMAN_Estevam_2023	0.315 [−0.124, 0.692]	25
S22A1_HUMAN_Yee_2023_activity	0.490 [0.122, 0.794]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.382 [0.000, 0.672]	25

0.385

[0.172, 0.547]

100

2 / 13

0.476

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.673 [0.447, 0.882]	25
MET_HUMAN_Estevam_2023	0.740 [0.519, 0.947]	25
S22A1_HUMAN_Yee_2023_activity	0.780 [0.560, 0.947]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.681 [0.448, 0.897]	25

0.718

[0.608, 0.823]

100

4 / 13

0.759

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.199 [−0.206, 0.678]	25
MET_HUMAN_Estevam_2023	0.439 [0.033, 0.846]	25
S22A1_HUMAN_Yee_2023_activity	0.294 [−0.028, 0.761]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.280 [−0.201, 0.678]	25

0.303

[0.116, 0.530]

100

5 / 13

0.426

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.547 [0.185, 0.824]	25
MET_HUMAN_Estevam_2023	0.816 [0.573, 0.985]	25
S22A1_HUMAN_Yee_2023_activity	0.898 [0.770, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.281 [0.094, 0.736]	25

0.635 best

[0.518, 0.782]

100

—

0.635

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.000 [0.000, 0.333]	25
MET_HUMAN_Estevam_2023	0.333 [0.000, 0.800]	25
S22A1_HUMAN_Yee_2023_activity	0.333 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.000 [0.000, 0.667]	25

0.167 best

[0.000, 0.450]

100

—

0.167

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Domain variant fitness, human PLM 0.9 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-pfam-09

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.352 [−0.066, 0.727]	25
MET_HUMAN_Estevam_2023	0.315 [−0.124, 0.692]	25
S22A1_HUMAN_Yee_2023_activity	0.490 [0.122, 0.794]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.382 [0.000, 0.672]	25

0.385

[0.172, 0.547]

100

2 / 13

0.476

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.673 [0.447, 0.882]	25
MET_HUMAN_Estevam_2023	0.740 [0.519, 0.947]	25
S22A1_HUMAN_Yee_2023_activity	0.780 [0.560, 0.947]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.681 [0.448, 0.897]	25

0.718

[0.608, 0.823]

100

4 / 13

0.759

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.199 [−0.206, 0.678]	25
MET_HUMAN_Estevam_2023	0.439 [0.033, 0.846]	25
S22A1_HUMAN_Yee_2023_activity	0.294 [−0.028, 0.761]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.280 [−0.201, 0.678]	25

0.303

[0.116, 0.530]

100

5 / 13

0.426

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.547 [0.185, 0.824]	25
MET_HUMAN_Estevam_2023	0.816 [0.573, 0.985]	25
S22A1_HUMAN_Yee_2023_activity	0.898 [0.770, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.281 [0.094, 0.736]	25

0.635 best

[0.518, 0.782]

100

—

0.635

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.000 [0.000, 0.333]	25
MET_HUMAN_Estevam_2023	0.333 [0.000, 0.800]	25
S22A1_HUMAN_Yee_2023_activity	0.333 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.000 [0.000, 0.667]	25

0.167 best

[0.000, 0.450]

100

—

0.167

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Domain variant fitness, human PLM Clans-0.5 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-clan-05

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
ADRB2_HUMAN_Jones_2020	0.272 [−0.146, 0.599]	25
CCR5_HUMAN_Gill_2023	0.352 [−0.066, 0.727]	25
OPSD_HUMAN_Wan_2019	0.363 [−0.086, 0.730]	25
S22A1_HUMAN_Yee_2023_activity	0.488 [0.122, 0.791]	25

0.369

[0.163, 0.552]

100

3 / 13

0.463

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
ADRB2_HUMAN_Jones_2020	0.630 [0.380, 0.840]	25
CCR5_HUMAN_Gill_2023	0.673 [0.447, 0.882]	25
OPSD_HUMAN_Wan_2019	0.722 [0.420, 1.000]	25
S22A1_HUMAN_Yee_2023_activity	0.780 [0.560, 0.947]	25

0.701

[0.576, 0.812]

100

2 / 13

0.751

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
ADRB2_HUMAN_Jones_2020	0.277 [−0.199, 0.614]	25
CCR5_HUMAN_Gill_2023	0.199 [−0.206, 0.678]	25
OPSD_HUMAN_Wan_2019	0.280 [−0.165, 0.787]	25
S22A1_HUMAN_Yee_2023_activity	0.294 [−0.028, 0.761]	25

0.263

[0.058, 0.499]

100

4 / 13

0.426

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
ADRB2_HUMAN_Jones_2020	0.375 [0.000, 0.836]	25
CCR5_HUMAN_Gill_2023	0.547 [0.185, 0.824]	25
OPSD_HUMAN_Wan_2019	0.620 [0.535, 0.912]	25
S22A1_HUMAN_Yee_2023_activity	0.898 [0.770, 1.000]	25

0.610

[0.504, 0.777]

100

—

0.709

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
ADRB2_HUMAN_Jones_2020	0.000 [0.000, 0.667]	25
CCR5_HUMAN_Gill_2023	0.000 [0.000, 0.333]	25
OPSD_HUMAN_Wan_2019	0.000 [0.000, 1.000]	25
S22A1_HUMAN_Yee_2023_activity	0.333 [0.000, 1.000]	25

0.083

[0.000, 0.438]

100

—

0.250

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Domain variant fitness, human PLM Clans-0.9 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-clan-09

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.315 [−0.124, 0.692]	25
MK01_HUMAN_Brenan_2016	0.331 [−0.128, 0.673]	25
SCN5A_HUMAN_Glazer_2019	0.023 [−0.477, 0.444]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.382 [0.000, 0.672]	25

0.263

[0.059, 0.458]

100

5 / 13

0.362

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.740 [0.519, 0.947]	25
MK01_HUMAN_Brenan_2016	0.801 [0.532, 0.991]	25
SCN5A_HUMAN_Glazer_2019	0.487 [0.226, 0.720]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.681 [0.448, 0.897]	25

0.677

[0.566, 0.789]

100

6 / 13

0.722

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.439 [0.033, 0.846]	25
MK01_HUMAN_Brenan_2016	0.487 [0.038, 0.784]	25
SCN5A_HUMAN_Glazer_2019	0.033 [−0.428, 0.371]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.280 [−0.201, 0.678]	25

0.310 best

[0.099, 0.501]

100

1 / 13

0.310

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.816 [0.573, 0.985]	25
MK01_HUMAN_Brenan_2016	0.409 [0.000, 0.920]	25
SCN5A_HUMAN_Glazer_2019	0.489 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.281 [0.094, 0.736]	25

0.499

[0.301, 0.767]

100

—

0.725

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.333 [0.000, 0.800]	25
MK01_HUMAN_Brenan_2016	0.000 [0.000, 0.667]	25
SCN5A_HUMAN_Glazer_2019	0.333 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.000 [0.000, 0.667]	25

0.167

[0.000, 0.458]

100

—

0.250

reproduced

overlap

Train/test overlap declared by the evaluator: protein-ml/uniref50

Board

Protein stability ΔΔG (Mega-scale) protein-ml/protein-stability-ddg

atlas-n100-v1

Metric	Value	95% CI	n	Rank	Best in catalogue	Source	Overlap	Board
Spearman ρ (zero-shot) ↑ higher is better · zero-shot	0.402	[0.172, 0.588]	100	2 / 12	0.410	reproduced	unknown	Board