Models / protein-ml / human-protein-language-model-0-5 / benchmarks

protein-ml/human-protein-language-model-0-5

Visibility: PUBLIC

Model card FilesBenchmarks

Group by

ProteinGym protein-ml/proteingym

Domain variant fitness, human PLM 0.5 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-pfam-05

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.053 [−0.353, 0.455]	25
MET_HUMAN_Estevam_2023	0.246 [−0.188, 0.644]	25
S22A1_HUMAN_Yee_2023_activity	0.147 [−0.238, 0.524]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.192 [−0.215, 0.553]	25

0.160

[−0.051, 0.372]

100

13 / 13

0.476

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.519 [0.273, 0.760]	25
MET_HUMAN_Estevam_2023	0.708 [0.479, 0.904]	25
S22A1_HUMAN_Yee_2023_activity	0.487 [0.214, 0.738]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.514 [0.267, 0.770]	25

0.557

[0.436, 0.687]

100

13 / 13

0.759

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.199 [−0.315, 0.554]	25
MET_HUMAN_Estevam_2023	0.439 [−0.053, 0.763]	25
S22A1_HUMAN_Yee_2023_activity	−0.033 [−0.523, 0.360]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.113 [−0.368, 0.510]	25

0.180

[−0.086, 0.359]

100

10 / 13

0.426

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.486 [0.331, 0.770]	25
MET_HUMAN_Estevam_2023	0.940 [0.419, 0.989]	25
S22A1_HUMAN_Yee_2023_activity	0.594 [0.107, 0.891]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.103 [0.061, 0.724]	25

0.530

[0.385, 0.720]

100

—

0.635

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.000 [0.000, 0.333]	25
MET_HUMAN_Estevam_2023	0.000 [0.000, 0.750]	25
S22A1_HUMAN_Yee_2023_activity	0.000 [0.000, 0.333]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.333 [0.000, 0.800]	25

0.083

[0.000, 0.333]

100

—

0.167

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

ProteinGym_v1 protein-ml/proteingym-v1

Domain variant fitness, human PLM 0.5 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-pfam-05

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.053 [−0.353, 0.455]	25
MET_HUMAN_Estevam_2023	0.246 [−0.188, 0.644]	25
S22A1_HUMAN_Yee_2023_activity	0.147 [−0.238, 0.524]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.192 [−0.215, 0.553]	25

0.160

[−0.051, 0.372]

100

13 / 13

0.476

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.519 [0.273, 0.760]	25
MET_HUMAN_Estevam_2023	0.708 [0.479, 0.904]	25
S22A1_HUMAN_Yee_2023_activity	0.487 [0.214, 0.738]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.514 [0.267, 0.770]	25

0.557

[0.436, 0.687]

100

13 / 13

0.759

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.199 [−0.315, 0.554]	25
MET_HUMAN_Estevam_2023	0.439 [−0.053, 0.763]	25
S22A1_HUMAN_Yee_2023_activity	−0.033 [−0.523, 0.360]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.113 [−0.368, 0.510]	25

0.180

[−0.086, 0.359]

100

10 / 13

0.426

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.486 [0.331, 0.770]	25
MET_HUMAN_Estevam_2023	0.940 [0.419, 0.989]	25
S22A1_HUMAN_Yee_2023_activity	0.594 [0.107, 0.891]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.103 [0.061, 0.724]	25

0.530

[0.385, 0.720]

100

—

0.635

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
CCR5_HUMAN_Gill_2023	0.000 [0.000, 0.333]	25
MET_HUMAN_Estevam_2023	0.000 [0.000, 0.750]	25
S22A1_HUMAN_Yee_2023_activity	0.000 [0.000, 0.333]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.333 [0.000, 0.800]	25

0.083

[0.000, 0.333]

100

—

0.167

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board