Models / protein-ml / human-protein-language-model-clans-0-9 / benchmarks

protein-ml/human-protein-language-model-clans-0-9

Visibility: PUBLIC

Model card FilesBenchmarks

Group by

ProteinGym protein-ml/proteingym

Domain variant fitness, human PLM Clans-0.9 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-clan-09

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.113 [−0.316, 0.507]	25
MK01_HUMAN_Brenan_2016	0.388 [−0.079, 0.722]	25
SCN5A_HUMAN_Glazer_2019	0.398 [0.015, 0.676]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	−0.008 [−0.445, 0.402]	25

0.223

[0.021, 0.412]

100

9 / 13

0.362

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.604 [0.365, 0.821]	25
MK01_HUMAN_Brenan_2016	0.846 [0.603, 1.000]	25
SCN5A_HUMAN_Glazer_2019	0.807 [0.596, 0.968]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.576 [0.333, 0.790]	25

0.708

[0.596, 0.818]

100

2 / 13

0.722

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.116 [−0.447, 0.529]	25
MK01_HUMAN_Brenan_2016	0.487 [0.045, 0.763]	25
SCN5A_HUMAN_Glazer_2019	0.523 [0.033, 0.840]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	−0.053 [−0.458, 0.445]	25

0.268

[0.018, 0.459]

100

2 / 13

0.310

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.637 [0.083, 0.976]	25
MK01_HUMAN_Brenan_2016	0.819 [0.394, 0.938]	25
SCN5A_HUMAN_Glazer_2019	0.774 [0.297, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.266 [0.078, 1.000]	25

0.624

[0.406, 0.872]

100

—

0.725

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.000 [0.000, 0.600]	25
MK01_HUMAN_Brenan_2016	0.000 [0.000, 0.667]	25
SCN5A_HUMAN_Glazer_2019	0.333 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.333 [0.000, 1.000]	25

0.167

[0.000, 0.475]

100

—

0.250

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

ProteinGym_v1 protein-ml/proteingym-v1

Domain variant fitness, human PLM Clans-0.9 (ProteinGym x Pfam) protein-ml/protein-domain-variant-fitness-clan-09

Drawn from: ProteinGym, ProteinGym_v1

atlas-n25-v2

Metric

Value

95% CI

Rank

Best in catalogue

Source

Overlap

Spearman ρ (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.113 [−0.316, 0.507]	25
MK01_HUMAN_Brenan_2016	0.388 [−0.079, 0.722]	25
SCN5A_HUMAN_Glazer_2019	0.398 [0.015, 0.676]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	−0.008 [−0.445, 0.402]	25

0.223

[0.021, 0.412]

100

9 / 13

0.362

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

AUC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.604 [0.365, 0.821]	25
MK01_HUMAN_Brenan_2016	0.846 [0.603, 1.000]	25
SCN5A_HUMAN_Glazer_2019	0.807 [0.596, 0.968]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.576 [0.333, 0.790]	25

0.708

[0.596, 0.818]

100

2 / 13

0.722

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

MCC (zero-shot, domain input) ↑ higher is better · zero-shot

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.116 [−0.447, 0.529]	25
MK01_HUMAN_Brenan_2016	0.487 [0.045, 0.763]	25
SCN5A_HUMAN_Glazer_2019	0.523 [0.033, 0.840]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	−0.053 [−0.458, 0.445]	25

0.268

[0.018, 0.459]

100

2 / 13

0.310

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

NDCG, top 10 % (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.637 [0.083, 0.976]	25
MK01_HUMAN_Brenan_2016	0.819 [0.394, 0.938]	25
SCN5A_HUMAN_Glazer_2019	0.774 [0.297, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.266 [0.078, 1.000]	25

0.624

[0.406, 0.872]

100

—

0.725

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board

Top-10 % recall (zero-shot, domain input) ↑ higher is better · zero-shot · secondary

By assay: 4 levels

assay	Value [95% CI]	n
MET_HUMAN_Estevam_2023	0.000 [0.000, 0.600]	25
MK01_HUMAN_Brenan_2016	0.000 [0.000, 0.667]	25
SCN5A_HUMAN_Glazer_2019	0.333 [0.000, 1.000]	25
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	0.333 [0.000, 1.000]	25

0.167

[0.000, 0.475]

100

—

0.250

reproduced

overlap

Train/test overlap declared by the evaluator: human-proteome Pfam/clan domain sequences (DS-UFT), protein-ml/uniref50

Board