Domain variant fitness, human PLM Clans-0.9 (ProteinGym x Pfam)

What is measured. Zero-shot ranking of single amino-acid substitutions by measured fitness, like protein-ml/proteingym-dms-zero-shot-spearman, but restricted to mutations inside one Pfam domain of a human protein, with the domain span alone as the model's input.

Why this benchmark exists, and why there are four of them. The four hub models of the Human-Protein-Language-Model family hold one checkpoint per Pfam family (-0.5, -0.9) or per Pfam clan (-Clans-0.5, -Clans-0.9), so each can only score a protein through a domain it has a checkpoint for, and only the domain span is in that checkpoint's distribution. At the pinned revisions no Pfam family is covered by all four repos, so there is one benchmark per repo instead of one benchmark for the family: this one's item pool is the coverage of zxcyfr/Human-Protein-Language-Model-Clans-0.9 (Pfam clan checkpoints, 171 clans at the pinned revision). protein-ml/protein-ml/human-protein-language-model-clans-0-9 is evaluated here and nowhere else (its config skips the other three), while ESM-2, ESM-C, ProtBert, ProtT5, AMPLIFY and the other protein LMs are scored on exactly these items — so the number next to the human PLM and the number next to ESM-2 650M are measured on the same variants, with the same inputs.

Data. ProteinGym v1 substitutions and reference file (as the reference benchmark pins them); the checkpoint listings of the four repos at the revisions the harness's model definitions pin; UniProt (release 2026_03) sequences of the 81 human ProteinGym proteins with their Pfam domains (InterPro 110.0) and clans. The three API answers are frozen under benchmarks/<slug>/ and sha256-pinned (the same two files in all four benchmarks), so a later InterPro or UniProt release cannot move the selection silently.

Selection (a function of the seed and those files). Human assays (UniProt_ID ending _HUMAN) whose ProteinGym target_seq occurs exactly once in the UniProt sequence, so the offset between the two numberings is unambiguous; per assay, the eligible Pfam domain location inside target_seq covered by the most of the four repos, then with the most single mutants (ties: Pfam accession, then start); assays with at least 100 such mutants; at most one assay per protein; ordered by how many of the four repos cover their domain (descending — shared families first, so two human PLMs can be compared with each other) and then by seed-hash rank; the first 4 assays are kept. That gives MET and SRC (PF07714), MK01 (PF00069) and SCN5A (PF00520).

Items. 25 in-domain single mutants per assay (100 in all), seed-hash rank — 25, not the usual 100 per group (exc: four benchmarks of four assays is a deliberate compute cost cap**)**. The wild type handed to the model is the domain span, and the mutation is re-indexed to it (mutant); mutant_protein keeps the original whole-protein string, and repo_coverage records which of the four repos can score the item's domain. No train split: zero-shot only.

Metric. Spearman ρ per assay against DMS_score, mean over the 4 assays; 95 % bootstrap interval, 1,000 resamples within assays. A result must be read with its coverage: n = 100 mutants over 4 assays, the assays this repo covers, not the human-PLM family's common ground. ProteinGym's NDCG and top-10 % recall (ndcg_top10_zero_shot, top10_recall_zero_shot, secondary) are computed the same way, per assay then the mean; the top decile of 25 mutants is 2-3 mutants, so they are coarse. ProteinGym's AUC and MCC (auc_zero_shot, mcc_zero_shot) read the assay's DMS_score_bin (ProteinGym's own per-assay cutoff, the label_bin column), per assay then the mean: the ROC AUC of the score, and the Matthews correlation of the score split at its median within the assay. Definition version 2 adds that column (and these two metrics) to version 1's items, which are otherwise the same.

Comparing the four human PLMs with each other. Only on shared assays. At the pinned revisions: -0.5 and -0.9 select the same four assays (CCR5, MET, S22A1, SRC — their coverage differs elsewhere, not here), -Clans-0.5 shares CCR5 and S22A1 with them, -Clans-0.9 shares MET and SRC. atlas-evals compare A B restricted to the two benchmarks of a pair gives that comparison; a mean over all four assays of two different benchmarks does not.

Caveats. Domain spans are 248-289 residues here, so this is not a "short peptide" benchmark; a whole-sequence model scored on the span loses the rest of the protein's context, which is the point (identical inputs) but also means numbers are not comparable with the whole-sequence ProteinGym benchmark, nor with the other three per-repo benchmarks. 25 mutants per assay is a deliberate cost cap (4 benchmarks x 4 assays x 25 = 400 scorings per model, about 20 distinct assays in all), so a per-assay ρ is noisy; the confidence interval says how much. Human proteins are in UniRef, which every protein LM pretrained on, and the human PLMs were trained on human Pfam domains, so the families themselves are in their training distribution — this measures within-family variant ranking, not generalisation to unseen families.

Data

• pg: hf_dataset OATML-Markslab/ProteinGym_v1 at revision 1ea2aa10a328cc085b64328e790c54755fe32b10, licence MIT, catalogued as protein-ml/proteingym-v1
• pg_ref: url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/reference_files/DMS_substitutions.csv, licence MIT, catalogued as protein-ml/proteingym
• hplm: api_frozen ?, licence listing only (file names of public HF model repos)
• pfam: api_frozen ?, licence CC-BY-4.0

Subset

• seed 20261006
• test: 25 item(s) per group
• train: all item(s)
• results at atlas-n25-v2 use subset c77f409d96d18b1f

Metrics

• spearman_zero_shot — Spearman ρ (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• auc_zero_shot — AUC (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• mcc_zero_shot — MCC (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• ndcg_top10_zero_shot — NDCG, top 10 % (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• top10_recall_zero_shot — Top-10 % recall (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

The four assays, with the Pfam domain each keeps and the repos that cover it (MET, MK01 and SRC are kinase domains of clan CL0016):

assay	Pfam (clan)	domain in target_seq	span	covered by
MET_HUMAN_Estevam_2023	PF07714 (CL0016)	20-278	259	0.5, 0.9, Clans-0.9
MK01_HUMAN_Brenan_2016	PF00069 (CL0016)	25-313	289	0.5, 0.9, Clans-0.9
SCN5A_HUMAN_Glazer_2019	PF00520 (CL0030)	1528-1782	255	0.5, 0.9, Clans-0.9
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	PF07714 (CL0016)	271-518	248	0.5, 0.9, Clans-0.9

Three items of MET_HUMAN_Estevam_2023 (MET kinase activity; the kept domain is the 259-residue kinase span VHFNEVIGRGHFGCVYHGTLLDNDGKKIHCAVKSLNRITD...), by seed-hash rank:

item_id	mutant (domain)	mutant (protein)	DMS_score
MET_HUMAN_Estevam_2023:K103P	K84P	K103P	-2.496
MET_HUMAN_Estevam_2023:P118Q	P99Q	P118Q	-2.962
MET_HUMAN_Estevam_2023:A266D	A247D	A266D	0.113

Input: the 259-residue domain span as the wild type plus the re-indexed mutation (e.g. K84P), and the mutated span. Expected output: one score per mutant whose order matches the measurement, here P99Q < K84P < A247D. MET and SRC are shared with the two family benchmarks, which is where -Clans-0.9 can be compared with -0.5 and -0.9.