Domain variant fitness, human PLM 0.9 (ProteinGym x Pfam)

What is measured. Zero-shot ranking of single amino-acid substitutions by measured fitness, like protein-ml/proteingym-dms-zero-shot-spearman, but restricted to mutations inside one Pfam domain of a human protein, with the domain span alone as the model's input.

Why this benchmark exists, and why there are four of them. The four hub models of the Human-Protein-Language-Model family hold one checkpoint per Pfam family (-0.5, -0.9) or per Pfam clan (-Clans-0.5, -Clans-0.9), so each can only score a protein through a domain it has a checkpoint for, and only the domain span is in that checkpoint's distribution. At the pinned revisions no Pfam family is covered by all four repos, so there is one benchmark per repo instead of one benchmark for the family: this one's item pool is the coverage of zxcyfr/Human-Protein-Language-Model-0.9 (Pfam family checkpoints, 254 families at the pinned revision). protein-ml/protein-ml/human-protein-language-model-0-9 is evaluated here and nowhere else (its config skips the other three), while ESM-2, ESM-C, ProtBert, ProtT5, AMPLIFY and the other protein LMs are scored on exactly these items — so the number next to the human PLM and the number next to ESM-2 650M are measured on the same variants, with the same inputs.

Data. ProteinGym v1 substitutions and reference file (as the reference benchmark pins them); the checkpoint listings of the four repos at the revisions the harness's model definitions pin; UniProt (release 2026_03) sequences of the 81 human ProteinGym proteins with their Pfam domains (InterPro 110.0) and clans. The three API answers are frozen under benchmarks/<slug>/ and sha256-pinned (the same two files in all four benchmarks), so a later InterPro or UniProt release cannot move the selection silently.

Selection (a function of the seed and those files). Human assays (UniProt_ID ending _HUMAN) whose ProteinGym target_seq occurs exactly once in the UniProt sequence, so the offset between the two numberings is unambiguous; per assay, the eligible Pfam domain location inside target_seq covered by the most of the four repos, then with the most single mutants (ties: Pfam accession, then start); assays with at least 100 such mutants; at most one assay per protein; ordered by how many of the four repos cover their domain (descending — shared families first, so two human PLMs can be compared with each other) and then by seed-hash rank; the first 4 assays are kept. That gives CCR5 (PF00001), MET (PF07714), S22A1 (PF00083) and SRC (PF07714).

Items. 25 in-domain single mutants per assay (100 in all), seed-hash rank — 25, not the usual 100 per group (exc: four benchmarks of four assays is a deliberate compute cost cap**)**. The wild type handed to the model is the domain span, and the mutation is re-indexed to it (mutant); mutant_protein keeps the original whole-protein string, and repo_coverage records which of the four repos can score the item's domain. No train split: zero-shot only.

Metric. Spearman ρ per assay against DMS_score, mean over the 4 assays; 95 % bootstrap interval, 1,000 resamples within assays. A result must be read with its coverage: n = 100 mutants over 4 assays, the assays this repo covers, not the human-PLM family's common ground. ProteinGym's NDCG and top-10 % recall (ndcg_top10_zero_shot, top10_recall_zero_shot, secondary) are computed the same way, per assay then the mean; the top decile of 25 mutants is 2-3 mutants, so they are coarse. ProteinGym's AUC and MCC (auc_zero_shot, mcc_zero_shot) read the assay's DMS_score_bin (ProteinGym's own per-assay cutoff, the label_bin column), per assay then the mean: the ROC AUC of the score, and the Matthews correlation of the score split at its median within the assay. Definition version 2 adds that column (and these two metrics) to version 1's items, which are otherwise the same.

Comparing the four human PLMs with each other. Only on shared assays. At the pinned revisions: -0.5 and -0.9 select the same four assays (CCR5, MET, S22A1, SRC — their coverage differs elsewhere, not here), -Clans-0.5 shares CCR5 and S22A1 with them, -Clans-0.9 shares MET and SRC. atlas-evals compare A B restricted to the two benchmarks of a pair gives that comparison; a mean over all four assays of two different benchmarks does not.

Caveats. Domain spans are 248-376 residues here, so this is not a "short peptide" benchmark; a whole-sequence model scored on the span loses the rest of the protein's context, which is the point (identical inputs) but also means numbers are not comparable with the whole-sequence ProteinGym benchmark, nor with the other three per-repo benchmarks. 25 mutants per assay is a deliberate cost cap (4 benchmarks x 4 assays x 25 = 400 scorings per model, about 20 distinct assays in all), so a per-assay ρ is noisy; the confidence interval says how much. Human proteins are in UniRef, which every protein LM pretrained on, and the human PLMs were trained on human Pfam domains, so the families themselves are in their training distribution — this measures within-family variant ranking, not generalisation to unseen families.

Data

• pg: hf_dataset OATML-Markslab/ProteinGym_v1 at revision 1ea2aa10a328cc085b64328e790c54755fe32b10, licence MIT, catalogued as protein-ml/proteingym-v1
• pg_ref: url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/reference_files/DMS_substitutions.csv, licence MIT, catalogued as protein-ml/proteingym
• hplm: api_frozen ?, licence listing only (file names of public HF model repos)
• pfam: api_frozen ?, licence CC-BY-4.0

Subset

• seed 20261006
• test: 25 item(s) per group
• train: all item(s)
• results at atlas-n25-v2 use subset 60318c8400cb3d13

Metrics

• spearman_zero_shot — Spearman ρ (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• auc_zero_shot — AUC (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• mcc_zero_shot — MCC (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• ndcg_top10_zero_shot — NDCG, top 10 % (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• top10_recall_zero_shot — Top-10 % recall (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

The four assays this repo covers are the same four as protein-ml/protein-domain-variant-fitness-pfam-05 at the pinned revisions (both family repos hold a checkpoint for PF00001, PF07714 and PF00083), so the two benchmarks share their frozen test items and the -0.5 and -0.9 columns are directly comparable:

assay	Pfam (clan)	domain in target_seq	span	covered by
CCR5_HUMAN_Gill_2023	PF00001 (CL0192)	47-297	251	0.5, 0.9, Clans-0.5
MET_HUMAN_Estevam_2023	PF07714 (CL0016)	20-278	259	0.5, 0.9, Clans-0.9
S22A1_HUMAN_Yee_2023_activity	PF00083 (CL0015)	147-522	376	0.5, 0.9, Clans-0.5
SRC_HUMAN_Chakraborty_2023_binding-DAS_25uM	PF07714 (CL0016)	271-518	248	0.5, 0.9, Clans-0.9

Three items of CCR5_HUMAN_Gill_2023 (the 251-residue 7TM span GNMLVILILINCKRLKSMTDIYLLNLAISDLFFLLTVPFW...), by seed-hash rank:

item_id	mutant (domain)	mutant (protein)	DMS_score
CCR5_HUMAN_Gill_2023:K228R	K182R	K228R	0.130
CCR5_HUMAN_Gill_2023:V243Q	V197Q	V243Q	-0.665
CCR5_HUMAN_Gill_2023:V51Q	V5Q	V51Q	0.770

Input: the 251-residue domain span as the wild type plus the re-indexed mutation (e.g. K182R), and the mutated span. Expected output: one score per mutant whose order matches the measurement, here V197Q < K182R < V5Q.