Domain variant fitness, human PLM Clans-0.5 (ProteinGym x Pfam)

What is measured. Zero-shot ranking of single amino-acid substitutions by measured fitness, like protein-ml/proteingym-dms-zero-shot-spearman, but restricted to mutations inside one Pfam domain of a human protein, with the domain span alone as the model's input.

Why this benchmark exists, and why there are four of them. The four hub models of the Human-Protein-Language-Model family hold one checkpoint per Pfam family (-0.5, -0.9) or per Pfam clan (-Clans-0.5, -Clans-0.9), so each can only score a protein through a domain it has a checkpoint for, and only the domain span is in that checkpoint's distribution. At the pinned revisions no Pfam family is covered by all four repos, so there is one benchmark per repo instead of one benchmark for the family: this one's item pool is the coverage of zxcyfr/Human-Protein-Language-Model-Clans-0.5 (Pfam clan checkpoints, 82 clans at the pinned revision). protein-ml/protein-ml/human-protein-language-model-clans-0-5 is evaluated here and nowhere else (its config skips the other three), while ESM-2, ESM-C, ProtBert, ProtT5, AMPLIFY and the other protein LMs are scored on exactly these items — so the number next to the human PLM and the number next to ESM-2 650M are measured on the same variants, with the same inputs.

Data. ProteinGym v1 substitutions and reference file (as the reference benchmark pins them); the checkpoint listings of the four repos at the revisions the harness's model definitions pin; UniProt (release 2026_03) sequences of the 81 human ProteinGym proteins with their Pfam domains (InterPro 110.0) and clans. The three API answers are frozen under benchmarks/<slug>/ and sha256-pinned (the same two files in all four benchmarks), so a later InterPro or UniProt release cannot move the selection silently.

Selection (a function of the seed and those files). Human assays (UniProt_ID ending _HUMAN) whose ProteinGym target_seq occurs exactly once in the UniProt sequence, so the offset between the two numberings is unambiguous; per assay, the eligible Pfam domain location inside target_seq covered by the most of the four repos, then with the most single mutants (ties: Pfam accession, then start); assays with at least 100 such mutants; at most one assay per protein; ordered by how many of the four repos cover their domain (descending — shared families first, so two human PLMs can be compared with each other) and then by seed-hash rank; the first 4 assays are kept. That gives ADRB2, CCR5 and OPSD (PF00001, clan CL0192) and S22A1 (PF00083, CL0015).

Items. 25 in-domain single mutants per assay (100 in all), seed-hash rank — 25, not the usual 100 per group (exc: four benchmarks of four assays is a deliberate compute cost cap**)**. The wild type handed to the model is the domain span, and the mutation is re-indexed to it (mutant); mutant_protein keeps the original whole-protein string, and repo_coverage records which of the four repos can score the item's domain. No train split: zero-shot only.

Metric. Spearman ρ per assay against DMS_score, mean over the 4 assays; 95 % bootstrap interval, 1,000 resamples within assays. A result must be read with its coverage: n = 100 mutants over 4 assays, the assays this repo covers, not the human-PLM family's common ground. ProteinGym's NDCG and top-10 % recall (ndcg_top10_zero_shot, top10_recall_zero_shot, secondary) are computed the same way, per assay then the mean; the top decile of 25 mutants is 2-3 mutants, so they are coarse. ProteinGym's AUC and MCC (auc_zero_shot, mcc_zero_shot) read the assay's DMS_score_bin (ProteinGym's own per-assay cutoff, the label_bin column), per assay then the mean: the ROC AUC of the score, and the Matthews correlation of the score split at its median within the assay. Definition version 2 adds that column (and these two metrics) to version 1's items, which are otherwise the same.

Comparing the four human PLMs with each other. Only on shared assays. At the pinned revisions: -0.5 and -0.9 select the same four assays (CCR5, MET, S22A1, SRC — their coverage differs elsewhere, not here), -Clans-0.5 shares CCR5 and S22A1 with them, -Clans-0.9 shares MET and SRC. atlas-evals compare A B restricted to the two benchmarks of a pair gives that comparison; a mean over all four assays of two different benchmarks does not.

Caveats. Domain spans are 251-376 residues here, so this is not a "short peptide" benchmark; a whole-sequence model scored on the span loses the rest of the protein's context, which is the point (identical inputs) but also means numbers are not comparable with the whole-sequence ProteinGym benchmark, nor with the other three per-repo benchmarks. 25 mutants per assay is a deliberate cost cap (4 benchmarks x 4 assays x 25 = 400 scorings per model, about 20 distinct assays in all), so a per-assay ρ is noisy; the confidence interval says how much. Human proteins are in UniRef, which every protein LM pretrained on, and the human PLMs were trained on human Pfam domains, so the families themselves are in their training distribution — this measures within-family variant ranking, not generalisation to unseen families.

Data

• pg: hf_dataset OATML-Markslab/ProteinGym_v1 at revision 1ea2aa10a328cc085b64328e790c54755fe32b10, licence MIT, catalogued as protein-ml/proteingym-v1
• pg_ref: url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/reference_files/DMS_substitutions.csv, licence MIT, catalogued as protein-ml/proteingym
• hplm: api_frozen ?, licence listing only (file names of public HF model repos)
• pfam: api_frozen ?, licence CC-BY-4.0

Subset

• seed 20261006
• test: 25 item(s) per group
• train: all item(s)
• results at atlas-n25-v2 use subset 4e4f5788dc537ad0

Metrics

• spearman_zero_shot — Spearman ρ (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• auc_zero_shot — AUC (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• mcc_zero_shot — MCC (zero-shot, domain input); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• ndcg_top10_zero_shot — NDCG, top 10 % (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• top10_recall_zero_shot — Top-10 % recall (zero-shot, domain input); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

The four assays, with the Pfam domain each keeps and the repos that cover it (three of them are in clan CL0192, which this repo has a checkpoint for):

assay	Pfam (clan)	domain in target_seq	span	covered by
ADRB2_HUMAN_Jones_2020	PF00001 (CL0192)	50-326	277	0.5, 0.9, Clans-0.5
CCR5_HUMAN_Gill_2023	PF00001 (CL0192)	47-297	251	0.5, 0.9, Clans-0.5
OPSD_HUMAN_Wan_2019	PF00001 (CL0192)	55-306	252	0.5, 0.9, Clans-0.5
S22A1_HUMAN_Yee_2023_activity	PF00083 (CL0015)	147-522	376	0.5, 0.9, Clans-0.5

Three items of ADRB2_HUMAN_Jones_2020 (beta-2 adrenergic receptor surface expression; the kept domain is the 277-residue 7TM span GNVLVITAIAKFERLQTVTNYFITSLACADLVMGLAVVPF...), by seed-hash rank:

item_id	mutant (domain)	mutant (protein)	DMS_score
ADRB2_HUMAN_Jones_2020:F104R	F55R	F104R	2.120
ADRB2_HUMAN_Jones_2020:T118F	T69F	T118F	0.935
ADRB2_HUMAN_Jones_2020:R228V	R179V	R228V	3.131

Input: the 277-residue domain span as the wild type plus the re-indexed mutation (e.g. F55R), and the mutated span. Expected output: one score per mutant whose order matches the measurement, here T69F < F55R < R179V. CCR5 and S22A1 are shared with the two family benchmarks, which is where -Clans-0.5 can be compared with -0.5 and -0.9.