ProteinGym DMS zero-shot (Spearman)

What is measured. How well a model ranks single amino-acid substitutions of a protein by their experimentally measured effect, on ten deep mutational scanning (DMS) assays of ProteinGym v1 (Notin et al., NeurIPS 2023). DMS_score is oriented by ProteinGym so that higher means fitter.

Assays. Two per ProteinGym coarse_selection_type, drawn by seed-hash rank among the substitution assays whose wild type has at most 1,022 residues and that measure at least 600 single mutants. The choice is a function of the seed and the pinned reference file, not a hand-picked list:

selection type	assays
Activity	ANCSZ_Hobbs_2022, CASP3_HUMAN_Roychowdhury_2020
Binding	CCR5_HUMAN_Gill_2023, SPG1_STRSG_Olson_2014
Expression	CP2C9_HUMAN_Amorosi_2021_abundance, KCNE1_HUMAN_Muhammad_2023_expression
OrganismalFitness	MLAC_ECOLI_MacRae_2023, MTH3_HAEAE_RockahShmuel_2015
Stability	ILF3_HUMAN_Tsuboyama_2023_2L33, RPC1_BP434_Tsuboyama_2023_1R69

Items. Single substitutions only; multi-mutants are dropped. Per assay, 100 test and 500 train mutants, disjoint, both drawn by seed-hash rank.

Zero-shot (spearman_zero_shot). The model scores each test mutant from the wild-type sequence and the mutation alone (for a masked language model, typically the masked-marginal log-ratio), higher meaning fitter, and never sees assay data. Spearman ρ against DMS_score per assay, then the mean over the ten assays.

Probe (spearman_probe). The model embeds each mutated sequence (last layer, mean over residues). Per assay, a ridge head is fit on the 500 train mutants and predicts the 100 test mutants; Spearman ρ per assay, mean over assays. Labels stay in the runner: the model only ever sees sequences.

ProteinGym's other metrics (both protocols, per assay, then the mean over the ten, as ProteinGym's performance_DMS_benchmarks.py computes them). auc_*: ROC AUC of the score against DMS_score_bin, the assay's binarised measurement (ProteinGym's per-assay DMS_binarization_cutoff), joined from the item table's label_bin. mcc_*: Matthews correlation between the score split at its median and DMS_score_bin. ndcg_top10_*: NDCG of the 10 % highest-scored mutants (10 of 100), min-max scaled DMS scores as gains. top10_recall_*: share of the true top 10 % that the score also puts in its top 10 %. A top decile of 10 mutants per assay makes the NDCG and recall intervals wide.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test mutants drawn within each assay.

Caveats. Train and test are disjoint by mutant, not by position, so a probe can learn from train mutants at a test mutant's position that the position is intolerant; the zero-shot metric is unaffected. Ten assays and 100 mutants each are a subset of ProteinGym's own protocol (every mutant of 217 assays): numbers compare models here, not with the official leaderboard.

Data

• pg: hf_dataset OATML-Markslab/ProteinGym_v1 at revision 1ea2aa10a328cc085b64328e790c54755fe32b10, licence MIT, catalogued as protein-ml/proteingym-v1
• pg_ref: url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/reference_files/DMS_substitutions.csv, licence MIT, catalogued as protein-ml/proteingym

Subset

• seed 20261006
• test: 100 item(s) per group
• train: 500 item(s) per group
• results at atlas-n100-v1 use subset 09197ac2b2f80dc3

Metrics

• spearman_zero_shot — Spearman ρ (zero-shot); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• auc_zero_shot — AUC (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• mcc_zero_shot — MCC (zero-shot); protocol zero_shot; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• ndcg_top10_zero_shot — NDCG, top 10 % (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• top10_recall_zero_shot — Top-10 % recall (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• spearman_probe — Spearman ρ (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• auc_probe — AUC (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• mcc_probe — MCC (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• ndcg_top10_probe — NDCG, top 10 % (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• top10_recall_probe — Top-10 % recall (ridge probe); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

Assay SPG1_STRSG_Olson_2014: binding of the protein G B1 domain to IgG-Fc (448-residue wild type MEKEKKVKYF…GALAVASKRKED, 1,045 single mutants). Three of its training mutants:

item_id	mutant	DMS_score
SPG1_STRSG_Olson_2014:A252K	A252K	−4.61
SPG1_STRSG_Olson_2014:G264S	G264S	0.03
SPG1_STRSG_Olson_2014:A250Y	A250Y	2.16

Input: the wild-type sequence, the mutant string and the mutated sequence. Expected output: one score per mutant whose order matches the measured one, here A252K < G264S < A250Y.