Protein stability ΔΔG (Mega-scale)

What is measured. How well a model predicts the change in folding stability (ΔΔG, kcal/mol) caused by a single amino-acid substitution, on the Mega-scale cDNA display proteolysis data (Tsuboyama et al., Nature 2023): domains of 40-72 residues, every single substitution measured.

Data. RosettaCommons/MegaScale on HF, config dataset3_single (single mutants of domains whose wild-type ΔG < 4.75 kcal/mol, so ΔΔG is measurable), pinned revision. Its split_name carries the ThermoMPNN split, built so that no wild-type cluster crosses it; the test rows of the test file are the test pool (28,172 mutants, 28 wild types, 17 clusters) and the train rows of the train files the train pool (215,731 mutants). The CV and validation rows are not used. ddG_ML is ΔG(mutant) − ΔG(wild type) with ΔG the unfolding free energy, so higher is more stable, the orientation of a zero-shot fitness score. Rows without a numeric ΔΔG are dropped. The wild-type sequence is rebuilt from each mutant and checked consistent across all mutants of a domain.

Items. Test: 100 single mutants, stratified by wild-type cluster with at least 2 per cluster (so all 17 test clusters are present), seed-hash rank inside each. Train: 500 single mutants drawn from wild-type clusters absent from the test subset (disjoint_by: wt_cluster).

Zero-shot (spearman_zero_shot). The model scores each test mutant from the wild-type sequence and the mutation (score.variant_effect, e.g. a masked-marginal log-ratio), higher meaning fitter; Spearman ρ against ΔΔG, pooled over the 100 mutants.

Probe (spearman_probe, rmse_probe; secondary pearson_probe, mae_probe). Mean-pooled last-layer embeddings of the mutated sequence; one ridge head (RidgeCV, alpha by leave-one-out) over all 500 train mutants predicts ΔΔG of the test mutants. RMSE and MAE are in kcal/mol; Pearson r beside Spearman and RMSE is ThermoMPNN's set. The zero-shot scores are log-ratios, not ΔΔG, so only the rank metric applies to them.

Uncertainty. 95 % cluster bootstrap: 1,000 resamples of whole wild-type clusters (a cluster's mutants are not independent).

Caveats. Pooled Spearman mixes within- and between-domain variation: a model that only ranks domains by overall stability gets some credit. Designed domains of the same topology family (HHH, HEEH, EHEE, hall) form one cluster each; train items from those families are excluded whenever the test subset contains them. ProteinGym includes 64 Tsuboyama assays (v1 reference file): a model tuned on ProteinGym's stability assays has seen this kind of data, and the natural domains are PDB/UniRef sequences that sequence models were pretrained on.

Data

• megascale: hf_dataset RosettaCommons/MegaScale at revision f27fb1f61a7e835a1c05aa7fd3c2ea9fbbdf8508, licence CC-BY-4.0

Subset

• seed 20261006
• test: 100 item(s), stratified by wt_cluster
• train: 500 item(s), disjoint by wt_cluster
• results at atlas-n100-v1 use subset d6e55e5e8b9eb5a7

Metrics

• spearman_zero_shot — Spearman ρ (zero-shot); protocol zero_shot; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by wt_cluster
• spearman_probe — Spearman ρ (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by wt_cluster
• rmse_probe — RMSE, kcal/mol (ridge probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples, grouped by wt_cluster
• pearson_probe — Pearson r (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by wt_cluster
• mae_probe — MAE, kcal/mol (ridge probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples, grouped by wt_cluster

Three of the 100 frozen test mutants, all from the same wild type (input: the wild-type sequence and the mutation; expected output: a score ordering them by ΔΔG):

item_id	wild type	cluster	mutation	ΔΔG (kcal/mol)	wild-type length
1GYZ.pdb_E29Y	1GYZ.pdb	100	E29Y	+0.288	59
1GYZ.pdb_L23W	1GYZ.pdb	100	L23W	−0.811	59
1GYZ.pdb_P43S	1GYZ.pdb	100	P43S	−0.467	59

The wild type is WIARINAAVRAYGLNYSTFINGLKKAGIELDRKILADMAVRDPQAFEQVVNKVKEALQV (59 residues). Expected output, zero-shot: one score per mutant whose order matches ΔΔG, here L23W < P43S < E29Y (E29Y is mildly stabilising, L23W destabilising). Probe: a ΔΔG in kcal/mol, so RMSE is readable directly — predicting 0.0 for every mutant gives an RMSE equal to the root-mean-square of the test ΔΔG values.

The frozen test subset covers all 17 test wild-type clusters with 4-9 mutants each, and its 500 train mutants come from 80 other clusters, disjoint from these 17.