Unconditional backbone design (designability)

What is measured. Whether a generative model's unconditional protein backbones are designable: does some sequence fold back to the backbone it proposed? This is the self-consistency protocol of the diffusion literature (Trippe et al. 2023; Watson et al., Nature 2023; Yim et al. 2023), and it needs no data at all — only the model and two fixed tools.

Requests. 100 backbones, lengths 70 (34), 100 (33) and 150 (33), listed in the committed design_specs.json (sha256-pinned). Items are grouped by length (L070, L100, L150), so the bootstrap keeps the length mix of every resample. Each item carries a seed derived from its id, which a stochastic generator seeds itself with, so a re-run asks for the same samples rather than new ones.

Stage 1, the model under test (generate.backbone). The worker gets length and seed and writes one structure per item (PDB or mmCIF, all four backbone atoms N/CA/C/O at every position) of exactly that length. Nothing else is given: this is unconditional generation. A Cα-only output is refused per item: stage 2's ProteinMPNN reads N, C and O and rejects a non-finite coordinate, and reconstructing them from the Cα trace would score the harness's ideal geometry rather than the model's output.

Stage 2, the fixed refold (identical for every model, so the numbers are comparable). For each backbone: ProteinMPNN designs refold.n_sequences = 8 sequences at temperature 0.1, and ESMFold folds each of the 8. The self-consistency RMSD (scRMSD) of a backbone is the lowest Cα RMSD, after Kabsch superposition, between the designed backbone and its 8 refolds. The evaluator exposes this stage as a second FeatureRequest (refold_requests(), capability refold, the spec above) and reads the tools' answer from the refolded field of the design manifest: K structure paths or K Cα arrays, in design order, and it refuses a count other than 8 so a half-finished stage cannot be scored.

Phase 3 note. The two refold tools are not wired into the harness yet; this definition fixes their protocol (which tools, how many sequences, which temperature, how scRMSD is reduced) and the evaluator interface they plug into, so adding them is an adapter, not a redefinition (and therefore not a definition_version bump).

Metrics. designability_zero_shot = share of backbones with scRMSD < 2 Å (the registry's designability, which takes the designed backbone as y_true and its 8 refolds as y_pred, the closest one counting); sc_rmsd_median_zero_shot = median scRMSD over backbones (the registry's rmsd computed per item, then the median: aggregate: {over: item_id, how: median}). 95 % bootstrap intervals, 1,000 resamples: stratified by length for designability, resampling whole items for the median.

Secondary metrics, each its own number, so nothing is folded into designability. sc_tm_median_zero_shot: median over backbones of the self-consistency TM-score (the highest of the 8 refolds). diversity_zero_shot: per length, the designable backbones clustered at TM-score ≥ 0.5 (a sequence-independent alignment, average linkage), clusters over designable backbones, then the mean over the lengths that have one. helix_fraction_zero_shot, strand_fraction_zero_shot: the share of residues in each state (P-SEA on the Cα trace) per backbone, then the mean: descriptive, not a ranking (their direction is nominal).

Caveats. Self-consistency measures agreement between three models (the generator, ProteinMPNN, ESMFold), not experimental foldability: a backbone ESMFold cannot fold may still be designable in the lab, and idealised helical bundles are easy for all three, which is why the length mix is fixed and reported. Designability alone does not see diversity: a model that emits the same designable helical bundle 100 times scores 1.0 on it, and only diversity_zero_shot and the helix content show it. Novelty (TM-score to the PDB) is not measured: it needs a structure database the harness does not have. scRMSD depends on the refold tools' versions, which the cell notes record.

Data

• specs: api_frozen ?, licence CC0-1.0

Subset

• seed 20261006
• test: 100 item(s)
• train: all item(s)

Metrics

• designability_zero_shot — Designability (scRMSD < 2 Å); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• sc_rmsd_median_zero_shot — Median scRMSD, Å; protocol zero_shot; lower is better; range ≥ 0.0; median over item_id; 95% bootstrap CI, 1000 resamples, grouped by item_id
• sc_tm_median_zero_shot — Median scTM; protocol zero_shot; higher is better; range [0.0, 1.0]; median over item_id; 95% bootstrap CI, 1000 resamples, grouped by item_id
• diversity_zero_shot — Diversity (TM-score clusters / designable); protocol zero_shot; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• helix_fraction_zero_shot — Helix content; protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• strand_fraction_zero_shot — Strand content; protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group

Three of the 100 requests (input: a length and a seed; expected output: a backbone of that length):

item_id	group	length	seed
len070-00	L070	70	derived from the item id
len100-07	L100	100	derived from the item id
len150-32	L150	150	derived from the item id

For len100-07 the model writes a 100-residue backbone. ProteinMPNN then designs 8 sequences for it, ESMFold folds each, and the scRMSD is the lowest of the 8 Cα RMSDs — say 1.2 Å, which makes this backbone designable (< 2 Å) and contributes 1.2 to the median. A backbone whose closest refold is 6 Å away contributes 0 to designability and 6.0 to the median.