Inverse folding (native sequence recovery)

protein

protein-ml/protein-inverse-folding Task: token-classification Visibility: PUBLIC

OverviewLeaderboard

Leaderboard

No results yet. When an evaluation of this benchmark is published, its ranked results will appear here, with their source and their uncertainty.

Protocol

What is measured. Given the backbone of a protein and nothing else, how often a model proposes the residue the crystal structure actually has — native sequence recovery, the standard inverse-folding number (ProteinMPNN, ESM-IF).

Data. Exactly the same 100 chains as protein-ml/protein-structure-prediction, from the same frozen RCSB search and the same sha256-pinned versioned mmCIF files (X-ray ≤ 2.5 Å, released in 2024, one protein entity of 50-300 residues, 30 % sequence-identity cluster representatives, backbone observed at ≥ 80 % of positions). Running both benchmarks on one model therefore costs one download, and the two numbers describe the same structures.

Items. One item per chain. The positions evaluated are those whose four backbone atoms (N, CA, C, O) are all observed — M positions of the L deposited ones.

What the worker sees. backbone (M, 4, 3) coordinates, residue_index (the 1-based position of each in the deposited chain, so chain breaks are visible as gaps) and length (M). Nothing that names a residue: no sequence, no residue type, and not the mmCIF file (the item table keeps its path for provenance, and a unit test asserts the worker table holds only those three columns). Side chains are not given either, which is the usual inverse-folding setting.

Protocol. Zero-shot only (generate.sequence). The manifest row carries sequence (M one-letter codes, one per given position) inline or a FASTA/text file. A model that samples several sequences per backbone reports them in sequences and the first is scored: recovery is a property of the model's single best guess, and scoring the best of K would reward sampling more. The spec asks for a low temperature (0.1), the common choice for a single deterministic-ish design; adapters that ignore it should say so in their notes. The row may also carry log_probs (M × A natural-log probabilities) with the alphabet its columns stand for, which is where perplexity comes from.

Second stage (fixed for every model). The scored design is folded by ESMFold — the same tool, at the same pinned revision, for every model under test, which is what makes the numbers comparable — and the TM-score of that refold against the crystal backbone's Cα trace (residue i to residue i) is this chain's scTM. It is the refold stage of protein-ml/protein-backbone-design with the design step left out: there ProteinMPNN designs the sequences, here the model under test already did, so the stage names no inverse-folding tool and only folds. The tools' identities ride in the cell key, so a new ESMFold revision invalidates the number.

Metrics. Recovery per chain = matching positions / M, then the mean over the 100 chains; 95 % bootstrap interval, 1,000 resamples of chains. sequence_recovery_median_zero_shot (secondary) is the median over the chains, the statistic ProteinMPNN reports. perplexity_zero_shot (secondary) is exp of the mean negative log-likelihood of the crystal residue, pooled over every evaluated position of the 100 chains. sc_tm_zero_shot (secondary) is the median over the chains of the scTM above.

What perplexity here is, and is not. ProteinMPNN's published perplexity comes from its --score_only path: the native sequence is an input, and each position is scored conditioned on the native residues decoded before it. This benchmark never sends a worker the sequence it is designing, so that number cannot be produced honestly here. What is reported instead is the backbone-only per-residue perplexity — each position's distribution conditioned on the coordinates alone, from upstream's own sequence-free scoring call (ProteinMPNN unconditional_probs, LigandMPNN score(use_sequence=False) over its random decoding orders, combined as its score.py combines them: the mean of the probabilities, then its log). It is higher than the autoregressive number and must not be compared with it; comparing two models' values on this board is what it is for. A model whose adapter returns no log_probs has the metric omitted from its cell (never written as a 0 or an NaN).

Caveats. Native recovery is not design quality: a different sequence may fold to the same backbone, and 100 % recovery is not the goal of a design model. Published numbers for ProteinMPNN (~50 % on CATH test sets) are not directly comparable — different chains, different masking of unobserved positions, one design instead of the best of several. Crystal backbones carry crystallisation artefacts, and the 30 % identity clustering does not remove homology to the PDB a model trained on, so a model trained on 2024+ PDB snapshots has seen these very chains (the export's overlap flag declares protein-ml/protein-data-bank as a whole, not by date). Positions with an incomplete backbone are dropped rather than designed, so M < L on most chains — and scTM compares the refold of those M residues with the M observed ones, so a chain with unobserved loops is refolded without them: ESMFold sees a sequence whose gaps were spliced out, which lowers its scTM for a reason that is the crystal's, not the designer's. scTM is also a one-design number: the self-consistency rates the design literature reports usually take the best of 8 sequences per backbone, which is systematically higher.

Data

rcsb: api_frozen ?, licence search answer (RCSB data are CC0-1.0), catalogued as protein-ml/protein-data-bank
pdb-7hll: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/hl/pdb_00007hll/pdb_00007hll_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8bp1: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/bp/pdb_00008bp1/pdb_00008bp1_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8cd5: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/cd/pdb_00008cd5/pdb_00008cd5_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8cmp: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/cm/pdb_00008cmp/pdb_00008cmp_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8cpq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/cp/pdb_00008cpq/pdb_00008cpq_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8gc6: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/gc/pdb_00008gc6/pdb_00008gc6_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8gi6: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/gi/pdb_00008gi6/pdb_00008gi6_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8hy9: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/hy/pdb_00008hy9/pdb_00008hy9_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8ii8: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ii/pdb_00008ii8/pdb_00008ii8_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8is9: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/is/pdb_00008is9/pdb_00008is9_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8izt: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/iz/pdb_00008izt/pdb_00008izt_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8j5h: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/j5/pdb_00008j5h/pdb_00008j5h_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8j5i: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/j5/pdb_00008j5i/pdb_00008j5i_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8j7q: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/j7/pdb_00008j7q/pdb_00008j7q_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8jnb: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/jn/pdb_00008jnb/pdb_00008jnb_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8jqu: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/jq/pdb_00008jqu/pdb_00008jqu_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8k4m: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/k4/pdb_00008k4m/pdb_00008k4m_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8k8f: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/k8/pdb_00008k8f/pdb_00008k8f_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8ka7: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ka/pdb_00008ka7/pdb_00008ka7_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8kac: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ka/pdb_00008kac/pdb_00008kac_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8kb6: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/kb/pdb_00008kb6/pdb_00008kb6_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8kc4: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/kc/pdb_00008kc4/pdb_00008kc4_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8kg4: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/kg/pdb_00008kg4/pdb_00008kg4_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8okq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ok/pdb_00008okq/pdb_00008okq_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8omi: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/om/pdb_00008omi/pdb_00008omi_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8ovq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ov/pdb_00008ovq/pdb_00008ovq_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8oxd: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ox/pdb_00008oxd/pdb_00008oxd_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8oxr: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ox/pdb_00008oxr/pdb_00008oxr_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8peb: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pe/pdb_00008peb/pdb_00008peb_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8pjx: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pj/pdb_00008pjx/pdb_00008pjx_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8pkb: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pk/pdb_00008pkb/pdb_00008pkb_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8pnx: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pn/pdb_00008pnx/pdb_00008pnx_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8por: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/po/pdb_00008por/pdb_00008por_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8prr: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pr/pdb_00008prr/pdb_00008prr_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8pvq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/pv/pdb_00008pvq/pdb_00008pvq_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8q1v: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/q1/pdb_00008q1v/pdb_00008q1v_xyz_v2-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qaz: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qa/pdb_00008qaz/pdb_00008qaz_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qdj: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qd/pdb_00008qdj/pdb_00008qdj_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qp7: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qp/pdb_00008qp7/pdb_00008qp7_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qua: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qu/pdb_00008qua/pdb_00008qua_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qup: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qu/pdb_00008qup/pdb_00008qup_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8qzh: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/qz/pdb_00008qzh/pdb_00008qzh_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8r39: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/r3/pdb_00008r39/pdb_00008r39_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8r5l: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/r5/pdb_00008r5l/pdb_00008r5l_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8r8o: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/r8/pdb_00008r8o/pdb_00008r8o_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8r9r: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/r9/pdb_00008r9r/pdb_00008r9r_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8rcy: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/rc/pdb_00008rcy/pdb_00008rcy_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8rij: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ri/pdb_00008rij/pdb_00008rij_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8rom: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ro/pdb_00008rom/pdb_00008rom_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8rqy: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/rq/pdb_00008rqy/pdb_00008rqy_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8sif: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/si/pdb_00008sif/pdb_00008sif_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8slr: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/sl/pdb_00008slr/pdb_00008slr_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8snq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/sn/pdb_00008snq/pdb_00008snq_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8t2m: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/t2/pdb_00008t2m/pdb_00008t2m_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8t5a: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/t5/pdb_00008t5a/pdb_00008t5a_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8tat: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ta/pdb_00008tat/pdb_00008tat_xyz_v2-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8tav: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ta/pdb_00008tav/pdb_00008tav_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8tct: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/tc/pdb_00008tct/pdb_00008tct_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8tqg: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/tq/pdb_00008tqg/pdb_00008tqg_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8twn: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/tw/pdb_00008twn/pdb_00008twn_xyz_v1-4.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8u57: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/u5/pdb_00008u57/pdb_00008u57_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8uan: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ua/pdb_00008uan/pdb_00008uan_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8uds: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ud/pdb_00008uds/pdb_00008uds_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8upb: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/up/pdb_00008upb/pdb_00008upb_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8uss: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/us/pdb_00008uss/pdb_00008uss_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8vrg: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/vr/pdb_00008vrg/pdb_00008vrg_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8w13: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/w1/pdb_00008w13/pdb_00008w13_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8w1d: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/w1/pdb_00008w1d/pdb_00008w1d_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8w4b: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/w4/pdb_00008w4b/pdb_00008w4b_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8w4m: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/w4/pdb_00008w4m/pdb_00008w4m_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8wcj: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/wc/pdb_00008wcj/pdb_00008wcj_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8wsv: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ws/pdb_00008wsv/pdb_00008wsv_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8xe7: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/xe/pdb_00008xe7/pdb_00008xe7_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8y8y: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/y8/pdb_00008y8y/pdb_00008y8y_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8yhv: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/yh/pdb_00008yhv/pdb_00008yhv_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8zj9: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/zj/pdb_00008zj9/pdb_00008zj9_xyz_v2-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8zmn: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/zm/pdb_00008zmn/pdb_00008zmn_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8zqf: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/zq/pdb_00008zqf/pdb_00008zqf_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-8zxq: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/zx/pdb_00008zxq/pdb_00008zxq_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9b3n: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/b3/pdb_00009b3n/pdb_00009b3n_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9bjw: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/bj/pdb_00009bjw/pdb_00009bjw_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9bka: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/bk/pdb_00009bka/pdb_00009bka_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9bkf: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/bk/pdb_00009bkf/pdb_00009bkf_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9c8l: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/c8/pdb_00009c8l/pdb_00009c8l_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9ce4: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ce/pdb_00009ce4/pdb_00009ce4_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9cj0: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/cj/pdb_00009cj0/pdb_00009cj0_xyz_v1-4.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9ctx: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ct/pdb_00009ctx/pdb_00009ctx_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9deu: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/de/pdb_00009deu/pdb_00009deu_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9dug: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/du/pdb_00009dug/pdb_00009dug_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9e7c: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/e7/pdb_00009e7c/pdb_00009e7c_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9e9m: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/e9/pdb_00009e9m/pdb_00009e9m_xyz_v1-3.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9eat: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/ea/pdb_00009eat/pdb_00009eat_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9f7j: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/f7/pdb_00009f7j/pdb_00009f7j_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9f96: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/f9/pdb_00009f96/pdb_00009f96_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9fsa: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/fs/pdb_00009fsa/pdb_00009fsa_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9fse: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/fs/pdb_00009fse/pdb_00009fse_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9ghy: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/gh/pdb_00009ghy/pdb_00009ghy_xyz_v1-1.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9h48: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/h4/pdb_00009h48/pdb_00009h48_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9kbt: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/kb/pdb_00009kbt/pdb_00009kbt_xyz_v1-2.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank
pdb-9mgj: url https://files-versioned.wwpdb.org/pdb_versioned/data/entries/mg/pdb_00009mgj/pdb_00009mgj_xyz_v1-0.cif.gz, licence CC0-1.0, catalogued as protein-ml/protein-data-bank

Subset

seed 20261006
test: 100 item(s)
train: all item(s)

Metrics

sequence_recovery_zero_shot — Native sequence recovery; protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
sequence_recovery_median_zero_shot — Median native sequence recovery; protocol zero_shot; higher is better; range [0.0, 1.0]; median over item_id; 95% bootstrap CI, 1000 resamples, grouped by item_id
perplexity_zero_shot — Native-sequence perplexity (backbone-only); protocol zero_shot; lower is better; range ≥ 1.0; 95% bootstrap CI, 1000 resamples
sc_tm_zero_shot — Self-consistency TM-score (median); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Sample

Three of the 100 chains (input: the backbone of M positions; expected output: M one-letter residues):

item_id	deposited residues	designed positions M	resolution
`7HLL_A`	188	185	1.15 Å
`8CMP_A`	66	58	1.06 Å
`8BP1_A`	242	231	1.72 Å

For 8CMP_A the worker receives a (58, 4, 3) array of N/CA/C/O coordinates and residue_index [6, 7, 8, 9, …] (the first five deposited residues, an expression tag, have no complete backbone). The expected output is the native sequence at those positions, VLVVTSKVKKLIKEKGQMNTSAET…; a model getting 29 of the 58 right scores 0.5 on this item.