Secondary structure (Q3)

What is measured. Per-residue three-state secondary structure — helix (H: DSSP G, H, I), strand (E: DSSP B, E) or coil (C: everything else) — from the amino-acid sequence alone, on the CB513 test set as distributed with TAPE (Rao et al., NeurIPS 2019), which takes its labels from NetSurfP-2.0 (Klausen et al. 2019). The probe also predicts the eight DSSP states themselves (Q8, the standard companion of Q3): G (3₁₀ helix), H (α helix), I (π helix), B (isolated β bridge), E (strand), S (bend), T (H-bonded turn) and C (anything else).

Data. The official TAPE archive secondary_structure.tar.gz (JSON flavour, sha256-pinned; the LMDB flavour is not used because its values are pickles). secondary_structure_cb513.json is the test pool (513 records; one PDB chain can carry several CB513 domains, so items are keyed by record position, cb513-0042), secondary_structure_train.json the train pool (NetSurfP-2.0 training chains). Each record carries ss8 (codes 0-7 in the order G H I B E S T C) and ss3, its 8→3 reduction; the loader keeps both (label, label_ss8) and refuses a record whose ss3 is not the reduction of its ss8, which every residue of the pinned archive satisfies (the archive does not name the codes; the reduction and DSSP's minimal segment lengths in the file pin the order). Each record's valid_mask marks the residues whose label is defined; the others are never scored. Proteins longer than 512 residues are left out (443 test and 8,115 train records remain).

Items. 100 test proteins and 500 train proteins, each drawn by seed-hash rank (seed 20261006) from its pool. Definition version 2 adds label_ss8 to the item table: its subset atlas-n100-v2 holds the same proteins as atlas-n100-v1, whose cells are kept as history.

Probe (q3_accuracy_probe). Frozen per-residue embeddings of the sequence (last layer, one vector per residue, special tokens excluded). A multinomial logistic head (C chosen by stratified 5-fold CV on log-loss inside the train residues) is fit on 10,000 labelled train residues, drawn by seed-hash rank of item_id:position from the 500 train proteins (deviation: the cap keeps the fit within laptop RAM for 5,120-dimensional embeddings), and predicts every test residue. Requires embed.protein.tokens.

Probe, 8 states (q8_accuracy_probe). A second multinomial logistic head, with the same recipe, on the same 10,000 train residues and the same embeddings, is fit on their 8-state labels and predicts every test residue; the Q3 head is fit and scored exactly as without it (Q3 is not derived from the 8-state prediction). The rarest state, I, is about 0.5 % of the labelled train residues (some 50 of the 10,000), enough for the 5-fold CV. Zero-shot Q8 is not offered: the CA-only fallback cannot separate G/H/I or S/T.

Zero-shot (q3_accuracy_zero_shot). The model folds each test sequence (generate.structure); the predicted backbone is assigned 3 states by a built-in re-implementation of DSSP's H-bond logic (Kabsch & Sander 1983: electrostatic H-bond energy < −0.5 kcal/mol, minimal 3/4/5 helices, parallel and antiparallel bridges, DSSP's priority, the 8→3 reduction above). It agrees with the deposited helix/sheet annotation on 91 % of residues of 40 PDB-2024 chains; it is not DSSP itself (no ladder/bulge bookkeeping), so the zero-shot ceiling is a little below 100 %. Predictions with CA atoms only fall back to P-SEA-like CA distance rules (69 % agreement on the same chains); the notes record which was used. No external DSSP/pydssp is used, so the number cannot change with the environment.

Metric. Q3 = correct labelled residues / labelled residues, pooled over the 100 proteins; Q8 the same on the 8-state labels (the same residues are labelled in both). 95 % bootstrap interval, 1,000 resamples of test proteins.

Caveats. NetSurfP-2.0 removed training chains with more than 25 % sequence identity to CB513, so train and test are homology-filtered; pretrained sequence models have usually seen CB513 sequences (PDB chains are in UniRef), and structure predictors were trained on the PDB chains themselves (AlphaFold-style models: overlap by construction), which makes the zero-shot number an upper bound. Labels are DSSP of crystal structures from the NetSurfP-2.0 pipeline, not re-computed here.

Data

• tape_ss: url https://s3.amazonaws.com/songlabdata/proteindata/data_raw_pytorch/secondary_structure.tar.gz, licence BSD-3-Clause, catalogued as protein-ml/tape

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s)
• results at atlas-n100-v2 use subset 3668a9ebbbacda02

Metrics

• q3_accuracy_probe — Q3 accuracy (per-residue probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• q8_accuracy_probe — Q8 accuracy (per-residue probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• q3_accuracy_zero_shot — Q3 accuracy (DSSP of predicted structure); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Three CB513 test records (input: the sequence; expected output: one state per residue; · = residue without a defined label, not scored):

item_id	PDB chain	length	sequence (first 40)	3-state label (first 40)	8-state label (first 40)
cb513-0000	1l58-A	164	MNIFEMLRIDEGLRLKIYKDTEGYYTIGIGHLLTKSPSLN	CCHHHHHHHHHCCEEEEEECCCCCEEEECCEEEECCCCHH	CCHHHHHHHHHCCEEEEEECTTSCEEEETTEEEESSSCHH
cb513-0001	1vca-B	202	FKIETTPESRYLAQIGDSVSLTCSTTGCESPFFSWRTQID	CEEEEECCCEEEEECCCCEEEEEEEECCCCCEEEEEECCC	CEEEEESSSEEEEETTCCEEEEEEEESCSSCEEEEEETTC
cb513-0002	1tig-A	94	KQKVINVKEVRLSPTIEEHDFNTKLRNARKFLEKGDKVKA	····CCEEEEEECCCCCHHHHHHHHHHHHHHHHCCCEEEE	····CCEEEEEECTTCCHHHHHHHHHHHHHHHHTTCEEEE