GO function prediction (Swiss-Prot 2025H2)

What is measured. Which Gene Ontology terms describe a protein, from its sequence alone, scored the way CAFA scores it: protein-centric Fmax, separately for the three ontologies (molecular function, biological process, cellular component).

Per-ontology scoring. Each ontology is scored over its own 50 columns of the 150-term vector: the registry's fmax takes params.label_range: [start, stop], so the three metrics of a protocol read one predictions frame. (This benchmark was shipped blocked on that parameter in phase 2 and unblocked when it landed; the definition did not change.)

Data. Two UniProtKB REST answers, frozen at release 2026_03 (02-September-2026) and sha256-pinned: reviewed (Swiss-Prot) entries with at least one experimental GO annotation (go_exp:*) created after 2025-07-01 — 167 entries, the test window — and the same query over 2022 — 660 entries, the train window. Every train protein therefore predates every test protein, the temporal split CAFA uses. Propagation uses go-basic.obo of the dated 2026-08-05 GO release.

Labels. A protein's experimental terms (CAFA's experimental evidence codes plus their high-throughput counterparts: EXP, IDA, IPI, IMP, IGI, IEP, TAS, IC, HTP, HDA, HMP, HGI, HEP) are mapped through alt_id, obsolete terms are dropped, and each is propagated to all its ancestors over is_a and part_of inside its own namespace. The three ontology roots and GO:0005515 (protein binding, the uninformative IPI catch-all) are excluded.

Term space. Fixed by the train window alone, so the label space cannot encode anything about the test proteins: per ontology, the 50 most frequent propagated terms among the 660 train-window entries (ties by GO id). The label vector is MF's 50 terms, then BP's, then CC's — the label_range slices above. go_terms on every item lists them in that order, and is what a native predictor is asked to score.

Items. Proteins with no term of the space in any ontology are left out (824 remain: 166 test, 658 train). 100 test and 500 train items by seed-hash rank. label_mask is True on an ontology's block only when the protein has an experimental annotation in that ontology, so a protein annotated only for localisation is scored on CC and ignored on MF and BP — CAFA's rule. In the frozen windows that means roughly 55 test proteins with MF terms, 33 with BP and 134 with CC (the counts a bootstrap interval has to live with).

Probe (fmax_*_probe). Frozen mean-pooled last-layer embeddings; one binary logistic head per term (C by stratified CV inside train), fit on the train proteins annotated in that term's ontology. A term with fewer than 2 positive or 2 negative such proteins gets that set's positive rate as a constant score, and the notes list those terms.

Zero-shot (fmax_*_zero_shot, capability predict.go). A model's own GO head scores the term space; the reply gives go_scores (a term → score mapping) or one column per GO id, and terms it does not score count as 0. The executor does not drive native predict ops for zero-shot cells yet, a second reason these cells cannot run today.

Metrics. Protein-centric Fmax per ontology: for each threshold τ in 0.01..1.00, precision averaged over the proteins with at least one prediction and recall over the proteins with at least one true term, then the maximum F1 over τ. 95 % bootstrap interval, 1,000 resamples of proteins.

Secondary metrics. aupr_micro_*: DeepFRI's micro-averaged AUPR over the scored (protein, term) pairs of one ontology's block, proteins without a true term in the block left out as for Fmax. smin_*: CAFA's Smin, the minimum over τ of sqrt(ru² + mi²) (remaining uncertainty and misinformation, averaged over proteins), each term weighted by its information content IC(f) = -log2 P(f | parents of f), in bits, lower is better. The IC is computed once from the propagated labels of the frozen subset's 500 train proteins annotated in the ontology, with each term's parents taken as its nearest ancestors on the 50-term axis (no parent on the axis: the ontology root), and written into the metric's params.ic (go_information_content.json next to this file, rebuilt by scripts/build_go_information_content.py). A native head's cell holds its own ontology's three metrics; the others are omitted, as for Fmax.

Caveats. 50 terms per ontology is a small, deliberately frequent slice of GO (CAFA scores thousands of terms), so these numbers are far above a full-GO Fmax and are not comparable with CAFA's. A model pretrained on UniProt text or on GO annotations may have seen the test proteins' functions — entries created after 2025-07-01 postdate most released protein LMs, but their sequences often exist in UniParc or in close homologues, so this is a temporal split, not a homology-filtered one. No sequence-identity filter is applied between the windows: a test protein may have a close train homologue, which a probe exploits (the standard, and standardly criticised, GO prediction setting).

Data

• uniprot: api_frozen ?, licence CC-BY-4.0, catalogued as protein-ml/uniprot-knowledgebase
• go: url https://release.geneontology.org/2026-08-05/ontology/go-basic.obo, licence CC-BY-4.0

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s)
• results at atlas-n100-v1 use subset 7e49b56db4673808

Metrics

• fmax_mf_zero_shot — Fmax (molecular function, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• fmax_bp_zero_shot — Fmax (biological process, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• fmax_cc_zero_shot — Fmax (cellular component, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• aupr_micro_mf_zero_shot — Micro AUPR (molecular function, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_mf_zero_shot — Smin (molecular function, native head); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• aupr_micro_bp_zero_shot — Micro AUPR (biological process, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_bp_zero_shot — Smin (biological process, native head); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• aupr_micro_cc_zero_shot — Micro AUPR (cellular component, native head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_cc_zero_shot — Smin (cellular component, native head); protocol zero_shot; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• fmax_mf_probe — Fmax (molecular function, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• fmax_bp_probe — Fmax (biological process, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• fmax_cc_probe — Fmax (cellular component, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• aupr_micro_mf_probe — Micro AUPR (molecular function, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_mf_probe — Smin (molecular function, probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• aupr_micro_bp_probe — Micro AUPR (biological process, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_bp_probe — Smin (biological process, probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• aupr_micro_cc_probe — Micro AUPR (cellular component, probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• smin_cc_probe — Smin (cellular component, probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples

Three test-window proteins (input: the sequence; expected output: a score per GO term of the 150-term space, scored only where the mask is True):

item_id	organism	residues	scored in	true terms in the space
Q04230	Escherichia coli	507	MF, CC	7 MF, 4 CC
J9VED5	Cryptococcus neoformans H99	483	MF, CC	8 MF, 4 CC
A0A1D8PJ72	Candida albicans SC5314	915	MF, BP, CC	8 MF, 14 BP, 10 CC

J9VED5's molecular-function block is 1 on GO:0003824 (catalytic activity), GO:0016740 (transferase), GO:0140096 (catalytic activity acting on a protein), GO:0016772, GO:0016773, GO:0016301 (kinase), GO:0004672 (protein kinase) and GO:0106310 — its one experimental annotation propagated up the ontology — and 0 on the other 42 MF terms; its BP block is masked out entirely, because the entry has no experimental BP annotation. A predictor scoring GO:0004672 at 0.9 and the unrelated terms below 0.3 reaches a high Fmax on this protein; one scoring every term 1.0 reaches recall 1 with precision 8/50.