Protein membrane association (DeepLoc)

protein

protein-ml/protein-membrane-localization Task: binary-classification Visibility: PUBLIC

OverviewLeaderboard

Leaderboard

Version

atlas-full-v1 34 · default

Q2 accuracy (logistic probe)

↑ higher is better · probe · atlas-full-v1

Share of the 1,728 DeepLoc test proteins with a membrane annotation whose class (membrane-bound or soluble) is predicted correctly by a logistic head fit on frozen mean-pooled embeddings of 300 training proteins: DeepLoc 1.0's Q2.

Best in the catalogue: 0.888 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (1728). Floors are trivial baselines scored on the same items, shown unranked.

#	Model	Value	95% CI	n	Source	Overlap
1	protein-ml/esm2-35m-protein-cellular-component	0.888 best	[0.871, 0.903]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
2	protein-ml/prot-t5-xl-half-uniref50-enc	0.861	[0.845, 0.878]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref50
2	protein-ml/prot-t5-xl-uniref50	0.861	[0.845, 0.878]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref50
4	protein-ml/esm-protein-localization	0.860	[0.844, 0.877]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
5	protein-ml/esm2-t30-150m-ur50d	0.855	[0.839, 0.872]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
6	protein-ml/esm2-150m-protein-molecular-function	0.849	[0.833, 0.866]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
6	protein-ml/esm2-t33-650m-ur50d	0.849	[0.834, 0.867]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
8	protein-ml/esm2-t36-3b-ur50d	0.848	[0.831, 0.866]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
9	protein-ml/esm2-t12-35m-ur50d	0.844	[0.828, 0.863]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
10	protein-ml/esm2-35m-protein-biological-process	0.837	[0.821, 0.854]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
11	protein-ml/esm2-t6-8m-ur50d	0.830	[0.813, 0.847]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
11	protein-ml/prot-bert	0.830	[0.813, 0.848]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UniRef100
13	protein-ml/amplify-350m	0.828	[0.811, 0.847]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)
14	protein-ml/progen2-base	0.824	[0.807, 0.843]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90
15	protein-ml/amplify-120m	0.822	[0.806, 0.841]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)
15	protein-ml/protgpt2	0.822	[0.805, 0.840]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
17	protein-ml/progen2-small	0.811	[0.794, 0.829]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90
—	kmer-protein Reference floor, not a hub model; unranked	0.723	[0.705, 0.744]	1728	floor

Secondary metrics (1) · probe · context, not ranked

MCC (logistic probe)

↑ higher is better · probe

Matthews correlation of the same probe's membrane-bound vs soluble calls: unlike Q2 it is 0 for a constant answer, whatever the class balance (57 % of the test proteins are soluble).

Model	Value	95% CI	n	Source	Overlap
protein-ml/esm2-35m-protein-cellular-component	0.771 best	[0.736, 0.802]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/prot-t5-xl-half-uniref50-enc	0.715	[0.681, 0.751]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref50
protein-ml/prot-t5-xl-uniref50	0.715	[0.681, 0.751]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref50
protein-ml/esm-protein-localization	0.714	[0.681, 0.748]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-t30-150m-ur50d	0.708	[0.676, 0.741]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-150m-protein-molecular-function	0.690	[0.657, 0.726]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-t33-650m-ur50d	0.690	[0.659, 0.727]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-t36-3b-ur50d	0.689	[0.653, 0.727]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-t12-35m-ur50d	0.680	[0.646, 0.718]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/esm2-35m-protein-biological-process	0.667	[0.632, 0.701]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/prot-bert	0.652	[0.617, 0.689]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UniRef100
protein-ml/esm2-t6-8m-ur50d	0.650	[0.615, 0.686]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/amplify-350m	0.648	[0.611, 0.686]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)
protein-ml/progen2-base	0.641	[0.605, 0.679]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90
protein-ml/amplify-120m	0.637	[0.601, 0.677]	1728	reproduced	overlap Train/test overlap declared by the evaluator: UR100P (UniRef100 + OAS + SCOP)
protein-ml/protgpt2	0.635	[0.600, 0.672]	1728	reproduced	overlap Train/test overlap declared by the evaluator: protein-ml/uniref50
protein-ml/progen2-small	0.620	[0.585, 0.657]	1728	reproduced	overlap Train/test overlap declared by the evaluator: BFD, protein-ml/uniref90
kmer-protein Reference floor, not a hub model; unranked	0.426	[0.387, 0.471]	1728	floor

Protocol

What is measured. Whether a eukaryotic protein is membrane-bound or soluble, from its amino-acid sequence alone: DeepLoc 1.0's second task (Q2), beside its ten-way localization (protein-subcellular-localization).

Data. DeepLoc 1.0 (Almagro Armenteros et al., Bioinformatics 2017) through the same AI4Protein/DeepLocMulti mirror and loader as the localization benchmark. Each protein's location string ends in M (membrane-bound), S (soluble) or U (unknown); only M and S proteins are items, U ones are left out (1,014 of the 2,742 test proteins). The flag is the source's label_membrane column, kept as is.

Items. atlas-full-v1: every test protein with a flag, 1,728 (736 membrane-bound, 992 soluble), and the 300 flagged proteins (119 membrane-bound, 181 soluble) of the 500 training proteins of the localization benchmark's frozen atlas-full-v1 subset: the same proteins, so a model's embeddings are the ones its localization probe reads. There is no smaller version.

Probe (q2_accuracy_probe; secondary mcc_probe). Frozen embeddings (last layer, mean over residues), as in the localization probe; a logistic head is fit on the 300 train proteins and predicts every test protein. Q2 is the share predicted correctly; MCC is 0 for a constant answer (Q2 0.574 when every protein is called soluble).

Uncertainty. 95 % bootstrap interval over test proteins, 1,000 resamples stratified by class.

Caveats. The class follows the location for most compartments (every cell-membrane protein is membrane-bound; every cytoplasmic and extracellular one soluble in this source), so a model that localizes well gets much of Q2 from that; the mixed compartments (mitochondrion, plastid, nucleus, ER) are the hard part. The probe trains on 300 proteins, not DeepLoc's ~5,700 flagged training proteins: its numbers are below the literature's (DeepLoc 1.0's abstract: 92 % for membrane-bound vs soluble). Pretraining overlap is the localization benchmark's (UniRef-trained models have seen the sequences, never these labels).

Data

deeploc: hf_dataset AI4Protein/DeepLocMulti at revision 1af82ded911c8b3faadf185d9357193de9f301c9, licence apache-2.0

Subset

seed 20261006
test: all item(s)
train: all item(s)
results at atlas-full-v1 use subset a0aa21c45b4fb0c6

Metrics

q2_accuracy_probe — Q2 accuracy (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_membrane
mcc_probe — MCC (logistic probe); protocol probe; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label_membrane

Sample

The leaderboard (atlas-full-v1) scores the 1,728 DeepLoc test proteins with a membrane annotation. Three training proteins (input: the sequence; expected output: the class; the location is shown, not used):

item_id	sequence	location	class
`P0C893`	`DRDSCIDKSRCSKYGYYQECQDCCKKAGHNGGTCMFFKCKCA` (42 aa)	Extracellular	S (soluble)
`O95867`	`MKALMLLTLSVLLCWVSADIRCHSCY…LAGLGLWLLH` (125 aa)	Cell membrane	M (membrane-bound)
`O96011`	`MDAWVRFSAQSQARERLCRAAQYACS…LIYPWLRLKP` (259 aa)	Peroxisome	M (membrane-bound)