Atlas
Sign in
Benchmarks / coding-sequence-protein-abundance / leaderboard

Protein abundance from coding sequence (E. coli)

codon
rna-ml/coding-sequence-protein-abundance Task: regression Visibility: PUBLIC
OverviewLeaderboard

Leaderboard

Version
Protocol

Spearman ρ (zero-shot pseudo-log-likelihood)

↑ higher is better · zero-shot · atlas-n100-v1

Spearman correlation between the model's mean per-token pseudo-log-likelihood of each coding sequence and the measured log10 protein abundance of its gene; no training.

Best in the catalogue: 0.552 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (100). Floors are trivial baselines scored on the same items, shown unranked.

# Model Value 95% CI n Source Overlap
1 rna-ml/nv-codonfm-encodon-1b-v1 0.552 best [0.371, 0.710] 100 reproduced
overlap

Train/test overlap declared by the evaluator: rna-ml/ncbi-refseq-coding-sequences
2 rna-ml/nv-codonfm-encodon-80m-v1 0.431 [0.231, 0.607] 100 reproduced
overlap

Train/test overlap declared by the evaluator: rna-ml/ncbi-refseq-coding-sequences
3 genomics/caduceus-ps-seqlen-131k-d-model-256-n-layer-16 0.349 [0.139, 0.537] 100 reproduced none declared
4 rna-ml/rnafm 0.344 [0.146, 0.526] 100 reproduced none declared
5 genomics/caduceus-ph-seqlen-131k-d-model-256-n-layer-16 0.304 [0.089, 0.499] 100 reproduced none declared
6 rna-ml/rinalmo-giga 0.207 [−0.003, 0.404] 100 reproduced none declared
7 rna-ml/rinalmo-mega 0.188 [−0.028, 0.381] 100 reproduced none declared
8 genomics/dnabert-2-117m 0.184 [−0.043, 0.384] 100 reproduced unknown
9 rna-ml/rinalmo-micro 0.160 [−0.057, 0.364] 100 reproduced none declared
10 genomics/hyenadna-medium-450k-seqlen-hf −0.022 [−0.238, 0.193] 100 reproduced none declared
11 genomics/hyenadna-tiny-1k-seqlen-hf −0.069 [−0.281, 0.139] 100 reproduced none declared

Protocol

What is measured. Whether a model reads how much protein a gene makes from its coding sequence alone: 3,333 Escherichia coli genes with their measured protein abundance in parts per million (PaxDb), the protein-abundance task of CaLM (Outeiral & Deane, Nature Machine Intelligence 6:170, 2024). Unlike the mRFP task, the genes encode different proteins, so a model may use both codon usage and what the protein is — which is what makes the same number comparable for codon, RNA, DNA and (through their own benchmark) protein models.

Why this source. It replaces the Sanofi/CodonBERT mRFP expression data (rna-ml/mrna-protein-expression, blocked: academic non-commercial only) with a chain that is permissive end to end: the file ships in the BSD-3-Clause CaLM repository at a pinned commit, and its labels are PaxDb abundances, published under CC-BY-4.0. Nothing here needs a licence confirmation before publication.

Data. data/protein_abundance/ecoli.csv of github.com/oxpig/CaLM at commit 24187ea4, pinned by sha256: 3,627 rows of gene_id, abundance (ppm) and cds. Dropped: 288 rows whose abundance is 0 (PaxDb has no measurement) and 6 duplicate gene ids, leaving 3,333 genes of 135-7,077 nt (45-2,359 codons, median 290, all whole codons).

Items. 100 test and 500 train genes by seed-hash rank (seed 20261006), train genes sharing a coding sequence with a test gene dropped first (disjoint_by: sequence). The label is log10(ppm), which spans seven decades (-2.52 to 4.92, i.e. 0.003 to 82,534 ppm).

Zero-shot (spearman_zero_shot). The model's mean per-token pseudo-log-likelihood of the coding sequence (score.pll.codon, .rna or .dna: codon models score codons, RNA models nucleotides with T read as U, DNA models the sequence as it is), against the measured label. The sign is not flipped: a model that finds abundant genes more likely scores positive.

Probe (spearman_probe, rmse_probe, pearson_probe, r2_probe). Last-layer embeddings, mean over tokens; a ridge head (StandardScaler + RidgeCV, leave-one-out over 13 alphas) fit on the 500 train genes.

Uncertainty. 95 % bootstrap interval over the 100 test genes, 1,000 resamples.

Caveats. Abundance is a property of the gene in one growth condition, not of the sequence alone: sequence length, operon context and the protein itself all carry signal, so a high ρ is not evidence of codon-level understanding by itself (compare with the mRFP task, where the protein is constant). PaxDb values are integrated over experiments and their dynamic range is uneven. CaLM reported this task with 5-fold cross-validation over the whole file; 100/500 by seed-hash rank is cheaper and is not comparable with its published numbers. Six other species (human, yeast, A. thaliana, D. melanogaster, plus transcript abundance) ship in the same repository under the same licence and can be added as groups later.

Data

  • calm_ecoli: url https://raw.githubusercontent.com/oxpig/CaLM/24187ea44744ca548b747ad262058f3463589f8e/data/protein_abundance/ecoli.csv, licence BSD-3-Clause

Subset

  • seed 20261006
  • test: 100 item(s)
  • train: 500 item(s), disjoint by sequence
  • results at atlas-n100-v1 use subset 5dcc79f59ca1a1b0

Metrics

  • spearman_zero_shot — Spearman ρ (zero-shot pseudo-log-likelihood); protocol zero_shot; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • spearman_probe — Spearman ρ (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • rmse_probe — RMSE (ridge probe); protocol probe; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
  • pearson_probe — Pearson r (ridge probe); protocol probe; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples
  • r2_probe — R² (ridge probe); protocol probe; higher is better; range ≤ 1.0; 95% bootstrap CI, 1000 resamples

Sample

Every item is one E. coli gene: its coding sequence, and the abundance of the protein it encodes. The first three test genes, by seed-hash rank:

item_idn_codonsabundance (ppm)label (log10 ppm)
511145.b27013629.6400.984
511145.b37842580.305-0.516
511145.b4179828272.0002.435

Input: the coding sequence, e.g. ATGTTCAAGCGTCGTTATGTAACATTGCTTCCCCTTTTTGTGTT… (1,086 nt for 511145.b2701). Expected output: one score per gene whose order follows abundance, here b3784 < b2701 < b4179, or, for the probe, a predicted log10 ppm.