Genomic Benchmarks accuracy

dna

genomics/genomic-benchmarks-accuracy Task: binary-classification Visibility: PUBLIC

OverviewLeaderboard

Leaderboard

Version

Accuracy (logistic probe, mean of 3 sets)

↑ higher is better · probe · atlas-n100-v1

Share of the published test sequences of each dataset (40,982 in all; 100 per dataset in atlas-n100-v1) classified correctly by a logistic head fit per dataset on frozen mean-pooled embeddings of 500 training sequences, averaged over the three Genomic Benchmarks datasets (per-dataset values in the details).

Best in the catalogue: 0.853 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (300). Floors are trivial baselines scored on the same items, shown unranked.

Model

Value

95% CI

Source

Overlap

genomics/caduceus-ph-seqlen-131k-d-model-256-n-layer-16

By dataset: 3 levels

dataset	Value [95% CI]	n
Coding vs intergenomic sequences (demo)	0.940 [0.890, 0.980]	100
Human enhancers (Cohn)	0.790 [0.700, 0.860]	100
Human non-TATA promoters	0.830 [0.750, 0.900]	100

0.853 best

[0.810, 0.887]

300

reproduced

overlap

Train/test overlap declared by the evaluator: genomics/ucsc-hg38

genomics/caduceus-ps-seqlen-131k-d-model-256-n-layer-16

By dataset: 3 levels

dataset	Value [95% CI]	n
Coding vs intergenomic sequences (demo)	0.910 [0.850, 0.960]	100
Human enhancers (Cohn)	0.800 [0.710, 0.870]	100
Human non-TATA promoters	0.800 [0.720, 0.880]	100

0.837

[0.793, 0.877]

300

reproduced

overlap

Train/test overlap declared by the evaluator: genomics/ucsc-hg38

Secondary metrics (1) · probe · context, not ranked

Macro F1 (logistic probe, mean of 3 sets)

↑ higher is better · probe

Unweighted mean over classes of the per-class F1 of the same per-dataset logistic heads on the same test sequences, averaged over the three Genomic Benchmarks datasets (per-dataset values in the details).

Model

Value

95% CI

Source

Overlap

genomics/caduceus-ph-seqlen-131k-d-model-256-n-layer-16

By dataset: 3 levels

dataset	Value [95% CI]	n
Coding vs intergenomic sequences (demo)	0.940 [0.889, 0.980]	100
Human enhancers (Cohn)	0.789 [0.694, 0.860]	100
Human non-TATA promoters	0.830 [0.750, 0.900]	100

0.853 best

[0.809, 0.886]

300

reproduced

overlap

Train/test overlap declared by the evaluator: genomics/ucsc-hg38

genomics/caduceus-ps-seqlen-131k-d-model-256-n-layer-16

By dataset: 3 levels

dataset	Value [95% CI]	n
Coding vs intergenomic sequences (demo)	0.910 [0.848, 0.960]	100
Human enhancers (Cohn)	0.799 [0.708, 0.870]	100
Human non-TATA promoters	0.799 [0.713, 0.878]	100

0.836

[0.791, 0.875]

300

reproduced

overlap

Train/test overlap declared by the evaluator: genomics/ucsc-hg38

Protocol

What is measured. Whether a DNA model's frozen embedding separates regulatory or coding sequence from background, on three binary datasets of Genomic Benchmarks (Grešová et al., BMC Genomic Data 2023), the three the hub catalogues:

dataset	positives	negatives	length
`human_enhancers_cohn`	human enhancers (Cohn et al. 2018)	random genomic windows	500 bp
`human_nontata_promoters`	non-TATA promoters (EPD)	non-promoter windows	251 bp
`demo_coding_vs_intergenomic_seqs`	human coding transcripts (label 1 = intergenic)	intergenic windows	200 bp

Data. The katarinagresova/Genomic_Benchmarks_* mirrors of the package's published train/test splits, each at a pinned commit. The benchmark is drawn from all three hub datasets (an N-to-N drawn_from).

Items. atlas-full-v1: per dataset, every sequence of its published test file (6,948 enhancers, 9,034 non-TATA promoters, 25,000 coding-vs-intergenomic; 40,982 in all) and 500 train sequences from its train file, stratified by label (largest-remainder quotas, seed-hash rank, seed 20261006). atlas-n100-v1: 100 test sequences per dataset, stratified the same way, and 500 train sequences. Train sequences identical to any test sequence are dropped before drawing (disjoint_by: sequence): the package's splits are random, not chromosome-held-out.

Test size. The leaderboard version is atlas-full-v1 on every profile: the published test sets whole, the numbers the package's paper reports on. Windows of 200-500 bp embed fast; the train (1,500 sequences) is unchanged. The earlier atlas-n100-v1 stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one. The mean over the three datasets weighs each dataset equally, whatever its test size.

Probe (accuracy_probe, macro_f1_probe). Last-layer embeddings, mean over tokens; one logistic head per dataset (StandardScaler + LogisticRegression, C by 5-fold CV on log-loss inside train), accuracy per dataset, mean over the three. All three are balanced or nearly so, so chance is about 0.5.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test sequences drawn within each dataset.

Caveats. Random train/test splits: near-duplicate windows can sit on both sides (only exact duplicates are removed). Models pretrained on the human reference genome have seen every sequence, without its label. demo_coding_vs_intergenomic_seqs labels coding sequence 0 and intergenic 1, as the package does.

Data

human_enhancers_cohn: hf_dataset katarinagresova/Genomic_Benchmarks_human_enhancers_cohn at revision 49ef8d0d80087ece7441096afcef0123b0781ce0, licence apache-2.0, catalogued as genomics/genomic-benchmarks-human-enhancers-cohn
human_nontata_promoters: hf_dataset katarinagresova/Genomic_Benchmarks_human_nontata_promoters at revision e0003669df0a180c8570ebe091ed91f20fa080ec, licence apache-2.0, catalogued as genomics/genomic-benchmarks-human-nontata-promote
demo_coding_vs_intergenomic_seqs: hf_dataset katarinagresova/Genomic_Benchmarks_demo_coding_vs_intergenomic_seqs at revision 85458172cab6cffad46cb172e64ca94780506ed6, licence apache-2.0, catalogued as genomics/genomic-benchmarks-demo-coding-vs-interg

Subset

seed 20261006
test: 100 item(s) per group, stratified by label
train: 500 item(s) per group, stratified by label, disjoint by sequence
leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)
results at atlas-full-v1 use subset 59b4e648e96e98bc
results at atlas-n100-v1 use subset e2e9a5a57b976913

Metrics

accuracy_probe — Accuracy (logistic probe, mean of 3 sets); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
macro_f1_probe — Macro F1 (logistic probe, mean of 3 sets); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

Sample

Three training sequences of human_nontata_promoters (251 bp; 1 = promoter), shown truncated:

item_id	sequence	label
`human_nontata_promoters:train:000008`	`TTGAGCAGATGTTGGGTTGGATGGTGAGATCACCACAGAGTA…CTTTTCTGTTTT`	0
`human_nontata_promoters:train:000029`	`TGATCCAACAGAAAATGTTGGCTAGAACCACTCTGCTGAGTC…CCGTTCATTATT`	0
`human_nontata_promoters:train:000101`	`TGGTTCTGATGAGGAAGAAGAAAAGAAGGATGGTGACAAGAA…TCATCTTTCTCC`	0

Input: the sequence. Expected output: its class. The leaderboard (atlas-full-v1) scores the 40,982 published test sequences (100 per dataset in atlas-n100-v1).