Genomic Benchmarks accuracy

What is measured. Whether a DNA model's frozen embedding separates regulatory or coding sequence from background, on three binary datasets of Genomic Benchmarks (Grešová et al., BMC Genomic Data 2023), the three the hub catalogues:

dataset	positives	negatives	length
human_enhancers_cohn	human enhancers (Cohn et al. 2018)	random genomic windows	500 bp
human_nontata_promoters	non-TATA promoters (EPD)	non-promoter windows	251 bp
demo_coding_vs_intergenomic_seqs	human coding transcripts (label 1 = intergenic)	intergenic windows	200 bp

Data. The katarinagresova/Genomic_Benchmarks_* mirrors of the package's published train/test splits, each at a pinned commit. The benchmark is drawn from all three hub datasets (an N-to-N drawn_from).

Items. atlas-full-v1: per dataset, every sequence of its published test file (6,948 enhancers, 9,034 non-TATA promoters, 25,000 coding-vs-intergenomic; 40,982 in all) and 500 train sequences from its train file, stratified by label (largest-remainder quotas, seed-hash rank, seed 20261006). atlas-n100-v1: 100 test sequences per dataset, stratified the same way, and 500 train sequences. Train sequences identical to any test sequence are dropped before drawing (disjoint_by: sequence): the package's splits are random, not chromosome-held-out.

Test size. The leaderboard version is atlas-full-v1 on every profile: the published test sets whole, the numbers the package's paper reports on. Windows of 200-500 bp embed fast; the train (1,500 sequences) is unchanged. The earlier atlas-n100-v1 stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one. The mean over the three datasets weighs each dataset equally, whatever its test size.

Probe (accuracy_probe, macro_f1_probe). Last-layer embeddings, mean over tokens; one logistic head per dataset (StandardScaler + LogisticRegression, C by 5-fold CV on log-loss inside train), accuracy per dataset, mean over the three. All three are balanced or nearly so, so chance is about 0.5.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test sequences drawn within each dataset.

Caveats. Random train/test splits: near-duplicate windows can sit on both sides (only exact duplicates are removed). Models pretrained on the human reference genome have seen every sequence, without its label. demo_coding_vs_intergenomic_seqs labels coding sequence 0 and intergenic 1, as the package does.

Data

• human_enhancers_cohn: hf_dataset katarinagresova/Genomic_Benchmarks_human_enhancers_cohn at revision 49ef8d0d80087ece7441096afcef0123b0781ce0, licence apache-2.0, catalogued as genomics/genomic-benchmarks-human-enhancers-cohn
• human_nontata_promoters: hf_dataset katarinagresova/Genomic_Benchmarks_human_nontata_promoters at revision e0003669df0a180c8570ebe091ed91f20fa080ec, licence apache-2.0, catalogued as genomics/genomic-benchmarks-human-nontata-promote
• demo_coding_vs_intergenomic_seqs: hf_dataset katarinagresova/Genomic_Benchmarks_demo_coding_vs_intergenomic_seqs at revision 85458172cab6cffad46cb172e64ca94780506ed6, licence apache-2.0, catalogued as genomics/genomic-benchmarks-demo-coding-vs-interg

Subset

• seed 20261006
• test: 100 item(s) per group, stratified by label
• train: 500 item(s) per group, stratified by label, disjoint by sequence
• leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)
• results at atlas-full-v1 use subset 59b4e648e96e98bc
• results at atlas-n100-v1 use subset e2e9a5a57b976913

Metrics

• accuracy_probe — Accuracy (logistic probe, mean of 3 sets); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• macro_f1_probe — Macro F1 (logistic probe, mean of 3 sets); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

Three training sequences of human_nontata_promoters (251 bp; 1 = promoter), shown truncated:

item_id	sequence	label
human_nontata_promoters:train:000008	TTGAGCAGATGTTGGGTTGGATGGTGAGATCACCACAGAGTA…CTTTTCTGTTTT	0
human_nontata_promoters:train:000029	TGATCCAACAGAAAATGTTGGCTAGAACCACTCTGCTGAGTC…CCGTTCATTATT	0
human_nontata_promoters:train:000101	TGGTTCTGATGAGGAAGAAGAAAAGAAGGATGGTGACAAGAA…TCATCTTTCTCC	0

Input: the sequence. Expected output: its class. The leaderboard (atlas-full-v1) scores the 40,982 published test sequences (100 per dataset in atlas-n100-v1).