Nucleotide Transformer downstream tasks (MCC)

Licence to confirm before publication on the hub. Source(s) nt declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. How much regulatory and splicing information a DNA model's frozen embedding carries, on the 18 downstream tasks of the Nucleotide Transformer (Dalla-Torre et al., Nature Methods 2024), in their revised release: histone-mark peaks of 10 marks in K562 (H2AFZ, H3K27ac, H3K27me3, H3K36me3, H3K4me1, H3K4me2, H3K4me3, H3K9ac, H3K9me3, H4K20me1; 1 kb windows, ENCODE), enhancers and enhancer types (400 bp, SCREEN; types: none / tissue-specific / tissue-invariant), promoters (all, TATA, non-TATA; 300 bp, EPD) and splice sites (acceptors, donors, all; 600 bp, GENCODE). Fifteen tasks are binary, enhancers_types and splice_sites_all have three classes.

Data. InstaDeepAI/nucleotide_transformer_downstream_tasks_revised at a pinned commit. Its test files hold out whole chromosomes; the loader refuses the data if any task's test chromosome appears in its train file.

Items. atlas-full-v1: per task, every sequence of the published test file (212 for promoter_tata to 3,000 for the largest; 38,822 in all), and 500 train sequences from the published train file, stratified by class (largest-remainder quotas, seed-hash rank inside each class, seed 20261006). atlas-n100-v1: 100 test sequences per task, stratified by class with at least 3 per class, and 500 train sequences. Train sequences identical to any test sequence are dropped before drawing (disjoint_by: sequence). The validation part of the original protocol (10-fold CV on the full train files) is not used and the probe trains on 500 sequences per task, so values are comparable across hub models, not with the paper's table.

Test size. The leaderboard version is atlas-full-v1 on every profile: the published test sets whole (the paper's test sets). The train (9,000 sequences) is unchanged, so the cost grows with the 38,822 test windows of 300 bp to 1 kb: minutes for models up to 500 M parameters, several hours for the 2.5 B model on the laptop (it runs on gpu-full). The earlier atlas-n100-v1 stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Probe (mcc_probe, macro_f1_probe, accuracy_probe). The model embeds each sequence (last layer, mean over tokens). Per task, a multinomial logistic head (StandardScaler + LogisticRegression, C chosen by 5-fold CV on log-loss inside train) is fit on the 500 train embeddings and predicts every test sequence. MCC per task (multi-class MCC for the two 3-class tasks), then the mean over the 18 tasks; the cell details list the per-task values.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test sequences drawn within each task.

Caveats. The NT, DNABERT-2, HyenaDNA and Caduceus families pretrained on the human reference genome (or multi-species genomes including it), which contains every test sequence: that is the usual pretraining overlap of genomic probes, not label leakage. Promoter-TATA has only 212 test sequences (its 100 in atlas-n100-v1 covered half of them), so its MCC is the noisiest of the 18.

Data

• nt: hf_dataset InstaDeepAI/nucleotide_transformer_downstream_tasks_revised at revision 851f9946252e90c665cdb3cc3eedb78f1f26197c, licence unknown, catalogued as genomics/nucleotide-transformer-downstream-tasks

Subset

• seed 20261006
• test: 100 item(s) per group, stratified by label
• train: 500 item(s) per group, stratified by label, disjoint by sequence
• leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)

Metrics

• mcc_probe — MCC (logistic probe, mean of 18 tasks); protocol probe; higher is better; range [-1.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• macro_f1_probe — Macro F1 (logistic probe, mean of 18 tasks); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group
• accuracy_probe — Accuracy (logistic probe, mean of 18 tasks); protocol probe; higher is better; range [0.0, 1.0]; mean over group; 95% bootstrap CI, 1000 resamples, stratified by group

The leaderboard (atlas-full-v1) scores the 38,822 published test sequences (100 per task in atlas-n100-v1). Three training sequences of task promoter_tata (300 bp; label 1 = a TATA-box promoter, 0 = a non-promoter window), shown truncated (each carries a TATA-like TATAAA motif):

item_id	sequence	label
promoter_tata:train:000003	TGTTTTACATAACCCCTGAATATAAATGCCAGATGTTGCTGA…GGCTAGAGCTAT (chr10:5196787-5197087)	1
promoter_tata:train:000014	GCTGCCAGCGTGGCTAGAATATAAACAGGCAGAAAAATGTGA…ACACCCCATCAC (chr10:15032145-15032445)	1
promoter_tata:train:000020	ATGAGCCATATTGTATTTATAAACATCACGTGTTTCCTTGAT…AGTCACCATGAA (chr10:24449364-24449664)	1

Input: the sequence. Expected output: the class; the probe sees train labels only.