Splice site prediction (top-k accuracy)

Licence to confirm before publication on the hub. SpliceAI's annotation file ships in a PolyForm-Strict-1.0.0 repository (noncommercial use, no redistribution of modified copies) and the UCSC hg19 genome states no licence. Local evaluation is fine; results and the proposed datasets must not be published on the hub before legal confirmation.

What is measured. Whether a model locates the splice sites of a whole human pre-mRNA, nucleotide by nucleotide, the task and metric of SpliceAI (Jaganathan et al., Cell 2019): for each class (acceptor, donor), with k the number of true sites of that class over all test pre-mRNAs, the share of true sites among the k highest-scoring positions; the reported value is the mean over the two classes (top_k_accuracy, ties at the k-th score sharing the remaining slots).

Data. Gene structures: SpliceAI's own annotation file grch37.txt (GENCODE V24lift37, one canonical transcript per gene, 20,274 genes) at a pinned commit of Illumina/SpliceAI. Sequence: UCSC hg19.2bit (sha256 pinned), read region by region by a streaming 2bit reader (no full load). The loader checks that ≥ 95 % of the sites carry GT (GC) / AG dinucleotides, which pins the coordinate convention.

Items. One per gene: the pre-mRNA (transcript start to end) plus 5,000 nt of genomic context on each side, in transcript orientation; labels on the pre-mRNA only (acceptor = first nucleotide of every exon but the first, donor = last nucleotide of every exon but the last). Eligible genes have a pre-mRNA of at most 20 kb and at least 2 exons (8,259 genes). Test: 100 genes of chr1, 3, 5, 7, 9 (SpliceAI's test chromosomes); train: 500 genes of the other chromosomes (2, 4, 6, 8, 10-22, X, Y), so train and test are disjoint by chromosome. Both drawn by seed-hash rank (seed 20261006). (exc) For memory, each pool is first cut to its 300 / 1,500 best-ranked genes by the same rank the subset builder uses, which leaves any subset of up to that size unchanged; atlas-full would mean those caps. The 100 test genes hold 1,001,688 nucleotides and 536 sites of each class.

Zero-shot (top_k_accuracy_zero_shot, pr_auc_zero_shot). A native splice model (predict.splice, SpliceAI) receives the whole window and returns [none, acceptor, donor] probabilities for every pre-mRNA nucleotide.

Probe (top_k_accuracy_probe, pr_auc_probe). A DNA model returns frozen per-nucleotide features (embed.dna.tokens, last layer, nucleotide resolution: a k-mer or BPE token's state is repeated over its nucleotides) with the window as context. For train genes, only every splice site and 20 times as many non-sites per gene (fixed integer hash of the position) are requested: 119,322 nucleotides with 5,682 sites. A three-class logistic head (StandardScaler + LogisticRegression, C by 5-fold CV on log-loss) is fit on them and scores every pre-mRNA nucleotide of the test genes. Negative subsampling changes the head's calibration, not the ranking top-k reads.

Uncertainty. 95 % bootstrap interval, 1,000 resamples of test genes.

Caveats. SpliceAI trained on GENCODE genes of the train chromosomes and selected its test genes the same way, so its zero-shot number is in its own held-out setting; SpliceAI also excluded test genes with paralogs on train chromosomes, which is not done here. DNA language models pretrained on the human genome have seen every test sequence (without labels). Models whose context is shorter than the 30 kb window embed it in windows chosen by their adapter. A test cell stores (L, 3) scores of about 1 M nucleotides.

Data

• annotation: url https://raw.githubusercontent.com/Illumina/SpliceAI/03f42437aaf56dc5dfd822c4ccee5aec1a705079/spliceai/annotations/grch37.txt, licence PolyForm-Strict-1.0.0
• genome: url https://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/hg19.2bit, licence unknown

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s)

Metrics

• top_k_accuracy_zero_shot — Top-k accuracy (native splice model); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• pr_auc_zero_shot — PR-AUC (native splice model); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• top_k_accuracy_probe — Top-k accuracy (per-nucleotide probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• pr_auc_probe — PR-AUC (per-nucleotide probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Limitations

• probe (top_k_accuracy_probe, pr_auc_probe): the frozen per-nucleotide probe sees one state per token. A k-mer or BPE token's state is copied over all its nucleotides, so a linear head cannot single out the splice nucleotide inside a token, and HyenaDNA is causal (left context only). Single-nucleotide bidirectional models (Caduceus) are not limited this way, so this ranking mixes model quality with tokenisation.

Three training genes (hg19, transcript orientation; label string over the pre-mRNA, 0 none, 1 acceptor, 2 donor):

item_id	locus	pre-mRNA	acceptors	donors
A1BG	chr19, minus strand	6,650 nt	7	7
ABCD1	chrX, plus strand	19,894 nt	9	9
AC007216.2	chr16, minus strand	334 nt	1	1

Input: the 5 kb + pre-mRNA + 5 kb window. Expected output: per pre-mRNA nucleotide, scores for acceptor and donor that rank the true sites (each acceptor preceded by AG, each donor followed by GT) above the others.