ClinVar variant pathogenicity (ROC-AUC)

Licence to confirm before publication on the hub. Source(s) genome declare no known licence (an unknown licence is gated until confirmed in a publication block).

What is measured. Whether a DNA model tells pathogenic from benign single-nucleotide variants of the human genome from sequence alone, without and with a light supervised probe.

Data. songlab/clinvar (test.parquet, MIT), the ClinVar set of the GPN-MSA paper (Benegas et al., Nature Biotechnology 2025): 40,976 SNVs on hg38, 22,252 pathogenic and 18,724 benign, mostly missense. Sequence: UCSC hg38.2bit (sha256 pinned), read region by region; every variant's ref is checked against the genome.

Items. One per variant: the 512 bp reference window with the variant at position 257 (1-based), and the same window carrying the alternate base. Test: 100 variants, 50 pathogenic and 50 benign (min_per_stratum: 50). Train: 500 variants, 250 / 250, from 1 Mb bins (chromosome, position // 10^6) holding no test variant (disjoint_by: bin_1mb), so no train variant sits next to a test one in the same gene neighbourhood. Seed-hash rank, seed 20261006.

Zero-shot (roc_auc_zero_shot, average_precision_zero_shot). The model scores the variant in the reference window (score.variant_effect.dna): the log-likelihood ratio log p(alt) − log p(ref) at the variant (a masked model masks it, a causal model compares the two windows' likelihoods, as its adapter documents). Pathogenic is the positive class, ranked by −LLR.

Probe (roc_auc_probe, average_precision_probe). The model embeds both windows (last layer, mean over tokens); a logistic head (StandardScaler + LogisticRegression, C by 5-fold CV on log-loss) is fit on emb(alt) − emb(ref) of the 500 train variants and ranks the test variants by P(pathogenic).

Validation fixture. songlab's reference scores are kept in the item table (y_ref_*, never sent to a worker). On the frozen 100 test variants, ROC-AUC of −score: GPN-MSA 0.956, CADD 0.964, ESM-1b 0.959 (90 scored), phyloP-241m 0.902, phyloP-100v 0.888, phastCons 0.858, NT 0.556, HyenaDNA 0.550 (on all 40,976: 0.970, 0.966, 0.944, 0.912, 0.927, 0.883, 0.601, 0.497). A phase-3 NT or HyenaDNA adapter should land near its column.

Uncertainty. 95 % bootstrap interval, 1,000 resamples stratified by label.

Caveats. ClinVar labels are curated, not measured, and missense-heavy: protein models can be strong here through the coding context. Models pretrained on the human genome (or on multi-species genomes including it) have seen every reference window. A 512 bp window is short for long-context models; it is the same for all, so numbers compare models, not contexts.

Data

• clinvar: hf_dataset songlab/clinvar at revision 4dd687ab6fd8f73b737148891afc4956bc42ab9a, licence mit
• genome: url https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.2bit, licence unknown

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by bin_1mb

Metrics

• roc_auc_zero_shot — ROC-AUC (zero-shot LLR); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_zero_shot — Average precision (zero-shot LLR); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_probe — ROC-AUC (logistic probe on alt - ref); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• average_precision_probe — Average precision (logistic probe on alt - ref); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

Three training variants (512 bp hg38 windows; label 1 = pathogenic):

item_id	mutant (in window)	label	CADD (songlab, −)	GPN-MSA
chr10:100183802:C>A	C257A	1	−3.88	−11.98
chr10:102832626:G>T	G257T	1	−3.81	−8.07
chr10:11742634:C>T	C257T	0	−2.62	−1.88

Input: the reference window and the mutant string (zero-shot), or both windows (probe). Expected output: a score ranking the pathogenic variants above the benign one.