Biomedical sentence similarity (BIOSSES)

Licence to confirm before publication on the hub. BIOSSES is GPL-3.0, a copyleft software licence applied to data: what it requires of a hub entry and of published numbers needs legal confirmation. Local evaluation is fine; results and the proposed dataset must not be published on the hub before that.

What is measured. Whether a model's sentence embeddings place biomedical sentences that say similar things close together, graded the way experts grade them.

Data. BIOSSES (Soğancıoğlu, Öztürk and Özgür, Bioinformatics 2017): 100 pairs of citing sentences from the TAC 2014 Biomedical Summarization track, each rated by five domain experts from 0 (unrelated) to 4 (equivalent); the gold similarity is their mean. Mirrored in tabilab/biosses (one file, no pair ids: items are numbered by row, biosses-000 to biosses-099).

Items (exc). All 100 pairs are the test set (atlas-full-v1) and there is no train split: the dataset is too small to split, and the only protocol fits nothing. Published BIOSSES numbers of fine-tuned models (BLURB) use a 64/16/20 split and are not comparable with these.

Zero-shot (pearson_zero_shot, spearman_zero_shot; capability embed.text.pooled). Each sentence is embedded on its own (last layer, mean over tokens; a sentence-embedding model applies its own pooling when its adapter says so) and the prediction is the cosine of the two vectors. Nothing is fit: the correlation is computed between raw cosines and gold means.

Uncertainty. 95 % bootstrap interval over the 100 pairs, 1,000 resamples.

Caveats. With 100 pairs the interval is wide (about ±0.07 for r near 0.8, ±0.15 near 0.5). The sentences come from PubMed Central full texts, which several hub encoders were pretrained on (unlabelled). Anisotropic embeddings (cosines all near 1) depress Pearson more than Spearman, which is why both are reported. The dataset is GPL-3.0 licensed; examples below are quoted under that licence.

Data

• biosses: hf_dataset tabilab/biosses at revision 2394a2eda8dae34a30f68f0770775fd5c2e863bd, licence gpl-3.0

Subset

• seed 20261006
• test: all item(s)
• train: all item(s)

Metrics

• pearson_zero_shot — Pearson r (embedding cosine); protocol zero_shot; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples
• spearman_zero_shot — Spearman ρ (embedding cosine); protocol zero_shot; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples

Three pairs (input: two sentences; expected output: a similarity that ranks pairs like the mean expert rating, 0-4):

item_id	sentence 1	sentence 2	gold
biosses-000	Here, looking for agents that could specifically kill KRAS mutant cells, they found that knockdown of GATA2 was synthetically lethal with KRAS mutation	Not surprisingly, GATA2 knockdown in KRAS mutant cells resulted in a striking reduction of active GTP-bound RHO proteins, including the downstream ROCK kinase	2.2
biosses-001	MLL-FKBP and MLL-AF9 transformed cells showed persistent expression of Hox a7 and Hox a9 as well as the Hox cofactor Meis 1.	Regardless of the mechanism for transcriptional activation, increasing data suggest that Hox a7, Hox a9, and Meis1 are pivotal targets for MLL fusion protein-mediated transformation.	3.2
biosses-002	The oncogenic activity of mutant Kras appears dependent on functional Craf.	Oncogenic KRAS mutations are common in cancer.	2.0