PubMedQA accuracy

What is measured. Whether a model can answer a biomedical research question with yes, no or maybe from the abstract that studied it, without the abstract's conclusion.

Data. PubMedQA PQA-L (Jin et al., EMNLP 2019): 1,000 questions derived from PubMed article titles, each labelled by experts from the abstract (552 yes, 338 no, 110 maybe), as published in qiaojin/PubMedQA (config pqa_labeled). The authors' official split (data/test_ground_truth.json of the pubmedqa GitHub repository, 500 pubids: 276 yes, 169 no, 55 maybe) is the test pool; the other 500 are the train pool. The input is the question and the abstract's sections other than its conclusion (context, the section texts joined by spaces); the conclusion (long_answer) states the answer and is never shown.

Items. atlas-full-v1: the whole official test set, 500 questions (276 yes, 169 no, 55 maybe). atlas-n100-v1: 100 test questions stratified by answer (largest-remainder quotas, seed-hash rank inside each answer: 55 yes, 34 no, 11 maybe). Either way the probe trains on all 500 train questions.

Test size. The leaderboard version is atlas-full-v1 on every profile: the authors' whole test set of 500 questions. It is the set papers report on, and 400 more prompts cost minutes. The earlier atlas-n100-v1 (100 test items, drawn as described above) stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Zero-shot (accuracy_zero_shot, macro_f1_zero_shot; capability score.options). A causal language model reads Abstract: <context>\nQuestion: <question>\nAnswer: (the prompt of the common evaluation harnesses) and scores each of " yes", " no", " maybe" as a continuation (summed token log-probabilities); the most likely one is the answer. The context is cut to its first 300 words (PubMedQA contexts average about 200), and a prompt longer than the model's window is cut from the left. No demonstration examples are given.

Probe (accuracy_probe, macro_f1_probe; capability embed.text.pooled). Frozen embeddings (last layer, mean over tokens) of Question: <question>\nContext: <context>, truncated by the encoder at its own maximum length (the question comes first, so it always survives); a multinomial logistic head is fit on the 500 train questions.

Uncertainty. 95 % bootstrap interval over test questions, 1,000 resamples stratified by answer.

Caveats. 55 maybe questions (11 in atlas-n100-v1) make the maybe F1, and so macro F1, noisier than accuracy. Most hub language models were pretrained on PubMed abstracts, which include these abstracts and their conclusions (BioGPT and BioMedLM report PubMedQA fine-tuning results; their released checkpoints evaluated here are the pretrained ones unless the model card says otherwise). Always answering yes scores 0.55 accuracy on either test set (276 / 500, 55 / 100).

Data

• pqa: hf_dataset qiaojin/PubMedQA at revision 9001f2853fb87cab8d220904e0de81ac6973b318, licence mit, catalogued as bio-nlp/pubmedqa
• test-ids: url https://raw.githubusercontent.com/pubmedqa/pubmedqa/1cbae8e92f72f20c8d3747cbb3bf5bc53554d997/data/test_ground_truth.json, licence mit, catalogued as bio-nlp/pubmedqa

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label
• leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)
• results at atlas-full-v1 use subset dda23642a2828bc1

Metrics

• accuracy_zero_shot — Accuracy (option log-likelihood); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• macro_f1_zero_shot — Macro F1 (option log-likelihood); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_probe — Accuracy (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• macro_f1_probe — Macro F1 (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

The leaderboard (atlas-full-v1) scores the 500 official test questions (100 in atlas-n100-v1). Three training questions, one per answer (input: question and context; expected output: the answer; the zero-shot prompt puts the same context after Abstract: and the question after it):

item_id	question	context (start)	answer
pubmedqa-24964725	Recovery Outcome Measures: Is There a Place for Culture, Attitudes, and Faith?	Utilization of the Recovery Knowledge Inventory (RKI) and Recovery Attitudes Questionnaire (RAQ) in southeastern Australia raised questions about the RAQ, including links between attitudes, faith, and culture in supporting the recovery journey. …	yes
pubmedqa-12068831	Do nontriploid partial hydatidiform moles exist?	To study whether nontriploid partial hydatidiform moles truly exist. We conducted a reevaluation of pathology and ploidy in 19 putative nontriploid partial hydatidiform moles … 53% (10/19) were diploid nonpartial moles (initially pathologically misclassified) …	no
pubmedqa-11296674	Prostatic syndrome and pleural effusion: are they different diseases?	To report an uncommon association of prostate and lung cancer. The characteristics of both tumors, their association with tumors in other sites and the time of presentation are analyzed. …	maybe