Clinical entity recognition (MACCROBAT)

What is measured. Finding the clinical problems, treatments and tests mentioned in sentences of clinical case reports, with exact boundaries, on reports never seen in training.

Data. MACCROBAT 2020 (Caufield et al., BMC Med Inform Decis Mak 2019): 200 PubMed Central case reports edited down to their clinical content, one sentence per line, with manual brat annotations of 41 entity types, as converted to JSON in singh-aditya/MACCROBAT_biomedical_ner (the converter drops discontinuous entities and, where entities overlap, keeps the earliest-starting one). 4,541 non-empty sentences.

Types. The 41 types are mapped onto the three concept types of the i2b2/VA 2010 challenge, the label space of the hub's clinical NER heads:

MACCROBAT type	type here
Sign_symptom, Disease_disorder	problem
Medication, Therapeutic_procedure	treatment
Diagnostic_procedure	test
the 36 others (Lab_value, Biological_structure, Dosage, Age, Date, ...)	O

Lab_value is a result ("elevated", "12 mg/dl"), not the name of a test. The corpus then holds 4,656 problem, 2,098 treatment and 4,574 test mentions.

Words. Each sentence is cut into words (runs of letters/digits, single punctuation marks, also cut at mention boundaries); gold tags are IOB2 per word; every model is scored on these words.

Items. No split is published. 100 test sentences are drawn by seed-hash rank over all sentences; every sentence of a case report that has a test sentence is then removed from the train pool, and 500 train sentences are drawn from the remaining reports (report-disjoint, column document).

Zero-shot (entity_f1_zero_shot; capability predict.ner). A model's own token-classification head tags the same words (each word takes its first sub-word's tag); native types are mapped by a fixed table (identity for i2b2-style heads, the table above for MACCROBAT-style heads, everything else O).

Probe (entity_f1_probe; capability embed.text.tokens, pooling word_mean). One vector per gold word (mean of last-layer sub-word states); a logistic head over the 7 IOB2 tags is fit on the words of the 500 train sentences and tags each test word independently; an I-X after O starts a mention (conlleval-style repair).

Metric. Strict micro F1 over (type, first word, last word) mentions, pooled over the 100 sentences. 95 % bootstrap interval, 1,000 resamples of whole case reports (sentences of one report are not independent).

Caveats — known overlap. bio-nlp/biomedical-ner-all (d4data/biomedical-ner-all) was trained on MACCROBAT: its zero-shot number is on its own training data and is flagged by the overlap marker of its results (it declares trained_on this dataset). Its mapped numbers are an upper bound, not a transfer measure. The i2b2-trained head (samrawal) transfers across annotation guidelines, which differ (MACCROBAT splits a finding into Sign_symptom and Biological_structure where i2b2 marks one problem span), so its F1 understates its recall of concepts.

Precision, recall and macro F1 (entity_precision_zero_shot, entity_precision_probe, entity_recall_zero_shot, entity_recall_probe, entity_macro_f1_zero_shot, entity_macro_f1_probe; metrics review). Precision and recall are the two halves of the micro F1, from the same strict span matching (0 precision when nothing is predicted); the macro F1 is the unweighted mean of the per-type F1 (seqeval average="macro", over the types present in the annotations or the predictions of each resample). The per-type values are stored with each result. Same cluster bootstrap over documents.

Data

• maccrobat: hf_dataset singh-aditya/MACCROBAT_biomedical_ner at revision 1470ebf27c9fdf65850ceb841ccd757e4aa79aa5, licence mit, catalogued as bio-nlp/maccrobat-biomedical-ner

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s), disjoint by document
• results at atlas-n100-v1 use subset c1e3a87a110e1e36

Metrics

• entity_f1_zero_shot — Entity F1, 3 types (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_f1_probe — Entity F1, 3 types (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_precision_zero_shot — Entity precision, 3 types (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_recall_zero_shot — Entity recall, 3 types (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_macro_f1_zero_shot — Entity macro F1, 3 types (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_precision_probe — Entity precision, 3 types (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_recall_probe — Entity recall, 3 types (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document
• entity_macro_f1_probe — Entity macro F1, 3 types (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, grouped by document

Three training sentences (input: the sentence and its words; expected output: one IOB2 tag per word, shown as the mentions they form):

item_id	sentence	mentions
maccrobat-d002-s005	One month earlier he had undergone a laparotomy to excise a complex ileo-cutaneous fistula with blind tracts, and two weeks earlier a defunctioning ileostomy had been created in view of persistent abdominal pain.	treatment: laparotomy; problem: fistula; treatment: ileostomy; problem: pain
maccrobat-d002-s016	Urea and electrolytes, complement, urine analysis and microscopy were normal.	test: Urea; test: electrolytes; test: complement; test: urine analysis; test: microscopy
maccrobat-d000-s005	The anterior tricuspid valve leaflet was elongated (Fig.2C, arrow), whereas the septal leaflet was rudimentary (Fig.2C, arrowhead).	problem: elongated; problem: rudimentary

In the second, urine analysis is two words tagged B-test I-test. In the first and third, MACCROBAT annotates the anatomy ("abdominal", "anterior tricuspid valve leaflet") and the qualifiers ("complex", "ileo-cutaneous", "persistent") as separate types, which map to O: a model tagging "abdominal pain" as one problem misses the gold "pain".