BC5CDR chemical NER

What is measured. Finding every chemical (drug, compound, element) mention in a PubMed title and abstract, with exact boundaries.

Data. The BioCreative V CDR corpus (Li et al., Database 2016): 1,500 PubMed abstracts annotated by MeSH indexers with chemical and disease mentions, read from the NCBI release archive (CDR_Data.zip, CDR.Corpus.v010516, PubTator format) mirrored in bigbio/bc5cdr. The official training set (500 abstracts, 5,203 chemical mentions) is the train pool and the official test set (500, 5,385) the test pool; the development set is not used. Only Chemical mentions are scored.

Words. Each abstract (title + space + abstract, the text the PubTator offsets index) is cut into words: runs of letters/digits and single punctuation marks, also cut at every mention boundary, so each mention is a run of whole words (5-fluorouracil -> 5 - fluorouracil). Gold tags are IOB2 per word (B-Chemical, I-Chemical, O). Every model is scored on these words, whatever its own tokenizer. The rare nested mentions keep the longest one.

Items. 100 test abstracts (seed-hash rank) and all 500 training abstracts (about 243 words each, 670 at most).

Probe (entity_f1_probe; capability embed.text.tokens, pooling word_mean). The model returns one vector per gold word (the mean of its last-layer states over the sub-word tokens of that word; abstracts longer than the model's window are encoded in windows). A logistic head over the three tags is fit on the ~120,000 training words and tags each test word independently; an I-Chemical that follows O starts a mention (conlleval-style repair).

Zero-shot (entity_f1_zero_shot; capability predict.ner). A model's own token-classification head tags the same words (each word takes the tag of its first sub-word). Its entity types are mapped by a fixed table: Medication (d4data/biomedical-ner-all, MACCROBAT types) and treatment (samrawal/bert-base-uncased_clinical-ner, i2b2 2010 types) become Chemical, every other type O. Neither head has a chemical type: the mapping is lossy by construction (medications only on one side, treatments that include procedures on the other), and its number says how far a clinical NER head transfers, not how well it was trained.

Metric. Strict micro F1 over (type, first word, last word) mentions, pooled over the 100 abstracts (seqeval strict IOB2). 95 % bootstrap interval over abstracts, 1,000 resamples.

Caveats. Several hub encoders (BioBERT, PubMedBERT, SciBERT) were pretrained on PubMed text, which includes these abstracts (unlabelled). Word-level scoring differs slightly from the official character-offset evaluation of the CDR task; numbers are comparable between models here, not with published BC5CDR-chem tables to the decimal.

Precision and recall (entity_precision_probe, entity_precision_zero_shot, entity_recall_probe, entity_recall_zero_shot; metrics review). The two halves of the entity F1, from the same strict span matching (seqeval strict IOB2): precision is the share of predicted Chemical spans that are exact annotated spans (0 when nothing is predicted), recall the share of annotated spans found exactly. BC5CDR has one entity type here, so there is no macro F1.

Data

• cdr: hf_dataset bigbio/bc5cdr at revision 6ba1463320d003e8232dfa0c3ee9aaa2559998ec, licence public-domain, catalogued as bio-nlp/bc5cdr

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s), disjoint by pmid
• results at atlas-n100-v1 use subset 45a9ead52526fb52

Metrics

• entity_f1_probe — Chemical entity F1 (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_f1_zero_shot — Chemical entity F1 (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_precision_probe — Chemical entity precision (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_recall_probe — Chemical entity recall (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_precision_zero_shot — Chemical entity precision (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_recall_zero_shot — Chemical entity recall (native NER head); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

One training abstract (input: the text and its words; expected output: one tag per word, here shown as the mentions they form):

bc5cdr-1009330 — Further studies on effects of irrigation solutions on rat bladders. Further studies on the effects of certain irrigating fluids on the rat bladder for 18 hours are reported. The results have shown that the degradation product p-choloroaniline is not a significant factor in chlorhexidine-digluconate associated erosive cystitis. A high percentage of kanamycin-colistin and povidone-iodine irrigations were associated with erosive cystitis and suggested a possible complication with human usage. Picloxydine irrigations appeared to have a lower incidence of erosive cystitis but further studies would have to be performed before it could be recommended for use in urological procedures.

mention	words	tags
p-choloroaniline	p - choloroaniline	B-Chemical I-Chemical I-Chemical
chlorhexidine-digluconate	chlorhexidine - digluconate	B-Chemical I-Chemical I-Chemical
kanamycin	kanamycin	B-Chemical (the - after it is O)
colistin	colistin	B-Chemical
povidone-iodine	povidone - iodine	B-Chemical I-Chemical I-Chemical
Picloxydine	Picloxydine	B-Chemical

Every other word is O ("erosive cystitis" is a Disease mention, not scored here).