Clinical negation detection (NegEx test kit)

Licence to confirm before publication on the hub. The NegEx test kit's Apache-2.0 is declared by the package README next to the file only; the repository that ships it carries no licence file, so the terms covering the sentences themselves are unclear. Local evaluation is fine; results and the proposed dataset must not be published on the hub before legal confirmation.

What is measured. Whether a clinical concept named in a sentence of a clinical report is negated ("Denies shortness of breath, stridor, or air hunger") or affirmed.

Data. The NegEx test kit (Annotations-1-120-random.txt, distributed with GenNegEx 1.2 in the chapmanbe/negex repository, Apache-2.0): 2,376 de-identified sentences from clinical reports, each with one target concept (written in upper case inside the sentence) and a gold decision, Negated (491) or Affirmed (1,885). 11 rows whose concept does not occur in the sentence (the kit cut the sentence before it) are dropped: 2,365 items.

Input. The sentence with the target's occurrence wrapped in [entity] ... [entity] (the upper-case occurrence when there is one, else the first), the marking the hub's assertion classifier was trained with. Report headers that the kit kept in some sentences stay as they are.

Items (exc). No split is published. atlas-n100-v1 draws 100 test items stratified by decision (21 negated, 79 affirmed), then 500 train items stratified the same way. A sentence that carries several concepts never sits on both sides (sentence_key, the hash of the normalised sentence). atlas-full-v1 keeps that train set (the same 500 items) and tests on every other item whose sentence is not a train sentence: 1,442 items (316 negated, 1,126 affirmed), which contain the 100 of atlas-n100-v1. The 423 remaining items share a sentence with a train item and are used by neither split.

Test size. The leaderboard version is atlas-full-v1 on every profile: every kit item outside the train sentences, 1,442 concepts. Encoders score a short sentence in milliseconds; with 21 negated test items the n100 negated F1 was mostly noise. The earlier atlas-n100-v1 (100 test items, drawn as described above) stays frozen and its cells stay current; it is a separate ranking, never mixed with the full one.

Zero-shot (macro_f1_zero_shot). Either capability: predict.assertion, a native assertion head on the marked sentence, mapped ABSENT -> negated, PRESENT and POSSIBLE -> affirmed (the kit has no possible class, and NegEx counts a possible mention as not negated); or score.options, a language model reading Sentence: <sentence>\nQuestion: In this sentence, is "<concept>" negated (stated as absent)?\nAnswer: and scoring " yes" (negated) against " no" (affirmed).

Probe (macro_f1_probe; capability embed.text.pooled). Frozen embeddings (last layer, mean over tokens) of the marked sentence; a logistic head is fit on the 500 train items. The markers are plain text, so the encoder knows which concept is asked about only through them.

Uncertainty. 95 % bootstrap interval over test items, 1,000 resamples stratified by decision.

Caveats. Rule-based NegEx scores about 0.95 negated F1 on the full kit; the kit was built to evaluate it. Macro F1 is used because 78-79 % of test items are affirmed (always answering affirmed scores 0.44 macro F1 on both test sets). Some sentences are long report fragments (up to ~900 characters). The hub's assertion classifier (bvanaken/clinical-assertion-negation-bert) was trained on i2b2 2010 assertions, not on this kit.

Sensitivity, specificity, precision (sensitivity_zero_shot, sensitivity_probe, specificity_zero_shot, specificity_probe, precision_zero_shot, precision_probe; metrics review). NegEx (Chapman et al. 2001) reports the sensitivity, specificity and positive predictive value of the negated class: the same predictions, negated as the positive class (pos_label: negated; precision is 0 when nothing is called negated). The probe also gets roc_auc_probe, from its probability of the negated class. A language model gets roc_auc_zero_shot, from its probability of " yes" (negated) against " no": the softmax over the two options' log-likelihoods, which the zero-shot frames keep as y_score. A native assertion head gives labels without scores, so its zero-shot cells have no ROC-AUC.

Data

• negex: url https://raw.githubusercontent.com/chapmanbe/negex/d136b972b5d214959807ad5de1881a3225541fe7/GeneralNegEx.Java.v.1.2.05092009/Annotations-1-120-random.txt, licence apache-2.0

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by sentence_key
• leaderboard version atlas-full-v1 on every profile: the whole eligible test pool (the sizes above define the n version)

Metrics

• macro_f1_zero_shot — Macro F1 (native head or prompt); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• macro_f1_probe — Macro F1 (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• sensitivity_zero_shot — Sensitivity, negated (native head or prompt); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• specificity_zero_shot — Specificity, negated (native head or prompt); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• precision_zero_shot — Precision (PPV), negated (native head or prompt); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• sensitivity_probe — Sensitivity, negated (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• specificity_probe — Specificity, negated (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• precision_probe — Precision (PPV), negated (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_probe — ROC-AUC, negated (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• roc_auc_zero_shot — ROC-AUC, negated (prompt); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

The leaderboard (atlas-full-v1) scores 1,442 test concepts (100 in atlas-n100-v1). Three training items (input: the concept-marked sentence; expected output: the decision):

item_id	concept	input text	decision
negex-0003	air hunger	Denies shortness of breath, stridor, or [entity] AIR HUNGER [entity].	negated
negex-1663	weakness	No [entity] WEAKNESS [entity].	negated
negex-0002	pulmonic regurgitation	There is trace [entity] PULMONIC REGURGITATION [entity].	affirmed

For a language model, the first item becomes Sentence: Denies shortness of breath, stridor, or AIR HUNGER.\nQuestion: In this sentence, is "air hunger" negated (stated as absent)?\nAnswer: and " yes" should be more likely than " no".