Spanish occupation entity recognition (ProfNER)

What is measured. Finding mentions of professions ("enfermera", "Presidente") and employment statuses ("desempleados") in Spanish tweets, with exact boundaries: occupational exposure mining for public health.

Data. ProfNER (Miranda-Escalada et al., SMM4H 2021 task 7): 10,000 Spanish COVID-19 tweets annotated with four types, of which the gold training (6,000) and validation (2,000) tweets were released, here in the BIO files of Biomedical-TeMU/ProfNER_corpus_NER (spaCy es_core_news_sm tokens with character offsets). The validation file is the test pool and the training file the train pool; the shared task's test labels were never published.

Types. PROFESION and SITUACION_LABORAL, the two types the ProfNER NER subtask evaluated (2,163 and 349 mentions over the 8,000 tweets); ACTIVIDAD and FIGURATIVA (a few dozen mentions) are tagged O. An I- tag orphaned by that removal begins a mention. Gold words are the corpus' tokens; the text given to models is rebuilt from their offsets (one space where tokens were not adjacent).

Items. 100 test tweets and 500 train tweets by seed-hash rank. Only 23 % of tweets mention a profession or status: the 100 test tweets hold 43 mentions (40 PROFESION, 3 SITUACION_LABORAL) in 24 tweets, the 500 train tweets 166.

Probe (entity_f1_probe; capability embed.text.tokens, pooling word_mean). One vector per gold word (mean of last-layer sub-word states); a logistic head over the 5 IOB2 tags is fit on the words of the 500 train tweets and tags each test word independently; an I-X after O starts a mention (conlleval-style repair).

Metric. Strict micro F1 over (type, first word, last word) mentions, pooled over the 100 tweets. 95 % bootstrap interval over tweets, 1,000 resamples.

Caveats (exc). With 43 test mentions the interval is wide; the benchmark is P1 and ranks encoders coarsely. The hub's Spanish models (roberta-base-biomedical-es, SapBERT-es) were pretrained on Spanish biomedical and clinical text, not on tweets; English encoders see Spanish through their sub-word vocabulary only. No zero-shot protocol: no hub model has a profession NER head.

Precision, recall and macro F1 (entity_precision_probe, entity_recall_probe, entity_macro_f1_probe; metrics review). ProfNER reports strict micro precision, recall and F1 on exact spans; precision and recall come from the same span matching as the F1 (0 precision when nothing is predicted), and the macro F1 is the unweighted mean of the PROFESION and SITUACION_LABORAL F1. The per-type values are stored with each result.

Data

• profner: hf_dataset Biomedical-TeMU/ProfNER_corpus_NER at revision de9bf1404880f4b7225e1cc0e9268192e57fefca, licence cc-by-4.0, catalogued as bio-nlp/profner-corpus-ner

Subset

• seed 20261006
• test: 100 item(s)
• train: 500 item(s)
• results at atlas-n100-v1 use subset 6a64cf58f0f13fda

Metrics

• entity_f1_probe — Entity F1, PROFESION + SITUACION_LABORAL (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_precision_probe — Entity precision, PROFESION + SITUACION_LABORAL (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_recall_probe — Entity recall, PROFESION + SITUACION_LABORAL (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• entity_macro_f1_probe — Entity macro F1, PROFESION + SITUACION_LABORAL (token probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

Two training tweets (input: the tweet and its words; expected output: one IOB2 tag per word, shown as the mentions they form):

item_id	tweet	mentions
profner-1243654656372494339	Primera demanda contra el Govern por el coronavirus. El Sindicato de Médicos denuncia a Salud por “poner en riesgo” a los profesionales https://t.co/KFPi9rT5OC	PROFESION: Médicos; PROFESION: profesionales
profner-1246759619873198080	¡INDIGNANTE! 📢 La policía nacional ha permitido que esto tuviera lugar esta mañana en #Madrid sin autorización de ningún tipo. Es evidente que la ley no es igual para todos. #Covid_19 #EsteVirusLoParamosUnidos #QuedateEnCasa #PoliciaNacional #CoronavirusARV @policia https://t.co/La8lq9crH5	PROFESION: policía nacional; PROFESION: PoliciaNacional; PROFESION: @policia

policía nacional is two words tagged B-PROFESION I-PROFESION; hashtags and handles naming a profession are mentions too.