Cell type annotation (macro F1)

What is measured. Assigning each cell its cell type from its gene expression alone, for cells of a donor the model never saw annotated. Data: Tabula Sapiens, liver (Tabula Sapiens Consortium, Science 2022), as curated by CZ CELLxGENE Discover (dataset version a4b3a49e, 22,214 cells, 4 donors, 10x 3' v3 and Smart-seq2); labels are CELLxGENE's cell_type (Cell Ontology terms, kept as label_ontology).

Cells (subset rule). Only 10x 3' v3 cells (21,398), so assay is not confounded with donor. A cell type is eligible for a donor when that donor has at least 20 of its cells and the other donors together at least 100. The held-out donor is the first, by seed-hash rank (seed 20261006), of the donors with at least 10 eligible types: TSP6; the cell types are the first 10 of its eligible types by seed-hash rank: CD8-positive alpha-beta T cell, classical monocyte, endothelial cell, hepatocyte, intermediate monocyte, intrahepatic cholangiocyte, macrophage, mature NK T cell, monocyte, natural killer cell. The test pool holds the 20 top-ranked TSP6 cells of each type, the train pool the 100 top-ranked cells of each type from TSP14, TSP17 and TSP19; the label-stratified draws give 100 test cells (10 per type, all from TSP6) and 500 train cells (50 per type), donor-disjoint by construction and re-checked (disjoint_by: batch). (exc) The default per-group sizes are replaced by per-type quotas, so the macro F1 weighs every type equally on 10 cells each.

Input. Workers never read the CELLxGENE file (it carries every cell's annotation). The loader writes a derived h5ad of the 1,200 pool cells with the raw UMI counts (raw/X), all 60,606 GENCODE v41 genes (Ensembl ids, feature_name), the cell id and the donor; an item points at its row (h5ad_path, cell_index). Normalisation, gene selection and the context length are the model's own.

Zero-shot (macro_f1_zero_shot, ari_zero_shot). Without any training on the benchmark: frozen mean-pooled embeddings of the 100 test cells are clustered by k-means (k = 10, 10 restarts, seed 0); ARI compares the clusters with the cell types, and each cluster is matched one-to-one to a cell type by the assignment that maximises agreement (Hungarian) to give the macro F1 of the best labelling of that clustering. A model with a native cell-type head (predict.cell_type) would be scored on its own labels instead; that path waits for the executor's zero-shot predict op and is not applicable until then. The method is in each result's notes.

Probe (macro_f1_probe). A logistic head (C chosen by stratified CV on log-loss inside train) on frozen mean-pooled embeddings of the 500 train cells, applied to the 100 test cells.

Supervised (macro_f1_supervised). Annotators with no pretrained weights (scANVI) are trained from scratch on the 500 train cells (counts, donor, cell type) and predict the 100 test cells of the unseen donor.

Uncertainty. 95 % bootstrap interval over test cells, 1,000 resamples stratified by cell type.

Caveats. Three of the ten types are nested monocyte annotations (monocyte, classical monocyte, intermediate monocyte) and mature NK T cells sit between NK and CD8 T cells, so a perfect score is not expected; every model faces the same ambiguity. Ambient RNA is visible (hepatocyte transcripts such as ALB in immune cells). Tabula Sapiens is in the CELLxGENE Census, the pretraining corpus of several single-cell foundation models (scGPT whole-human, UCE, SE-600M; possibly Geneformer V2's Genecorpus-104M and scFoundation's corpus): those models have seen these very cells, without the labels, and the exported overlap flag says so (training_overlap). SCimilarity is different: it is metric-learned on author cell-type annotations that include Tabula Sapiens (Heimberg et al. 2025, Methods; donors TSP1-TSP15, so TSP6 and TSP14 here), so it has seen these cells with their labels. Its cells on this benchmark are marked "in training data": computed and shown in compare with a dagger, never ranked and never exported.

Accuracy and NMI (accuracy_zero_shot, accuracy_probe, accuracy_supervised, nmi_zero_shot; metrics review). scGPT reports accuracy next to macro F1. Balanced accuracy (the mean per-type recall) is not declared: with 10 test cells of every type it equals accuracy by construction. The zero-shot NMI compares the k-means clusters with the cell types (scIB's bio conservation), and does not depend on the cluster-to-type matching. The zero-shot cell-type silhouette (asw_celltype_zero_shot, scIB's (ASW + 1) / 2, euclidean) reads the frozen embeddings the k-means clusters, which the zero-shot frames keep.

Data

• cellxgene: url https://datasets.cellxgene.cziscience.com/a4b3a49e-062b-4e3f-8915-02f40607651f.h5ad, licence cc-by-4.0, catalogued as single-cell/cellxgene-census

Subset

• seed 20261006
• test: 100 item(s), stratified by label
• train: 500 item(s), stratified by label, disjoint by batch
• results at atlas-n100-v1 use subset 96640c03008bc579

Metrics

• macro_f1_zero_shot — Macro F1 (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• ari_zero_shot — ARI (zero-shot clustering); protocol zero_shot; higher is better; range [-0.5, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• macro_f1_probe — Macro F1 (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• macro_f1_supervised — Macro F1 (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_zero_shot — Accuracy (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• nmi_zero_shot — NMI (zero-shot clustering); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• asw_celltype_zero_shot — Cell-type ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_probe — Accuracy (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label
• accuracy_supervised — Accuracy (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by label

Two training cells of the frozen subset (input: the cell's raw counts, row cell_index of the derived h5ad; expected output: its cell type):

item_id	donor	total UMIs	top genes (counts)	label
TSP14_Liver_NA_10X_1_1_AAAGTCCCACACGGTC	TSP14	9,092	NEAT1 174, MALAT1 173, ALB 70, CPS1 49, GPHN 46	hepatocyte (CL:0000182)
TSP14_Liver_NA_10X_1_1_ACAGGGATCCATCAGA	TSP14	11,867	MALAT1 242, FTL 117, FTH1 115, MT-RNR2 108, IL1B 85	intermediate monocyte (CL:0002393)

Data: Tabula Sapiens Consortium, CZ CELLxGENE Discover, CC BY 4.0.