Single-cell batch integration (scIB)

What is measured. Whether a cell representation keeps cell types apart while mixing donors: the batch-integration task of scIB (Luecken et al., Nat. Methods 2022), with donor as the batch, on Tabula Sapiens liver (CELLxGENE dataset version a4b3a49e, the file of cell-type-annotation-macro-f1; four donors TSP6, TSP14, TSP17, TSP19).

Cells (subset rule). 10x 3' v3 cells only. Cell types with at least 30 cells in at least 3 donors (12 types: CD8-positive alpha-beta T cell, classical monocyte, endothelial cell, erythrocyte, hepatocyte, intermediate monocyte, intrahepatic cholangiocyte, macrophage, mature NK T cell, monocyte, natural killer cell, neutrophil); for each (type, donor) the 30 top-ranked cells by seed-hash rank (seed 20261006), fewer when the donor has fewer: 1,342 candidate cells. The subset draws 500 test cells and 500 train cells from them, disjoint by cell, both stratified by cell type; every donor is on both sides (integration corrects batches the model has seen). (exc) n_test is 500, not 100: a silhouette over 100 cells of 12 types and 4 donors leaves 2 to 3 cells per (type, donor) and is noise.

Input. As for cell-type-annotation-macro-f1: a derived, label-free h5ad of the candidate cells (raw UMI counts, Ensembl ids and feature_name, cell id, donor); each item carries h5ad_path, cell_index and batch (the donor).

Zero-shot (*_zero_shot). The frozen mean-pooled embedding of each test cell is the integrated representation, with no correction step.

Supervised (*_supervised). Integration models with no pretrained weights (scVI; fit.batch_integration) are fit on the 500 train cells with their donor as batch key (their cell types are in the fit table too, for semi-supervised methods such as scANVI) and return a latent vector for each of the 500 test cells.

Metrics (scIB definitions, euclidean). Cell-type ASW = (mean silhouette w.r.t. cell type + 1) / 2; batch ASW = per cell type, the mean of 1 - |silhouette w.r.t. donor| over its cells, then the mean over types (types seen in one donor only are skipped); graph connectivity = per cell type, the fraction of its cells in the largest connected component of the symmetrised 15-nearest-neighbour graph restricted to that type, then the mean. No overall score is formed: ADR 17 keeps one scalar per metric.

Uncertainty. 95 % bootstrap interval over test cells (1,000 resamples, stratified by cell type), conditional on the geometry: each cell's silhouette and component membership are computed once on all 500 cells and resamples reweight them (resampling cells would put duplicates into every neighbourhood).

Caveats. Batch ASW rewards mixing even where biology differs between donors (the four donors differ in age and sex), and a representation that collapses everything mixes perfectly: read it together with cell-type ASW. Erythrocytes and neutrophils are missing from some donors. The cells are in the CELLxGENE Census, the pretraining corpus of several single-cell foundation models (cell identity seen, not these labels).

scIB bio and batch sets (metrics review). The scib-metrics Benchmarker defaults on the same embeddings: bio conservation = isolated-label ASW, k-means NMI and ARI (k = the number of cell types, seed 0), cell-type ASW, cLISI; batch correction = batch ASW (scIB's, not BRAS), iLISI, kBET per cell type, graph connectivity and PCR comparison; total = 0.6 bio + 0.4 batch. Two departures, both stated in the keys' descriptions: the composites are unscaled means (the Benchmarker's min-max scaling over the compared methods would make a value depend on which models are in the table), and the unintegrated representation PCR comparison needs is computed here: the raw counts of the same test cells log-normalised with scanpy's normalize_total defaults (each cell scaled to the median of the cells' non-zero total counts, not to 10,000), log1p, every gene, first 50 principal components (scanpy's defaults, the PCA scib-metrics computes when given no pre-integration embedding; no gene selection), in the frames as embedding_pre. The first composites (batch_correction_*, scib_total_*, pcr: false) leave PCR out, as the frames scored before held no unintegrated PCA; they keep their keys and values and are secondary. The full ones (pcr_comparison_*, batch_correction_with_pcr_*, scib_total_with_pcr_*) are the scib-metrics set. kBET tests every cell (R kBET samples 10 %) without a diffusion-map distance. The k-means clusterings, LISI neighbourhoods and kBET verdicts are fixed on the full data; resamples re-weight them, as for the silhouettes. Keys: nmi_kmeans_zero_shot, ari_kmeans_zero_shot, isolated_label_asw_zero_shot, clisi_zero_shot, ilisi_zero_shot, kbet_zero_shot, bio_conservation_zero_shot, batch_correction_zero_shot, scib_total_zero_shot, nmi_kmeans_supervised, ari_kmeans_supervised, isolated_label_asw_supervised, clisi_supervised, ilisi_supervised, kbet_supervised, bio_conservation_supervised, batch_correction_supervised, scib_total_supervised, pcr_comparison_zero_shot, batch_correction_with_pcr_zero_shot, scib_total_with_pcr_zero_shot, pcr_comparison_supervised, batch_correction_with_pcr_supervised, scib_total_with_pcr_supervised.

Data

• cellxgene: url https://datasets.cellxgene.cziscience.com/a4b3a49e-062b-4e3f-8915-02f40607651f.h5ad, licence cc-by-4.0, catalogued as single-cell/cellxgene-census

Subset

• seed 20261006
• test: 500 item(s), stratified by label
• train: 500 item(s), stratified by label
• results at atlas-n500-v1 use subset 2dcecb1aacb26f56

Metrics

• asw_celltype_zero_shot — Cell-type ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• asw_batch_zero_shot — Batch ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• graph_connectivity_zero_shot — Graph connectivity (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• asw_celltype_supervised — Cell-type ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• asw_batch_supervised — Batch ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• graph_connectivity_supervised — Graph connectivity (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• nmi_kmeans_zero_shot — NMI, k-means (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• ari_kmeans_zero_shot — ARI, k-means (zero-shot); protocol zero_shot; higher is better; range [-0.5, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• isolated_label_asw_zero_shot — Isolated-label ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• clisi_zero_shot — cLISI (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• ilisi_zero_shot — iLISI (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• kbet_zero_shot — kBET acceptance (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• bio_conservation_zero_shot — scIB bio conservation, unscaled (zero-shot); protocol zero_shot; higher is better; range [-0.1, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• batch_correction_zero_shot — scIB batch correction, unscaled, without PCR (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• scib_total_zero_shot — scIB total, unscaled, without PCR (zero-shot); protocol zero_shot; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• pcr_comparison_zero_shot — PCR comparison (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• batch_correction_with_pcr_zero_shot — scIB batch correction, unscaled (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• scib_total_with_pcr_zero_shot — scIB total, unscaled (zero-shot); protocol zero_shot; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• nmi_kmeans_supervised — NMI, k-means (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• ari_kmeans_supervised — ARI, k-means (trained from scratch); protocol supervised; higher is better; range [-0.5, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• isolated_label_asw_supervised — Isolated-label ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• clisi_supervised — cLISI (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• ilisi_supervised — iLISI (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• kbet_supervised — kBET acceptance (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• bio_conservation_supervised — scIB bio conservation, unscaled (trained from scratch); protocol supervised; higher is better; range [-0.1, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• batch_correction_supervised — scIB batch correction, unscaled, without PCR (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• scib_total_supervised — scIB total, unscaled, without PCR (trained from scratch); protocol supervised; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• pcr_comparison_supervised — PCR comparison (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• batch_correction_with_pcr_supervised — scIB batch correction, unscaled (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
• scib_total_with_pcr_supervised — scIB total, unscaled (trained from scratch); protocol supervised; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true

Two training cells of the frozen subset (input: the cell's raw counts and its donor; expected output: an embedding in which cells of one type sit together whatever their donor):

item_id	batch (donor)	total UMIs	top genes (counts)	label
TSP14_Liver_NA_10X_1_1_AACCATGCATAGACTC	TSP14	5,329	HBB 2042, HBA2 216, HBA1 156, CA1 66, HBD 63	erythrocyte (CL:0000232)
TSP14_Liver_NA_10X_1_1_AAGGTAAAGCCTCAAT	TSP14	7,638	S100A9 116, MALAT1 90, MT-RNR2 72, S100A8 64, MT-CO1 61	classical monocyte (CL:0000860)

Data: Tabula Sapiens Consortium, CZ CELLxGENE Discover, CC BY 4.0.