Single-cell batch integration (scIB)
single-cellLeaderboard
Cell-type ASW (zero-shot)
scIB cell-type silhouette, (ASW + 1) / 2, of the frozen mean-pooled embeddings of 500 test cells from four donors: higher means cell types are better separated.
Best in the catalogue: 0.624 (the best visible result here, not a state of the art). Ranks are the server's: ties share a rank. A ≠ marks a row whose n differs from the board's usual n (500). Floors are trivial baselines scored on the same items, shown unranked.
| # | Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|---|
| 1 | single-cell/scimilarity | 0.624 best | [0.618, 0.631] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| 2 | single-cell/uce | 0.557 | [0.554, 0.561] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| 3 | single-cell/scgpt | 0.553 | [0.548, 0.557] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| 4 | single-cell/geneformer | 0.549 | [0.544, 0.553] | 500 | reproduced | unknown |
| — | hvg-pca-cells Reference floor, not a hub model; unranked | 0.574 | [0.568, 0.580] | 500 | floor |
Secondary metrics (9) · zero-shot · context, not ranked
NMI, k-means (zero-shot)
scib-metrics NMI between the cell types and a k-means clustering (k = the number of cell types, n_init 10, seed 0) of the frozen mean-pooled embeddings of the 500 test cells.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.764 best | [0.752, 0.787] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.720 | [0.705, 0.749] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.690 | [0.676, 0.723] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.685 | [0.671, 0.721] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.702 | [0.687, 0.735] | 500 | floor |
ARI, k-means (zero-shot)
Adjusted Rand index between the cell types and the same k-means clustering of the frozen mean-pooled embeddings of the 500 test cells.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.534 best | [0.511, 0.569] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.512 | [0.483, 0.554] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.455 | [0.426, 0.502] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.438 | [0.413, 0.477] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.481 | [0.453, 0.524] | 500 | floor |
Isolated-label ASW (zero-shot)
scIB isolated-label silhouette of the frozen mean-pooled embeddings of the 500 test cells: (silhouette + 1) / 2 over the cells of the cell types seen in the fewest donors, per type, then the mean.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.681 best | [0.673, 0.689] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.606 | [0.596, 0.615] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.604 | [0.593, 0.614] | 500 | reproduced | unknown |
| single-cell/uce | 0.576 | [0.569, 0.583] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.646 | [0.635, 0.656] | 500 | floor |
cLISI (zero-shot)
scIB cell-type LISI of the frozen mean-pooled embeddings of the 500 test cells, scaled: (L - median cLISI) / (L - 1) on a 90-nearest-neighbour graph, L the number of cell types; 1 means pure neighbourhoods.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.952 best | [0.943, 0.961] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.913 | [0.902, 0.923] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.909 | [0.897, 0.915] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.896 | [0.886, 0.905] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.913 | [0.904, 0.922] | 500 | floor |
iLISI (zero-shot)
scIB donor LISI of the frozen mean-pooled embeddings of the 500 test cells, scaled: (median iLISI - 1) / (B - 1) on a 90-nearest-neighbour graph, B = 4 donors; 1 means donors perfectly mixed.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.245 best | [0.218, 0.267] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.185 | [0.165, 0.208] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.138 | [0.121, 0.168] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.120 | [0.103, 0.135] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.078 | [0.068, 0.088] | 500 | floor |
kBET acceptance (zero-shot)
kBET per cell type of the frozen mean-pooled embeddings of the 500 test cells: the share of cells whose nearest same-type neighbourhood has the type's donor mix (chi-square, alpha 0.05), then the mean over cell types.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/uce | 0.632 best | [0.596, 0.669] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scimilarity | 0.621 | [0.583, 0.661] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.513 | [0.479, 0.548] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.505 | [0.470, 0.541] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.407 | [0.374, 0.440] | 500 | floor |
scIB batch correction, unscaled, without PCR (zero-shot)
Unweighted mean of four scIB batch-correction metrics of the frozen mean-pooled embeddings of the 500 test cells (batch ASW, iLISI, kBET, graph connectivity); PCR comparison left out (pcr: false), the batch score of the cells scored before the frames held the unintegrated PCA (batch_correction_with_pcr_zero_shot has all five).
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/uce | 0.673 best | [0.662, 0.684] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scimilarity | 0.662 | [0.650, 0.674] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.610 | [0.599, 0.622] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.606 | [0.596, 0.617] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.552 | [0.542, 0.562] | 500 | floor |
scIB total, unscaled, without PCR (zero-shot)
0.6 x bio conservation + 0.4 x batch correction (without PCR) of the frozen mean-pooled embeddings of the 500 test cells, both unscaled (scib_total_with_pcr_zero_shot includes PCR comparison).
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.692 best | [0.684, 0.702] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.663 | [0.655, 0.674] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/scgpt | 0.630 | [0.621, 0.642] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.623 | [0.615, 0.636] | 500 | reproduced | unknown |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.619 | [0.609, 0.631] | 500 | floor |
PCR comparison (zero-shot)
scIB principal-component regression comparison: (PCR before - PCR after) / PCR before, 0 when the representation holds more donor variance than the unintegrated data; PCR before is on the first 50 principal components of the log-normalised counts of the same 500 test cells (scanpy's normalize_total defaults: each cell scaled to the median of the cells' non-zero total counts, then log1p, every gene), PCR after on every component of the frozen mean-pooled embeddings.
| Model | Value | 95% CI | n | Source | Overlap |
|---|---|---|---|---|---|
| single-cell/scimilarity | 0.573 best | [0.466, 0.650] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/uce | 0.270 | [0.213, 0.313] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| single-cell/geneformer | 0.122 | [0.062, 0.174] | 500 | reproduced | unknown |
| single-cell/scgpt | 0.097 | [0.040, 0.143] | 500 | reproduced | overlapTrain/test overlap declared by the evaluator: single-cell/cellxgene-census |
| hvg-pca-cells Reference floor, not a hub model; unranked | 0.000 | [0.000, 0.000] | 500 | floor |
Protocol
What is measured. Whether a cell representation keeps cell types
apart while mixing donors: the batch-integration task of scIB (Luecken et
al., Nat. Methods 2022), with donor as the batch, on Tabula Sapiens liver
(CELLxGENE dataset version a4b3a49e, the file of
cell-type-annotation-macro-f1; four donors TSP6, TSP14, TSP17, TSP19).
Cells (subset rule). 10x 3' v3 cells only. Cell types with at least 30 cells in at least 3 donors (12 types: CD8-positive alpha-beta T cell, classical monocyte, endothelial cell, erythrocyte, hepatocyte, intermediate monocyte, intrahepatic cholangiocyte, macrophage, mature NK T cell, monocyte, natural killer cell, neutrophil); for each (type, donor) the 30 top-ranked cells by seed-hash rank (seed 20261006), fewer when the donor has fewer: 1,342 candidate cells. The subset draws 500 test cells and 500 train cells from them, disjoint by cell, both stratified by cell type; every donor is on both sides (integration corrects batches the model has seen). (exc) n_test is 500, not 100: a silhouette over 100 cells of 12 types and 4 donors leaves 2 to 3 cells per (type, donor) and is noise.
Input. As for cell-type-annotation-macro-f1: a derived, label-free
h5ad of the candidate cells (raw UMI counts, Ensembl ids and
feature_name, cell id, donor); each item carries h5ad_path,
cell_index and batch (the donor).
Zero-shot (*_zero_shot). The frozen mean-pooled embedding of each
test cell is the integrated representation, with no correction step.
Supervised (*_supervised). Integration models with no pretrained
weights (scVI; fit.batch_integration) are fit on the 500 train cells
with their donor as batch key (their cell types are in the fit table too,
for semi-supervised methods such as scANVI) and return a latent vector for
each of the 500 test cells.
Metrics (scIB definitions, euclidean). Cell-type ASW = (mean silhouette w.r.t. cell type + 1) / 2; batch ASW = per cell type, the mean of 1 - |silhouette w.r.t. donor| over its cells, then the mean over types (types seen in one donor only are skipped); graph connectivity = per cell type, the fraction of its cells in the largest connected component of the symmetrised 15-nearest-neighbour graph restricted to that type, then the mean. No overall score is formed: ADR 17 keeps one scalar per metric.
Uncertainty. 95 % bootstrap interval over test cells (1,000 resamples, stratified by cell type), conditional on the geometry: each cell's silhouette and component membership are computed once on all 500 cells and resamples reweight them (resampling cells would put duplicates into every neighbourhood).
Caveats. Batch ASW rewards mixing even where biology differs between donors (the four donors differ in age and sex), and a representation that collapses everything mixes perfectly: read it together with cell-type ASW. Erythrocytes and neutrophils are missing from some donors. The cells are in the CELLxGENE Census, the pretraining corpus of several single-cell foundation models (cell identity seen, not these labels).
scIB bio and batch sets (metrics review). The scib-metrics
Benchmarker defaults on the same embeddings: bio conservation =
isolated-label ASW, k-means NMI and ARI (k = the number of cell types,
seed 0), cell-type ASW, cLISI; batch correction = batch ASW (scIB's, not
BRAS), iLISI, kBET per cell type, graph connectivity and PCR comparison;
total = 0.6 bio + 0.4 batch. Two departures, both stated in the keys'
descriptions: the composites are unscaled means (the Benchmarker's min-max
scaling over the compared methods would make a value depend on which
models are in the table), and the unintegrated representation PCR
comparison needs is computed here: the raw counts of the same test cells
log-normalised with scanpy's normalize_total defaults (each cell scaled to
the median of the cells' non-zero total counts, not to 10,000), log1p, every
gene, first 50 principal components (scanpy's defaults, the PCA scib-metrics
computes when given no pre-integration embedding; no gene selection), in the
frames as
embedding_pre. The first composites
(batch_correction_*, scib_total_*, pcr: false) leave PCR out, as
the frames scored before held no unintegrated PCA; they keep their keys
and values and are secondary. The full ones (pcr_comparison_*,
batch_correction_with_pcr_*, scib_total_with_pcr_*) are the
scib-metrics set. kBET tests
every cell (R kBET samples 10 %) without a diffusion-map distance. The
k-means clusterings, LISI neighbourhoods and kBET verdicts are fixed on
the full data; resamples re-weight them, as for the silhouettes. Keys:
nmi_kmeans_zero_shot, ari_kmeans_zero_shot,
isolated_label_asw_zero_shot, clisi_zero_shot, ilisi_zero_shot,
kbet_zero_shot, bio_conservation_zero_shot,
batch_correction_zero_shot, scib_total_zero_shot,
nmi_kmeans_supervised, ari_kmeans_supervised,
isolated_label_asw_supervised, clisi_supervised, ilisi_supervised,
kbet_supervised, bio_conservation_supervised,
batch_correction_supervised, scib_total_supervised,
pcr_comparison_zero_shot, batch_correction_with_pcr_zero_shot,
scib_total_with_pcr_zero_shot, pcr_comparison_supervised,
batch_correction_with_pcr_supervised, scib_total_with_pcr_supervised.
Data
cellxgene: urlhttps://datasets.cellxgene.cziscience.com/a4b3a49e-062b-4e3f-8915-02f40607651f.h5ad, licence cc-by-4.0, catalogued assingle-cell/cellxgene-census
Subset
- seed 20261006
- test: 500 item(s), stratified by label
- train: 500 item(s), stratified by label
- results at
atlas-n500-v1use subset2dcecb1aacb26f56
Metrics
asw_celltype_zero_shot— Cell-type ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueasw_batch_zero_shot— Batch ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truegraph_connectivity_zero_shot— Graph connectivity (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueasw_celltype_supervised— Cell-type ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueasw_batch_supervised— Batch ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truegraph_connectivity_supervised— Graph connectivity (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truenmi_kmeans_zero_shot— NMI, k-means (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueari_kmeans_zero_shot— ARI, k-means (zero-shot); protocol zero_shot; higher is better; range [-0.5, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueisolated_label_asw_zero_shot— Isolated-label ASW (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueclisi_zero_shot— cLISI (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueilisi_zero_shot— iLISI (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truekbet_zero_shot— kBET acceptance (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebio_conservation_zero_shot— scIB bio conservation, unscaled (zero-shot); protocol zero_shot; higher is better; range [-0.1, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebatch_correction_zero_shot— scIB batch correction, unscaled, without PCR (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truescib_total_zero_shot— scIB total, unscaled, without PCR (zero-shot); protocol zero_shot; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truepcr_comparison_zero_shot— PCR comparison (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebatch_correction_with_pcr_zero_shot— scIB batch correction, unscaled (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truescib_total_with_pcr_zero_shot— scIB total, unscaled (zero-shot); protocol zero_shot; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truenmi_kmeans_supervised— NMI, k-means (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueari_kmeans_supervised— ARI, k-means (trained from scratch); protocol supervised; higher is better; range [-0.5, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueisolated_label_asw_supervised— Isolated-label ASW (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueclisi_supervised— cLISI (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_trueilisi_supervised— iLISI (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truekbet_supervised— kBET acceptance (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebio_conservation_supervised— scIB bio conservation, unscaled (trained from scratch); protocol supervised; higher is better; range [-0.1, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebatch_correction_supervised— scIB batch correction, unscaled, without PCR (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truescib_total_supervised— scIB total, unscaled, without PCR (trained from scratch); protocol supervised; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truepcr_comparison_supervised— PCR comparison (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truebatch_correction_with_pcr_supervised— scIB batch correction, unscaled (trained from scratch); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_truescib_total_with_pcr_supervised— scIB total, unscaled (trained from scratch); protocol supervised; higher is better; range [-0.06, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by y_true
Sample
Two training cells of the frozen subset (input: the cell's raw counts and its donor; expected output: an embedding in which cells of one type sit together whatever their donor):
| item_id | batch (donor) | total UMIs | top genes (counts) | label |
|---|---|---|---|---|
TSP14_Liver_NA_10X_1_1_AACCATGCATAGACTC | TSP14 | 5,329 | HBB 2042, HBA2 216, HBA1 156, CA1 66, HBD 63 | erythrocyte (CL:0000232) |
TSP14_Liver_NA_10X_1_1_AAGGTAAAGCCTCAAT | TSP14 | 7,638 | S100A9 116, MALAT1 90, MT-RNR2 72, S100A8 64, MT-CO1 61 | classical monocyte (CL:0000860) |
Data: Tabula Sapiens Consortium, CZ CELLxGENE Discover, CC BY 4.0.