Secondary metrics (3) · supervised · context, not ranked
MSE, all genes (trained)
↓ lower is better · log1p-normalised expression² · supervised Mean squared error between predicted and measured mean expression over the 5,045 genes, per perturbation, then the mean over the 107 test perturbations (GEARS mse); lower is better.
Discrimination score (trained)
↑ higher is better · supervised cell-eval's discrimination score: per test perturbation, 1 - rank / n of its own measured change from control among the measured changes of all 107 test perturbations, by L1 distance to its predicted change (half for ties), then the mean; 1 means each prediction is closest to its own perturbation.
DE-gene overlap, top 20, pseudobulk proxy (trained)
↑ higher is better · supervised Pseudobulk proxy of cell-eval's DE overlap, NOT cell-eval's overlap (which re-runs a Wilcoxon DE test on predicted single cells): per test perturbation, the share of its 20 top DE genes (the ones of Pearson Δ DE20 and MSE DE20) found among the 20 genes with the largest predicted |ŷ - c| (ties by gene index, unchanged genes never counted), then the mean over the 107 test perturbations.
What is measured. Predicting the transcriptional response of K562
cells to CRISPR activation of one or two genes that the model was never
trained on in that combination: Norman et al., Science 2019 (GEO
GSE133344), as processed and released by GEARS (Roohani et al., Nat.
Biotechnol. 2023; Harvard Dataverse doi:10.7910/DVN/Q2ZV3E, norman.zip):
91,205 cells, 5,045 genes (log-normalised), 284 conditions: control, 152
single-gene conditions over 105 genes (some genes under both spellings,
A+ctrl and ctrl+A, which GEARS keeps apart) and 131 double
activations.
Split. GEARS' simulation split with GEARS' defaults and seed 1
(PertData.prepare_split(split="simulation", seed=1),
train_gene_set_size 0.75, combo_seen2_train_frac 0.75), reimplemented
with GEARS' numpy calls. As GEARS does, the 7 conditions whose genes are
not in its perturbation graph are dropped first (IER5L+ctrl, ctrl+IER5L,
KIAA1804+ctrl, LYL1+IER5L, RHOXF2BB+ctrl, RHOXF2BB+SET, RHOXF2BB+ZBTB25;
graph = GEARS' essential_all_data_pert_genes.pkl genes that have GO terms
in gene2go_all.pkl, both pinned). Test = the 107 held-out
perturbations (combo_seen0 9, combo_seen1 43, combo_seen2 19,
unseen_single 36, the subgroup column), train = the other 169 (GEARS'
train and validation parts together; a model holds out its own
validation from them). (exc) The test set is the whole held-out set
(atlas-full-v1), not 100 drawn items: the split, not a sample, defines
the benchmark, and GEARS numbers are reported on it.
Input. Workers get a derived h5ad (never the GEARS file, which holds
the test cells): the 7,353 control cells and, for each train
perturbation, its 100 top-ranked cells by seed-hash rank (seed 20261006;
fewer if it has fewer), with condition, condition_name, the control
flag, gene ids and names, and GEARS' DE genes of the train conditions
(uns/top_non_dropout_de_20). A train item's cell_index lists its rows;
a test item has none, only its perturbation (GEARS spelling) and
cell_line. The fit table also carries each train item's mean profile
(label).
Supervised (only protocol). Perturbation models (GEARS, CPA, scGen;
fit.perturbation) are trained on the train items and predict the mean
log-normalised expression of each test perturbation over the 5,045 genes.
Metrics (GEARS definitions). With c the mean expression of all control
cells, y the mean over all cells of the perturbation and ŷ the
prediction: Pearson Δ = corr(ŷ - c, y - c) over all genes; Pearson Δ DE20
the same on the perturbation's 20 top non-dropout DE genes (GEARS
uns["top_non_dropout_de_20"]: Welch t-test against control, ranked by
|t|, genes the perturbation drops out removed); MSE DE20 = mean of
(ŷ - y)² on those genes. Each is computed per perturbation and averaged
over the 107; an undefined correlation counts as 0, as in GEARS.
Uncertainty. 95 % bootstrap interval over test perturbations, 1,000
resamples stratified by GEARS subgroup.
Caveats. Train perturbations are capped at 100 cells (GEARS trains on
all of them; median 272 cells per perturbation): numbers are close to, not identical with,
GEARS' published ones. Pearson Δ over all genes is dominated by the
shared response of K562 cells to any activation (erythroid genes such as
HBZ, HBG2): on this split, predicting the mean profile of the train
perturbations for every test perturbation scores Pearson Δ 0.58, Pearson
Δ DE20 0.65 and MSE DE20 0.41 (the control profile: MSE DE20 0.56), the
floor any model has to clear. Single-cell foundation
models pretrained on CELLxGENE have not seen these cells (GEO, not
CELLxGENE); GEARS, CPA and scGen checkpoints trained on Norman 2019 must
not be used as is (they saw the test perturbations).
Direction accuracy, MSE, discrimination
(direction_accuracy_de20_supervised, mse_supervised,
discrimination_score_supervised; metrics review). GEARS reports the
share of the 20 top DE genes whose predicted change from control has the
measured sign (frac_correct_direction_20; a gene unchanged on both sides
counts as right) and the MSE over all genes. cell-eval's discrimination
score ranks each perturbation's own measured change among all test
perturbations' by L1 distance to its prediction (1 - rank / n, half for
ties, then the mean): Δ-correlations reward a systematic shift shared by
every perturbation (Ahlmann-Eltze et al. 2025), discrimination rewards
perturbation-specific predictions. Each score is computed once against the
full test set; the bootstrap re-weights perturbations.
DE-gene overlap, pseudobulk proxy (overlap_de20_pseudobulk_supervised,
secondary). Per test perturbation, the share of its 20 top DE genes
(the GEARS genes above) that are among the 20 genes with the largest
predicted |ŷ - c|, then the mean. Ties go to the lower gene index, and a
gene whose |ŷ - c| is within the float32 resolution of the profiles
(10^-6 + 2^-16 x the larger of |ŷ| and |c|: round-off, not a change)
is never counted; with fewer than 20 reference genes both lists are cut
at that number (cell-eval's arithmetic). This is a proxy, not cell-eval's DE
overlap (overlap_at_k, github.com/ArcInstitute/cell-eval at 6928cf8;
reported in the State preprint, Adduri et al. 2025): cell-eval re-runs a
Wilcoxon test of predicted single cells against control cells, keeps the
genes with FDR < 0.05 and ranks them by |log2 fold change|, while this
protocol stores one predicted mean per perturbation, so the proxy ranks
the predicted mean change, with no test and no FDR. The no-change
baseline scores 0: its prediction differs from the stored control mean by
float32 round-off only (|Δ| < 1e-5, within the tolerance), so it ranks no
gene.
Data
norman: url https://dataverse.harvard.edu/api/access/datafile/6154020, licence cc0-1.0gene2go: url https://dataverse.harvard.edu/api/access/datafile/6153417, licence cc0-1.0essential_genes: url https://dataverse.harvard.edu/api/access/datafile/6934320, licence cc0-1.0
Subset
- seed 20261006
- test: all item(s)
- train: all item(s)
- results at
atlas-full-v1 use subset d889ab4560e46f19
Metrics
delta_pearson_supervised — Pearson Δ, all genes (trained); protocol supervised; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroupdelta_pearson_de20_supervised — Pearson Δ, top-20 DE genes (trained); protocol supervised; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroupmse_de20_supervised — MSE, top-20 DE genes (trained); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples, stratified by subgroupdirection_accuracy_de20_supervised — Direction accuracy, top-20 DE genes (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroupmse_supervised — MSE, all genes (trained); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples, stratified by subgroupdiscrimination_score_supervised — Discrimination score (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroupoverlap_de20_pseudobulk_supervised — DE-gene overlap, top 20, pseudobulk proxy (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroup
One test perturbation (input: the perturbation and the control cells of
the derived file; expected output: the mean expression of each of the
5,045 genes) and one train perturbation:
| item_id | split | cells | top DE genes: measured mean (control mean) |
|---|
norman:CBL+CNN1 | test, combo_seen1 | 288 | HBZ 4.885 (1.793), SLC25A37 2.890 (0.858), HBG2 3.813 (1.893), PRSS57 0.133 (0.774), ALAS2 1.671 (0.173) |
norman:KLF1+ctrl | train | 100 (capped) | TMSB10 3.541 (2.418), HBZ 3.547 (1.793), HBG2 3.229 (1.893), PNMT 1.189 (0.284), KLF1 1.347 (0.479) |
A model that predicted the control profile for CBL+CNN1 would have
Pearson Δ 0 (an undefined correlation of a zero change counts as 0).