Perturbation response in an unseen cell line (Replogle CRISPRi)

Licence to confirm before publication on the hub. The Arc State release of the Replogle/Nadig CRISPRi Perturb-seq data declares no licence (no dataset card) and the upstream papers state no redistribution terms. Local evaluation is fine; results and the proposed dataset must not be published on the hub before legal confirmation.

What is measured. Transferring perturbation effects to a cell line the model never saw perturbed: CRISPRi knockdowns measured by Perturb-seq in K562, RPE1, Jurkat and HepG2 cells (Replogle et al., Cell 2022; Nadig et al. 2025), in the Arc Institute's filtered concatenation used to train and evaluate State (arcinstitute/State-Replogle-Filtered, replogle_concat.h5ad: 643,413 cells, 6,546 genes, log-normalised X; K562 188,590, Jurkat 184,470, RPE1 173,737, HepG2 96,616 cells).

Split. The pinned hepg2_zeroshot.toml (State's zero-shot configuration) sends HepG2 to test: HepG2 is held out. Among the 639 knockdowns present in all four lines (each with at least 30 cells per line), 100 are chosen by seed-hash rank (seed 20261006). Test = those 100 knockdowns in HepG2 (atlas-n100-v1, the whole test pool); train = the same 100 knockdowns in K562, RPE1 and Jurkat (300 items).

Input. A derived h5ad (never the source, which holds the HepG2 perturbed cells): up to 500 non-targeting control cells per line (all four lines, HepG2 included, by seed-hash rank) and, for each train item, at most 50 cells (seed-hash rank); 16,547 cells, with cell_line, gene (the knocked-down gene) and the control flag. A train item's cell_index lists its rows, a test item has none; perturbation is the gene symbol and cell_line the line. (exc) The 50-cell and 500-control caps and the restriction to 100 knockdowns keep the file at 110 MB; State trains on every knockdown and cell of the three lines.

Labels. y = mean expression of all cells of the knockdown in HepG2 (30 to 275 cells); c = mean of all HepG2 control cells (4,976); DE genes (the source has none): Welch t statistic of the knockdown's cells against the controls of its line, ranked by |t|, genes the knockdown drops out while controls express them skipped (GEARS' top_non_dropout_de_20 rule), the top 20.

Supervised (only protocol). Perturbation models (fit.perturbation: State, GEARS, CPA, scGen) are trained on the train items and predict the mean HepG2 expression of each test knockdown, given HepG2 control cells.

Metrics (GEARS definitions, per knockdown then the mean): Pearson Δ = corr(ŷ - c, y - c) over all genes; MSE DE20 = mean of (ŷ - y)² on the 20 DE genes.

Uncertainty. 95 % bootstrap interval over the 100 test knockdowns, 1,000 resamples.

Caveats. Adding the knockdown's average shift in the three training lines to the HepG2 control profile already scores Pearson Δ 0.33 and MSE DE20 0.031 (the control profile itself: MSE DE20 0.111): that is the bar. The target gene's own knockdown dominates the DE genes of many items. The released State and SE-600M checkpoints were trained on Replogle and Nadig data, HepG2 included unless a zero-shot checkpoint is used: they must declare that overlap (or use a HepG2-held-out checkpoint).

Further metrics (metrics review). delta_pearson_de20_supervised is the top-20 DE Pearson Δ of the Norman benchmark, on the same DE genes as the MSE. direction_accuracy_de20_supervised is GEARS' share of the 20 top DE genes whose predicted change from control has the measured sign; mae_supervised the all-gene MAE (cell-eval); discrimination_score_supervised cell-eval's rank of each knockdown's own measured change among all test knockdowns' by L1 distance to its prediction (1 - rank / n, then the mean), which, unlike Δ-correlations, does not reward a shift shared by every knockdown. Each score is computed once against the full test set; the bootstrap re-weights knockdowns.

DE-gene overlap, pseudobulk proxy (overlap_de20_pseudobulk_supervised, secondary). Per test knockdown, the share of its 20 top DE genes (the Welch-t genes above) that are among the 20 genes with the largest predicted |ŷ - c|, then the mean. Ties go to the lower gene index, and a gene whose |ŷ - c| is within the float32 resolution of the profiles (10^-6 + 2^-16 x the larger of |ŷ| and |c|: round-off, not a change) is never counted; with fewer than 20 reference genes both lists are cut at that number (cell-eval's arithmetic). This is a proxy, not cell-eval's DE overlap (overlap_at_k, github.com/ArcInstitute/cell-eval at 6928cf8; reported in the State preprint, Adduri et al. 2025): cell-eval re-runs a Wilcoxon test of predicted single cells against control cells, keeps the genes with FDR < 0.05 and ranks them by |log2 fold change|, while this protocol stores one predicted mean per knockdown, so the proxy ranks the predicted mean change, with no test and no FDR. The no-change baseline predicts the mean of the 500 HepG2 controls of the derived file, not of all 4,976, so its |Δ| is control sub-sampling noise (overlap 0.006, chance being 20 / 6,546 = 0.003).

Data

• replogle: hf_dataset arcinstitute/State-Replogle-Filtered at revision d790193bb2c93726541a75ca3fa873a92ed44da5, licence unknown

Subset

• seed 20261006
• test: 100 item(s)
• train: all item(s)

Metrics

• delta_pearson_supervised — Pearson Δ, all genes (trained); protocol supervised; higher is better; range [-1.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mse_de20_supervised — MSE, top-20 DE genes (trained); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• delta_pearson_de20_supervised — Pearson Δ, top-20 DE genes (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• direction_accuracy_de20_supervised — Direction accuracy, top-20 DE genes (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• mae_supervised — MAE, all genes (trained); protocol supervised; lower is better; range ≥ 0.0; 95% bootstrap CI, 1000 resamples
• discrimination_score_supervised — Discrimination score (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples
• overlap_de20_pseudobulk_supervised — DE-gene overlap, top 20, pseudobulk proxy (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples

One test knockdown (input: perturbation EIF6, cell_line hepg2 and the HepG2 control cells of the derived file; expected output: its mean expression over the 6,546 genes):

item_id	cells	top DE genes: measured mean (HepG2 control mean)
hepg2:EIF6	91	EIF6 0.13 (1.72), BAX 0.36 (1.08), MDM2 1.66 (0.94), NEAT1 2.87 (2.06), CWF19L2 0.02 (0.08)
k562:EIF6 (train)	50 (capped)	EIF6 0.05 (0.92), MT-ND2 3.33 (3.81), FBXO25 0.01 (0.08), ETAA1 0.01 (0.09), OAZ1 2.42 (2.75)

In HepG2 (p53 wild type) the EIF6 knockdown induces the p53 targets BAX and MDM2; in K562 (p53 null) it does not: the effect a model has to transfer is line-dependent.