Perturbation DE-gene overlap from predicted cells (Norman CRISPRa)

What is measured. The DE-gene overlap cell-eval reports for perturbation models, on the items, split and training data of single-cell/perturbation-response-delta-pearson (107 test perturbations of the GEARS simulation split (combo_seen0 9, combo_seen1 43, combo_seen2 19, unseen_single 36), bootstrap stratified by subgroup). The model is trained exactly as there (same recipe), then returns, for each test perturbation, the predicted single cells its mean prediction is the average of.

Wilcoxon DE of the predicted cells (cell-eval's recipe: github.com/ArcInstitute/cell-eval at 6928cf8bd7a7, its pdex 0.2.5 call, pdex(mode="ref", is_log1p=True, epsilon=0); reported in the State preprint, Adduri et al. 2025). Per test perturbation, every gene: a two-sided Mann-Whitney U test of the predicted cells against the control cells of the line in the derived file (normal approximation, continuity and tie correction, as numba-mwu and scipy.stats.mannwhitneyu(method= "asymptotic"); a gene with no variance gets p = 1), Benjamini-Hochberg FDR over the genes, and log2 fold change of the pseudobulks log2(expm1(mean ŷ cells) / expm1(mean control cells)), a NaN set to 0 as pdex 0.2.5 does (0/0, or a negative predicted mean log1p value). Predicted cells with a NaN or inf fail the cell. The stored frame keeps, per perturbation, that FDR and fold change for every gene (de_pred_fdr, de_pred_lfc), never the cells.

Metrics (both secondary). The predicted DE list is the genes with FDR < 0.05 by |log2 FC| descending (float32, ties to the lower gene index, as cell-eval's ordinal rank keeps pdex's gene order). The reference list is this benchmark's own 20 DE genes (the first 20 distinct of de_idx, the ones the parent benchmark's Pearson Δ DE20 and MSE DE20 read), not a second Wilcoxon run on the real cells as cell-eval does (the real cells of a test perturbation never leave the loader; its DE genes do). overlap_de20_wilcoxon_supervised is cell-eval's overlap_at_k at k = 20: |true[:k'] ∩ predicted[:k']| / k' with k' = min(20, |true|); precision_de20_wilcoxon_supervised is its precision_at_k, with k' = min(20, |predicted|) (0 when nothing is significant), which tells a low overlap from too few significant predictions. Both come from the same DE pass at no extra cost; cell-eval's other DE metrics need the real cells' DE table (de_spearman_sig, de_sig_genes_recall, PR/ROC AUC) and are left out. Each is computed per perturbation and averaged; bootstrap as the parent's sibling metrics.

Which models. A model takes part when its adapter can return the cells its mean prediction is made of (predict.perturbation_cells): State (its n_predict_cells = 256 cells per perturbation from state tx infer, sampled basal cells through a set model), CPA (the 300 sampled control cells decoded under the perturbation: per-cell conditional means, no decoder noise) and scGen (every control cell of the line decoded after the latent shift). GEARS is not applicable: its output is each control cell plus one per-perturbation shift (the network's residual), so its "cells" are the control cells moved together and their Wilcoxon test measures the shift against the control spread, not a predicted cell distribution. The reference baselines predict means only.

Opt-in. The cells are planned on gpu-full only (opt_in): each one re-trains its model (the parent's recipe, so the same fit) and runs the Wilcoxon pass on the CPU (~5 s per perturbation at Norman's size). Elsewhere they are SMOKE_ONLY with the gpu-full command. The sources are the parent's, read from its data directory (sources_from).

Data

• norman: url https://dataverse.harvard.edu/api/access/datafile/6154020, licence cc0-1.0
• gene2go: url https://dataverse.harvard.edu/api/access/datafile/6153417, licence cc0-1.0
• essential_genes: url https://dataverse.harvard.edu/api/access/datafile/6934320, licence cc0-1.0

Subset

• seed 20261006
• test: all item(s)
• train: all item(s)

Metrics

• overlap_de20_wilcoxon_supervised — DE-gene overlap, top 20, Wilcoxon on predicted cells (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroup
• precision_de20_wilcoxon_supervised — DE-gene precision, top 20, Wilcoxon on predicted cells (trained); protocol supervised; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by subgroup