Cell Painting knockout retrieval (RxRx3-core)

What is measured. Whether an image model's embedding of a Cell Painting well captures the phenotype of a gene knockout well enough to recognise the same knockout on another plate, with another guide. Data: RxRx3-core (Recursion, Kraus et al., LMRL @ ICLR 2025), CRISPR knockouts in HUVEC cells, one 6-channel field of view per well (Hoechst, ConA, phalloidin, SYTO14, MitoTracker, WGA; 512 x 512, 8-bit JPEG 2000).

Selection (seed 20261006, no image download beyond what is used). Gene experiments are ranked by seed-hash of their name and taken in order, among those whose every well sits in the one pinned image shard, until they hold 20 knockouts with at least 30 query-guide wells over at least 3 plates; that is gene-023 and gene-026 (9 plates each). 20 of their eligible knockouts are chosen by seed-hash rank. In each experiment, 3 of the 9 plates (seed-hash rank) are held out: their wells are the test pool, the wells of the 6 other plates the train pool, so test and train never share a plate (disjoint_by: plate re-checks it). Every knockout has 6 guides x 9 plates = 54 wells or a few fewer.

Items. 5 test wells per knockout (100) and 25 train wells per knockout (500), by seed-hash rank. (exc) The test set is 5 wells for each of 20 knockouts rather than 100 independent items: replicate retrieval needs several wells per perturbation.

Centring. Every well's embedding is centred on the mean embedding of its plate's EMPTY_control wells (no guide; 16 per plate), which are embedded with the items and never scored. This removes the plate effect that otherwise dominates Cell Painting embeddings.

Zero-shot (replicate_map_zero_shot). Each test well queries the other 99 by cosine similarity of the centred embeddings; its replicates are the 4 other test wells of its knockout (other plates or guides). The metric is the mean average precision over the 100 queries (chance is about 0.05).

Probe (accuracy_probe). A multinomial logistic head on the centred embeddings of the 500 train wells (6 plates) predicts the knockout of the 100 test wells (3 other plates): 20-way accuracy, chance 0.05.

Uncertainty. 95 % bootstrap interval over test wells, 1,000 resamples stratified by knockout.

Caveats. OpenPhenom (cell-imaging/openphenom) was trained on RxRx3, which contains these images: its numbers include training data (a known overlap, reported with the results). Some knockouts have no visible phenotype in HUVEC, which caps the reachable mAP below 1. The OpenPhenom_rxrx3_core_embeddings.parquet Recursion ships (532 MB, not fetched) gives reference OpenPhenom embeddings for these wells, a validation fixture for the OpenPhenom adapter (see benchmarks/cell-painting-perturbation-retrieval/openphenom_reference.json). Licence: "We used the RxRx3-core dataset, available from Recursion Pharmaceuticals at www.rxrx.ai, pursuant to Recursion Pharmaceutical's licensing terms"; no use in the neuroscience field.

Fraction retrieved, macro F1, top-5 accuracy (fraction_retrieved_zero_shot, macro_f1_probe, top5_accuracy_probe; metrics review). The fraction retrieved (Kalinin et al. 2025, copairs) is the share of knockouts whose mAP beats a permutation null: per well 10,000 average precisions of random rankings with its number of replicates and candidates (fixed seed 0), a knockout's null the mean of its wells', p = (#null >= mAP + 1) / 10,001, Benjamini-Hochberg over knockouts, retrieved at q < 0.05. The verdicts are decided once on the full data; the bootstrap resamples wells, so its interval covers which knockouts were tested, not permutation noise. The probe's macro F1 weighs the 20 knockouts equally; the top-5 accuracy counts a well right when its knockout is among the head's 5 most probable.

Data

• rxrx3_meta: hf_dataset recursionpharma/rxrx3-core at revision 89aedc798bf33e6f51cc8c6363009d9ffed69e31, licence Recursion RxRx3-core EULA: any use except in the neuroscience field; attribution; share-alike, catalogued as cell-imaging/rxrx3-core
• rxrx3_images: hf_dataset recursionpharma/rxrx3-core at revision 89aedc798bf33e6f51cc8c6363009d9ffed69e31, licence Recursion RxRx3-core EULA: any use except in the neuroscience field; attribution; share-alike, catalogued as cell-imaging/rxrx3-core

Subset

• seed 20261006
• test: 100 item(s), stratified by group
• train: 500 item(s), stratified by group, disjoint by plate
• results at atlas-n100-v1 use subset 6131bd7fd63d963d

Metrics

• replicate_map_zero_shot — Replicate mAP (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• accuracy_probe — Knockout accuracy, 20-way (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• fraction_retrieved_zero_shot — Fraction retrieved (zero-shot); protocol zero_shot; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• macro_f1_probe — Knockout macro F1, 20-way (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group
• top5_accuracy_probe — Knockout top-5 accuracy, 20-way (logistic probe); protocol probe; higher is better; range [0.0, 1.0]; 95% bootstrap CI, 1000 resamples, stratified by group

Wells are 6-channel images (input); the expected output is an embedding in which wells of one knockout lie close together. Three test wells:

item_id	knockout	guide	plate
rxrx3-gene-023_5_N33	TNFSF10	TNFSF10_guide_6	gene-023 plate 5
rxrx3-gene-023_2_U04	TNFSF10	TNFSF10_guide_1	gene-023 plate 2
rxrx3-gene-026_2_P36	PDE3A	PDE3A_guide_1	gene-026 plate 2

The first two are replicates of each other (same knockout, other guide and plate): a good embedding ranks each near the top of the other's neighbours, above the PDE3A well.