PDS
Perturbation discrimination. Whether each predicted profile is closer, by cosine similarity, to its own true profile than to those of other targets. It rewards predictions that are distinct and correctly matched.
Fourteen tries at a virtual cell
Virtual Cell Challenge 2026: a version-by-version retrospective of my first perturbation-modelling project
Retrospective of the Pedrito Amor entry to the Virtual Cell Challenge 2026: the reasoning, implementation, results and failure modes of each version, from the first baseline to the final submission.
Competition retrospective · October 2026 Team name: Pedrito Amor
This was my first project with perturbation data and my first attempt at the “virtual cell” problem. I am not a specialist in this area. I entered the Virtual Cell Challenge mainly to work with these datasets directly and understand how they behave in practice, beyond what the papers report.
This page documents every version in chronological order, from the first baseline (v0) to the final delivery (v14): what each one tested, how it was implemented, what the official leaderboard returned, and what did not work. Negative results are reported alongside positive ones, because most of what I learned came from them.
14validation versions (v0 to v13), plus the final v14-DEF
0.2166best official validation score (v12)
14,218logged results from local experiments
12alternative model routes in a matched comparison
Summary
- Best result. v12 scored 0.2166 on the validation leaderboard. It combined multi-source effect transfer with removal of 15 control principal components. The final test delivery (v14-DEF) reused this recipe; its score is not visible.
- Where the gains came from. The largest improvement (0.151 to 0.208, v5 to v8) came from choosing better-matched source data for each hidden context, not from more complex models. The expression-accuracy (MSE) component stayed at zero until v6.
- Main trade-off. Methods that made targets easier to tell apart (PDS) tended to cost expression accuracy (MSE). v12 struck the best balance; v13 pushed discrimination further and lost overall.
- Local evaluation was optimistic. In all three cases where a local estimate was recorded before the official score (v5, v10 and v11), the local number was higher, and one early local gain was produced by a bug in my own evaluator.
- Alternative models. None of twelve alternative routes (CPA, scGen, STATE, Geneformer, scGPT and others) was clearly better than my v11 recipe on error in matched panels; on the H1 panel all but one were clearly worse, with 95% paired bootstrap intervals. PIE, tested end to end, reproduced its published benchmark but carried almost no target-specific signal on my data.
The task
A CRISPR interference (CRISPRi) knockdown silences one gene. The effect rarely stays local: through regulatory networks, hundreds or thousands of other genes change expression. The full transcriptome-wide change is the perturbation response.
The challenge asks for that response in a cell context that has never been perturbed. For each hidden context, participants receive only its untreated control cells (non-targeting controls, NTC) and a list of target genes. Everything else has to come from other experiments, usually run in other cell types. The submission is not a summary but a count matrix: hundreds of simulated single cells per target, with non-negative integer counts for every measured gene.
In practice, this splits into two problems. The first is estimating the average expression change for each target in a context where it was never measured. The second is turning those averages into realistic single-cell counts, because several metrics are computed on cell-level differential expression tests and are sensitive to how counts are distributed across cells.
How the score works
The leaderboard combines six metrics into one overall score. The four differential-expression (DE) metrics are computed from Wilcoxon rank-sum tests between perturbed and control cells. Much of this project is about how the metrics pull in different directions.
MSE
Expression accuracy. An unbiased, capped and normalised mean squared error between predicted and observed pseudobulk expression: whether the predicted levels are right.
NMAE
DE log-fold-change accuracy. Normalised mean absolute error of the predicted fold changes for differentially expressed genes.
Fidelity
DE direction fidelity. For genes that change, whether the predicted direction (up or down) matches.
Reach
DE direction reach. How many of the truly changed genes the prediction recovers with the correct direction.
Jaccard
DE significance overlap. Overlap between the predicted and observed sets of significant genes. It stayed close to zero for every version.
Each metric is normalised against a reference prediction, so 0 means “no better than the reference” and higher is better. The overall 0.2166 obtained with v12 is therefore not an accuracy of 21.66%; it is an aggregate of six relative improvements. When the raw error is worse than the reference, the MSE component is clipped to zero. That was the case for every submission up to v5.
How one prediction is built
Almost every version used the same pipeline. Versions differ in one or two stages, and the chapters below refer back to these steps.
- 1
Characterise the receiver
Untreated controls of the new context give baseline expression and natural cell-to-cell variation.
all versions - 2
Collect source effects
Measured knockdown effects for the same targets in other cell types and studies.
v1, v2, v6, v8–v10 - 3
Combine sources
Weight each source per gene according to its estimated reliability.
v6, v8, v10, v13 - 4
Remove shared structure
Centre out the response common to all targets and project out control-variation axes.
v6, v12, v13 - 5
Set the amplitude
Scale the predicted change (lambda) to a calibrated magnitude.
v7, v10–v13 - 6
Calibrate DE calls
Enforce a minimum number of genes called as changed, with consistent direction.
v7, v10, v12 - 7
Emit cells
Convert pseudobulk predictions into integer counts for each simulated cell.
v4, v5
The last stage matters more than it appears. I refer to the starting count matrix as the carrier and to the procedure that modifies it as the emitter. Two predictions with identical pseudobulk means can score differently depending on how counts are distributed across cells, because differential expression tests depend on variance as well as means.
The whole journey in two charts
The first chart shows the official overall score of every validation submission. The hollow point (v7) is a score recovered only as a rounded value. Hover over a point to see the exact number. The two points before v1 are an upload test and a baseline (v0).
The second chart breaks the overall score into its six components. Each row is coloured from the version that scored worst on that metric (lightest) to the one that scored best (darkest), so trade-offs between metrics are visible at a glance.
The versions fall into six phases, each refining one idea. Every chapter follows the same structure: the hypothesis, the implementation, what improved, what failed, what was tried and rejected, and what carried over into the next version.
The record is detailed for the later versions and incomplete for the earliest ones. Where the archive does not state the reason for a decision, I give my reconstruction and mark it “In retrospect”. Terms are defined in the glossary.
Chapter 1 · v0 to v3: transferring measured responses
27 September to 10 October · the starting point
test 0.023v0 0.129v1 0.134v2 0.144v3 0.148
Hypothesis
The simplest defensible baseline: if knocking down a gene produces a given response in well-characterised cell lines, a similar response is a reasonable first estimate in a new context. Take measured effects from other experiments, combine them, and adapt them to the receiver’s baseline.
Implementation
The first submission, on 27 September, only checked that the upload pipeline worked. It used an identity-style baseline (identity75) and scored 0.0230.
The first real baseline, universal_5 (v0), used five heavily perturbed cell types: H1 embryonic stem cells, K562, RPE1, HepG2 and Jurkat. It scored 0.1290 the following day.
v1 separated each response into its direction (which genes go up or down) and its magnitude. Related stem-cell experiments (KOLF2.1J, HipSci and hESC) contributed to direction with a low weight of 0.25 each, while magnitude remained determined by the five-cell core. A common multinomial generator then produced the simulated cells. v1 scored 0.1338 on 30 September.
v2 expanded the source catalogue considerably: Orion HCT116 and HEK293T, CD4 T cells under several conditions, Jiang et al. by cell line and cytokine, PerturbME, Frangieh, Tian, Papalexi and additional KOLF samples. It also introduced leave-one-source-out routing, shrinkage of uncertain estimates and alternative cell emitters. v2 scored 0.1441 on 6 October, with PDS rising from 0.516 to 0.601.
v3, submitted on 10 October, consolidated this work into the first published reference: 0.1482.
Provenance of the early scores. The test, v0, v1 and v2 scores are taken from the official result emails, with the four decimals they report. The emails do not identify the model or entry, so each was matched to its version by date and local logs.
What improved
- An end-to-end pipeline producing valid submissions.
- The direction/magnitude split allowed related data to contribute without letting uncertain sources set effect sizes.
- The direction-bank configuration improved several local contexts.
What failed
- The MSE component was 0.0000 in every early submission: predicted levels were no better than the reference.
- A cache of control cells was being reused across contexts. Errors of this kind propagate silently into every downstream result.
- Some reference panels had an undefined normalised MSE, which made comparisons unreliable.
- Version tracking was weak: v2’s score could only be recovered from an email, and two different entry IDs are recorded for it.
Tried and rejected
- Transferring KOLF, Jurkat, HipSci and hESC effects to HepG2. Each scored slightly below the baseline; a source that helps one receiver can hurt another.
- The direction-bank recipe lost a little in HepG2 while improving other contexts.
Carried forward. In retrospect, the pseudobulk estimates were roughly in range but the differential-expression metrics were weak. The next hypothesis targeted the last pipeline stage: perhaps the limitation was not the average prediction but how it was distributed across simulated cells.
Chapter 2 · v4 and v5: redistributing counts across cells
Cell-level emission · 10 to 12 October
v4 0.147v5 0.151
Hypothesis
For a given gene, the same total count can be spread evenly across 400 simulated cells or with realistic cell-to-cell variability. The pseudobulk mean is identical in both cases, but differential-expression tests, which compare distributions, are not. Making the cell-level distribution resemble real controls should improve the DE-based metrics without changing average expression.
Implementation
v4 applied a copula transform (joint_control_copula050_r1) to the v3-era predictions. A copula models the joint dependence structure of variables; here it transferred the co-variation pattern of real control cells to the simulated cells while preserving each gene’s total count exactly.
v5 restricted redistribution to weak effects (absolute log pseudobulk effect below 0.085). It protected the 54 strongest effects, the perturbed gene itself and very low-count genes, then repaired integer row and column totals so that weighted cell means were approximately preserved.
An evaluator bug
The local evaluation reported a large gain for v4: 0.2005 versus 0.1614 for the previous recipe. The official score was 0.1465, lower than v3.
The cause was in my evaluator. It scored only min(400, number of real cells) predicted cells. Redistribution changed which cells fell into that subset, and that alone produced an apparent PDS gain. After switching to all 400 predicted cells, most of the gain disappeared; what remained came from the DE metrics, not from PDS or MSE.
This was one of the most useful findings of the project: a better local score is not evidence of a better official score unless the evaluator is shown to be faithful. The same lesson recurred until the final submission.
For v5 I therefore fixed a forecast of the official score before uploading: 0.1585, from a proxy that rebuilt predicted cells by transporting real control variation. The official result was 0.1507, an absolute error of 0.0079. The forecast got the direction of the change right (a small gain over v3) but overestimated the level. A second proxy, fixed at the same time with uniform cells, predicted 0.1679 and was further off (mean absolute error across the six metrics 0.029 against 0.011). One submission does not validate a proxy, but it was the first check of one against the leaderboard.
What improved
- A real evaluator bug was found and fixed, which made every later comparison more trustworthy.
- v5 raised the official score to 0.1507.
- The v5 count matrix became the carrier reused by all later versions, giving a stable starting point.
What failed
- v4 regressed officially despite a strong local result.
- The MSE component remained at zero.
- v5’s gains were supported by four development contexts, not by an independent confirmation.
Tried and rejected
- Genomic coordinates (GENCODE). Neighbouring genes are sometimes co-regulated, so position could plausibly help. The effect was −0.0006 with an interval including zero.
- Gradient-boosted trees and TabICL pair models. Locally 0.1646 and 0.1696 against 0.1614 for the previous recipe, both below the copula candidate.
- Starting from resampled controls instead of the v5 carrier. This lost the predictive starting point that v5 provided.
Carried forward. With cell emission stabilised, the remaining lever was the response estimate itself. If the transferred effects are weak or come from poorly matched cell types, redistribution cannot compensate. The next phase focused on source selection.
Chapter 3 · v6 to v9: context-specific source branches
The steady climb
v6 0.171v7 ≈0.194v8 0.208v9 0.208
Hypothesis
A single universal average ignores that some sources match some receivers better. Building specialised branches, each combining sources suited to a family of cell types, and routing each hidden validation context (A, B and C) to the branch whose controls it most resembles should transfer stronger and more relevant effects.
Implementation
v6 used two branches. The H1 branch averaged effects from the genome-wide K562 screen (GWPS), CD4 T cells, HCT116 and HEK293T, with an effect-mixing weight of 0.5. The CD4 branch combined centred GWPS effects with a ridge-regression prediction from Jurkat. Context A was routed to the CD4 branch and B and C to the H1 branch, using frozen control-similarity weights. v6 also introduced centring: subtracting part of the response shared by all targets, so that predictions do not collapse towards a common profile. v6 scored 0.1705, and the MSE component rose above zero for the first time (0.150).
v7 targeted the DE metrics. A post-processing floor ensured that about 3,000 genes per target were called as changed, with the significance threshold relaxed from z = 8 towards 4.5, plus an explicit cis component for the targeted gene. In context A, the amplitude was recalculated after removing 10 control principal components (lambda 0.625). v7 scored about 0.194, and direction fidelity became positive for the first time.
v8 changed only context A. It moved A to the H1 multi-source branch and added a genome-wide Jurkat screen (GSE249595), blended per gene through a noise gate, w·z²/(z²+9) with w = 0.5: the stronger a gene’s effect relative to its noise, the more weight Jurkat received. v8 scored 0.2084.
v9 added the Yao et al. THP-1 screen (GSE221321), using the single-guide arm after three hours of LPS stimulation. It covered 598 targets, but only 12 overlapped the official list. v9 scored 0.20844, a change of +0.00006.
What improved
- The largest gain of the project: from 0.151 to 0.208 over four versions.
- The MSE component rose from zero to 0.221; predicted levels finally beat the reference.
- v8 changed a single context, which made it a cleaner experiment than most.
What failed
- v6 changed sources, branches and emission together, so the contribution of each could not be isolated.
- Routing A to the CD4 branch reflects control similarity, not evidence that A is a CD4 population.
- v9 added substantial work for almost no gain; LPS-stimulated THP-1 cells are a poor proxy for resting T cells.
- A linear score forecast was fitted on two points (v6 and v7). Two points always fit a line, so it carried no predictive information.
Tried and rejected
- Adding HipSci and PerturbFate to the best H1 recipe. The local score fell from 0.3006 to 0.2993 and 0.2987; related data can dilute a good signal.
- Glioma (A172, T98G, U87MG) and TeloHAEC variants. They did not pass the initial promotion criteria and were kept only as reference data.
- Jiang 2025. The first audit found nine targets matching the official list, so it was never given a full performance test. Low coverage alone does not show it would have hurt.
Carried forward. Each added source appeared to help slightly. The natural next step was to include all available sources, weighted by reliability, and to adjust each effect to the receiver’s baseline expression.
Chapter 4 · v10 and v11: expansion and rollback
More data, worse score · v11 submitted 16 October
v10 0.198v11 0.211
Hypothesis
Use substantially more data, combine it according to estimated source reliability, and transform each borrowed effect to account for differences in baseline expression between source and receiver.
Implementation
v10 was the most ambitious version. On top of the four base sources it enabled Orion and CD4 profiles with no minimum cell count (to recover rare targets), added VIPerturb and DLD1 for B and C, Jurkat for A, and eleven GEO studies plus nine L1000 cell-line banks for all contexts.
Sources were combined by inverse-variance weighting in B and C and by a random-effects model in A, which also allows for genuine between-source differences. Three receiver-baseline transformations were enabled: fold-change transfer, a transfer map, and a mask excluding genes detected in fewer than 5% of receiver controls. The amplitude target rose to 1.10 and the DE floor to 6,000 genes.
Locally, the GEO plus L1000 configuration was the best result so far, at about 0.2314. The official score was 0.1985, a clear regression. The MSE component fell from 0.221 to 0.169.
v11 reverted the risky part. It kept the wider source set but disabled all three baseline transformations and returned the amplitude to 0.99. v11, submitted on 16 October, scored 0.2113, above both v10 (+0.0128) and v9 (+0.0028), and ranked 241st on the validation leaderboard at the time. Its MSE component, 0.239, was the highest of the project.
What improved
- v11 recovered from v10 and set a new best, with the highest expression-accuracy score of the project.
- Keeping the extra sources while removing the transformations gave a fairly clean contrast.
- v11 became the reference for expression accuracy in all later comparisons.
What failed
- v10 changed many components at once. Baseline scaling, the higher amplitude and adaptive proxy selection are all plausible causes; no single-cause ablation was run officially.
- The local forecast for v11 was 0.22685 against an official 0.2113. Local estimates remained optimistic.
- v11’s reach and Jaccard were worse than v9’s.
Tried and rejected
- An all-source bulk mixture: about 0.2300 locally against 0.2314 for GEO plus L1000.
- Transferring a common perturbation response (the component shared by most knockdowns). It agreed poorly across contexts and worsened local metrics.
- Removing the minimum cell count everywhere. It recovers rare targets but does not correct weak knockdowns, mismatched controls, double perturbations or incompatible units; it admits them.
One implementation detail mattered more than expected. Under inverse-variance weighting, an effect reported with zero variance receives infinite precision and can dominate a gene. An explicit uncertainty floor was required. Data plumbing of this kind took a large share of the work.
Side study: benchmarking PIE against v11
From mid-October, a large part of the work went into testing whether PIE, a perturbation-effect model from Arc Institute, could beat v11. PIE received more effort than any other alternative model, so it is documented here in full. Every comparison below is against v11.
Pilots (10 to 18 October). I downloaded PIE’s preprocessed public datasets and ran a high-memory Colab pilot on CD4 T cells, followed by a version trained only on my own data banks. Both runs had problems with scale and with DE calling, and their numbers were not used.
Joint training from scratch (18 to 19 October). I kept PIE’s architecture (a Perceiver with 512 latents and 4 + 6 + 4 encoder, processor and decoder layers) but initialised every response parameter from scratch; no checkpoint trained on responses was loaded. Training combined five public datasets (Replogle/Nadig, Tahoe-100M, Jiang, ARC VCC 2025 and X-Atlas/Orion; about 76,000 public profiles) with about 45,000 profiles from my banks: roughly 68 GB of inputs and 274,000 task rows. It used PIE’s eleven knowledge sources, including ESM2 protein embeddings, STRING, DepMap, gene and perturbation text, L1000, PRISM and JUMP. Each measurement unit had its own output head, so units were never mixed.
6,250GPU training steps before the run was stopped
0.864best validation error ratio (1.0 = predicting no change)
300VIPerturb K562 targets predicted with the frozen model
The validation error ratio fell from 5.22 at step 50 to about 0.87 by step 500, then remained between 0.86 and 0.92 for the next 5,750 steps. I stopped the run manually and froze the best checkpoint. It had covered every training task once, but not every gene of every profile, and it did not complete its planned budget.
Integration with the pipeline (19 October). A bridge converted PIE’s predictions into count cells and mixed them with v11 at four weights (0%, 25%, 50% and 100% PIE). These were scored on held-out VIPerturb targets: 21 for selection and 27 sealed for confirmation. The outcome was “no comparable six-metric score, keep v11”. One local anchor failed: the Jaccard anchor was higher for a generic prediction than for real replicates, so no valid local overall score could be computed.
Arc’s published checkpoint (19 to 20 October). To rule out my training as the cause, I also ran the checkpoint Arc published for the Replogle K562 benchmark. It first had to reproduce the published results:
| Arc metric | Published | My run |
|---|---|---|
| AUPRC for DE genes | 0.1984 | 0.1984 |
| Jaccard of significant genes | 0.1004 | 0.1004 |
| Direction match | 0.7083 | 0.7082 |
| Spearman, significance | 0.4749 | 0.4749 |
| Spearman, log fold change | 0.4010 | 0.4003 |
| Discrimination (L1) | 0.6815 | 0.6814 |
The largest difference was 0.0007, within the pre-declared tolerance of 0.005, so the model was running as published. The problem appeared on transfer. On 170 VIPerturb K562 targets, PIE’s predicted changes were barely better than the same predictions shuffled between targets (median error ratio 4.26 against 4.33). A generic average response did better (2.32), and the cosine similarity between predicted and observed changes was essentially zero. About 37% of the prediction energy was a component common to all targets. PIE performed well on the benchmark it was trained for but carried almost no target-specific signal for this study.
The final comparison with v11 used a rule fixed before it was run: a paired bootstrap over targets, a multiple-comparison correction across candidates and non-inferiority margins for every metric. The candidates were v11, pure PIE, two PIE/v11 mixtures (10% and 25% PIE) and a null mixture with no PIE signal. The result was “keep v11”, because the metrics did not share enough targets for a fair comparison (fidelity covered 69% of targets against the 80% required).
PIE was therefore tested end to end: trained from scratch, run as published, integrated into the pipeline and compared under a pre-specified rule. It never outperformed v11, and no PIE-based version was submitted. Two caveats remain: the from-scratch run stopped before its full budget, and overlap between PIE’s pretraining data and my evaluation data was not verified.
Carried forward. v11 had the best expression accuracy, and PIE could not improve on it, but v11 was weak at discriminating targets. The next question was whether target discrimination could be improved without losing too much accuracy, by removing shared background structure from the predictions.
Chapter 5 · v12 and v13: removing control-variation axes
Best version, and an over-amplified successor · second half of October
v12 0.2166v13 0.212
Hypothesis
Untreated cells vary along a few dominant, recurring axes, such as cell-cycle stage, size and stress. When these axes leak into predicted effects, predictions for different targets become more similar and PDS suffers. Estimating the main axes of variation in the receiver’s controls and projecting them out of the predicted effects should make target-specific differences stand out.
Implementation
v12 took up to 3,000 official control cells per context, transformed them to log1p CP50K and centred them per gene. A seeded singular value decomposition (SVD) gave the top 15 principal components, which were projected out of the predicted effects after centring (0.75) and before recalculating the amplitude at ratio 0.99. The resulting lambdas were 0.90, 0.925 and 0.975 for A, B and C. The DE floor rose from 6,000 to 8,000 genes.
v12 scored 0.2166, the best official result. The six metrics show what changed relative to v11: PDS rose from 0.655 to 0.743 while the MSE component fell from 0.239 to 0.211. Removing the control axes made targets much easier to discriminate at a moderate cost in expression accuracy, and the trade improved the overall score.
v13 (candidate c12133, selected from three parallel configuration searches described below) pushed further. It weighted genes by cross-source reliability, clip(0.4 + r/0.05, 0.4, 1); removed 8 receiver PCs plus an additional 15-PC subspace estimated from VIPerturb controls; raised the amplitude ratio to 1.05; and allowed lambda up to 6.0. v13 scored 0.2124, below v12.
The component scores explain the result. PDS rose slightly to 0.757 (v12: 0.743), but the MSE component fell further to 0.192 (v12: 0.211), and NMAE and reach also declined. A small gain in discrimination was paid for with a larger loss in accuracy, and the overall score went down.
The same trade-off appeared locally on a held-out panel (VIPerturb B/C, with disjoint control sets for generation and scoring). These are raw local values, not official scores:
| Variant | PDS ↑ | MSE ↓ | Reach ↑ |
|---|---|---|---|
| v11 | 0.578 | 1.114 | 0.510 |
| 50/50 mixture | 0.580 | 1.026 | 0.538 |
| 15 PCs removed, DE floor 6,000 | 0.602 | 1.123 | 0.504 |
| 15 PCs removed, DE floor 8,000 | 0.602 | 1.166 | 0.530 |
Removing control PCs improved discrimination, and the mixture had the lowest raw error. No variant was best on every metric, and this panel could not determine the behaviour on the hidden contexts.
Paired bootstrap intervals on the two historical 24-target panels (2,000 target resamples, 95% CI) point the same way. On CD4, v12’s PDS was higher than v11’s (+0.034 to +0.221), and v13’s was higher again than v12’s (+0.009 to +0.123). On H1, where PDS is already close to its ceiling (0.93 to 0.96), the intervals cross zero. In native MSE on H1, v12 was slightly but clearly worse than v11 (+0.001 to +0.020), and v13 was worse than v12 (+0.003 to +0.028); on CD4 the MSE intervals cross zero. These intervals reflect target sampling only, not training variability or the hidden contexts.
What improved
- v12 was the best official result (0.2166), and its package and generator are fully preserved.
- Projecting out control axes did what was intended for discrimination: PDS 0.655 to 0.743 in one step.
- v13 quantified the cost of over-amplification clearly.
What failed
- Projecting out control axes also removed some real signal: the MSE component fell from 0.239 to 0.211.
- v13 changed reliability weighting, projection axes and amplitude together. Excess amplitude is the most likely cause of the loss, but it was not isolated.
- Two different recipes were labelled “v13” at different times; the submitted one was c12133. Consistent naming would have avoided this.
Tried and rejected
- Removing more PCs, or giving VIPerturb more weight. Small, inconsistent changes or regressions.
- Reliability scores computed with the receiver’s own source included. The agreement was inflated by contamination, and the comparison was rejected.
- Direct effects without the carrier. Promising against a control-only baseline, worse against realistic H1, HepG2 and K562 baselines; changing the starting point can manufacture an apparent gain.
Carried forward. The newest version was not the best. Before the final round, I fixed a rule: unless an alternative could be shown to beat v12, the final submission would use v12.
Chapter 6 · v14-DEF: the final submission
Final test delivery · 24 October
v14-DEF published · score not visible
Hypothesis
The final round introduced three new hidden contexts, D, E and F, again with only untreated controls available. The aim was to use the recipe with the strongest official evidence rather than the most recent one.
Implementation
I rebuilt the controls and target lists for the final panel and compared each context’s control cells with a reference atlas of control profiles, using Spearman correlation of expression ranks across roughly 17,700 measured genes. The nearest references were:
| Context | Nearest | Second | Third |
|---|---|---|---|
| D | HL60, 0.877 | THP-1, 0.859 | CD4, stimulated 48 h, 0.858 |
| E | U87MG, 0.871 | UMRC3 (kidney), 0.867 | KMRC20 (kidney), 0.867 |
| F | KMRC20 (kidney), 0.868 | UMRC3 (kidney), 0.867 | AC16, 0.866 |
D is clearly myeloid-like and E is closest to the glioma line U87MG, while F has a mesenchymal or fibroblast-like profile with several references within 0.002 of each other. These are indications, not identities: the margins are small, and expression similarity does not provide perturbation responses or establish the exact cell line.
D used the arm developed for context A, and E and F the arm for B and C. A quick comparison on analogue datasets (Yao THP-1 with LPS, U87MG and T98G, with 18, 21 and 21 evaluable targets) could not produce a comparable six-metric ranking, because some corrected MSE values were undefined.
The pre-specified fallback therefore selected v12. The final delivery used v12’s recipe (15 control PCs removed, amplitude ratio 0.99, DE floor 8,000) with the new controls and targets, under the name pedrito-amor-v14-DEF.
The upload (3.2 GB, MD5 verified) completed on 24 October at 14:21 UTC. The server lists it as published, with no visible score.
What went right
- The fallback rule was fixed before any comparison, which prevented a last-minute choice driven by noise.
- The delivery is a new prediction with the new controls and targets, not a relabelled validation matrix.
- The upload was verified byte for byte.
What remains unknown
- Whether it worked. No score is visible, and “published” indicates neither a zero score nor success.
- Whether v12’s recipe transfers to D, E and F. No perturbed data from these contexts were available to check.
Behind the scenes: three parallel searches
The submitted versions are a small fraction of the work. Between v12 and v13, I ran three independent configuration searches around the best recipes, each with its own evaluator and search strategy, and compared their shortlists before choosing what to submit.
Claude · about 1,000
Four campaigns (CELLSEARCH 364, FASTSEARCH 315, RANDSEARCH 333 and STACK 22 rows) scored with a fast local evaluator. They varied gene-reliability weighting, control subspaces, amplitude and cellular carrier variants.
Codex · 10,000
A continuous grid over version mixtures, partial control-PC removal, amplitude and expression masks, scored with an approximate bulk evaluator.
Colab · 10,000
Proposals around v12 (15 PCs, DE floor 8,000) in 16 families. 16 were re-scored natively on two metrics and four finalists on all six, from emitted cells.
Each search produced a shortlist of about five candidates. I compared the shortlists and selected the Claude candidate, c12133, which was submitted as v13. Officially it scored 0.2124, below v12 (0.2166). The candidate favoured discrimination and amplitude, and the extra amplitude cost more in expression accuracy than it gained in PDS (see Chapter 5).
The other two shortlists also stayed close to their baselines. The Codex grid’s best configuration (90% v10 plus 10% v11, additional partial PC removal, a basal-expression quantile mask and amplitude 0.85) improved PDS on 60 internal validation targets from 0.592 to 0.605, but it was never emitted as integer counts, so its native MSE was unknown. 2,857 of its 10,000 rows met the eligibility criterion, which is not the same as 2,857 improvements. The Colab finalists, scored natively against a local reproduction of v12, changed the metrics only marginally:
| Candidate | H1 PDS ↑ | H1 MSE ↓ | CD4 PDS ↑ | CD4 MSE ↓ |
|---|---|---|---|---|
| v12 (local reproduction) | 0.949 | 0.851 | 0.697 | 1.002 |
| Common-response subtraction | 0.957 | 0.853 | 0.701 | 0.997 |
| Soft control-PC removal | 0.949 | 0.852 | 0.712 | 1.001 |
| Common/specific rescaling | 0.957 | 0.854 | 0.697 | 0.997 |
| Weak-target PC removal | 0.947 | 0.852 | 0.716 | 1.002 |
None of the finalists improved PDS and MSE together on both panels, and the run’s own report warned against inferring an official gain from 10,000 retrospective trials.
The logged result rows from the local campaigns (14,218 in eleven journals, which do not include the Colab run) are distributed as follows. They are rows, not independent models or validated experiments; many are small variations of one another.
One further result is worth noting. A separate source-combiner configuration was emitted as real counts on a training panel and outperformed both v11 and the 15-PC recipe on PDS (0.611 against 0.566 and 0.598) and on raw error (1.081 against 1.165 and 1.211). It is the most promising untested local result, but it was selected on training data and its normalised MSE was still zero.
Alternative models
Twelve alternative routes completed a matched comparison against my recipes; GEARS was only partly run, and PIE was tested separately (Chapter 4). They are not twelve equally fine-tuned foundation models: they include native perturbation models, effect regressors and supervised readouts of frozen pretrained representations.
CPA
Compositional perturbation autoencoder: separates basal cell state from the perturbation effect and recombines them. No MSE improvement over v11.
scGen
Models a perturbation as a shift in a learned latent space. No improvement.
GEARS
Uses a gene-interaction graph. Some fits completed, but the queue was stopped for cost and no complete comparison row exists.
STATE
Arc's model trained on sets of observed cells. Trained here on seven selected strata, with worse MSE than v11. A full-data run over all 95 groups (338,676 cells) started on 24 October; three group fits completed before it was stopped, so no full comparison exists.
PRESAGE
Predicts responses from aggregated gene-knowledge embeddings. Run with a reduced knowledge setup. No gain.
scLAMBDA
Latent-space model for single and multi-gene perturbations. No gain.
Ridge
A linear baseline on GenePT and source-PCA features, to test whether model complexity adds anything. No gain.
STRING & GenePT readers
Linear and MLP readers that predict a gene's response from similar genes, using protein-interaction or text-derived gene embeddings.
Geneformer & scGPT
Pretrained expression models used frozen, with a supervised readout on top. Not fine-tuned for perturbation prediction.
PIE
Arc's perturbation-effect model, the most extensively tested: trained from scratch on public data plus my banks, and run from Arc's published checkpoint. Neither beat v11. Details in Chapter 4.
The chart shows the difference in native MSE between each route and v11 on two historical 24-target panels (H1 and CD4), with 95% confidence intervals from 2,000 paired bootstrap resamples of target genes. Negative values favour the candidate; an interval that crosses zero does not establish a clear difference.
No route was clearly better than v11 on either panel. On H1, every alternative except scGen had an interval entirely above zero, i.e. clearly higher error. On CD4 the panel is less decisive: the intervals for CPA, scGen, STATE, GenePT MLP, GenePT linear, STRING MLP, scGPT and Geneformer all cross zero, so they cannot be separated from v11 there, while Ridge, PRESAGE, scLAMBDA and STRING linear are clearly worse. v12 and v13 have slightly but clearly higher error than v11 on H1 and are indistinguishable from it on CD4, consistent with the discrimination-for-accuracy trade-off described in Chapter 5.
These intervals describe target-sampling uncertainty for the fitted models. They do not include training-seed variability, do not correct for comparing many models, and are not an independent validation.
This is also a result about my adapters, training scope and panels, not evidence that these methods are inferior in general. Several routes produced finite predictions for only a small share of the target-by-gene matrix (about 16.5% for CPA and 12.5% for STATE on H1). Four were trained on seven selected strata, about 71,000 cells out of roughly 339,000 available. A frozen foundation model with a supervised readout is also not equivalent to fine-tuning it for perturbation prediction. A fair comparison would need matched coverage, training scope and compute.
Local evaluation versus the leaderboard
The most persistent difficulty was that local evaluation did not reliably predict the official ranking.
local estimateofficial result
A local evaluator can reproduce the reference arithmetic exactly and still rank candidates differently from the hidden panel. The choice of targets, source exclusions, control splits and starting carrier all affect the result. Examples from this project:
- The official order was v11 > v9 > v10. On the 300-target VIPerturb panel, v10 was also worst, but v11 did not beat v9 (native MSE 1.114 against 1.043). The 24-target CD4 panel did not reproduce the official order either. The orderings were reported as found, not forced to agree.
- Stage-by-stage diagnostics showed that the final redistribution step alone could worsen corrected MSE: for v10 on VIPerturb, native MSE rose from 1.083 to 1.199 at that step with identical gene totals.
- A suspected factor-of-100 scaling error in the CD4 data turned out not to exist. The relevant distinction was cell-wise normalisation versus normalising summed counts.
- Fourteen official Ultima-platform targets always had an undefined normalised MSE and were analysed separately. Adding 544 other targets did not validate them.
Engineering and AI assistants
I used Codex and Claude extensively to find datasets, write extractors and adapters, propose ablations, coordinate audits and job queues, and prepare Colab notebooks. I also wrote self-contained prompts for ChatGPT Pro. Their suggestions were treated as hypotheses to test, not as evidence. I set the priorities and authorised every submission.
A large share of the time went into engineering: incompatible package versions, Google Drive cache locks, lost state after Colab reconnects, repeated hashing of large files, slow graph preparation for GEARS and fits that did not finish. The fixes were isolated environments, checkpoints, content-based cache keys, per-phase logs, timing counters and reuse only of artefacts marked complete. A passing smoke test was never counted as a scientific result.
Data, models and links
The main public resources used in this project. Only links that could be confirmed are given; a few smaller sources in my banks (VIPerturb, PerturbME, Tian, DLD1 and some CD4 sets) are not linked.
The challenge
- Virtual Cell Challenge, the competition site
- Arc Institute’s launch announcement and the Virtual Cell Initiative
- A primer on the challenge on the Hugging Face blog
Perturbation datasets
- Replogle et al. 2022, genome-scale Perturb-seq in K562 and RPE1 (Cell)
- Nadig et al. 2025, Perturb-seq in HepG2 and Jurkat (Nature Genetics)
- X-Atlas/Orion, genome-wide Perturb-seq from Xaira (preprint, data)
- Tahoe-100M, a drug-perturbation atlas used in PIE training
- Jiang et al., perturbations across cell lines and cytokine contexts (preprint)
- Frangieh et al. 2021, Perturb-CITE-seq (Nature Genetics)
- Papalexi et al. 2021, ECCITE-seq in THP-1 (Nature Genetics)
- Yao et al., THP-1 with LPS (GEO GSE221321) and the genome-wide Jurkat screen (GEO GSE249595)
- The eleven GEO banks: GSE103851, GSE115784, GSE163819, GSE210093, GSE30815, GSE44601, GSE58403, GSE68297, GSE79255, GSE87428, GSE89940
- L1000 / LINCS banks for nine cell lines, and HipSci iPSC lines
Models compared
Knowledge and reference resources
Glossary
- Control (NTC)
- Cells carrying a non-targeting guide: they go through the experiment without any gene being knocked down and define the baseline.
- Source and receiver
- A source is an experiment with measured responses. The receiver is the context being predicted.
- Perturbation
- Here, CRISPRi knockdown of a single target gene, and the resulting transcriptome-wide change.
- Pseudobulk
- The aggregate of many cells treated as one sample. Useful for comparing means, but it hides cell-to-cell variation.
- Carrier and emitter
- The starting count matrix, and the procedure that modifies it into the final simulated cells.
- Control PCs
- Principal components of variation among untreated cells. Removing them can reduce shared noise but can also remove real signal.
- Lambda / amplitude
- The scale applied to the predicted change. Larger values separate targets more but can worsen expression accuracy.
- CP50K
- Counts rescaled to 50,000 per cell, usually followed by log1p. Mixing normalisation conventions silently is a common source of error.
- Proxy
- A local stand-in for the hidden official evaluation. It is useful only to the extent that its conclusions transfer.
- Native vs normalised MSE
- Native MSE is the raw error computed locally from emitted cells (lower is better). The leaderboard's MSE score is normalised against a reference and clipped at zero (higher is better).
- Paired bootstrap interval
- Targets are resampled with replacement 2,000 times and the difference between two models is recomputed on each resample. The central 95% of those differences gives the interval.
- Leakage
- Using evaluation answers, or anything derived from them, to build a prediction. It inflates results and invalidates them.