🧬🔮 Single cell foundation models have been a recent hot topic in bio-ML! A few of the recent methods and some thoughts 🧬🔮. 1) Geneformer.2) scGPT.3) scFoundation.4) Exceiver.

RT @lizbwood: The biggest challenge for AI in biology isn't just models, it's the data used to train them. Standard biological data isn't b….

I highlight a recent paper from @BoWang87 and colleagues at @ShiftBioscience that mark new progress towards the larger goal.

Second, the field has been rallying to describe metrics that provide high signal and prediction "utility". Good measurements, confidence metrics, and directly actionable outcomes are crucial for utility. 3/n.

First of all, it's important to delineate between cis- and trans- gene regulatory modeling. 2/n.

What is the state of research on the emerging grand challenge of virtual cell modeling? 1/n.

I highlight a recent paper from @BoWang87 and colleagues at @ShiftBioscience that mark new progress towards the larger goal.

Second, the field has been rallying to describe metrics that provide high signal and prediction "utility". Good measurements, confidence metrics, and directly actionable outcomes are crucial for utility.

First of all, it's important to delineate between cis- and trans- gene regulatory modeling. 2/n.

Further, some simple changes may greatly improve the many available methods, particularly when compared against these new well-calibrated baselines. There is much more progress to follow from this. Great job @BoWang87 and the @shiftbioscience team!.

The field has been circling the paradox of poor method performance wrt to simple baselines. My (optimistic) takeaway is that the methods aren’t necessarily bad, but our measuring sticks have been. We’ve understood this – but the study does a great job rigorously documenting why. .

Unfortunately, many studies have favored a narrative over actionable conclusions — for example, notably omitting R², a classical regression metric that captures variance and scale relative to baseline. There is much more nuance explored in the study. .

To my relief, they highlight a blatant failure of Pearson correlation for evaluating simulation of trans-gene expression. PC ignores magnitude of deviation from baseline, yet it’s often reported in place of more informative metrics. .

1) new metrics that appropriately consider biased baselines. 2) a loss function that calibrates the model to learn the most important, sparse signals. .

👏👏👏. High-impact results here, a thorough study on the pitfalls of current “trans-gene expression” virtual cell modeling metrics and proposals for. .

It’s early days, but this is the kind of effort that helps structure the space. Looking forward to seeing how the field responds—and how the benchmark evolves, which is a laudable goal in itself. n/n.

You could also blend these approaches. These aren’t fancy models—but they may be strong baselines. For the comp they may help tease apart what’s truly generalized vs cleverly matched. 8/n.

2️⃣ Submit perturbation profiles from a cell line whose perturbed states match the X% of H1 perturbations that are revealed (if this exists). Align in output space, then map across. 7/n.

1️⃣ Directly submit perturbation profiles from a public dataset where the basal (unperturbed) expression is most similar to H1 (if this exists). Basal transcriptional proximity might make the transfer surprisingly effective. 6/n.

Based on past comps, I’d expect strong submissions to blend ML techniques with smart data reuse. A couple baseline “hacks” come to mind: 5/n.

It’s not zero-shot: participants get expression profiles for X% of perturbations in H1 (in Arc's State paper it was 30%). So it’s really a few-shot adaptation task, not a full generalization eval. 4/n