Don’t trust any imaging-based spatial transcriptomics results that: – hide false positive rates – skip rigorous cell segmentation QC – rely solely on label transfer for cell type annotation #SpatialTranscriptomics #SpatialBiology #SingleCell

We are seeking a computational biologist to join our assay development team at Resolve Biosciences. If you are interested, please apply: https://t.co/Xm0RYk39py

I understand the need in the community to benchmark technologies, but the current results are highly limited in their significance. Overall, there is much more to do on the benchmarking side, but I am looking forward to the things to come with these great technologies.

Detailed evaluation of 1) FPs and 2) segmentation should be separated IMO. The former would require relaxing of parameters for spot detection/transcript decoding which is also difficult as it requires raw data and a unified image processing pipeline. But again, that’s how it is.

2) To mitigate the effect of different segmentation performances, the authors re-segmented all datasets using Baysor. However, Baysor depends on detected transcripts and is therefore panel/multiplexing dependent. Thus, it should not be used as “objective” segmentation tool here

MECR is not useful for this IMO, because the influence of segmentation on this value is much more pronounced. Still, MECR is a great way to systematically evaluate segmentation performance.

1) Analysis on FP counts is under-represented in this paper. I would love to see much more numbers and analysis on this topic, because it gives a very good idea how specific the oligo-chemistry of a particular technology is.

However, this kind of error can come from two sources. 1) false positive (FP) de-coding of the transcript and 2) segmentation errors.

The usage of the term specificity is confusing in this pre-print. Their final measure for specificity is the MECR value, which (in simple words) is a measure for the detection of a marker gene in the wrong cell type.

I am looking forward to a study that compares different technologies using the same panels at increasing levels of multiplexing (with consecutive sections).

Therefore, mean counts per cell are not the right measure to assess sensitivity when different panels are used. The ratio of negative control probe counts to gene probe counts is not useful for the same reasons.

This alone already severely confounds any precise technical evaluation of the technologies. If you want to benchmark iST technologies, use the same gene panel! I know that’s hard to achieve and expensive, but that’s how it is.

When different (numbers of) genes with different expression levels are profiled, these cause different levels of optical crowding which affect sensitivity and specificity in different ways. (More explanation on the influence of optical crowding here: https://t.co/HJPXZYhnS2)

BTW, this also the case for the other two recent benchmark papers or iST methods on FFPE sections: https://t.co/pFjkUGoiNM https://t.co/QDPd6e1WK8

The most critical limitation of this study (that also affects all other discussed aspects below) are the different gene panels used for the different technologies with very different levels of multiplexing.

However, I disagree with the presented measures for sensitivity/specificity, and with the statements that are made regarding the performance of the technologies:

It´s great to see that labs start to technically evaluate different aspects of such technologies/platforms, and the pre-print raises important points, including the influence of segmentation errors on downstream analysis.

A few thoughts on the recent comparison paper for imaging-based #spatialtranscriptomics (iST) methods from the @satijalab: https://t.co/cuoOqLrqFT

Spatial transcriptomics trade offs that sometimes are not realized. Here are some examples, simplified for twitter

Although I seem a bit negative here, I truly believe that iST methods will have a tremendous impact on bio-medical research. But there is still a long way to go to. I am very much open for discussion, and I also like to learn from your experience. Thanks for reading this thread!