QuantISH: RNA in situ hybridization image analysis framework for quantifying cell type-specific target RNA expression and variability

RNA in situ hybridization (RNA-ISH) is a powerful spatial transcriptomics technology to characterize target RNA abundance and localization in individual cells. This allows analysis of tumor heterogeneity and expression localization, which are not readily obtainable through transcriptomic data analysis. RNA-ISH experiments produce large amounts of data and there is a need for automated analysis methods. Here we present QuantISH, a comprehensive open-source RNA-ISH image analysis pipeline that quantifies marker expressions in individual carcinoma, immune, and stromal cells on chromogenic or fluorescent in situ hybridization images. QuantISH is designed to be modular and can be adapted to various image and sample types and staining protocols. We show that in chromogenic RNA in situ hybridization images of high-grade serous carcinoma (HGSC) QuantISH cancer cell classification has high precision, and signal expression quantification is in line with visual assessment. We further demonstrate the power of QuantISH by showing that CCNE1 average expression and DDIT3 expression variability, as captured by the variability factor developed herein, act as candidate biomarkers in HGSC. Altogether, our results demonstrate that QuantISH can quantify RNA expression levels and their variability in carcinoma cells, and thus paves the way to utilize RNA-ISH technology.