Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning

by   Richard J. Chen, et al.

The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq abundance, copy number variation, and mutation data from 5,720 patients across 14 major cancer types. Our interpretable, weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities for predicting outcomes and discover prognostic features from these modalities that corroborate with poor and favorable outcomes via multimodal interpretability. We compared our model with unimodal deep learning models trained on histology slides and molecular profiles alone, and demonstrate performance increase in risk stratification on 9 out of 14 cancers. In addition, we analyze morphologic and molecular markers responsible for prognostic predictions across all cancer types. All analyzed data, including morphological and molecular correlates of patient prognosis across the 14 cancer types at a disease and patient level are presented in an interactive open-access database ( to allow for further exploration and prognostic biomarker discovery. To validate that these model explanations are prognostic, we further analyzed high attention morphological regions in WSIs, which indicates that tumor-infiltrating lymphocyte presence corroborates with favorable cancer prognosis on 9 out of 14 cancer types studied.


page 8

page 10

page 24

page 25

page 26

page 28

page 29

page 41


Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H E: A Practical Review

Molecular and genomic properties are critical in selecting cancer treatm...

Comprehensive and user-analytics-friendly cancer patient database for physicians and researchers

Nuanced cancer patient care is needed, as the development and clinical c...

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Cancer diagnosis, prognosis and therapeutic response predictions are bas...

Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images

The rapidly emerging field of computational pathology has the potential ...

Regression-based Deep-Learning predicts molecular biomarkers from pathology slides

Deep Learning (DL) can predict biomarkers from cancer histopathology. Se...

Please sign up or login with your details

Forgot password? Click here to reset