Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

12/18/2019
by   Richard J. Chen, et al.
18

Cancer diagnosis, prognosis and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complimentary information in an intuitive manner. In this work, we propose Pathomic Fusion, a strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, mRNAseq) features for survival outcome prediction and patient stratification, where histologic features are also learned via graph convolutional networks (GCNs). Our approach models pairwise feature interactions across modalities by taking the Kronecker product of gated feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. The proposed framework is able to model pairwise interactions across features in different modalities and control their relative importance. We validate our approach using glioma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. Based on a rigorous 15-fold cross validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from grading and molecular subtyping as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment.

READ FULL TEXT

page 1

page 2

page 4

page 5

research
08/18/2023

Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction

Cancer prognosis and survival outcome predictions are crucial for therap...
research
01/06/2023

Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction

The integration of multi-modal data, such as pathological images and gen...
research
08/04/2021

Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning

The rapidly emerging field of deep learning-based computational patholog...
research
04/01/2023

Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision

Prostate cancer is one of the leading causes of cancer-related death in ...
research
03/09/2021

Multimodal fusion using sparse CCA for breast cancer survival prediction

Effective understanding of a disease such as cancer requires fusing mult...
research
04/13/2023

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

Integrating whole-slide images (WSIs) and bulk transcriptomics for predi...

Please sign up or login with your details

Forgot password? Click here to reset