Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data

by   Huy Hoang Nguyen, et al.

Deep neural networks are often applied to medical images to automate the problem of medical diagnosis. However, a more clinically relevant question that practitioners usually face is how to predict the future trajectory of a disease. Current methods for prognosis or disease trajectory forecasting often require domain knowledge and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many prediction problem. Inspired by a clinical decision-making process with two agents – a radiologist and a general practitioner – we predict prognosis with two transformer-based components that share information with each other. The first transformer in this framework aims to analyze the imaging data, and the second one leverages its internal states as inputs, also fusing them with auxiliary clinical data. The temporal nature of the problem is modeled within the transformer states, allowing us to treat the forecasting problem as a multi-task classification, for which we propose a novel loss. We show the effectiveness of our approach in predicting the development of structural knee osteoarthritis changes and forecasting Alzheimer's disease clinical status directly from raw multi-modal data. The proposed method outperforms multiple state-of-the-art baselines with respect to performance and calibration, both of which are needed for real-world applications. An open-source implementation of our method is made publicly available at <>.


page 1

page 2

page 8

page 9

page 12

page 14

page 15


DeepProg: A Transformer-based Framework for Predicting Disease Prognosis

A vast majority of deep learning methods are built to automate diagnosti...

TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

The prediction of mild cognitive impairment (MCI) conversion to Alzheime...

Predicting Density of States via Multi-modal Transformer

The density of states (DOS) is a spectral property of materials, which p...

Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting

Hand trajectory forecasting from egocentric views is crucial for enablin...

NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecasting

Financial forecasting has been an important and active area of machine l...

Cross-Modal Health State Estimation

Individuals create and consume more diverse data about themselves today ...

Please sign up or login with your details

Forgot password? Click here to reset