Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation

by   Fenglin Liu, et al.

Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approaches mainly adopt a supervised manner and heavily rely on coupled image-report pairs. However, in the medical domain, building a large-scale image-report paired dataset is both time-consuming and expensive. To relax the dependency on paired data, we propose an unsupervised model Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images and reports in training. KGAE consists of a pre-constructed knowledge graph, a knowledge-driven encoder and a knowledge-driven decoder. The knowledge graph works as the shared latent space to bridge the visual and textual domains; The knowledge-driven encoder projects medical images and reports to the corresponding coordinates in this latent space and the knowledge-driven decoder generates a medical report given a coordinate in this space. Since the knowledge-driven encoder and decoder can be trained with independent sets of images and reports, KGAE is unsupervised. The experiments show that the unsupervised KGAE generates desirable medical reports without using any image-report training pairs. Moreover, KGAE can also work in both semi-supervised and supervised settings, and accept paired images and reports in training. By further fine-tuning with image-report pairs, KGAE consistently outperforms the current state-of-the-art models on two datasets.


page 1

page 2

page 3

page 4


Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation

Automatic generation of medical reports from X-ray images can assist rad...

Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

Automatic radiology report generation is critical in clinics which can r...

Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation

Beyond the common difficulties faced in the natural image captioning, me...

Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation

Automatic radiology reporting has great clinical potential to relieve ra...

Competence-based Multimodal Curriculum Learning for Medical Report Generation

Medical report generation task, which targets to produce long and cohere...

KabOOM: Unsupervised Crash Categorization through Timeseries Fingerprinting

Modern mobile applications include instrumentation that sample internal ...

Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation

Medical report generation is a challenging task since it is time-consumi...

Please sign up or login with your details

Forgot password? Click here to reset