PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion

by   Yu Fu, et al.

The Transformer architecture has achieved rapiddevelopment in recent years, outperforming the CNN archi-tectures in many computer vision tasks, such as the VisionTransformers (ViT) for image classification. However, existingvisual transformer models aim to extract semantic informationfor high-level tasks such as classification and detection, distortingthe spatial resolution of the input image, thus sacrificing thecapacity in reconstructing the input or generating high-resolutionimages. In this paper, therefore, we propose a Patch PyramidTransformer(PPT) to effectively address the above issues. Specif-ically, we first design a Patch Transformer to transform theimage into a sequence of patches, where transformer encodingis performed for each patch to extract local representations.In addition, we construct a Pyramid Transformer to effectivelyextract the non-local information from the entire image. Afterobtaining a set of multi-scale, multi-dimensional, and multi-anglefeatures of the original image, we design the image reconstructionnetwork to ensure that the features can be reconstructed intothe original input. To validate the effectiveness, we apply theproposed Patch Pyramid Transformer to the image fusion task.The experimental results demonstrate its superior performanceagainst the state-of-the-art fusion approaches, achieving the bestresults on several evaluation indicators. The underlying capacityof the PPT network is reflected by its universal power in featureextraction and image reconstruction, which can be directlyapplied to different image fusion tasks without redesigning orretraining the network.


page 1

page 2

page 3

page 4

page 5

page 6

page 7


Multi-scale Efficient Graph-Transformer for Whole Slide Image Classification

The multi-scale information among the whole slide images (WSIs) is essen...

Patch Is Not All You Need

Vision Transformers have achieved great success in computer visions, del...

Breaking Free from Fusion Rule: A Fully Semantic-driven Infrared and Visible Image Fusion

Infrared and visible image fusion plays a vital role in the field of com...

Localizing Semantic Patches for Accelerating Image Classification

Existing works often focus on reducing the architecture redundancy for a...

Pyramid Transformer for Traffic Sign Detection

Traffic sign detection is a vital task in the visual system of self-driv...

Localizing Multi-scale Semantic Patches for Image Classification

Deep convolutional neural networks (CNN) always non-linearly aggregate t...

Research on Patch Attentive Neural Process

Attentive Neural Process (ANP) improves the fitting ability of Neural Pr...

Please sign up or login with your details

Forgot password? Click here to reset