SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images

by   Zhishe Wang, et al.
Jiangnan University
NetEase, Inc

The existing deep learning fusion methods mainly concentrate on the convolutional neural networks, and few attempts are made with transformer. Meanwhile, the convolutional operation is a content-independent interaction between the image and convolution kernel, which may lose some important contexts and further limit fusion performance. Towards this end, we present a simple and strong fusion baseline for infrared and visible images, namely Residual Swin Transformer Fusion Network, termed as SwinFuse. Our SwinFuse includes three parts: the global feature extraction, fusion layer and feature reconstruction. In particular, we build a fully attentional feature encoding backbone to model the long-range dependency, which is a pure transformer network and has a stronger representation ability compared with the convolutional neural networks. Moreover, we design a novel feature fusion strategy based on L_1-norm for sequence matrices, and measure the corresponding activity levels from row and column vector dimensions, which can well retain competitive infrared brightness and distinct visible details. Finally, we testify our SwinFuse with nine state-of-the-art traditional and deep learning methods on three different datasets through subjective observations and objective comparisons, and the experimental results manifest that the proposed SwinFuse obtains surprising fusion performance with strong generalization ability and competitive computational efficiency. The code will be available at https://github.com/Zhishe-Wang/SwinFuse.


page 2

page 4

page 6

page 7

page 8

page 9

page 10

page 12


DenseFuse: A Fusion Approach to Infrared and Visible Images

In this paper, we present a novel deep learning architecture for infrare...

RFN-Nest: An end-to-end residual fusion network for infrared and visible images

In the image fusion field, the design of deep learning-based fusion meth...

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

The existing generative adversarial fusion methods generally concatenate...

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

The end-to-end image fusion framework has achieved promising performance...

Infrared and Visible Image Fusion with ResNet and zero-phase component analysis

In image fusion task, feature extraction and processing are keys for fus...

ABC: Attention with Bilinear Correlation for Infrared Small Target Detection

Infrared small target detection (ISTD) has a wide range of applications ...

WITT: A Wireless Image Transmission Transformer for Semantic Communications

In this paper, we aim to redesign the vision Transformer (ViT) as a new ...

Please sign up or login with your details

Forgot password? Click here to reset