Local Distortion Aware Efficient Transformer Adaptation for Image Quality Assessment

by   Kangmin Xu, et al.

Image Quality Assessment (IQA) constitutes a fundamental task within the field of computer vision, yet it remains an unresolved challenge, owing to the intricate distortion conditions, diverse image contents, and limited availability of data. Recently, the community has witnessed the emergence of numerous large-scale pretrained foundation models, which greatly benefit from dramatically increased data and parameter capacities. However, it remains an open problem whether the scaling law in high-level tasks is also applicable to IQA task which is closely related to low-level clues. In this paper, we demonstrate that with proper injection of local distortion features, a larger pretrained and fixed foundation model performs better in IQA tasks. Specifically, for the lack of local distortion structure and inductive bias of vision transformer (ViT), alongside the large-scale pretrained ViT, we use another pretrained convolution neural network (CNN), which is well known for capturing the local structure, to extract multi-scale image features. Further, we propose a local distortion extractor to obtain local distortion features from the pretrained CNN and a local distortion injector to inject the local distortion features into ViT. By only training the extractor and injector, our method can benefit from the rich knowledge in the powerful foundation models and achieve state-of-the-art performance on popular IQA datasets, indicating that IQA is not only a low-level problem but also benefits from stronger high-level features drawn from large-scale pretrained models.


page 1

page 7

page 8


Domain-Aware No-Reference Image Quality Assessment

No-reference image quality assessment (NR-IQA) is a fundamental yet chal...

TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment

Image Quality Assessment (IQA) is a fundamental task in computer vision ...

Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild

Automatic Perceptual Image Quality Assessment is a challenging problem t...

MUSIQ: Multi-scale Image Quality Transformer

Image quality assessment (IQA) is an important research topic for unders...

Data-Efficient Image Quality Assessment with Attention-Panel Decoder

Blind Image Quality Assessment (BIQA) is a fundamental task in computer ...

Learning Transformer Features for Image Quality Assessment

Objective image quality evaluation is a challenging task, which aims to ...

Local Relation Networks for Image Recognition

The convolution layer has been the dominant feature extractor in compute...

Please sign up or login with your details

Forgot password? Click here to reset