A New Image Codec Paradigm for Human and Machine Uses

by   Sien Chen, et al.

With the AI of Things (AIoT) development, a huge amount of visual data, e.g., images and videos, are produced in our daily work and life. These visual data are not only used for human viewing or understanding but also for machine analysis or decision-making, e.g., intelligent surveillance, automated vehicles, and many other smart city applications. To this end, a new image codec paradigm for both human and machine uses is proposed in this work. Firstly, the high-level instance segmentation map and the low-level signal features are extracted with neural networks. Then, the instance segmentation map is further represented as a profile with the proposed 16-bit gray-scale representation. After that, both 16-bit gray-scale profile and signal features are encoded with a lossless codec. Meanwhile, an image predictor is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features. Finally, the residual map between the original image and the predicted one is compressed with a lossy codec, used for high-quality image reconstruction. With such designs, on the one hand, we can achieve scalable image compression to meet the requirements of different human consumption; on the other hand, we can directly achieve several machine vision tasks at the decoder side with the decoded 16-bit gray-scale profile, e.g., object classification, detection, and segmentation. Experimental results show that the proposed codec achieves comparable results as most learning-based codecs and outperforms the traditional codecs (e.g., BPG and JPEG2000) in terms of PSNR and MS-SSIM for image reconstruction. At the same time, it outperforms the existing codecs in terms of the mAP for object detection and segmentation.


page 1

page 3

page 5

page 8


DSSLIC: Deep Semantic Segmentation-based Layered Image Compression

Deep learning has revolutionized many computer vision fields in the last...

Learned Image Compression for Machine Perception

Recent work has shown that learned image compression strategies can outp...

Rethinking Implicit Neural Representations for Vision Learners

Implicit Neural Representations (INRs) are powerful to parameterize cont...

Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

Video Coding for Machines (VCM) is committed to bridging to an extent se...

Revisit Visual Representation in Analytics Taxonomy: A Compression Perspective

Visual analytics have played an increasingly critical role in the Intern...

Measuring uncertainty in human visual segmentation

Segmenting visual stimuli into distinct groups of features and visual ob...

Neural Photofit: Gaze-based Mental Image Reconstruction

We propose a novel method that leverages human fixations to visually dec...

Please sign up or login with your details

Forgot password? Click here to reset