Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition

by   Nan Li, et al.
University of Oulu

This paper studies the computational offloading of video action recognition in edge computing. To achieve effective semantic information extraction and compression, following semantic communication we propose a novel spatiotemporal attention-based autoencoder (STAE) architecture, including a frame attention module and a spatial attention module, to evaluate the importance of frames and pixels in each frame. Additionally, we use entropy encoding to remove statistical redundancy in the compressed data to further reduce communication overhead. At the receiver, we develop a lightweight decoder that leverages a 3D-2D CNN combined architecture to reconstruct missing information by simultaneously learning temporal and spatial information from the received data to improve accuracy. To fasten convergence, we use a step-by-step approach to train the resulting STAE-based vision transformer (ViT_STAE) models. Experimental results show that ViT_STAE can compress the video dataset HMDB51 by 104x with only 5 DeepISC. The proposed ViT_STAE achieves faster inference and higher accuracy than the DeepISC-based ViT model under time-varying wireless channel, which highlights the effectiveness of STAE in guaranteeing higher accuracy under time constraints.


Attention-based Feature Compression for CNN Inference Offloading in Edge Computing

This paper studies the computational offloading of CNN inference in devi...

Local Memory Attention for Fast Video Semantic Segmentation

We propose a novel neural network module that transforms an existing sin...

End-to-end Transformer for Compressed Video Quality Enhancement

Convolutional neural networks have achieved excellent results in compres...

Streaming Video Model

Video understanding tasks have traditionally been modeled by two separat...

Blockwise Temporal-Spatial Pathway Network

Algorithms for video action recognition should consider not only spatial...

DNN-Driven Compressive Offloading for Edge-Assisted Semantic Video Segmentation

Deep learning has shown impressive performance in semantic segmentation,...

Semantic Communication Enabling Robust Edge Intelligence for Time-Critical IoT Applications

This paper aims to design robust Edge Intelligence using semantic commun...

Please sign up or login with your details

Forgot password? Click here to reset