Multi-encoder attention-based architectures for sound recognition with partial visual assistance

by   Wim Boes, et al.

Large-scale sound recognition data sets typically consist of acoustic recordings obtained from multimedia libraries. As a consequence, modalities other than audio can often be exploited to improve the outputs of models designed for associated tasks. Frequently, however, not all contents are available for all samples of such a collection: For example, the original material may have been removed from the source platform at some point, and therefore, non-auditory features can no longer be acquired. We demonstrate that a multi-encoder framework can be employed to deal with this issue by applying this method to attention-based deep learning systems, which are currently part of the state of the art in the domain of sound recognition. More specifically, we show that the proposed model extension can successfully be utilized to incorporate partially available visual information into the operational procedures of such networks, which normally only use auditory features during training and inference. Experimentally, we verify that the considered approach leads to improved predictions in a number of evaluation scenarios pertaining to audio tagging and sound event detection. Additionally, we scrutinize some properties and limitations of the presented technique.


page 1

page 3

page 4

page 5


Audiovisual transfer learning for audio tagging and sound event detection

We study the merit of transfer learning for two sound recognition proble...

Impact of visual assistance for automated audio captioning

We study the impact of visual assistance for automated audio captioning....

Surrey-cvssp system for DCASE2017 challenge task4

In this technique report, we present a bunch of methods for the task 4 o...

Exploiting Attention-based Sequence-to-Sequence Architectures for Sound Event Localization

Sound event localization frameworks based on deep neural networks have s...

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

Recognizing sounds is a key aspect of computational audio scene analysis...

A Squeeze-and-Excitation and Transformer based Cross-task System for Environmental Sound Recognition

Environmental sound recognition (ESR) is an emerging research topic in a...

An Open-set Recognition and Few-Shot Learning Dataset for Audio Event Classification in Domestic Environments

The problem of training a deep neural network with a small set of positi...

Please sign up or login with your details

Forgot password? Click here to reset