Visual-Assisted Sound Source Depth Estimation in the Wild

by   Wei Sun, et al.

Depth estimation enables a wide variety of 3D applications, such as robotics, autonomous driving, and virtual reality. Despite significant work in this area, it remains open how to enable accurate, low-cost, high-resolution, and large-range depth estimation. Inspired by the flash-to-bang phenomenon (i.e. hearing the thunder after seeing the lightning), this paper develops FBDepth, the first audio-visual depth estimation framework. It takes the difference between the time-of-flight (ToF) of the light and the sound to infer the sound source depth. FBDepth is the first to incorporate video and audio with both semantic features and spatial hints for range estimation. It first aligns correspondence between the video track and audio track to locate the target object and target sound in a coarse granularity. Based on the observation of moving objects' trajectories, FBDepth proposes to estimate the intersection of optical flow before and after the sound production to locate video events in time. FBDepth feeds the estimated timestamp of the video event and the audio clip for the final depth estimation. We use a mobile phone to collect 3000+ video clips with 20 different objects at up to 50m. FBDepth decreases the Absolute Relative error (AbsRel) by 55% compared to RGB-based methods.


page 1

page 5

page 6

page 8

page 10


Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction

Reflective and textureless surfaces such as windows, mirrors, and walls ...

Evaluation of short range depth sonifications for visual-to-auditory sensory substitution

Visual to auditory sensory substitution devices convert visual informati...

Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization

Learning to localize the sound source in videos without explicit annotat...

EVEN: An Event-Based Framework for Monocular Depth Estimation at Adverse Night Conditions

Accurate depth estimation under adverse night conditions has practical i...

Circular Rectifiction of 3D Video and Efficient Modification of 3D-HEVC

Video acquired from multiple cameras located along a line is often recti...

Self-Supervised Generation of Spatial Audio for 360 Video

We introduce an approach to convert mono audio recorded by a 360 video c...

Please sign up or login with your details

Forgot password? Click here to reset