ViSE: Vision-Based 3D Real-Time Shape Estimation of Continuously Deformable Robots

by   Hehui Zheng, et al.

The precise control of soft and continuum robots requires knowledge of their shape. The shape of these robots has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built-in sensors resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on placing reflective markers on all tracked components and triangulating their position using multiple motion-tracking cameras. Tracking systems are expensive and infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, we present a regression approach for 3D shape estimation using a convolutional neural network. The proposed approach takes advantage of data-driven supervised learning and is capable of real-time marker-less shape estimation during inference. Two images of a robotic system are taken simultaneously at 25 Hz from two different perspectives, and are fed to the network, which returns for each pair the parameterized shape. The proposed approach outperforms marker-less state-of-the-art methods by a maximum of 4.4% in estimation accuracy while at the same time being more robust and requiring no prior knowledge of the shape. The approach can be easily implemented due to only requiring two color cameras without depth and not needing an explicit calibration of the extrinsic parameters. Evaluations on two types of soft robotic arms and a soft robotic fish demonstrate our method's accuracy and versatility on highly deformable systems in real-time. The robust performance of the approach against different scene modifications (camera alignment and brightness) suggests its generalizability to a wider range of experimental setups, which will benefit downstream tasks such as robotic grasping and manipulation.


page 1

page 3

page 5

page 6


Robust Shape Estimation for 3D Deformable Object Manipulation

Existing shape estimation methods for deformable object manipulation suf...

Sensing and Reconstruction of 3D Deformation on Pneumatic Soft Robots

Real-time proprioception is a challenging problem for soft robots, which...

Real-time Soft Robot 3D Proprioception via Deep Vision-based Sensing

The soft robots are welcomed in many robotic applications because of the...

Vision-Based Shape Reconstruction of Soft Continuum Arms Using a Geometric Strain Parametrization

Interest in soft continuum arms has increased as their inherent material...

On Model Adaptation for Sensorimotor Control of Robots

In this article, we address the problem of computing adaptive sensorimot...

Please sign up or login with your details

Forgot password? Click here to reset