Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation

by   Dean Zadok, et al.

A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (US) technologies to analyze the state of the muscles. In this work, we study the inference problem of identifying the activation of specific fingers from a sequence of US images when performing dexterous tasks such as keyboard typing or playing the piano. While estimating finger gestures has been done in the past by detecting prominent gestures, we are interested in classification done in the context of fine motions that evolve over time. We consider this task as an important step towards higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. Our key observation, motivating this work, is that modeling the hand as a robotic manipulator allows to encode an intermediate representation wherein US images are mapped to said configurations. Given a sequence of such learned configurations, coupled with a neural-network architecture that exploits temporal coherence, we are able to infer fine finger motions. We evaluated our method by collecting data from a group of subjects and demonstrating how our framework can be used to replay music played or text typed. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.


page 1

page 2

page 3

page 7

page 9


Ultrasound Based Prosthetic Arm Control

The creation of unique control methods for a hand prosthesis is still a ...

Simultaneous Estimation of Hand Configurations and Finger Joint Angles using Forearm Ultrasound

With the advancement in computing and robotics, it is necessary to devel...

Prediction of Metacarpophalangeal joint angles and Classification of Hand configurations based on Ultrasound Imaging of the Forearm

With the advancement in computing and robotics, it is necessary to devel...

An Overview of Robotic Grippers

The development of robotic grippers is driven by the need to execute par...

Towards Complex and Continuous Manipulation: A Gesture Based Anthropomorphic Robotic Hand Design

Most current anthropomorphic robotic hands can realize part of the human...

Leaf-like Origami with Bistability for Self-Adaptive Grasping Motions

The leaf-like origami structure was inspired by geometric patterns found...

Recognition and Localisation of Pointing Gestures using a RGB-D Camera

Non-verbal communication is part of our regular conversation, and multip...

Please sign up or login with your details

Forgot password? Click here to reset