Real-Time Shape Tracking of Facial Landmarks

by   Hyungjoon Kim, et al.

Detection of facial landmarks and accurate tracking of their shape are essential in real-time virtual makeup applications, where users can see the makeups effect by moving their face in different directions. Typical face tracking techniques detect diverse facial landmarks and track them using a point tracker such as the Kanade-Lucas-Tomasi (KLT) point tracker. Typically, 5 or 64 points are used for tracking a face. Even though these points are sufficient to track the approximate locations of facial landmarks, they are not sufficient to track the exact shape of facial landmarks. In this paper, we propose a method that can track the exact shape of facial landmarks in real-time by combining a deep learning technique and a point tracker. We detect facial landmarks accurately using SegNet, which performs semantic segmentation based on deep learning. Edge points of detected landmarks are tracked using the KLT point tracker. In spite of its popularity, the KLT point tracker suffers from the point loss problem. We solve this problem by executing SegNet periodically to calculate the shape of facial landmarks. That is, by combining the two techniques, we can avoid the computational overhead of SegNet for real-time shape tracking and the point loss problem of the KLT point tracker. We performed several experiments to evaluate the performance of our method and report some of the results herein.


page 4

page 5

page 6


Facial Action Unit Detection using 3D Facial Landmarks

In this paper, we propose to detect facial action units (AU) using 3D fa...

Deep Multi-Center Learning for Face Alignment

Facial landmarks are highly correlated with each other since a certain l...

Improving Real-time Score Following in Opera by Combining Music with Lyrics Tracking

Fully automatic opera tracking is challenging because of the acoustic co...

A Functional Regression approach to Facial Landmark Tracking

Linear regression is a fundamental building block in many face detection...

A weighting strategy for Active Shape Models

Active Shape Models (ASM) are an iterative segmentation technique to fin...

Robust Performance-driven 3D Face Tracking in Long Range Depth Scenes

We introduce a novel robust hybrid 3D face tracking framework from RGBD ...

SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral Cameras

Facial motion tracking in head-mounted displays (HMD) has the potential ...

Please sign up or login with your details

Forgot password? Click here to reset