C-Face: Continuously Reconstructing Facial Expressions by Deep Learning Contours of the Face with Ear-mounted Miniature Cameras
Published on the Association for Computing Machinery Symposium on User Software and Technology (UIST) ’20
Tuochao Chen, Benjamin Steeper, Kinan Alsheikh, Songyun Tao, Francois Guimbretiere, Cheng Zhang
Selected Media Coverage: Cornell Chronicle , Engadget , New Scientist, NowThis (Twitter, Youtube, Facebook) Yahoo , Mashable, New Atlas, South China Morning Post.
C-Face was demoed using an earphone, a headphone and when the user is wearing a face mask
C-Face (Contour-Face) is an ear-mounted wearable sensing technology that uses two miniature cameras to continuously reconstruct facial expressions by deep learning contours of the face. When facial muscles move, the contours of the face change from the point of view of the ear-mounted cameras. These subtle changes are fed into a deep learning model which continuously outputs 42 facial feature points representing the shapes and positions of the mouth, eyes, and eyebrows. To evaluate C-Face, we embedded our technology into headphones and earphones. We conducted a user study with nine participants. In this study, we compared the output of our system to the feature points outputted by a state of the art computer vision library (Dlib) from a front-facing camera.}We found that the mean error of all 42 feature points was 0.77 mm for earphones and 0.74 mm for headphones. The mean error for 20 major feature points capturing the most active areas of the face was 1.43 mm for earphones and 1.39 mm for headphones. The ability to continuously reconstruct facial expressions introduces new opportunities in a variety of applications. As a demonstration, we implemented and evaluated C-Face for two applications: facial expression detection (outputting emojis) and silent speech recognition. We further discuss the opportunities and challenges of deploying C-Face in real-world applications.
An earphone and headphone which are embedded with C-Face
Demo Video of C-Face