ENLIGHTEN3D

By Christian Laforte

Look directly at the camera, or the computer can’t see you. Or at least, it can’t recognize that it is you. The major drawback of all facial recognition systems in commercial use today is that they require people to face the camera directly, like a passport photo. A newly published paper takes aim at this problem, helping computers better recognize a face in more natural conditions.

Through Tied Factor Analysis, computers can recognize a face seen from the side by comparing the side image with the passport picture, achieving an accuracy of 92%, a major technological leap when compared with 60% in the previous state-of-the-art technique. The algorithm learns how to extrapolate other views by analyzing thousands of pictures of varied people, taken from many angles. The approach is relatively simple to implement and reportedly much faster than other state-of-the-art techniques.

Side photos (bottom) automatically generated from “Passport” photos (top)

This new model assumes very little about the structure of a face, geometry or lighting, so it could easily be adapted to applications such as recognizing vehicles or animals in a semi-controlled environment.

Technical details

The tied factor analysis (TFA) technique uses machine learning techniques such as Expectation Maximization to automatically learn the relationship between frontal and nonfrontal faces, e.g. pictures taken from the side or at an angle.

Along the way, the system automatically learns an identity space to represent a face in a few hundred parameters. This identity space doesn’t vary significantly with pose, angle or lighting, so in theory, all images of an individual would map to the identity position in that space.

To perform this feat, the researchers started from a large number of pictures, like the 320 individuals from the FERET database taken with multiple poses and angles. These faces were manually altered and annotated to increase accuracy. After running TFA on these pictures, the system can extrapolate from a known facing face (e.g. passport photo) to an unknown non-facing face (e.g. side photo), as shown in the image shown above.

The authors then significantly increased the accuracy by combining 21 TFAs applied around manually-specified positions, identifying standard facial features like the left eye corner in the following figure:

Prince and his colleagues also integrated a relatively simple face part detector inspired from Viola and Jones. This made the system more automatic but reduced the precision in the worst case by 6%. Still, they are confident that this gap could be filled with a more sophisticated detector.

Conclusion

I find these results exciting and promising, but we are still a long way from human-like recognition. Here are the most significant limitations with this approach, along with potential solutions:

Discretized poses: this paper shows how to support many poses, but it doesn’t address how to support arbitrary poses efficiently and accurately, like seeing a face slightly from above or from the bottom. This severely limits the use in non-intrusive monitoring. Building a full 3D model may be better suited to this.

Resistance to occlusions: The paper assumes that the user is cooperative and won’t get partially occluded, e.g. put his hand in front of his face. There are possible solutions that we’ll explore in future posts.

Automatic head pose detection: To make the system completely automatic and unobtrusive, the faces would need to be detected and registered (identified and located) automatically.

Tied factor analysis for face recognition across large pose differences
Simon J.D. Prince, James H. Elder, Jonathan Warrell, Fatima M. Felisberti

IEEE Transactions on pattern analysis and machine intelligence, Vol. 30, No. 6, June 2008
(http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4459336)
You need an IEEE Explore membership to access the paper.

An older, less complete version of the paper is publicly available: http://www.macs.hw.ac.uk/bmvc2006/papers/292.pdf