Impact of Plastic Surgery on Face Recognition

From Antispoofing Wiki

Problem Overview

Plastic surgery is a medically induced face alteration, which allows to both enhance aesthetic nuances of one’s appearance or correct gained/innate defects, including the craniofacial anomalies: cleft lip, hemifacial microsomia, and others. According to Statista, 4,667,931 plastic surgeries were performed in the US alone in 2019. As stated by Plasticsurgery.org, facial surgeries led the top 5 cosmetic procedures in 2020. Among them were nose reshaping, eyelid surgery and rhytidectomy (facelift).



While cosmetic and reconstructive plastic surgeries are mostly benign procedures, they can impede performance of facial recognition systems or man-operated security checkpoints. This was proved by a Hongqiao airport incident, in which a group of women was not identified using their passport due to the plastic surgeries they had undergone.

Plastic surgery can also be used for malevolent purposes. A widespread practice is to permanently alter appearance to avoid capture and imprisonment: a Mexican drug trafficker Amado Fuentes died because of plastic surgery complications. Another possible threat is impersonation through plastic surgery — a practice sporadically used, among all else, for unethical entertainment purposes.


Types of Facial Plastic Surgery

Apart from the general cosmetic and reconstructive surgery, other types include Local and Global surgery.

Local type

Local plastic surgeries aim to correct aesthetic flaws — scars, wrinkles, birthmarks, facial asymmetry or some natural anomalies — in a specific region of the body or face. This type of procedure focuses on removing defects that are constitutional (by-birth), gained in an accident and those defects that have developed over the time.

Global type

This type of surgery mostly addresses functional damage received as a result of some critical trauma. In this case, the entire facial structure — features, contour, skin texture — is being reconstructed to normalize the patient’s appearance. However, the same procedure can be used to permanently alter a person’s look making it hard or even impossible to identify them.



Plastic Surgery Databases

Plastic surgeons involved in facial reconstruction amass a considerable library of pictures and slides captured prior and afterwards surgeries. However, this data has to be kept private due to medical confidentiality, which is practiced in most countries. Besides, patients are often reluctant to publicly share their surgery cases. Therefore, assembling a plastic surgery database for facial recognition training is a challenging task. Virtually, there are no publicly available datasets of this kind. The only known example is the standard Plastic Surgery Database (PSD), which was used in the work of Singh et al. covering a probabilistic model for a locally changed face.


Plastic Surgery Detection

An intuitive method of plastic surgery detection (PSD) proposed, is based on the direct matching of corresponding face patches around the possible plastic surgery regions. It includes a component relative threshold setting, which is capable of spotting the exact region to which a surgical alteration has been applied.



Usually, such an algorithm includes the following steps:

  • Decomposition. Subject’s face is decomposed into regions that could possibly be surgically altered. There are 5 regions: forehead, two eyelids, nose, middle face.
  • Feature extraction. A uniform LBP feature is extracted for every subregion, which is represented as a 59-dimension normalized histogram.
  • Distance measuring. Single subregion components (e.g. forehead) are measured with a Chi-square distance. For the others (eyelids or nose) the maximum Chi-square distance is used.



The PSD method, while following a similar algorithm, is based on the partial matching technique. It includes localization and normalization of the eye images, facial decomposition into 10 subregions, distance measuring of the subregions with the Chi-square distance, LBP histograms calculation, and threshold filtering.


The Effect of Plastic Surgery on Face Recognition

An experiment was hosted to estimate the impact of plastic surgery over a face recognition system. Training data for the experiment was taken from public non-surgery datasets: GTAV Face, CMU Multi-PIE, FERET, and others. For testing a plastic surgery dataset (PSD) was used. The test results revealed that efficacy of a face recognition algorithm significantly decreases by approximately 26-30%. The tested algorithms included Fisher Discriminant Analysis (FDA), Local Feature Analysis (LFA), Principal Component Analysis (PCA), and others.


Methods for Recognizing Faces with Surgical Alterations

A number of techniques have been proposed for facial recognition in the context of facial surgeries.

Granular approach

The granular approach implies extraction of non-disjoint features at different granular levels. Next, assimilated information should be procured with the help of synergistic combination with multiobjective evolutionary learning. This method provides more flexibility when it comes to analyzing information pertaining to the nose, ears, forehead, and so on. It includes three levels of granularity.



Additionally, this method employs Extended Uniform Circular Local Binary Patterns, Scale Invariant Feature Transform, Multiobjective Evolutionary Learning, and other tools for feature extraction, face granules combining, etc.

Using sparse representation

Sparse signal representation theory suggests that it is possible to identify a person even with a sparse number of identity features. Unconventional features — such as random projections — can be on par with such standard features as Laplacianfaces and help identify a subject successfully (only if the dimension of the feature space surpasses a specific threshold).


Combination of Information From the Facial & Ocular Regions

Another proposed method is a fusion approach, which analyzes data retrieved from the facial and ocular regions. It is claimed that it achieves an 87.4% accuracy on the Plastic Surgery Dataset. The method employs a Viola-Jones Adaboost face-detecting algorithm — it helps to detect and crop eye regions of the images. Scale Invariant Feature Transform (SIFT) and Local Binary Patterns (LBP), in turn, are responsible for feature extraction both locally and globally. At the final stage, facial and ocular information are combined with the score-fusion technique.


GPROF

Gabor Patch classifiers via Rank-Order list Fusion or GPROF method suggests that even a global face surgery cannot completely change the initial information naturally intrinsic to a face in question. Therefore, an after-surgery face can be identified with the help of Gabor feature together with Fisher Linear Discriminant Analysis since Gabor feature is acknowledged as a highly effective descriptor for facial recognition.



The GPROF method includes eye centers localization, facial alignment, normalization, illumination preprocessing, face separation into 2x4 non-overlapping patches, Gabor magnitude features extraction, patch rank-order calculation with the cosine similarity, and so on.

FARO & FACE

FAce Recognition against Occlusions and expression variations (FARO) and Face Analysis for Commercial Entities (FACE) are effectively used in tandem. FARO — based on Partitioned Iterated Function Systems — is an efficient tool for analyzing partially occluded facial images, as well as emotional expressions. FACE focuses on image matching with the help of a localized version of the correlation index between a pair of images. As a result, this combination can derive the local region of interest (ROI) representations and provide accurate recognition and matching.

References

  1. Overview of Craniofacial Anomalies
  2. Countries with the largest total number of cosmetic procedures in 2020
  3. Plastic Surgery Statistics Report
  4. Amado Fuentes by Wikipedia
  5. He Used Plastic Surgery to Raise Rock Stars From the Dead
  6. Amado Fuentes (center) tried to alter his appearance with plastic surgery, which resulted in his death due to complications
  7. Plastic Surgery: A New Dimension to Face Recognition
  8. What is the ‘criminal’ plastic surgery in The Tinder Swindler and is it real?
  9. Medical confidentiality
  10. Plastic Surgery: A New Dimension to Face Recognition
  11. Face Recognition using Probabilistic Model for Locally Changed Face
  12. Face Recognition after Plastic Surgery: a Comprehensive Study
  13. Impact and Detection of Facial Beautification in Face Recognition: An Overview
  14. Example of a face from the GTAV Face database
  15. CMU Multi-PIE
  16. FERET
  17. Recognizing Surgically Altered Face Images Using Multiobjective Evolutionary Algorithm
  18. Robust Face Recognition via Sparse Representation
  19. Mitigating Effects of Plastic Surgery: Fusing Face and Ocular Biometrics
  20. Viola–Jones object detection framework by Wikipedia
  21. Fisher’s Linear Discriminant: Intuitively Explained
  22. FARO: FAce Recognition Against Occlusions and Expression Variations
  23. Face: face analysis for Commercial Entities