Burger menu
-


Impact of Plastic Surgery on Face Recognition

Facial plastic surgery is considered as one of the vital threats to facial recognition and liveness detection

Problem Overview

Plastic surgery is a medically induced face alteration, which allows to both enhance aesthetic nuances of one’s appearance or correct gained/innate defects, including the craniofacial anomalies: cleft lip, hemifacial microsomia, and others. According to Statista, 4,667,931 plastic surgeries were performed in the US alone in 2019. As stated by Plasticsurgery.org, facial surgeries led the top 5 cosmetic procedures in 2020. Among them were nose reshaping, eyelid surgery and rhytidectomy (facelift). Consequently, plastic surgeries can impact facial anti-spoofing types and countermeasures, and cause new challenges.

Plastic surgery statistics provided by Plasticsurgery.org
Plastic surgery statistics provided by Plasticsurgery.org

While cosmetic and reconstructive plastic surgeries are mostly benign procedures, they can impede performance of facial recognition systems or man-operated security checkpoints. This was proved by a Hongqiao airport incident, in which a group of women was not identified using their passport due to the plastic surgeries they had undergone.

Plastic surgery can also be used for malevolent purposes to bypass face liveness systems. Another widespread practice is to permanently alter appearance to avoid capture and imprisonment: a Mexican drug trafficker Amado Fuentes died because of plastic surgery complications. Another possible threat is impersonation through plastic surgery — a practice sporadically used, among all else, for unethical entertainment purposes.

Mexican drug trafficker Amado Fuentes tried to alter his appearance with plastic surgery, resulting in his death due to complications
Amado Fuentes (center) tried to alter his appearance with plastic surgery, which resulted in his death due to complications

Types of Facial Plastic Surgery

Apart from the general cosmetic and reconstructive surgery, other types include Local and Global surgery.

Local type

Local plastic surgeries aim to correct aesthetic flaws — scars, wrinkles, birthmarks, facial asymmetry or some natural anomalies — in a specific region of the body or face. This type of procedure focuses on removing defects that are constitutional (by-birth), gained in an accident and those defects that have developed over the time.

Global type

This type of surgery mostly addresses functional damage received as a result of some critical trauma. In this case, the entire facial structure — features, contour, skin texture — is being reconstructed to normalize the patient’s appearance. However, the same procedure can be used to permanently alter a person’s look making it hard or even impossible to identify them.

Types of plastic surgery procedures separated into Local and Global
Plastic surgery procedures separated into Local and Global
Israeli conman Simon Leviev attempted to change his appearance with plastic surgery to avoid arrest
An Israeli conman Simon Leviev attempted to change his appearance with plastic surgery to avoid arrest

Plastic Surgery Databases

Plastic surgeons involved in facial reconstruction amass a considerable library of pictures and slides captured prior and afterward surgeries. However, this data has to be kept private due to medical confidentiality, which is practiced in most countries. Besides, patients are often reluctant to publicly share their surgery cases. Therefore, assembling a plastic surgery database for face anti spoofing training is a challenging task. Virtually, there are no publicly available datasets of this kind. The only known example is the standard Plastic Surgery Database (PSD), which was used in the work of Singh et al. covering a probabilistic model for a locally changed face.

Images of people with different types of plastic surgery from Plastic Surgery Database (PSD)
Plastic Surgery Database samples

Plastic Surgery Detection

An intuitive method of plastic surgery detection (PSD) proposed, is based on the direct matching of corresponding face patches around the possible plastic surgery regions. It includes a component relative threshold setting, which is capable of spotting the exact region to which a surgical alteration has been applied. A passive liveness solution can benefit from such an approach.

Before and after images of popular facial plastic surgeries
Popular facial plastic surgeries in ‘before’ and ‘after’ images

Usually, such an algorithm includes the following steps:

  • Decomposition. Subject’s face is decomposed into regions that could possibly be surgically altered. There are 5 regions: forehead, two eyelids, nose, middle face.
  • Feature extraction. A uniform LBP feature is extracted for every subregion, which is represented as a 59-dimension normalized histogram.
  • Distance measuring. Single subregion components (e.g. forehead) are measured with a Chi-square distance. For the others (eyelids or nose) the maximum Chi-square distance is used.
Facial decomposition into potential surgical regions for anti-spoofing and recognition
Example of facial decomposition into potential surgical regions

The PSD method, while following a similar algorithm, is based on the partial matching technique. It includes localization and normalization of the eye images, facial decomposition into 10 subregions, distance measuring of the subregions with the Chi-square distance, LBP histograms calculation, and threshold filtering.

The PSD method based on partial matching is used to detect plastic surgery for anti-spoofing
PSD based on partial matching method

The Effect of Plastic Surgery on Face Recognition

An experiment was hosted to estimate the impact of plastic surgery over a face recognition system. Training data for the experiment was taken from various public facial datasets: GTAV Face, CMU Multi-PIE, FERET, and others. For testing a plastic surgery dataset (PSD) was used. The test results revealed that efficacy of a face recognition algorithm significantly decreases by approximately 26-30%. The tested algorithms included Fisher Discriminant Analysis (FDA), Local Feature Analysis (LFA), Principal Component Analysis (PCA), and others. Not only passive, but also active liveness detection can be jeopardized this way.

Comparison of performance of facial detection algorithms in detecting faces with and without plastic surgery
Results of facial recognition algorithms tested on a PSD

Methods for Recognizing Faces with Surgical Alterations

A number of techniques have been proposed for facial recognition in the context of facial surgeries.

Granular approach

The granular approach implies extraction of non-disjoint features at different granular levels. Next, assimilated information should be procured with the help of synergistic combination with multiobjective evolutionary learning. This method provides more flexibility when it comes to analyzing information pertaining to the nose, ears, forehead, and so on. It includes three levels of granularity.

Facial detection algorithm using first level facial granules for plastic surgery detection
Facial granules in the first level of granularity

Additionally, this method employs Extended Uniform Circular Local Binary Patterns, Scale Invariant Feature Transform, Multiobjective Evolutionary Learning, and other tools for feature extraction, face granules combining, etc.

Using sparse representation

Sparse signal representation theory suggests that it is possible to identify a person even with a sparse number of identity features. Unconventional features — such as random projections — can be on par with such standard features as Laplacianfaces and help identify a subject successfully (only if the dimension of the feature space surpasses a specific threshold).

Sparse representation is used to identify people with occluded faces, for anti-spoofing
Sparse representation can be used to identify people with occluded faces among all else

Combination of Information From the Facial & Ocular Regions

Another proposed method is a fusion approach, which analyzes data retrieved from the facial and ocular regions. It is claimed that it achieves an 87.4% accuracy on the Plastic Surgery Dataset. The method employs a Viola-Jones Adaboost face-detecting algorithm — it helps to detect and crop eye regions of the images. Scale Invariant Feature Transform (SIFT) and Local Binary Patterns (LBP), in turn, are responsible for feature extraction both locally and globally. At the final stage, facial and ocular information are combined with the score-fusion technique.

Facial and ocular regions of a person analyzed for facial detection
Scheme of the facial & ocular regions analysis method

GPROF

Gabor Patch classifiers via Rank-Order list Fusion or GPROF method suggests that even a global face surgery cannot completely change the initial information naturally intrinsic to a face in question. Therefore, an after-surgery face can be identified with the help of Gabor feature together with Fisher Linear Discriminant Analysis since Gabor feature is acknowledged as a highly effective descriptor for facial recognition.

GPROF algorithm used for detecting plastic surgery in anti-spoofing
GPROF algorithm

The GPROF method includes eye centers localization, facial alignment, normalization, illumination preprocessing, face separation into 2x4 non-overlapping patches, Gabor magnitude features extraction, patch rank-order calculation with the cosine similarity, and so on.

FARO & FACE

FAce Recognition against Occlusions and expression variations (FARO) and Face Analysis for Commercial Entities (FACE) are effectively used in tandem. FARO — based on Partitioned Iterated Function Systems — is an efficient tool for analyzing partially occluded facial images, as well as emotional expressions. FACE focuses on image matching with the help of a localized version of the correlation index between a pair of images. As a result, this combination can derive the local region of interest (ROI) representations and provide accurate recognition and matching.

FAQ

Does plastic surgery influence face recognition?

Influence of plastic surgery on facial recognition is poorly explored. But the spoofing potential of this procedure cannot be denied.

Whether plastic surgery spoofing attack is a serious threat to facial recognition (FR) or not is yet to be figured out. However, at least one instance of personal misidentification due to surgical facial alteration took place in China: a group of women who had undergone the procedure weren’t recognized.

A laboratory experiment showed that accuracy of a FR algorithm decreased by 26-30% after the plastic surgery data was presented to it. It’s also stated that even if such a procedure can drastically alter one’s appearance, the human face still retains fundamental, initial parameters that cannot be changed. (The GPROF method.)

What are the main plastic surgery datasets?

Plastic surgery datasets are sparse in quantity due to privacy protection and ethical issues.

Medical institutions involved in plastic surgeries assemble a lot of pre/after-procedure pictures. At the same time, the plastic surgery data cannot be redistributed to preserve medical patient’s privacy.

Another issue arises due to patients being unwilling to share this data with the public. The only known plastic surgery dataset — Plastic Surgery Database (PSD) — was introduced together with a Singh et al. research, which studied a probabilistic model for the locally changed faces. However, It’s unknown whether this sample collection is available for researchers or not. The issue of assembling such antispoofing data is yet to be solved.

Is it possible to detect if a person has had plastic surgery?

The GPROF method allows detecting plastic facial surgeries.

Gabor Patch classifiers via Rank-Order list Fusion (GPROF) used together with the Fisher Linear Discriminant Analysis (FLDA) is proposed as a method to detect facial surgery spoofing attacks.

It implies that the human face has constitutional parameters that cannot be altered surgically. It includes eye centers localization, facial alignment, normalisation, face separation, patch rank-order calculation, Gabor magnitude features extraction, and other techniques. Alternative anti-spoofing know-hows suggest analyzing facial/ocular region information, non-disjoint features extraction at various granular levels, and other concepts.

References

  1. Overview of Craniofacial Anomalies
  2. Countries with the largest total number of cosmetic procedures in 2020
  3. Plastic Surgery Statistics Report
  4. Amado Fuentes by Wikipedia
  5. He Used Plastic Surgery to Raise Rock Stars From the Dead
  6. Amado Fuentes (center) tried to alter his appearance with plastic surgery, which resulted in his death due to complications
  7. Plastic Surgery: A New Dimension to Face Recognition
  8. What is the ‘criminal’ plastic surgery in The Tinder Swindler and is it real?
  9. Medical confidentiality
  10. Plastic Surgery: A New Dimension to Face Recognition
  11. Face Recognition using Probabilistic Model for Locally Changed Face
  12. Face Recognition after Plastic Surgery: a Comprehensive Study
  13. Impact and Detection of Facial Beautification in Face Recognition: An Overview
  14. Example of a face from the GTAV Face database
  15. CMU Multi-PIE
  16. FERET
  17. Recognizing Surgically Altered Face Images Using Multiobjective Evolutionary Algorithm
  18. Robust Face Recognition via Sparse Representation
  19. Mitigating Effects of Plastic Surgery: Fusing Face and Ocular Biometrics
  20. Viola–Jones object detection framework by Wikipedia
  21. Fisher’s Linear Discriminant: Intuitively Explained
  22. FARO: FAce Recognition Against Occlusions and Expression Variations
  23. Face: face analysis for Commercial Entities

Avatar Antispoofing

1 Followers

Editors at Antispoofing Wiki thoroughly review all featured materials before publishing to ensure accuracy and relevance.

Contents

Hide