Impact of Plastic Surgery on Face Recognition
Problem Overview
Plastic surgery is a medically induced face alteration, which allows to both enhance aesthetic nuances of one’s appearance or correct gained/innate defects, including the craniofacial anomalies: cleft lip, hemifacial microsomia, and others. According to Statista, 4,667,931 plastic surgeries were performed in the US alone in 2019. As stated by Plasticsurgery.org, facial surgeries led the top 5 cosmetic procedures in 2020. Among them were nose reshaping, eyelid surgery and rhytidectomy (facelift).
While cosmetic and reconstructive plastic surgeries are mostly benign procedures, they can impede performance of facial recognition systems or man-operated security checkpoints. This was proved by a Hongqiao airport incident, in which a group of women was not identified using their passport due to the plastic surgeries they had undergone.
Plastic surgery can also be used for malevolent purposes. A widespread practice is to permanently alter appearance to avoid capture and imprisonment: a Mexican drug trafficker Amado Fuentes died because of plastic surgery complications. Another possible threat is impersonation through plastic surgery — a practice sporadically used, among all else, for unethical entertainment purposes.
Types of Facial Plastic Surgery
Apart from the general cosmetic and reconstructive surgery, other types include Local and Global surgery.
Local type
Local plastic surgeries aim to correct aesthetic flaws — scars, wrinkles, birthmarks, facial asymmetry or some natural anomalies — in a specific region of the body or face. This type of procedure focuses on removing defects that are constitutional (by-birth), gained in an accident and those defects that have developed over the time.
Global type
This type of surgery mostly addresses functional damage received as a result of some critical trauma. In this case, the entire facial structure — features, contour, skin texture — is being reconstructed to normalize the patient’s appearance. However, the same procedure can be used to permanently alter a person’s look making it hard or even impossible to identify them.
Plastic Surgery Databases
Plastic surgeons involved in facial reconstruction amass a considerable library of pictures and slides captured prior and afterwards surgeries. However, this data has to be kept private due to medical confidentiality, which is practiced in most countries. Besides, patients are often reluctant to publicly share their surgery cases. Therefore, assembling a plastic surgery database for facial recognition training is a challenging task. Virtually, there are no publicly available datasets of this kind. The only known example is the standard Plastic Surgery Database (PSD), which was used in the work of Singh et al. covering a probabilistic model for a locally changed face.
Plastic Surgery Detection
An intuitive method of plastic surgery detection (PSD) proposed, is based on the direct matching of corresponding face patches around the possible plastic surgery regions. It includes a component relative threshold setting, which is capable of spotting the exact region to which a surgical alteration has been applied.
Usually, such an algorithm includes the following steps:
- Decomposition. Subject’s face is decomposed into regions that could possibly be surgically altered. There are 5 regions: forehead, two eyelids, nose, middle face.
- Feature extraction. A uniform LBP feature is extracted for every subregion, which is represented as a 59-dimension normalized histogram.
- Distance measuring. Single subregion components (e.g. forehead) are measured with a Chi-square distance. For the others (eyelids or nose) the maximum Chi-square distance is used.
The PSD method, while following a similar algorithm, is based on the partial matching technique. It includes localization and normalization of the eye images, facial decomposition into 10 subregions, distance measuring of the subregions with the Chi-square distance, LBP histograms calculation, and threshold filtering.
The Effect of Plastic Surgery on Face Recognition
An experiment was hosted to estimate the impact of plastic surgery over a face recognition system. Training data for the experiment was taken from public non-surgery datasets: GTAV Face, CMU Multi-PIE, FERET, and others. For testing a plastic surgery dataset (PSD) was used. The test results revealed that efficacy of a face recognition algorithm significantly decreases by approximately 26-30%. The tested algorithms included Fisher Discriminant Analysis (FDA), Local Feature Analysis (LFA), Principal Component Analysis (PCA), and others.
Methods for Recognizing Faces with Surgical Alterations
A number of techniques have been proposed for facial recognition in the context of facial surgeries.
Granular approach
The granular approach implies extraction of non-disjoint features at different granular levels. Next, assimilated information should be procured with the help of synergistic combination with multiobjective evolutionary learning. This method provides more flexibility when it comes to analyzing information pertaining to the nose, ears, forehead, and so on. It includes three levels of granularity.
Additionally, this method employs Extended Uniform Circular Local Binary Patterns, Scale Invariant Feature Transform, Multiobjective Evolutionary Learning, and other tools for feature extraction, face granules combining, etc.
Using sparse representation
Sparse signal representation theory suggests that it is possible to identify a person even with a sparse number of identity features. Unconventional features — such as random projections — can be on par with such standard features as Laplacianfaces and help identify a subject successfully (only if the dimension of the feature space surpasses a specific threshold).
Combination of Information From the Facial & Ocular Regions
Another proposed method is a fusion approach, which analyzes data retrieved from the facial and ocular regions. It is claimed that it achieves an 87.4% accuracy on the Plastic Surgery Dataset. The method employs a Viola-Jones Adaboost face-detecting algorithm — it helps to detect and crop eye regions of the images. Scale Invariant Feature Transform (SIFT) and Local Binary Patterns (LBP), in turn, are responsible for feature extraction both locally and globally. At the final stage, facial and ocular information are combined with the score-fusion technique.
GPROF
Gabor Patch classifiers via Rank-Order list Fusion or GPROF method suggests that even a global face surgery cannot completely change the initial information naturally intrinsic to a face in question. Therefore, an after-surgery face can be identified with the help of Gabor feature together with Fisher Linear Discriminant Analysis since Gabor feature is acknowledged as a highly effective descriptor for facial recognition.
The GPROF method includes eye centers localization, facial alignment, normalization, illumination preprocessing, face separation into 2x4 non-overlapping patches, Gabor magnitude features extraction, patch rank-order calculation with the cosine similarity, and so on.
FARO & FACE
FAce Recognition against Occlusions and expression variations (FARO) and Face Analysis for Commercial Entities (FACE) are effectively used in tandem. FARO — based on Partitioned Iterated Function Systems — is an efficient tool for analyzing partially occluded facial images, as well as emotional expressions. FACE focuses on image matching with the help of a localized version of the correlation index between a pair of images. As a result, this combination can derive the local region of interest (ROI) representations and provide accurate recognition and matching.
FAQ
Does plastic surgery influence face recognition?
Influence of plastic surgery on facial recognition is poorly explored. But the spoofing potential of this procedure cannot be denied.
Whether plastic surgery spoofing attack is a serious threat to facial recognition (FR) or not is yet to be figured out. However, at least one instance of personal misidentification due to surgical facial alteration took place in China: a group of women who had undergone the procedure weren’t recognized.
A laboratory experiment showed that accuracy of a FR algorithm decreased by 26-30% after the plastic surgery data was presented to it. It’s also stated that even if such a procedure can drastically alter one’s appearance, the human face still retains fundamental, initial parameters that cannot be changed. (The GPROF method.)
What are the main plastic surgery datasets?
Plastic surgery datasets are sparse in quantity due to privacy protection and ethical issues.
Medical institutions involved in plastic surgeries assemble a lot of pre/after-procedure pictures. At the same time, the plastic surgery data cannot be redistributed to preserve medical patient’s privacy.
Another issue arises due to patients being unwilling to share this data with the public. The only known plastic surgery dataset — Plastic Surgery Database (PSD) — was introduced together with a Singh et al. research, which studied a probabilistic model for the locally changed faces. However, It’s unknown whether this sample collection is available for researchers or not. The issue of assembling such anti-spoofing data is yet to be solved.
Is it possible to detect if a person has had plastic surgery?
The GPROF method allows detecting plastic facial surgeries.
Gabor Patch classifiers via Rank-Order list Fusion (GPROF) used together with the Fisher Linear Discriminant Analysis (FLDA) is proposed as a method to detect facial surgery spoofing attacks.
It implies that the human face has constitutional parameters that cannot be altered surgically. It includes eye centers localization, facial alignment, normalisation, face separation, patch rank-order calculation, Gabor magnitude features extraction, and other techniques. Alternative anti-spoofing know-hows suggest analyzing facial/ocular region information, non-disjoint features extraction at various granular levels, and other concepts.
References
- Overview of Craniofacial Anomalies
- Countries with the largest total number of cosmetic procedures in 2020
- Plastic Surgery Statistics Report
- Amado Fuentes by Wikipedia
- He Used Plastic Surgery to Raise Rock Stars From the Dead
- Amado Fuentes (center) tried to alter his appearance with plastic surgery, which resulted in his death due to complications
- Plastic Surgery: A New Dimension to Face Recognition
- What is the ‘criminal’ plastic surgery in The Tinder Swindler and is it real?
- Medical confidentiality
- Plastic Surgery: A New Dimension to Face Recognition
- Face Recognition using Probabilistic Model for Locally Changed Face
- Face Recognition after Plastic Surgery: a Comprehensive Study
- Impact and Detection of Facial Beautification in Face Recognition: An Overview
- Example of a face from the GTAV Face database
- CMU Multi-PIE
- FERET
- Recognizing Surgically Altered Face Images Using Multiobjective Evolutionary Algorithm
- Robust Face Recognition via Sparse Representation
- Mitigating Effects of Plastic Surgery: Fusing Face and Ocular Biometrics
- Viola–Jones object detection framework by Wikipedia
- Fisher’s Linear Discriminant: Intuitively Explained
- FARO: FAce Recognition Against Occlusions and Expression Variations
- Face: face analysis for Commercial Entities