Impact of Partial Occlusion on Face Recognition

From Antispoofing Wiki

Problem Overview

Occlusion of a certain anatomic trait is a recurring problem in biometrics. While it drew mainstream attention only in 2020 with the onset of face masks, the issue has been addressed since at least 2009 by Hazım Kemal Ekenel & Rainer Stiefelhagen who explored lower performance of facial recognition (FR) with occluded face images.

Occlusions are basically observed in every biometric domain. For example, a higher cholesterol level can disrupt vein recognition. Loud noises, earphones, cholesterol levels, band-aid or dirt may negatively affect voice, ear, vein, and fingerprint recognition respectively. However, facial recognition still remains the most sensitive area as frequent facial occlusions — eyeglasses, scarfs, makeup — can be used for both benign and malicious purposes.

To both tackle potentially criminal behavior and decrease customer friction, a number of solutions and techniques to identify users with partially occluded faces are proposed under a collective moniker OFR — Occluded Facial Recognition.

Influence of Occlusion on Face Recognition

The Ekenel-Stiefelhagen 2009 research began with a number of experiments. They demonstrated that FR accuracy varies with occluded facial images depending on:

  • FR model in use.
  • Location of the occluded facial area.
  • Facial feature misalignment.
  • Texture of an occlusion.

Authors conducted an experiment with the AR database, in which subjects are wearing sunglasses and scarves. The results showed that upper face occlusions — the eye region — cause a significant dropdown in facial recognition: 92.7%-37.3%. Meanwhile, the lower face occlusion — mouth and chin — show a less dramatic accuracy decline: 91.8%-83.6%.

Authors assume that this phenomenon can be explained by misalignment sensitivity that common FR approaches — elastic bunch graph matching, Fisherfaces — are susceptible to. A face alignment algorithm is proposed as a possible remedy: it minimizes the closest distance at the classification step, while also providing rough estimation of the facial features positioning. Interestingly, authors also pointed out that an occlusion’s surface texture may affect recognition accuracy.

Types of Face Occlusion

Commonly, there are two general types of occlusion: Real and Synthetic. Natural occlusions — such as eyelashes, facial hair, skin conditions — are sometimes mentioned as an independent type as well.

A more detailed classification includes:

Real type

  • Systematic. Facial hair, pigmentation, makeup, and others.
  • Temporary. Pose variations, hand covering, telephones, blood or bruises, cups, cosmetic masks.
  • Special case. Blurred facial image, low resolution, overly bright/extremely low illumination.
  • Mixed. Involves more than one type of real occlusions.

Synthetic type

This category contains synthetic — digitally produced — occlusions, which simulate the real ones. Various filters used on social media — mustache, whiskers, headwear —can also be counted among synthetic occlusions.

Approaches to Recognizing Occluded Faces

As of now, there are three primary method categories of solving the issue.

Occlusion Robust Feature Extraction (ORFE)

This group is based on the concept of extracting facial features less affected with the occlusions. At the same time, the discriminative capability should stay preserved. These methods are separated into two groups:

  • Engineered features. They allow extracting features from the precisely outlined facial regions without a learning stage.
  • Learning-based-features. Here feature extraction is performed with learning-based methods: sparse representation, nonlinear deep learning techniques, etc.

They comprise such techniques as patch-based engineered features, Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG) descriptors, and so on.

Occlusion Aware Face Recognition (OAFR)

In this group FR uses only the visible parts of the face, while the occluded area is ignored. It incorporates two groups of techniques. The first, Occlusion Detection Based Face Recognition, initially detects the occluded area and then withdraws necessary data from the non-occluded parts. It uses a 1-NN (nearest neighbor) threshold classifier, selective local non-negative matrix factorization, and other know-hows.

The second, Partial Face Recognition, uses the partially visible face areas. It utilizes feature extraction with Multiscale Double Supervision Convolutional Neural Network (MDSCNN), Radial Basis Function components, multi-keypoint descriptors, and so on.

Occlusion Recovery-Based Face Recognition (ORBFR)

ORBFR is based on recreating the missing facial parts. It includes Reconstruction and Inpainting (fixing the image) approaches. They employ Structured Sparse Representation-Based Classification (SSRC), Long Short-Term Memory (LSTM), Weighted Principal Component Analysis (FW-PCA), etc.

Methods to Solve Partial Occlusion

A set of techniques are offered to solve the issue.

Part Based Methods

Part based methods imply that an image can be segregated into a group of overlapping and non-overlapping elements, which undergo the recognition process. An array of methods is proposed in this context:

  • Non-metric partial similarity. While two images are compared, differences between them are removed. This is done to retrieve intra-personal details. It employs a Self-Organizing Map (SOM), various distance measures, etc.
  • 2D-PCA. Two-dimensional principal component analysis explores the image rows, which in turn, correspond to the rows in the feature matrix.
  • SVM classification. The partial (e.g. modified) Support Vector Machine is used for reconstructing the missing parts.
  • LGBP. Local Gabor Binary Pattern is coupled with Kullback-Leibler Divergence (KLD) to estimate probability of occluded and non-occluded parts of a face, multiscale image conversion, histogram calculation for every local component, etc.

Other techniques are based on subspace learning, Selective Local Non-Negative Matrix Factorization (S-LNMF), Posterior Union Model (PUM), and so forth.

Fractal Based Methods

A noteworthy example is the Partitioned Iterated Function System (PIFS). PIFS on its own can be hindered with a distortion caused by occlusions, so a face should be divided into separate regions: nose, mouth, eyes, etc. Then, an ad-hoc distance measure is used to remove any occurring distortions and PIFS can focus on searching for correspondences that small square regions may share.

Feature Based Methods

This set of methods concentrates on individual features — like an area encircling the eyes — and ignores other features, which may be unique among different people. They rely on isolated Eigenspace analysis, soft masking for outlier classification, guided label learning, SVM + Gaussian summation kernel tandem, etc.

Periocular and Masked Face Recognition

Periocular approach focuses on specific regions: in Near-infrared images eyelids, tear ducts, and eye shape play a huge role, while for VW images skin and blood vessels are of importance. This method extracts a) Global features pertaining to face or its region b) Local features formed by a group of discrete points. Plus, it analyzes color, texture, and shape features.

Deep Learning in Partial Face Recognition

A promising method based on deep learning is called dynamic Feature Matching. It relies on the Fully Convolutional Network (FCN), which comprises convolutional and pooling layers. Another vital component is Sparse Representation Classification (SRC) applied to achieve alignment-free dynamic feature matching. The FCN itself is optimized with the sliding loss — it reduces face patch and face image intra-variation.

Classifying Facial Races with Partial Occlusions

A specific CNN model that features 10 layers and utilizes receptive field and weight sharing is proposed for race detection. It was trained with 96,000 sample images, which proportionately represent 4 human races: Caucasian, Mongolian, Indian, and African. It’s reported to demonstrate 95.1% accuracy at identifying a race shown in images with facial occlusions and pose variations.


  1. Face masks give facial recognition software an identity crisis
  2. Why Is Facial Occlusion a Challenging Problem?
  3. Efficient Detection of Occlusion prior to Robust Face Recognition
  4. How to Identify Customer Friction
  5. Example of the noise louder than 10 dB playing a role of the voice occlusion
  6. AR Face Database
  7. A robust face recognition algorithm for real-world applications
  8. Image samples from the AR face database
  9. Real examples with different types of facial occlusion
  10. Stage microphone is an example of a temporary facial occlusion
  11. Synthetic occlusions imitating real-life items
  12. A survey of face recognition techniques under occlusion
  13. HOG (Histogram of Oriented Gradients): An Overview
  14. Facial Inpainting
  15. A set of techniques
  16. FARO: FAce Recognition Against Occlusions and Expression Variations
  17. A Survey on Periocular Biometrics Research
  18. Dynamic Feature Learning for Partial Face Recognition
  19. FCN's architecture
  20. On Classifying Facial Races with Partial Occlusions and Pose Variations