Unsupervised Machine Learning based rapid Pose-Invariant Three-Dimensional Facial Reconstruction for Biomedical Extended Reality and Phantom Generation in Oncologic Cranio-Maxillofacial Surgery

Abstract

Cranio-maxillofacial surgery is a surgical specialty that focuses on the reconstructive surgery of the entire cranio-maxillofacial complex: the anatomical area of the mouth, jaws, face, and skull, head and neck as well as associated structures. A highly precise and complex procedure, it demands a well-structured and comprehensive treatment planning process. Biomedical extended reality (XR) has proven to be beneficial in the pre-surgery visualization step as it enables interaction in the three-dimensional (3D) space relative to the patient, however the tedious nature of the construction of unique, patient-specific facial models makes it difficult to implement XR and commercial physical patient phantoms are far too generic and expensive for the task. In the case of cancerous tumors on the head and face, the numbers of which have increased tremendously over the past few years, a suitable oncologic therapy needs to be delivered in the shortest possible period of time, as they spread rapidly and the possible complications can be fatal to patients. While PET/CT scans do help in accurately visualizing such tumors, and are the universally-accepted approach in surgically examining cases of craniofacial trauma and oral cancer, they are time-consuming and only assist in two-dimensional (2D) visualization. They have to be further mapped by a surgical or radiological expert to the 3D anatomy of the patient, a task that can be eliminated in both oncologic cranio-maxillofacial surgery and cosmetic surgery.

In this study, we explore near-instantaneous external 3D facial reconstruction from landmarks obtained specifically from a single 2D image of the human face, which is unchallenging to sample, making the system deployable as an application even on mobile devices. We also aim to eliminate the limitations encountered due to pose variations, through the frontalization of these images, while maintaining focus on the affected area, with the implementation of an illumination-preserving Generative Adversarial Network (GAN), integrated with a light Convolutional Neural Network (CNN) for feature extraction, trained and tested on the Multi-PIE and LFW (Labelled Faces In-the-Wild) datasets. The output is then processed to obtain 2D landmarks (disregarding external features) that are crucial to project the image on a 3D morphable model (3DMM). The final output is a digital, printable 3D mesh in both the OBJ and STL format, which can be used for generating custom patient phantoms as well as be implemented in biomedical XR for pre-surgical planning and evaluation. It can also be integrated with a CT scan for internal skeletal 3D imaging in other cranio-maxillofacial cases.

We ultimately evaluate the performance of the system on images captured from several angles, to show its generalization and robustness to novel facial configurations and unseen data.