Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition

1 CyberAgent, AI Lab  2 University of Tsukuba
Computer Graphics Forum (Proc. of Eurographics 2023)

3D Avatar is widely used in movies and advertisements, with facial makeup being a crucial aspect of creating these lifelike digital characters. However, the process of editing makeup on a 3D face or using specialized equipment can be time-consuming. To obtain facial makeup for 3D characters, we propose a method to extract makeup from a single face image in a UV format that can be used for 3D face models. We unwarp the input image to UV texture so that it can be applied to a 3D face model. Then, we decompose the UV texture to bare skin, makeup, and lighting effects.


Facial makeup enriches the beauty of not only real humans but also virtual characters; therefore, makeup for 3D facial models is highly in demand in productions. However, painting directly on 3D faces and capturing real-world makeup are costly, and extracting makeup from 2D images often struggles with shading effects and occlusions. This paper presents the first method for extracting makeup for 3D facial models from a single makeup portrait. Our method consists of the following three steps. First, we exploit the strong prior of 3D morphable models via regression-based inverse rendering to extract coarse materials such as geometry and diffuse/specular albedos that are represented in the UV space. Second, we refine the coarse materials, which may have missing pixels due to occlusions. We apply inpainting and optimization. Finally, we extract the bare skin, makeup, and an alpha matte from the diffuse albedo. Our method offers various applications for not only 3D facial models but also 2D portrait images. The extracted makeup is well-aligned in the UV space, from which we build a large-scale makeup dataset and a parametric makeup model for 3D faces. Our disentangled materials also yield robust makeup transfer and illumination-aware makeup interpolation/removal without a reference image.

Banner image

Makeup-aware facial inverse rendering and component-wise reconstruction. The top row displays a makeup portrait input and overlaid rendering images (from left to right: bare skin only, bare skin plus makeup, bare skin multiplied by diffuse shading, and plus specular reconstruction) whereas the bottom row shows disentangled materials in the UV space.


Computer Graphics Forum
          author = {Yang, Xingchao and Taketomi, Takafumi and Kanamori, Yoshihiro},
          title = {Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition},
          journal = {Computer Graphics Forum},
          volume = {42},
          number = {2},
          pages = {293-307},
          year = {2023}