3D GAN Inversion with Facial Symmetry Prior

Abstract

Although with the facial prior preserved in pre-trained 3D GANs, 3D GAN Inversion, i.e., reconstructing a 3D portrait with only one monocular image, is still an ill-pose problem. The straightforward application of 2D GAN inversion methods focuses on texture similarity only while ignoring the correctness of 3D geometry shapes. It may raise geometry collapse effects, especially when reconstructing a side face under an extreme pose. Besides, the synthetic results in novel views are prone to be blurry. In this work, we propose a novel method to promote 3D GAN inversion by introducing facial symmetry prior. We design a pipeline and constraints to make full use of the pseudo auxiliary view obtained via image flipping, which helps obtain a view-consistent and well-structured geometry shape during the inversion process. To enhance texture fidelity in unobserved viewpoints, pseudo labels from depth-guided 3D warping can provide extra supervision. We design constraints to filter out conflict areas for optimization in asymmetric situations.

Reconstruction

Video: Comparisons with state-of-the-art methods on novel view synthesis.

Editing

Video: Stylization in novel views.

Citation 

@article{yin2022spi,
      author = {Yin, Fei and Zhang, Yong and Wang, Xuan and Wang, Tengfei and Li, Xiaoyu and Gong, Yuan and Fan, Yanbo and Cun, Xiaodong and Cengiz, Öztireli and Yang, Yujiu},
      title = {3D GAN Inversion with Facial Symmetry Prior},
      journal = {arxiv:2211.16927},
      year = {2022}
}