We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait by predicting its appearance under disturbing shadows and highlights. Portrait shadow removal is a highly ill-posed problem where multiple plausible solutions can be found based on a single image. For example, disentangling complex environmental lighting from original skin color is a non-trivial problem. While existing works have solved this problem by predicting the appearance residuals that can propagate local shadow distribution, such methods are often incomplete and lead to unnatural predictions, especially for portraits with hard shadows. We overcome the limitations of existing local propagation methods by formulating the removal problem as a generation task where a diffusion model learns to globally rebuild the human appearance from scratch as a condition of an input portrait image. For robust and natural shadow removal, we propose to train the diffusion model with a compositional repurposing framework: a pre-trained text-guided image generation model is first fine-tuned to harmonize the lighting and color of the foreground with a background scene by using a background harmonization dataset; and then the model is further fine-tuned to generate a shadow-free portrait image via a shadow-paired dataset. To overcome the limitation of losing fine details in the latent diffusion model, we propose a guided-upsampling network to restore the original high-frequency details (e.g. , wrinkles and dots) from the input image. To enable our compositional training framework, we construct a high-fidelity and large-scale dataset using a lightstage capturing system and synthetic graphics simulation. Our generative framework effectively removes shadows caused by both self and external occlusions while maintaining original lighting distribution and high-frequency details. Our method also demonstrates robustness to diverse subjects captured in real environments.

中文翻译：

---

生成式人像阴影去除


我们介绍了一种高保真人像阴影去除模型，该模型可以通过预测人像在令人不安的阴影和高光下的外观来有效增强人像的图像。人像阴影去除是一个高度病态的问题，可以根据单个图像找到多个合理的解决方案。例如，将复杂的环境光照与原始肤色分开是一个重要的问题。虽然现有的工作通过预测可以传播局部阴影分布的外观残差解决了这个问题，但这样的方法往往是不完整的，会导致不自然的预测，尤其是对于具有硬阴影的肖像。我们通过将去除问题表述为生成任务来克服现有局部传播方法的局限性，其中扩散模型学习从头开始全局重建人类外观作为输入肖像图像的条件。为了实现稳健和自然的阴影去除，我们建议使用合成再利用框架来训练扩散模型：首先对预先训练的文本引导图像生成模型进行微调，以使用背景协调数据集来协调前景的照明和颜色与背景场景;然后进一步微调模型，通过阴影配对数据集生成无阴影的肖像图像。为了克服潜在扩散模型中丢失精细细节的限制，我们提出了一个引导式上采样网络，以恢复输入图像中的原始高频细节（例如，皱纹和点）。为了实现我们的构图训练框架，我们使用 lightstage 捕获系统和合成图形模拟构建了一个高保真和大规模的数据集。 我们的生成式框架可有效消除由自身和外部遮挡引起的阴影，同时保持原始的照明分布和高频细节。我们的方法还证明了对真实环境中捕获的不同主题的鲁棒性。