We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors. Code and model weights are available at https://doubiiu.github.io/projects/ToonCrafter

中文翻译：

---

ToonCrafter：生成卡通插值


我们介绍了 ToonCrafter，这是一种超越传统基于通信的卡通视频插值的新方法，为生成插值铺平了道路。隐含地假设线性运动且不存在消除遮挡等复杂现象的传统方法，通常难以处理卡通中常见的夸张非线性和大运动以及遮挡，从而导致难以置信甚至失败的插值结果。为了克服这些限制，我们探索了调整真人视频先验的潜力，以更好地适应生成框架内的卡通插值。ToonCrafter 有效地解决了将真人视频运动先验应用于生成式卡通插值时面临的挑战。首先，我们设计了一个卡通整顿学习策略，将真人视频先验无缝地适应卡通领域，解决了领域差距和内容泄露问题。接下来，我们引入了一个基于双参考的 3D 解码器，以补偿由于高度压缩的潜在先验空间而丢失的细节，确保在插值结果中保留精细细节。最后，我们设计了一个灵活的草图编码器，使用户能够对插值结果进行交互式控制。实验结果表明，我们提出的方法不仅产生了视觉上令人信服和更自然的动态，而且有效地处理了去遮挡。比较评估表明，我们的方法优于现有竞争对手。代码和模型权重可在 https://doubiiu.github.io/projects/ToonCrafter