Style-guided texture generation aims to generate a texture that is harmonious with both the style of the reference image and the geometry of the input mesh, given a reference style image and a 3D mesh with its text description. Although diffusion-based 3D texture generation methods, such as distillation sampling, have numerous promising applications in stylized games and films, it requires addressing two challenges: 1) decouple style and content completely from the reference image for 3D models, and 2) align the generated texture with the color tone, style of the reference image, and the given text prompt. To this end, we introduce StyleTex, an innovative diffusion-model-based framework for creating stylized textures for 3D models. Our key insight is to decouple style information from the reference image while disregarding content in diffusion-based distillation sampling. Specifically, given a reference image, we first decompose its style feature from the image CLIP embedding by subtracting the embedding's orthogonal projection in the direction of the content feature, which is represented by a text CLIP embedding. Our novel approach to disentangling the reference image's style and content information allows us to generate distinct style and content features. We then inject the style feature into the cross-attention mechanism to incorporate it into the generation process, while utilizing the content feature as a negative prompt to further dissociate content information. Finally, we incorporate these strategies into StyleTex to obtain stylized textures. We utilize Interval Score Matching to address over-smoothness and over-saturation, in combination with a geometry-aware ControlNet that ensures consistent geometry throughout the generative process. The resulting textures generated by StyleTex retain the style of the reference image, while also aligning with the text prompts and intrinsic details of the given 3D mesh. Quantitative and qualitative experiments show that our method outperforms existing baseline methods by a significant margin.

中文翻译：

---

StyleTex： 为 3D 模型生成样式图像引导纹理


样式导向纹理生成旨在生成与参考图像的样式和输入网格的几何图形相协调的纹理，给定参考样式图像和带有文本描述的 3D 网格。尽管基于扩散的 3D 纹理生成方法（例如蒸馏采样）在风格化游戏和电影中有许多前景广阔的应用，但它需要解决两个挑战：1） 将样式和内容与 3D 模型的参考图像完全分离，以及 2） 将生成的纹理与色调、参考图像的样式和给定的文本提示对齐。为此，我们推出了 StyleTex，这是一个创新的基于扩散模型的框架，用于为 3D 模型创建风格化的纹理。我们的主要见解是将样式信息与参考图像解耦，同时忽略基于扩散的蒸馏采样中的内容。具体来说，给定一个参考图像，我们首先通过减去嵌入在内容特征方向上的正交投影，从图像 CLIP 嵌入中分解其样式特征，该投影由文本 CLIP 嵌入表示。我们解开参考图像的样式和内容信息的新方法使我们能够生成不同的样式和内容特征。然后，我们将样式特征注入到交叉注意力机制中，将其纳入生成过程，同时利用内容特征作为负面提示来进一步解离内容信息。最后，我们将这些策略整合到 StyleTex 中以获得风格化的纹理。我们利用区间分数匹配来解决过度平滑和过度饱和的问题，并结合几何感知 ControlNet，确保在整个生成过程中几何图形的一致性。 StyleTex 生成的纹理保留了参考图像的样式，同时也与给定 3D 网格的文本提示和内在细节保持一致。定量和定性实验表明，我们的方法明显优于现有的基线方法。