Large pre-trained vision language models (VLMs) have demonstrated impressive representation learning capabilities, but their transferability across various downstream tasks heavily relies on prompt learning. Since VLMs consist of text and visual sub-branches, existing prompt approaches are mainly divided into text and visual prompts. Recent text prompt methods have achieved great performance by designing input-condition prompts that encompass both text and image domain knowledge. However, roughly incorporating the same image feature into each learnable text token may be unjustifiable, as it could result in learnable text prompts being concentrated on one or a subset of characteristics. In light of this, we propose a fine-grained text prompt (FTP) that decomposes the single global image features into several finer-grained semantics and incorporates them into corresponding text prompt tokens. On the other hand, current methods neglect valuable text semantic information when building the visual prompt. Furthermore, text information contains redundant and negative category semantics. To address this, we propose a text-reorganized visual prompt (TVP) that reorganizes the text descriptions of the current image to construct the visual prompt, guiding the image branch to attend to class-related representations. By leveraging both FTP and TVP, we enable mutual prompting between the text and visual modalities, unleashing their potential to tap into the representation capabilities of VLMs. Extensive experiments on 11 classification benchmarks show that our method surpasses existing methods by a large margin. In particular, our approach improves recent state-of-the-art CoCoOp by 4.79% on new classes and 3.88% on harmonic mean over eleven classification benchmarks.

大型预训练视觉语言模型（VLM）已经展示了令人印象深刻的表示学习能力，但它们在各种下游任务中的可迁移性在很大程度上依赖于即时学习。由于VLM由文本和视觉两个分支组成，现有的提示方法主要分为文本和视觉提示。最近的文本提示方法通过设计包含文本和图像领域知识的输入条件提示而取得了出色的性能。然而，将相同的图像特征粗略地合并到每个可学习文本标记中可能是不合理的，因为它可能导致可学习文本提示集中在一个或一个特征子集上。鉴于此，我们提出了一种细粒度文本提示（FTP），它将单个全局图像特征分解为多个更细粒度的语义，并将它们合并到相应的文本提示标记中。另一方面，当前的方法在构建视觉提示时忽略了有价值的文本语义信息。此外，文本信息包含冗余和否定的类别语义。为了解决这个问题，我们提出了一种文本重组视觉提示（TVP），它重新组织当前图像的文本描述来构建视觉提示，引导图像分支关注与类相关的表示。通过利用 FTP 和 TVP，我们实现了文本和视觉模式之间的相互提示，释放了它们挖掘 VLM 表示能力的潜力。对 11 个分类基准的大量实验表明，我们的方法大幅超越了现有方法。特别是，我们的方法将最新最先进的 CoCoOp 在新类别上提高了 4.79%，在 11 个分类基准上将调和平均值提高了 3.88%。