Pretrained language models (PLMs) are frequently employed in Decision Support Systems (DSSs) due to their strong performance. However, recent studies have revealed that these PLMs can exhibit social biases, leading to unfair decisions that harm vulnerable groups. Sensitive information contained in sentences from training data is the primary source of bias. Previously proposed debiasing methods based on contrastive disentanglement have proven highly effective. In these methods, PLMs can disentangle sensitive information from non-sensitive information in sentence embedding, and then adapts non-sensitive information only for downstream tasks. Such approaches hinge on having good sentence embedding as input. However, recent research found that most non-fine-tuned PLMs such as BERT produce poor sentence embedding. Disentangling based on these embedding will lead to unsatisfactory debiasing results. Taking a finer-grained perspective, we propose PCFR (Prompt and Contrastive-based Fair Representation), a novel disentanglement method integrating prompt and contrastive learning to debias PLMs. We employ prompt learning to represent information as sensitive embedding and subsequently apply contrastive learning to contrast these information embedding rather than the sentence embedding. PCFR encourages similarity among different non-sensitive information embedding and dissimilarity between sensitive and non-sensitive information embedding. We mitigate gender and religion biases in two prominent PLMs, namely BERT and GPT-2. To comprehensively assess debiasing efficacy of PCFR, we employ multiple fairness metrics. Experimental results consistently demonstrate the superior performance of PCFR compared to representative baseline methods. Additionally, when applied to specific downstream decision tasks, PCFR not only shows strong de-biasing capability but also significantly preserves task performance.

中文翻译：

---

迈向公平决策：一种用于消除预训练模型偏差的新颖表示方法

预训练语言模型 (PLM) 由于其强大的性能而经常用于决策支持系统 (DSS)。然而，最近的研究表明，这些 PLM 可能会表现出社会偏见，导致做出伤害弱势群体的不公平决策。训练数据句子中包含的敏感信息是偏见的主要来源。先前提出的基于对比解缠的去偏方法已被证明非常有效。在这些方法中，PLM 可以在句子嵌入中将敏感信息与非敏感信息分开，然后仅针对下游任务调整非敏感信息。这种方法取决于良好的句子嵌入作为输入。然而，最近的研究发现，大多数未经微调的 PLM（例如 BERT）都会产生较差的句子嵌入。基于这些嵌入的解开将导致不令人满意的去偏结果。从更细粒度的角度来看，我们提出了 PCFR（基于即时和对比的公平表示），这是一种新颖的解开方法，它将即时和对比学习结合起来以消除 PLM 的偏差。我们采用即时学习将信息表示为敏感嵌入，然后应用对比学习来对比这些信息嵌入而不是句子嵌入。 PCFR 鼓励不同非敏感信息嵌入之间的相似性以及敏感和非敏感信息嵌入之间的差异。我们在两个著名的 PLM（即 BERT 和 GPT-2）中减轻了性别和宗教偏见。为了全面评估 PCFR 的去偏功效，我们采用了多种公平性指标。实验结果一致证明 PCFR 与代表性基线方法相比具有优越的性能。此外，当应用于特定的下游决策任务时，PCFR不仅表现出强大的去偏能力，而且还显着保持了任务性能。