当前位置: X-MOL 学术J. Cheminfom. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
PETA: evaluating the impact of protein transfer learning with sub-word tokenization on downstream applications
Journal of Cheminformatics ( IF 7.1 ) Pub Date : 2024-08-02 , DOI: 10.1186/s13321-024-00884-3
Yang Tan 1, 2, 3, 4 , Mingchen Li 1, 2, 3, 4 , Ziyi Zhou 2 , Pan Tan 2, 3 , Huiqun Yu 1 , Guisheng Fan 1 , Liang Hong 2, 3, 4
Affiliation  

Protein language models (PLMs) play a dominant role in protein representation learning. Most existing PLMs regard proteins as sequences of 20 natural amino acids. The problem with this representation method is that it simply divides the protein sequence into sequences of individual amino acids, ignoring the fact that certain residues often occur together. Therefore, it is inappropriate to view amino acids as isolated tokens. Instead, the PLMs should recognize the frequently occurring combinations of amino acids as a single token. In this study, we use the byte-pair-encoding algorithm and unigram to construct advanced residue vocabularies for protein sequence tokenization, and we have shown that PLMs pre-trained using these advanced vocabularies exhibit superior performance on downstream tasks when compared to those trained with simple vocabularies. Furthermore, we introduce PETA, a comprehensive benchmark for systematically evaluating PLMs. We find that vocabularies comprising 50 and 200 elements achieve optimal performance. Our code, model weights, and datasets are available at https://github.com/ginnm/ProteinPretraining . This study introduces advanced protein sequence tokenization analysis, leveraging the byte-pair-encoding algorithm and unigram. By recognizing frequently occurring combinations of amino acids as single tokens, our proposed method enhances the performance of PLMs on downstream tasks. Additionally, we present PETA, a new comprehensive benchmark for the systematic evaluation of PLMs, demonstrating that vocabularies of 50 and 200 elements offer optimal performance.

中文翻译:


PETA:评估蛋白质转移学习与子词标记化对下游应用的影响



蛋白质语言模型(PLM)在蛋白质表示学习中发挥着主导作用。大多数现有的 PLM 将蛋白质视为 20 种天然氨基酸的序列。这种表示方法的问题在于,它只是将蛋白质序列划分为单个氨基酸的序列,而忽略了某些残基经常一起出现的事实。因此,将氨基酸视为孤立的标记是不合适的。相反,PLM 应该将频繁出现的氨基酸组合识别为单个标记。在这项研究中,我们使用字节对编码算法和一元语法来构建用于蛋白质序列标记化的高级残基词汇表,并且我们已经证明,与使用这些高级词汇表进行预训练的 PLM 相比,使用这些高级词汇表进行预训练的 PLM 在下游任务中表现出卓越的性能。简单的词汇。此外,我们还推出了 PETA,这是一个用于系统评估 PLM 的综合基准。我们发现包含 50 和 200 个元素的词汇表可以实现最佳性能。我们的代码、模型权重和数据集可在 https://github.com/ginnm/ProteinPretraining 获取。本研究引入了先进的蛋白质序列标记化分析,利用字节对编码算法和一元组。通过将频繁出现的氨基酸组合识别为单个标记,我们提出的方法增强了 PLM 在下游任务上的性能。此外,我们还推出了 PETA,这是一种用于系统评估 PLM 的新综合基准,证明包含 50 和 200 个元素的词汇表可提供最佳性能。
更新日期:2024-08-02
down
wechat
bug