Nowadays, web robots are predominantly used for auto-accessing web content, sharing almost one-third of the total web traffic and often posing threats to various web applications’ security, privacy, and performance. Detecting these robots is essential, and both online and offline methods are employed. One popular offline method is the use of weblog feature-based automated learning. However, this method alone cannot accurately identify web robots that continuously evolve and camouflage. Web content features combined with weblog features are used to detect such robots based on the assumption that human users exhibit specific interests while robots randomly navigate web pages. State-of-the-art web content-based feature methods lack the ability to generate coherent topics, which can confound the performance of classification models. Therefore, we propose a new content semantic feature extraction method that uses the LDA2Vec topic model, combining the strengths of LDA and the Word2Vec model to produce more semantically coherent topics by exploiting website content for a web session. To effectively detect web robots, web resource content semantic features are combined with log-based features in the proposed web robot detection approach. The proposed approach is evaluated in an e-commerce website access logs and content data. The F-score, balanced accuracy, G-mean, and Jaccard similarity are used for performance measures, and the coherence score metric is used to determine the number of topics for a session. Experimental results demonstrate that a combination of weblogs and content semantic features is effective in web robot detection.

中文翻译：

---

利用网络内容语义特征从博客中检测网络机器人


如今，网络机器人主要用于自动访问网络内容，共享近三分之一的网络总流量，并且经常对各种网络应用程序的安全、隐私和性能构成威胁。检测这些机器人是必不可少的，并且采用在线和离线方法。一种流行的离线方法是使用基于博客特征的自动学习。然而，仅靠这种方法无法准确识别不断进化和伪装的网络机器人。基于人类用户表现出特定兴趣而机器人随机浏览网页的假设，将网络内容特征与博客特征相结合来检测此类机器人。最先进的基于网络内容的特征方法缺乏生成连贯主题的能力，这可能会影响分类模型的性能。因此，我们提出了一种新的内容语义特征提取方法，该方法使用 LDA2Vec 主题模型，结合 LDA 和 Word2Vec 模型的优点，通过利用 Web 会话的网站内容来产生更多语义连贯的主题。为了有效地检测网络机器人，在所提出的网络机器人检测方法中，将网络资源内容语义特征与基于日志的特征相结合。所提出的方法在电子商务网站访问日志和内容数据中进行评估。 F 分数、平衡准确度、G 均值和 Jaccard 相似度用于绩效衡量，连贯性分数指标用于确定会话的主题数量。实验结果表明，网络博客和内容语义特征的结合在网络机器人检测中是有效的。