Man-at-the-end (MATE) attackers have full control over the system on which the attacked software runs, and try to break the confidentiality or integrity of assets embedded in the software. Both companies and malware authors want to prevent such attacks. This has driven an arms race between attackers and defenders, resulting in a plethora of different protection and analysis methods. However, it remains difficult to measure the strength of protections because MATE attackers can reach their goals in many different ways and a universally accepted evaluation methodology does not exist. This survey systematically reviews the evaluation methodologies of papers on obfuscation, a major class of protections against MATE attacks. For 571 papers, we collected 113 aspects of their evaluation methodologies, ranging from sample set types and sizes, over sample treatment, to performed measurements. We provide detailed insights into how the academic state of the art evaluates both the protections and analyses thereon. In summary, there is a clear need for better evaluation methodologies. We identify nine challenges for software protection evaluations, which represent threats to the validity, reproducibility, and interpretation of research results in the context of MATE attacks and formulate a number of concrete recommendations for improving the evaluations reported in future research papers.

中文翻译：

---

软件保护研究中的评估方法


MATE 攻击者可以完全控制运行受攻击软件的系统，并试图破坏软件中嵌入的资产的机密性或完整性。公司和恶意软件作者都希望防止此类攻击。这导致了攻击者和防御者之间的军备竞赛，导致了大量不同的保护和分析方法。但是，仍然很难衡量保护措施的强度，因为 MATE 攻击者可以通过多种不同的方式实现其目标，并且不存在普遍接受的评估方法。本调查系统地回顾了关于混淆的论文的评估方法，混淆是抵御 MATE 攻击的主要保护措施。对于 571 篇论文，我们收集了他们评估方法的 113 个方面，范围从样本组类型和大小、样本处理到执行的测量。我们提供了关于学术最新技术如何评估保护措施和分析的详细见解。总之，显然需要更好的评估方法。我们确定了软件保护评估的九大挑战，这些挑战代表了在 MATE 攻击背景下对研究结果的有效性、可重复性和解释的威胁，并提出了一些具体建议，以改进未来研究论文中报告的评估。