As software systems continuously grow in size and complexity, performance and load related issues have become more common than functional issues. Load testing is usually performed before software releases to ensure that the software system can still provide quality service under a certain load. Therefore, one of the common challenges of load testing is to design realistic workloads that can represent the actual workload in the field. In particular, one of the most widely adopted and intuitive approaches is to directly replay the field workloads in the load testing environment. However, replaying a lengthy, e.g., 48 hours, field workloads is rather resource- and time-consuming, and sometimes even infeasible for large-scale software systems that adopt a rapid release cycle. On the other hand, replaying a short duration of the field workloads may still result in unrealistic load testing. In this work, we propose an automated approach to reduce the length of load testing that is driven by replaying the field workloads. The intuition of our approach is: if the measured performance associated with a particular system behaviour is already stable, we can skip subsequent testing of this system behaviour to reduce the length of the field workloads. In particular, our approach first clusters execution logs that are generated during the system runtime to identify similar system behaviours during the field workloads. Then, we use statistical methods to determine whether the measured performance associated with a system behaviour has been stable. We evaluate our approach on three open-source projects (i.e., OpenMRS , TeaStore , and Apache James ). The results show that our approach can significantly reduce the length of field workloads while the workloads-after-reduction produced by our approach are representative of the original set of workloads. More importantly, the load testing results obtained by replaying the workloads after the reduction have high correlation and similar trend with the original set of workloads. Practitioners can leverage our approach to perform realistic field-replay based load testing while saving the needed resources and time. Our approach sheds light on future research that aims to reduce the cost of load testing for large-scale software systems.

中文翻译：

---

缩短基于现场回放的负载测试的长度


随着软件系统的规模和复杂性不断增长，性能和负载相关问题变得比功能问题更常见。负载测试通常在软件发布前进行，以确保软件系统在一定负载下仍能提供优质的服务。因此，负载测试的常见挑战之一是设计能够代表现场实际工作负载的实际工作负载。特别是，最广泛采用和直观的方法之一是直接重放负载测试环境中的现场工作负载。然而，重放长时间（例如48小时）的现场工作负载相当耗费资源和时间，有时对于采用快速发布周期的大型软件系统来说甚至是不可行的。另一方面，重放短时间的现场工作负载仍可能导致不切实际的负载测试。在这项工作中，我们提出了一种自动化方法来缩短通过重放现场工作负载驱动的负载测试的长度。我们方法的直觉是：如果与特定系统行为相关的测量性能已经稳定，我们可以跳过对该系统行为的后续测试，以减少现场工作负载的长度。特别是，我们的方法首先对系统运行时生成的执行日志进行集群，以识别现场工作负载期间的类似系统行为。然后，我们使用统计方法来确定与系统行为相关的测量性能是否稳定。我们在三个开源项目（即 OpenMRS 、 TeaStore 和 Apache James ）上评估我们的方法。 结果表明，我们的方法可以显着减少现场工作负载的长度，而我们的方法产生的减少后的工作负载代表了原始的工作负载集。更重要的是，通过重放缩减后的工作负载得到的负载测试结果与原始工作负载集具有较高的相关性和相似的趋势。从业人员可以利用我们的方法执行基于现场重放的真实负载测试，同时节省所需的资源和时间。我们的方法为未来的研究提供了启示，旨在降低大规模软件系统的负载测试成本。