Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1 % of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort. To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT in ProTI , a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6 081 programs from GitHub and artificial benchmarks shows that ProTI can directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.

中文翻译：

---

自动化基础设施即代码程序测试


基础设施即代码 (IaC) 可实现高效的部署和操作，这对于快速发布软件至关重要。由于设置可能很复杂，开发人员使用 Pulumi 和 AWS CDK 等 PL-IaC 解决方案，以 TypeScript 和 Python 等通用编程语言实施 IaC 程序。此类 IaC 程序的可靠性比传统软件更为重要，因为 IaC 中的错误会影响整个系统。然而，尽管测试是一种标准的开发实践，但它很少用于 IaC 程序。例如，2022 年 8 月，GitHub 上只有不到 1% 的公开 Pulumi IaC 项目实施了测试。可用的 IaC 程序测试技术严重限制了开发速度或需要大量的开发工作。为了解决这些问题，我们提出了自动配置测试 (ACT)，这是一种快速、轻松地在多种配置中测试 IaC 程序的方法。 ACT 自动模拟 IaC 程序中的所有资源定义，并使用生成器和预言机插件进行测试生成和验证。我们在 ProTI 中实现 ACT，ProTI 是 Pulumi TypeScript 的测试工具，具有基于类型的生成器和预言机，并支持应用程序规范。我们对 GitHub 上的 6081 个程序和人工基准进行的评估表明，ProTI 可以直接应用于现有的 IaC 程序，快速发现当前技术不可行的错误，并且由于其可插拔架构，可以重用现有的生成器和预言机。