Direct imaging of Earth-like exoplanets is one of the most prominent scientific drivers of the next generation of ground-based telescopes. Typically, Earth-like exoplanets are located at small angular separations from their host stars, making their detection difficult. Consequently, the adaptive optics (AO) system’s control algorithm must be carefully designed to distinguish the exoplanet from the residual light produced by the host star. A promising avenue of research to improve AO control builds on data-driven control methods, such as reinforcement learning (RL). RL is an active branch of the machine learning research field, where control of a system is learned through interaction with the environment. Thus, RL can be seen as an automated approach to AO control, where its usage is entirely a turnkey operation. In particular, model-based RL has been shown to cope with temporal and misregistration errors. Similarly, it has been demonstrated to adapt to nonlinear wavefront sensing while being efficient in training and execution. In this work, we implement and adapt an RL method called policy optimization for AO (PO4AO) to the GPU-based high-order adaptive optics testbench (GHOST) test bench at ESO headquarters, where we demonstrate a strong performance of the method in a laboratory environment. Our implementation allows the training to be performed parallel to inference, which is crucial for on-sky operation. In particular, we study the predictive and self-calibrating aspects of the method. The new implementation on GHOST running PyTorch introduces only around 700 μs of in addition to hardware, pipeline, and Python interface latency. We open-source well-documented code for the implementation and specify the requirements for the RTC pipeline. We also discuss the important hyperparameters of the method and how they affect the method. Further, the paper discusses the source of the latency and the possible paths for a lower latency implementation.

中文翻译：

---

基于模型的强化学习自适应光学控制的实验室实验

对类地系外行星的直接成像是下一代地面望远镜最重要的科学驱动力之一。通常，类地系外行星与其主恒星的角距较小，这使得它们的探测变得困难。因此，自适应光学（AO）系统的控制算法必须仔细设计，以区分系外行星和主星产生的残余光。改进 AO 控制的一个有前途的研究途径是建立在数据驱动的控制方法的基础上，例如强化学习 (RL)。强化学习是机器学习研究领域的一个活跃分支，其中系统的控制是通过与环境的交互来学习的。因此，RL 可以被视为 AO 控制的自动化方法，其使用完全是交钥匙操作。特别是，基于模型的强化学习已被证明可以应对时间错误和配准错误。同样，它已被证明能够适应非线性波前传感，同时能够高效地进行训练和执行。在这项工作中，我们实现了一种称为 AO 策略优化 (PO4AO) 的 RL 方法，并将其应用于 ESO 总部基于 GPU 的高阶自适应光学测试台 (GHOST) 测试台，我们在该测试台上展示了​​该方法的强大性能。实验室环境。我们的实现允许训练与推理并行进行，这对于空中操作至关重要。我们特别研究了该方法的预测和自校准方面。运行 PyTorch 的 GHOST 上的新实现除了硬件、管道和 Python 接口延迟外，仅引入约 700 μs 的延迟。我们开源了记录完善的实施代码，并指定了 RTC 管道的要求。我们还讨论了该方法的重要超参数以及它们如何影响该方法。此外，本文还讨论了延迟的来源以及实现较低延迟的可能路径。