Purpose

The premise of this research is that the coupling of reinforcement learning algorithms and computational dynamics can be used to design efficient control strategies and to improve the cooling of hot components by quenching, a process that is classically carried out based on professional experience and trial-error methods. Feasibility and relevance are assessed on various 2-D numerical experiments involving boiling problems simulated by a phase change model. The purpose of this study is then to integrate reinforcement learning with boiling modeling involving phase change to optimize the cooling process during quenching.

Design/methodology/approach

The proposed approach couples two state-of-the-art in-house models: a single-step proximal policy optimization (PPO) deep reinforcement learning (DRL) algorithm (for data-driven selection of control parameters) and an in-house stabilized finite elements environment combining variational multi-scale (VMS) modeling of the governing equations, immerse volume method and multi-component anisotropic mesh adaptation (to compute the numerical reward used by the DRL agent to learn), that simulates boiling after a phase change model formulated after pseudo-compressible Navier–Stokes and heat equations.

Findings

Relevance of the proposed methodology is illustrated by controlling natural convection in a closed cavity with aspect ratio 4:1, for which DRL alleviates the flow-induced enhancement of heat transfer by approximately 20%. Regarding quenching applications, the DRL algorithm finds optimal insertion angles that adequately homogenize the temperature distribution in both simple and complex 2-D workpiece geometries, and improve over simpler trial-and-error strategies classically used in the quenching industry.

Originality/value

To the best of the authors’ knowledge, this constitutes the first attempt to achieve DRL-based control of complex heat and mass transfer processes involving boiling. The obtained results have important implications for the quenching cooling flows widely used to achieve the desired microstructure and material properties of steel, and for which differential cooling in various zones of the quenched component will yield irregular residual stresses that can affect the serviceability of critical machinery in sensitive industries.

中文翻译：

---

淬火过程中冷却速率控制的强化学习

 目的

这项研究的前提是，强化学习算法和计算动力学的耦合可以用来设计有效的控制策略，并通过淬火来改善热部件的冷却，这是一个基于专业经验和试错的经典过程方法。通过涉及相变模型模拟的沸腾问题的各种二维数值实验来评估可行性和相关性。这项研究的目的是将强化学习与涉及相变的沸腾建模相结合，以优化淬火过程中的冷却过程。

设计/方法论/途径

所提出的方法结合了两种最先进的内部模型：单步近端策略优化（PPO）深度强化学习（DRL）算法（用于数据驱动的控制参数选择）和内部稳定有限元环境，结合了控制方程的变分多尺度 (VMS) 建模、浸没体积法和多分量各向异性网格自适应（计算 DRL 代理用于学习的数值奖励），模拟相变模型后的沸腾根据赝可压缩纳维-斯托克斯方程和热方程制定。

 发现

通过控制纵横比为 4:1 的封闭腔中的自然对流说明了所提出方法的相关性，其中 DRL 将流动引起的传热增强减轻了约 20%。关于淬火应用，DRL 算法找到了最佳插入角度，可以充分均匀化简单和复杂的二维工件几何形状中的温度分布，并改进了淬火行业中传统使用的简单试错策略。

 原创性/价值

据作者所知，这是对涉及沸腾的复杂传热传质过程实现基于 DRL 的控制的首次尝试。所获得的结果对于广泛用于实现钢的所需微观结构和材料性能的淬火冷却流程具有重要意义，并且淬火部件各个区域的差异冷却将产生不规则的残余应力，从而影响关键机械的适用性敏感行业。