When faced with a novel situation, people often spend substantial periods of time contemplating possible futures. For such planning to be rational, the benefits to behavior must compensate for the time spent thinking. Here, we capture these features of behavior by developing a neural network model where planning itself is controlled by the prefrontal cortex. This model consists of a meta-reinforcement learning agent augmented with the ability to plan by sampling imagined action sequences from its own policy, which we call ‘rollouts’. In a spatial navigation task, the agent learns to plan when it is beneficial, which provides a normative explanation for empirical variability in human thinking times. Additionally, the patterns of policy rollouts used by the artificial agent closely resemble patterns of rodent hippocampal replays. Our work provides a theory of how the brain could implement planning through prefrontal–hippocampal interactions, where hippocampal replays are triggered by—and adaptively affect—prefrontal dynamics.

当面对新的情况时，人们常常花大量时间思考可能的未来。为了使这种计划变得合理，行为的好处必须补偿花费在思考上的时间。在这里，我们通过开发神经网络模型来捕获这些行为特征，其中计划本身由前额叶皮层控制。该模型由一个元强化学习代理组成，该代理具有通过从其自己的策略中采样想象的动作序列来进行计划的能力，我们称之为“推出”。在空间导航任务中，智能体学会在有益的时候进行计划，这为人类思维时代的经验变异性提供了规范的解释。此外，人工智能代理使用的策略推出模式与啮齿动物海马回放模式非常相似。我们的工作提供了一种关于大脑如何通过前额叶-海马体相互作用来实施计划的理论，其中海马体重放是由前额叶动态触发并适应性影响的。