This paper analyzes finite-state Markov Decision Processes (MDPs) with nonstationary and uncertain parameters via set-based fixed point theory. Given compact parameter ambiguity sets, we demonstrate that a family of contraction operators, including the Bellman operator and the policy evaluation operator, can be extended to set-based contraction operators with a unique fixed point—a compact value function set. For non-stationary MDPs, we show that while the value function trajectory diverges, its Hausdorff distance from this fixed point converges to zero. In parameter uncertain MDPs, the fixed point’s extremum value functions are equivalent to the min–max value function in robust dynamic programming under the rectangularity condition. Furthermore, we show that the rectangularity condition is a sufficient condition for the fixed point to contain its own extremum value functions. Finally, we derive novel guarantees for probabilistic path planning in capricious wind fields and stratospheric station-keeping.

中文翻译：

---

用于非平稳和不确定马尔可夫决策过程的基于集合的值运算符


本文通过基于集合的不动点理论分析了具有非平稳和不确定参数的有限态马尔可夫决策过程 （MDP）。给定紧凑的参数模糊集，我们证明了一系列收缩运算符，包括 Bellman 运算符和策略评估运算符，可以扩展到具有唯一固定点（紧凑值函数集）的基于集合的收缩运算符。对于非平稳 MDP，我们表明，当值函数轨迹发散时，它与该固定点的 Hausdorff 距离收敛为零。在参数不确定的 MDP 中，在矩形条件下，不动点的极值函数等效于稳健动态规划中的 min-max 值函数。此外，我们表明矩形条件是不动点包含其自己的极值函数的充分条件。最后，我们为反复无常的风场和平流层站位保持中的概率路径规划推导出了新的保证。