Flood maps based on Earth Observation (EO) data inform critical decision-making in almost every stage of the disaster management cycle, directly impacting the ability of affected individuals and governments to receive aid as well as informing policies on future adaptation. However, flood map validation also presents a challenge in the form of class imbalance between flood and non-flood classes, which has rarely been investigated. There are currently no established best practices for addressing this issue, and the accuracy of these maps is often viewed as a mere formality, which leads to a lack of user trust in flood map products and a limitation in their operational use and uptake. This paper provides the first comprehensive assessment of the impact of current EO-based flood map validation practices. Using flood inundation maps derived from Sentinel-1 synthetic aperture radar data with synthetically generated controlled errors and Copernicus Emergency Management Service flood maps as the ground truth, binary metrics were statistically evaluated for the quantification of flood detection accuracy for events under varying flood conditions. Especially, class specific metrics were found to be sensitive to the class imbalance, i.e. larger flood magnitudes result in higher metric scores, thus being naturally biased towards overpredicting classifiers. Metric stability across error percentiles and flood magnitudes was assessed through standard deviation calculated by bootstrapping to quantify the impact of sample selection subjectivity, where stratified sampling schemes exhibited the lowest standard deviation consistently. Thoughtful sample and response design were critical, with probability-based random sampling and proportional or equal class allocation vital to producing robust accuracy estimates comparable across study sites, error classes, and flood magnitudes. Results suggest that popular evaluation metrics such as the F1-Score are in fact unsuitable for accurate characterization of map quality and are not comparable across different study sites or events. Overall accuracy and MCC are shown to be the most robust performance metrics when sampling designs are optimized, and bootstrapping is demonstrated to be a necessary tool for estimating variability in map accuracy observed due to the spatial sampling of validation points. Results presented herein pave the way for the development of global flood map validation guidelines, to support wider use of and trust in EO-derived flood risk and recovery products, eventually allowing us to unlock the full potential of EO for improved flood resilience.

中文翻译：

---

实现 EO 洪水图的稳健验证策略


基于地球观测 （EO） 数据的洪水地图为灾害管理周期的几乎每个阶段的关键决策提供信息，直接影响受影响个人和政府获得援助的能力，并为未来适应政策提供信息。但是，洪水地图验证也带来了洪水和非洪水类之间的类不平衡的挑战，这种情况很少被研究。目前还没有解决此问题的既定最佳实践，这些地图的准确性通常被视为一种形式，这导致用户对洪水地图产品缺乏信任，并且其操作使用和采用受到限制。本文首次全面评估了当前基于 EO 的洪水地图验证实践的影响。使用从 Sentinel-1 合成孔径雷达数据得出的洪水泛滥地图，合成生成的受控误差和哥白尼应急管理服务洪水地图作为地面实况，对二元指标进行统计评估，以量化不同洪水条件下事件的洪水检测准确性。特别是，发现特定于类的指标对类不平衡很敏感，即较大的洪水量级会导致更高的度量分数，因此自然而然地偏向于高估分类器。通过自举计算的标准差来评估误差百分位数和洪水幅度的度量稳定性，以量化样本选择主观性的影响，其中分层抽样方案始终表现出最低的标准差。 深思熟虑的样本和响应设计至关重要，基于概率的随机抽样和成比例或相等的类别分配对于生成跨研究地点、误差类别和洪水量级的可比性稳健的准确性估计至关重要。结果表明，F1 分数等流行的评估指标实际上并不适合准确描述地图质量，并且在不同的研究地点或事件之间没有可比性。在优化采样设计时，总体精度和 MCC 被证明是最稳健的性能指标，并且自举被证明是估计由于验证点的空间采样而观察到的地图精度变化的必要工具。本文提供的结果为制定全球洪水地图验证指南铺平了道路，以支持更广泛地使用和信任 EO 衍生的洪水风险和恢复产品，最终使我们能够释放 EO 的全部潜力，以提高洪水抵御能力。