Abstract

Data acquisition forms the primary step in all empirical research. The availability of data directly impacts the quality and extent of conclusions and insights. In particular, larger and more detailed datasets provide convincing answers even to complex research questions. The main problem is that large and detailed usually imply costly and difficult, especially when the data medium is paper and books. Human operators and manual transcription has been the traditional approach for collecting historical data. We instead advocate the use of modern machine learning techniques to automate the digitization and transcription process. We propose a customizable end-to-end transcription pipeline to perform layout classification, table segmentation, and transcribe handwritten text that is suitable for tabular data, as is common in, e.g., census lists and birth and death records. We showcase our pipeline through two applications: The first demonstrates that unsupervised layout classification applied to raw scans of nurse journals can be used to obtain valuable insights into an extended nurse home visiting program. The second application uses attention-based neural networks for handwritten text recognition to transcribe age and birth and death dates and includes a comparison to automated transcription using Transkribus in the regime of tabular data. We describe each step in our pipeline and provide implementation insights.

摘要

数据采集​​是所有实证研究的首要步骤。数据的可用性直接影响结论和见解的质量和范围。特别是，更大、更详细的数据集甚至可以为复杂的研究问题提供令人信服的答案。主要问题是，大而详细通常意味着成本高和难度大，尤其是当数据介质是纸张和书籍时。人工操作员和人工转录一直是收集历史数据的传统方法。相反，我们提倡使用现代机器学习技术来自动化数字化和转录过程。我们提出了一个可定制的端到端转录管道来执行布局分类、表格分割和转录适用于表格数据的手写文本，例如，人口普查名单以及出生和死亡记录。我们通过两个应用程序展示我们的管道：第一个应用程序表明，应用于护士期刊原始扫描的无监督布局分类可用于获得对扩展护士家访计划的宝贵见解。第二个应用程序使用基于注意力的神经网络进行手写文本识别，以转录年龄、出生和死亡日期，并包括与在表格数据范围内使用 Transkribus 的自动转录的比较。我们描述了管道中的每个步骤并提供了实施见解。第二个应用程序使用基于注意力的神经网络进行手写文本识别，以转录年龄、出生和死亡日期，并包括与在表格数据范围内使用 Transkribus 的自动转录的比较。我们描述了管道中的每个步骤并提供了实施见解。第二个应用程序使用基于注意力的神经网络进行手写文本识别，以转录年龄、出生和死亡日期，并包括与在表格数据范围内使用 Transkribus 的自动转录的比较。我们描述了管道中的每个步骤并提供了实施见解。