In this paper, an automatic vision-guided micromanipulation approach to facilitate versatile deployment and portable setup is proposed. This paper is motivated by the importance of micromanipulation and the limitations in existing automation technology in micromanipulation. Despite significant advancements in micromanipulation techniques, there remain bottlenecks in integrating and adopting automation for this application. An underlying reason for the gaps is the difficulty in deploying and setting up such systems. To address this, we identified two important design requirements, namely, portability and versatility of the micromanipulation platform. A self-contained vision-guided approach requiring no complicated preparation or setup is proposed. This is achieved through an uncalibrated self-initializing workflow algorithm also capable of assisted targeting. The feasibility of the solution is demonstrated on a low-cost portable microscope camera and compact actuated microstages. Results suggest subpixel accuracy in localizing the tool tip during initialization steps. The self-focus mechanism could recover intentional blurring of the tip by autonomously manipulating it 95.3% closer to the focal plane. The average error in visual servo is less than a pixel with our depth compensation mechanism showing better maintaining of similarity score in tracking. Cell detection rate in a 1637-frame video stream is 97.7% with subpixels localization uncertainty. Our work addresses the gaps in existing automation technology in the application of robotic vision-guided micromanipulation and potentially contributes to the way cell manipulation is performed. Note to Practitioners—This paper introduces an automatic method for micromanipulation using visual information from microscopy. We design an automatic workflow, which consists of: 1) self-initialization; 2) vision-guided manipulation; and 3) assisted targeting, and demonstrate versatile deployment of the micromanipulator on a portable microscope camera setup. Unlike existing systems, our proposed method does not require any tedious calibration or expensive setup making it mobile and low cost. This overcomes the constraints of traditional practices that confine automated cell manipulation to a laboratory setting. By extending the application beyond the laboratory environment, automated micromanipulation technology can be made more ubiquitous and expands readily to facilitate field study.

中文翻译：

---

自动视觉引导的显微操作，可实现多种部署和便携式设置

在本文中，提出了一种自动视觉引导的显微操纵方法，以促进通用部署和便携式设置。本文受到微操纵的重要性以及现有自动化技术在微操纵中的局限性的启发。尽管微操纵技术取得了重大进步，但在针对此应用程序集成和采用自动化方面仍然存在瓶颈。造成这种差距的根本原因是部署和设置此类系统的困难。为了解决这个问题，我们确定了两个重要的设计要求，即微操纵平台的便携性和多功能性。提出了一种独立的视觉引导方法，不需要复杂的准备或设置。这是通过未校准的自初始化工作流程算法实现的，该算法也可以辅助目标定位。该解决方案的可行性在低成本便携式显微镜相机和紧凑型致动微型平台上得到了证明。结果表明在初始化步骤中定位工具提示时的亚像素精度。自聚焦机制可以通过自动操纵靠近焦平面的95.3％来恢复尖端的故意模糊。视觉伺服的平均误差小于一个像素，而我们的深度补偿机制可更好地保持跟踪中的相似度得分。1637帧视频流中的单元检测率为97.7％，具有亚像素定位不确定性。我们的工作解决了在机器人视觉引导的微操作应用中现有自动化技术的空白，并可能有助于执行细胞操作的方式。给从业者的注意-本文介绍了一种自动方法，该方法利用显微镜的视觉信息进行显微操作。我们设计了一个自动工作流程，它包括：1）自初始化；2）视觉引导操作；和3）辅助瞄准，并演示了微型操纵器在便携式显微镜相机设置上的多功能部署。与现有系统不同，我们提出的方法不需要任何繁琐的校准或昂贵的设置，使其变得可移动且成本低廉。这克服了将自动细胞操作限制在实验室环境中的传统做法的限制。