In this paper, we propose a multimodal flexible sensory interface for interactively teaching soft robots to perform skilled locomotion using bare human hands. First, we develop a flexible bimodal smart skin (FBSS) based on triboelectric nanogenerator and liquid metal sensing that can perform simultaneous tactile and touchless sensing and distinguish these two modes in real time. With the FBSS, soft robots can react on their own to tactile and touchless stimuli. We then propose a distance control method that enabled humans to teach soft robots movements via bare hand-eye coordination. The results showed that participants can effectively teach a self-reacting soft continuum manipulator complex motions in three-dimensional space through a “shifting sensors and teaching” method within just a few minutes. The soft manipulator can repeat the human-taught motions and replay them at different speeds. Finally, we demonstrate that humans can easily teach the soft manipulator to complete specific tasks such as completing a pen-and-paper maze, taking a throat swab, and crossing a barrier to grasp an object. We envision that this user-friendly, non-programmable teaching method based on flexible multimodal sensory interfaces could broadly expand the domains in which humans interact with and utilize soft robots.

在这篇论文中，我们提出了一种多模态灵活的感官接口，用于交互式教学软机器人使用裸手进行熟练的运动。首先，我们开发了一种基于摩擦纳米发电机和液态金属传感的柔性双峰智能皮肤（FBSS），可以同时进行触觉和非接触式传感，并实时区分这两种模式。借助 FBSS，软体机器人可以自行对触觉和非接触式刺激做出反应。然后，我们提出了一种距离控制方法，使人类能够通过裸手眼协调来教授软机器人运动。结果表明，参与者可以在短短几分钟内通过“移动传感器和教学”方法有效地教授自反应软连续体机械手在三维空间中的复杂运动。软机械手可以重复人类教导的动作，并以不同的速度重放它们。最后，我们证明人类可以轻松地教软机械手完成特定任务，例如完成纸笔迷宫、采集咽拭子以及穿过障碍物抓住物体。我们设想，这种基于灵活的多模态感官接口的用户友好、不可编程的教学方法可以广泛扩展人类与软体机器人交互和使用软体机器人的领域。