This article presents a novel vision-based proprioception approach for a soft robotic finger that can estimate and reconstruct tactile interactions in terrestrial and aquatic environments. The key to this system lies in the finger's unique metamaterial structure, which facilitates omnidirectional passive adaptation during grasping, protecting delicate objects across diverse scenarios. A compact in-finger camera captures high-framerate images of the finger's deformation during contact, extracting crucial tactile data in real time. We present a volumetric discretized model of the soft finger and use the geometry constraints captured by the camera to find the optimal estimation of the deformed shape. The approach is benchmarked using a motion capture system with sparse markers and a haptic device with dense measurements. Both results show state-of-the-art accuracy, with a median error of 1.96 mm for overall body deformation, corresponding to 2.1 $\%$ of the finger's length. More importantly, the state estimation is robust in both on-land and underwater environments as we demonstrate its usage for underwater object shape sensing. This combination of passive adaptation and real-time tactile sensing paves the way for amphibious robotic grasping applications.

中文翻译：

---

两栖触觉感知的本体感觉状态估计


本文提出了一种基于视觉的新型本体感觉方法，用于软机器人手指，可以估计和重建陆地和水生环境中的触觉交互。该系统的关键在于手指独特的超材料结构，便于在抓取过程中进行全方位的被动适应，在各种场景中保护精致的物体。紧凑的手指内摄像头可捕捉手指接触过程中变形的高帧率图像，实时提取关键的触觉数据。我们提出了一个软手指的体积离散模型，并使用相机捕获的几何约束来找到变形形状的最佳估计。该方法使用具有稀疏标记的动作捕捉系统和具有密集测量的触觉设备进行基准测试。两个结果都显示了最先进的精度，整体体变形的中位数误差为 1.96 毫米，相当于手指长度的 2.1 $\%$。更重要的是，状态估计在陆地和水下环境中都是稳健的，因为我们展示了它在水下物体形状传感中的应用。这种被动适应和实时触觉传感的结合为水陆两用机器人抓取应用铺平了道路。