Multimodal Neural Radiance Field for In-Hand Robot Manipulation Tasks of Transparent Objects

Camille Taglione (1) , Carlos Mateo (1) , Christophe Stolz (1)

1 VIBOT - Equipe VIBOT - VIsion pour la roBOTique [ImViA EA7535 - ERL CNRS 6000]

In recent years, there has been significant progress in robot manipulation research, but challenges persist when it comes to tracking transparent objects using RGB-D sensors. To address this, polarimetric imaging technology has gained popularity due to cost-effective sensors becoming more accessible. This technology, which analyzes the polarization properties of light, offers valuable insights into object properties, aids in material differentiation, depth estimation, and precise pose estimation, even in challenging lighting conditions. A novel multimodal perception approach is introduced in this study, utilizing a learning-based neural radiance field method based on multimodal data to overcome the limitations of RGB-D imaging when dealing with transparent objects during in-hand manipulation (Kerr et al.)

Mots clés: Neural radiance field; Multi-modal polarimetric perception; Transparent object; Robotic manipulation

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Robotique [cs.RO]

Camille Taglione, Carlos Mateo, Christophe Stolz. Multimodal Neural Radiance Field for In-Hand Robot Manipulation Tasks of Transparent Objects. Journées Nationales de la Recherche en Robotique (JNRR 2023), Oct 2023, Moliets, France. . ⟨hal-04255537⟩