Real-to-sim Robotic Scene Generator

Inhalt:
It is cumbersome to collect data using a real robot in a physical environment, as it is very time consuming, not to mention costly in terms of human labor and hardware. As such, the use of simulation has become more prevalent in various robotic domains. A simulated robotic environment is a good alternative to collect the necessary data for training any machine learning method. However, training a model in simulation suffers from the well-known sim-to-real gap when transferred to the real world, in which it often fails to generalize properly. Here, we propose a novel approach to close this gap by translating real-world images into their equivalent simulated images. Using an image-to-image translation with a conditional GAN technique we develop a real-to-sim robotic scene generator that is not only capable of generating visually similar results to the ground truth, but is also able to preserve main characteristics of the image, namely, object and robot position and shape. Our work here uncovers opportunities to use machine learning models trained in simulation directly in the real world by transferring real-world style to simulation.