Researchers at Carnegie Mellon University (CMU) in Pittsburgh, Pennsylvania, USA have used machine learning and colour cameras to teach robots how to better grasp transparent and reflective objects.
The technique doesn’t require sensors, training or human guidance, but relies primarily on a colour camera, could lead to a breakthrough in the development of so-called kitchen robots capable of picking up items such as a clear measuring cup or a shiny knife.
David Held, an assistant professor in CMU’s Robotics Institute, said depth cameras, which shine infrared light on an object to determine its shape, work well for identifying opaque objects. But infrared light passes right through clear objects and scatters off reflective surfaces. Thus, depth cameras can’t calculate an accurate shape, resulting in largely flat or hole-riddled shapes for transparent and reflective objects.
However, a colour camera can see transparent and reflective objects as well as opaque ones. Thus, CMU scientists developed a camera system to recognise shapes based on colour. The researchers were able to train the new system to imitate the depth system and implicitly infer shape to grasp objects. They did so using depth camera images of opaque objects paired with colour images of those same objects.
Once trained, the colour camera system was applied to transparent and shiny objects. Based on those images, along with whatever scant information a depth camera could provide, the system was able to grasp a range challenging objects with a high degree of success.
“We do sometimes miss,” Held said, “but for the most part it did a pretty good job, much better than any previous system for grasping transparent or reflective objects.”
The system can’t pick up transparent or reflective objects as efficiently as opaque objects, said Thomas Weng, a Ph.D. student in robotics. But it is far more successful than depth camera systems alone. And the multimodal transfer learning used to train the system was so effective that the colour system proved almost as good as the depth camera system at picking up opaque objects.
“Our system not only can pick up individual transparent and reflective objects, but it can also grasp such objects in cluttered piles,” Weng added.
Other attempts at robotic grasping of transparent objects have relied on training systems based on exhaustively repeated attempted grasps – on the order of 800,000 attempts – or on expensive human labelling of objects.
The CMU system uses a commercial RGB-D camera that’s capable of both colour images and depth images. The system can use this single sensor to sort through recyclables or other collections of objects – some opaque, some transparent, some reflective.
