Crows fashion insect-grabbing hooks out of twigs and sea otters crack open shellfish with the help of jagged rocks. What sets human intelligence apart is our capacity for abstract thinking, problem-solving and innovation, allowing us to intuitively develop and use tools in a plethora of novel situations.
Despite monumental progress in machine learning sophistication, robots still don’t share the same instincts when it comes to tool use.
Samuel Cheong, a Research Engineer with A*STAR’s Institute for Infocomm Research (I2R) explained that it’s been a protracted struggle to develop robots with the capacity to pick the right tool for the job without prior training.
While humans can quickly identify objects and understand its potential as tools based on their shape, texture and context, these thought processes are tougher for robots to replicate. “For example, when a coin rolls under a sofa, we can immediately use a stick to retrieve it,” illustrated Cheong.
Training robots on the potential of every object in the toolbox is a possible, albeit extremely time-consuming and impractical approach. Instead, I2R researchers, Cheong and Jun Li, together with corresponding author, Keng Peng Tee, worked with collaborators from France’s Centre National de la Recherche Scientifique (CNRS) to develop a new framework for tool learning and skill transfer in robots based on how humans recognise and use tools.
The team used a robot, which they named Olivia, to build and test their computational framework. As Cheong explained, Olivia was first trained to explore key functionality features of her limb, such as the ability to lift, pinch or push. After acquiring these functionality features, Olivia was then taught to leverage this information as a visual template to identify previously unseen objects that can serve as tools. For example, a frying pan could be lifted and used as a hammer to drive a nail into wood.
The third component of the framework is tool augmentation optimisation, which guides Olivia to find a way to grab the recognised tool at the right spot and plan her movements effectively to complete the task without running into obstacles.
Although Olivia demonstrated remarkable progress in her ability to use objects as tools on the fly, Cheong said that their framework still has barriers to its widespread commercial use.
“Our framework relies heavily on 3D visual perception and is limited to the recognition of tools to extend the kinematic abilities of a limb,” said Cheong, who provided the example of robots having to recognise properties of objects beyond what they can ‘see’, such as the heavy weight of a hammer.
Cheong commented that this add-on feature will form the basis of the team’s future work. “We may explore visuo-dynamic associations to enable the association of texture and shape to mass to extend our framework.”
The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research (I2R).