In brief

Experiments on a robot named Olivia demonstrate how the framework—inspired by how humans use tools for specific tasks without needing to learn how to use them in advance—ushers in a new era of autonomous problem solving and adaptability in unstructured environments.

© A*STAR Research

Robot Olivia’s lessons in tool mastery

24 Nov 2023

A pioneering computational framework empowers robots to acquire, recognise and effectively use previously unseen tools.

Crows fashion insect-grabbing hooks out of twigs and sea otters crack open shellfish with the help of jagged rocks. What sets human intelligence apart is our capacity for abstract thinking, problem-solving and innovation, allowing us to intuitively develop and use tools in a plethora of novel situations.

Despite monumental progress in machine learning sophistication, robots still don’t share the same instincts when it comes to tool use.

Samuel Cheong, a Research Engineer with A*STAR’s Institute for Infocomm Research (I2R) explained that it’s been a protracted struggle to develop robots with the capacity to pick the right tool for the job without prior training.

While humans can quickly identify objects and understand its potential as tools based on their shape, texture and context, these thought processes are tougher for robots to replicate. “For example, when a coin rolls under a sofa, we can immediately use a stick to retrieve it,” illustrated Cheong.

Training robots on the potential of every object in the toolbox is a possible, albeit extremely time-consuming and impractical approach. Instead, I2R researchers, Cheong and Jun Li, together with corresponding author, Keng Peng Tee, worked with collaborators from France’s Centre National de la Recherche Scientifique (CNRS) to develop a new framework for tool learning and skill transfer in robots based on how humans recognise and use tools.

The team used a robot, which they named Olivia, to build and test their computational framework. As Cheong explained, Olivia was first trained to explore key functionality features of her limb, such as the ability to lift, pinch or push. After acquiring these functionality features, Olivia was then taught to leverage this information as a visual template to identify previously unseen objects that can serve as tools. For example, a frying pan could be lifted and used as a hammer to drive a nail into wood.

The third component of the framework is tool augmentation optimisation, which guides Olivia to find a way to grab the recognised tool at the right spot and plan her movements effectively to complete the task without running into obstacles.

Although Olivia demonstrated remarkable progress in her ability to use objects as tools on the fly, Cheong said that their framework still has barriers to its widespread commercial use.

“Our framework relies heavily on 3D visual perception and is limited to the recognition of tools to extend the kinematic abilities of a limb,” said Cheong, who provided the example of robots having to recognise properties of objects beyond what they can ‘see’, such as the heavy weight of a hammer.

Cheong commented that this add-on feature will form the basis of the team’s future work. “We may explore visuo-dynamic associations to enable the association of texture and shape to mass to extend our framework.”

The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research (I2R).

Want to stay up to date with breakthroughs from A*STAR? Follow us on Twitter and LinkedIn!


Tee, K.P., Cheong, S., Li, J. and Ganesh, G. A framework for tool cognition in robots without prior tool learning or observation. Nature Machine Intelligence 4, 533–543 (2022). | article

About the Researchers

View articles

Samuel Min Ting Cheong

Research Engineer

Institute for Infocomm Research (I2R)
Samuel Min Ting Cheong is currently a Research Engineer at A*STAR’s Institute for Infocomm Research (I2R). He graduated from Nanyang Technological University, Singapore in 2018, specialising in robot control systems. In 2018, Cheong and Keng Peng Tee developed a tool cognition framework which allows robots to recognise an object as a potential tool for performing a task. In 2019, he became a part of the Pholus project research team that was also collaborating with Istituto Italiano di Tecnologia (IIT) Italy. The team has since contributed to various research areas including the hybrid wheeled-legged quadrupedal robot and a supervised autonomy framework for remote teleoperation. Together with the I2R family, he is currently developing more applications for robots, and pushing the boundary of research for quadruped mobile manipulator robots.
Jun Li is a Senior Scientist at A*STAR’s Institute for Infocomm Research (I2R) and has extensive experience in the field of computer vision and robotics. He received his BEng (1997) and MEng degree in Circuit & System (2002) from the University of Science and Technology of China and PhD degree in computer vision (2007) from Nanyang Technological University, Singapore. Li has worked on several awarded projects such as APICS, MATAR and SML, and has filed patents related to these projects. Li has published high-quality papers in the domain of robot perception and navigation, including a co-authored paper that was selected as the front cover of Nature Machine Intelligence in June 2022, where he contributed to perception modules for unseen tool detection and analyses.

This article was made for A*STAR Research by Wildtype Media Group