Recognizing the right tool for the job

14 Jan 2019

An algorithm gives robots an instinctive understanding of how to use tools

Researchers from A*STAR working with colleagues in Japan have developed a method by which robots can automatically recognize an object as a potential tool and use it, despite never having seen it before.

For humans, the ability to recognize and use tools is almost instinctive. There are also many examples in which tool use seems hardwired into the brain of animals: some birds and primates use sticks or stones to obtain food, for example. One proposed reason for this neurologically embedded ability to use tools is that the animal’s brain perceives the external object as an extension of its own body. Inspired by this idea, Keng Peng Tee and his colleagues from the A*STAR Institute for Infocomm Research, along with Gowrishankar Ganesh from the CNRS-AIST Joint Robotics Laboratory located in Tsukuba, Japan developed an algorithm that enables robots to recognize, and immediately use tools that they have never seen before.

Previous research by Ganesh has indicated that the human brain recognizes a limb not just by its physical features, but by its functionality. Building on this insight, Ganesh and the A*STAR team proposed that a robot can recognize the potential of a tool by comparing its shape with that of its own hand and arm when used to achieve this same task. They wrote an algorithm based on this idea and tested it by setting a robot the task of moving, without grasping, a disk to a desired end point on a table. This involved the robot either pulling the disk toward itself, pushing it away, or shifting it sideways.

The robot realizes when the goal (the black dot) is out of its reach and uses spatial recognition to choose a tool to achieve the task.

© 2018 A*STAR Institute for Infocomm Research

“The robot is programmed with the ability to complete the task by itself, represented by a hand shape template,” explains Tee. “When the task cannot be completed, because its arms aren’t long enough for example, the robot is able to recognize a viable tool by matching the tool shape with the hand shape template.”

Thus, the algorithm enabled the robot to successfully recognize when a task cannot be performed without a tool, to identify an unknown object as a potential tool, and then to use the tool to achieve the task. “Next we will look into automatic learning of the features that represent existing skills, rather than pre-defining hand shape templates,” says Tee.

The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research.

Want to stay up-to-date with A*STAR’s breakthroughs? Follow us on Twitter and LinkedIn!


Tee, K. P., Li, J., Chen, L. T. P., Wan, K. W. & Ganesh, G. Towards emergence of tool use in robots: automatic tool recognition and use without prior tool learning. IEEE International Conference on Robotics and Automation (ICRA), 2018. | article

About the Researcher

View articles

Keng Peng Tee

Senior Scientist

Institute for Infocomm Research
Dr Keng Peng Tee is currently a Senior Scientist at A*STAR's Institute for Infocomm Research, where he holds an appointment as Head of Manipulation Unit in the Robotics Department. His active research interests include manipulation learning, adaptive systems, and human–robot collaboration.

This article was made for A*STAR Research by Nature Research Custom Media, part of Springer Nature