In brief

Researchers at A*STAR developed G-CRD, a contrastive learning model that captures wide-ranging global relationships, surpassing traditional methods in performance, robustness and efficiency.

© Unsplash

Bridging nodes to embrace connections

16 Apr 2024

A new machine learning method enables smaller models to understand and replicate complex relationships within large networks.

Picture a large company with multiple departments. If each employee is likened to a node, they would be interconnected through relationships such as collaborations or reporting lines.

Graph neural networks, or GNNs, constitute a class of machine learning methods capable of leveraging relationships between nodes to make predictions, such as predicting employee performance or identifying key influencers.

Traditional methods for distilling GNNs tend to focus on preserving local structures, potentially overlooking preserving global interactions and latent relationships. For instance, the distilled graph might only consider day-to-day interactions among employees who work closely yet miss the broader perspective when predicting the performance of employees.

A*STAR researchers Fayao Liu, Xun Xu and Chuan-Sheng Foo from the Institute for Infocomm Research (I2R) explored a new approach to preserving global topology in knowledge distillation of GNNs to enhance their performance.

The team introduced Graph Contrastive Representation Distillation (G-CRD) to assist the 'student' (smaller, less complex GNN models) in learning from a 'teacher', a large, pre-trained GNN capturing intricate node relationships. The researchers explained that instead of merely matching features one-to-one, G-CRD accentuates similarities and differences across the entire network to bolster learning efficiency.

“Teacher and student embeddings belonging to the same node were encouraged to be pulled closer, while those from different nodes were pushed apart,” the team commented.

The researchers assessed G-CRD by applying it across various GNN architectures and datasets, including large-scale networks, batches of small graphs and 3-D point clouds, focusing on real-world applications where data might be noisy and constantly evolving. They compared G-CRD with existing distillation techniques for node classification and molecular graph property prediction.

Their findings revealed that G-CRD surpassed existing methods of preserving local structure by effectively capturing both immediate and broader network connections, thereby significantly enhancing robustness and performance. G-CRD emerged as a faster, more accurate means of preserving complex relationships, even with noisy data or in resource-constrained environments.

“Lightweight and efficient GNN models trained with our distillation techniques may be useful for deployment on edge devices or in scenarios where resources are limited,” the researchers explained. “For instance, the low GPU memory consumption and fast inference speed of smaller models can be very useful for resource-constrained robotics applications.”

The introduction of G-CRD signifies a significant advancement in resource-efficient artificial intelligence research, offering a more nuanced and effective approach for smaller models to learn intricate patterns, with far-reaching implications for technology advancement in areas such as drug discovery, autonomous vehicle navigation and beyond.

The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research (I2R).

Want to stay up to date with breakthroughs from A*STAR? Follow us on Twitter and LinkedIn!


Joshi, C.K., Liu, F., Xu, X., Lin, J. and Foo, C.S. On representation knowledge distillation for graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 1-12 (2022). | article

About the Researchers

Fayao Liu is a Research Scientist at the Institute for Infocomm Research (I2R), A*STAR. She received her PhD degree in computer science from the University of Adelaide, Australia in 2015. She works mainly on machine learning and computer vision problems, with interests in scene understanding and 3D generative learning. She is an honoree of the Singapore 100 Women in Tech (SG100WiT) 2023.
Xun Xu is a Senior Scientist at the Institute for Infocomm Research (I2R), A*STAR. He received his PhD degree from Queen Mary University of London in 2016. He was a research fellow at the National University of Singapore between 2016 and 2019. He is now an IEEE senior member. His research interests include semi-supervised learning, domain adaptation and zero-shot learning with applications to 3D point cloud data.
View articles

Chuan-Sheng Foo

Assistant Head of Department (Research), Machine Intellection Department

Institute for Infocomm Research (I2R)
Chuan-Sheng Foo is the Assistant Head of Department (Research), Machine Intellection Department at the Institute for Infocomm Research (I2R), and an Investigator at the Centre for Frontier AI Research (CFAR), A*STAR. His research focuses on data-related challenges in machine learning: data-efficient learning, learning models robust to data shifts and incentivising data sharing. His work is inspired by applications in engineering, manufacturing and healthcare where many of these challenges arise. He received BS, MS and PhD degrees in Computer Science from Stanford University in the US.

This article was made for A*STAR Research by Wildtype Media Group