In brief

Reallocating underutilised attention heads to encode coreference resolution information significantly enhanced the model's dialogue summarisation capabilities while maintaining computational efficiency.

© Shutterstock

Summarising human conversations by intelligent machines

27 Mar 2024

Researchers developed a more efficient AI model for summarising conversations by innovatively enhancing underutilised computational components.

It seems like we’ve reached a pivotal moment in bridging the gap between complex human communication and computational understanding. Advances in Natural Language Processing (NLP) have spawned innovative platforms such as ChatGPT, which facilitate remarkably natural and seamless human-computer interactions.

At the heart of these NLP breakthroughs is a transformer-based model known for its ‘multi-head attention mechanism’. Zhengyuan Liu, a Lead Research Engineer at A*STAR’s Institute for Infocomm Research (I2R), explained that this works much like how our brain simultaneously processes different types of information—transformers focus on different parts of the input data at the same time, significantly enhancing context understanding and task efficiency.

However, in task-specific modelling like dialogue summarisation, the attention heads are not uniformly used. Liu, alongside A*STAR Senior Principal Scientist, Nancy Chen, developed a novel technique for repurposing these underused heads to infuse new capabilities into transformers and bump up their computational efficiencies.

“Redundant attention heads can be replaced with featured weights and it’s much more computationally efficient than introducing additional neural components,” explained Liu.

Their method involved training a base model to identify the attentive parts that were not contributing much during the task of summarising conversations. They then improved these underperforming parts by giving them additional information about how personal named entities in a conversation refer to each other; this helped the model to better understand the flow and context of the dialogue.

The researchers then experimented with a benchmark dataset and found that their enhanced transformer model not only improved upon the base model but also held its own against state-of-the-art models, all while being more computationally economical. Additionally, of the coreference information integration techniques tested, the nearest-neighbour approach proved superior.

In practical terms, this can lead to more effective summarisation of legal documents, medical records or customer service interactions, where both clarity and context are crucial.

“There are many directions that we are exploring,” said Liu, speaking on next steps. These include investigating the effectiveness of attention mechanisms in multiple modalities such as vision, speech and text, as well as improving the accountability of these models, which are crucial for sensitive applications such as healthcare.

The A*STAR-affiliated researchers contributing to this research are from the A*STAR’s Institute for Infocomm Research (I2R).

Want to stay up to date with breakthroughs from A*STAR? Follow us on Twitter and LinkedIn!


Liu Z. and Chen N.F. Picking the underused heads: a network pruning perspective of attention head selection for fusing dialogue coreference information. IEEE International Conference on Acoustics, Speech, and Signal Processing (2023). | article

About the Researchers

Zhengyuan Liu is currently a Lead Research Engineer at A*STAR’s Institute for Infocomm Research (I2R). His research interests include multilingual foundation models, conversational AI, controllable language generation, and AI for healthcare and education scenarios. He has published 30+ research papers in top-tier AI and Natural Language Processing conferences including ACL, NAACL, EMNLP, COLING, ICASSP and INTERSPEECH. He serves as the reviewer at conferences including NeurIPS, ICLR and ACL, and journals including IEEE TASLP, ACM CSUR and Neurocomputing. He has been elected as an IEEE senior member for his significant professional achievements, and won the Best Paper Award at SIGDIAL 2021 and the Outstanding Paper Award at EMNLP 2023.
View articles

Nancy F. Chen

Senior Principal Scientist and Group Leader

Institute for Infocomm Research (I2R)
Nancy F. Chen is an A*STAR fellow, Senior Principal Scientist, Principal Investigator and Group Leader at A*STAR’s Institute for Infocomm Research (I2R) and Principal Investigator at the Centre for Frontier AI Research (CFAR). Her group works on generative AI in speech, language and conversational technology. Her research has been applied to education, defence, healthcare and media/journalism. Chen has published 180+ papers and supervised 100+ students and staff. She has won awards from IEEE, Microsoft, NIH, P&G, UNESCO, L’Oréal, SIGDIAL, APSIPA and MICCAI. She is an IEEE SPS Distinguished Lecturer (2023-2024), Program Chair of ICLR 2023, Board Member of ISCA (2021-2025) and Singapore 100 Women in Tech (2021). Technology from her team has led to commercial spinoffs and government deployment.

This article was made for A*STAR Research by Wildtype Media Group