With developing brains and curious minds, children learn new languages more easily than most adults. In their first few years, however, most children struggle with linguistic nuances, such as how a noun can point to different identities—a concept known as co-referencing.
Nancy Chen, an A*STAR Fellow, Senior Principal Scientist, Lead Principal Investigator and Group Leader at A*STAR’s Institute for Infocomm Research (I2R), offers an illustration. “My son identifies himself as ‘didi’ (‘younger brother’ in Mandarin Chinese), but he might be confused if he’s told that his mum has a didi who is not him,” said Chen, who leads a team working on generative artificial intelligence (AI) and natural language generation in I2R's Aural and Language Intelligence (ALI) Department.
For computers, conversations can be equally confusing as they involve speakers who make references from their own perspectives, which may not line up with previously encoded information. Over time, children naturally learn how to intuitively interpret such linguistic nuances under different contexts. Computers, on the other hand, still lack the ability to contextualise well even with major advancements in AI, Chen added.
The decades-long quest to help machines comprehend human languages and use them in meaningful ways spans various disciplines, from computer science to linguistics to statistics. These fields are drawn under the umbrella of natural language processing (NLP), a branch of AI that aims to facilitate human-machine communication.
Since the 1950s, NLP has gone through several stages of evolution. Symbolic NLP was the first to enter the scene, using a rules-based approach that matched questions with programmed answers. The 1990s saw a shift to data-driven statistical techniques in machine learning (ML); today, deep learning approaches have led to large language models (LLMs) like ChatGPT, which mimic human language with more sophistication.
“We have witnessed technologies scale from supporting very narrow domain-specific applications to LLMs trained with huge datasets to solve a wide range of NLP tasks,” said Ai Ti Aw, Head of I2R's ALI Department. “All these are made possible with advances in computational resources.”
At A*STAR, NLP technologies have also evolved over time, supporting various applications from speech-to-text transcription and language translation, to information extraction and virtual chatbots. Research at A*STAR has played a prominent role in bridging the human-computer communication divide, revolutionising how we relate to machines.
AI's building blocks
Many technical leaps fuelling today’s deep learning revolution came from lessons learned in natural language generation, said Chen. “These developments, which were first exploited in LLMs, are making rapid inroads into other fields in ML, accelerating the evolution of generative AI and foundation models."
A few developments integral to NLP work at A*STAR include:
Transformers: State-of-the-art neural network architectures with self-attention mechanisms. These allow Al models to learn context by tracking how different parts of a data series (eg, the different words in a sentence) influence each other. Transformers can tokenise these elements in parallel, speeding up the learning process versus older step-by-step models.
Self-supervised learning(SSL): ML approaches where models are trained to predict a section of data based on another section (e.g., completing a sentence based on a few starting words). SSL allows a model to develop ’rules’ based on the structures it observes in large datasets, then apply those rules to its own further learning, reducing the need for well-labelled training data or explicit algorithms.
Representation learning: ML approaches where models identify meaningful patterns from a dataset to create more easily understood representations (e.g., ’red’ and ’orange’ are ‘colour names’). Often used with SSL, representation learning allows models to apply knowledge from one task to another related one (e.g., from classifying colour names to retrieving them), making it easier to learn new tasks.
Decoding more than words
Machine translation is a key NLP application which uses algorithms to convert text from one language to another. With the emergence of AI and ML, machine translation tools are no longer limited to simple word substitution, but can accurately translate jargon, slang and culturally-derived expressions into a target language. This progress has led to their widespread adoption and usage in various domains.
One of Singapore’s major achievements in machine translation through I2R is SGTranslate, an online engine that allows users to translate the country’s four commonly used languages—English, Malay, Mandarin Chinese and Tamil—while maintaining their local context.
“Singapore is a melting pot of diverse cultures and different languages,” explained Aw, a pioneering NLP researcher in Singapore and co-developer of SGTranslate. “Machine translation plays a critical role in helping people from different linguistic backgrounds to understand each other and read media in other languages.”
A collaborative project with the Ministry of Communications and Information, SGTranslate was supported by the Translational Research and Development for Application to Smart Nation (TRANS) Grant to help develop and deliver a localised translation engine. SGTranslate also aims to aid public officers in democratising public communication materials for Singaporeans across multiple languages.
“Our unique capabilities in understanding and processing Singaporean and Southeast Asian languages set us apart from other global players, giving us a competitive edge in deploying our technologies,” said Sumei Sun, I2R Acting Executive Director. Sun added that I2R is working with ecosystem partners in the LLM initiative to build national-level capabilities in multimodal foundation models.
Processing healthcare
At A*STAR’s Institute of High Performance Computing (IHPC), NLP is making waves in healthcare applications. One major work is the Radiology Pathology Information Exchange Resource (RAPIER), which processes medical scans and reports through AI to detect and diagnose liver abnormalities.
Using NLP, RAPIER extracts key information from clinical reports and automatically annotates radiologic and pathologic images, relieving clinicians from previously tedious tasks. “This project has the potential to convert previously untapped medical material into AI-ready data,” explained Rick Goh, Director of IHPC’s Computing and Intelligence (CI) Department. “Besides hepatology, many other medical specialties can benefit from this technology.”
Led by Yong Liu, IHPC CI Deputy Director, and developed with partners from Singapore General Hospital (SGH) and the National Cancer Centre Singapore, RAPIER was awarded the AI Singapore (AISG) Open Theme Technology Challenge grant in 2022.
“By harnessing multimodal inputs from medical images and reports, we plan to develop AI systems that can comprehend and reason with intricate medical knowledge, going beyond just pattern recognition,” said Goh. The team aims to harness such capabilities for an AI system that will answer patient queries and explain their medical scans and reports, effectively providing doctors more time directly with patients, Goh added.
In another collaboration with SGH, Feng Yang, an IHPC Principal Scientist, leads the Safe Medication Management Platform Augmented by ARTificial Intelligence for Prescribers [Rx] (SmartRx) project, which aims to enhance medication safety for improved patient care. Awarded the AISG Technology Challenge grant in 2022, SmartRx employs NLP to extract and process medical information for the detection and prediction of potential safety issues, such as medication errors and adverse drug reactions.
Machine educators
NLP is transforming the way we communicate not only with machines, but also with each other. A key sector feeling the boost from NLP-powered tools is education, where personalised AI tutoring and automatic grading are changing practices for teachers and schools.
At I2R, Chen’s team works on using deep learning approaches to fuel natural language generation, with applications in report generation, meeting summarisation and virtual tutors. One project underway to support educators is an automated audio highlighter that allows a virtual tutor’s voice to attract more cognitive attention from students.
“This tech can be applied in many areas. For example, you can use it to pinpoint a student’s errors and provide suggestive, highlighted audio feedback, making it easier for them to discover errors on their own and self-correct,” said Chen, who is leading AI for Education initiatives and driving research in controllable neural generation at A*STAR.
By fostering a self-discovery approach to learning, the tech can bring long-term benefits for learners. “We believe it’s a more effective approach than explicitly telling a student when there’s a mistake; it makes them a more active participant in the learning process,” Chen added.
The utility of digital learning tools came to the fore amid the 2019 SARS-CoV-2 pandemic, as many schools switched to online learning. In 2021, I2R worked with CommonTown, an A*STAR spinoff, to deploy an innovative AI speech evaluation system that supported home-based Malay and Tamil learning through the Ministry of Education’s Student Learning Space.
Following on this work, Chen’s team was recently awarded AISG’s AI in Education Grand Challenge grant to build multimodal, multilingual virtual tutors to converse and guide children aged 6-7 years with picture description tasks in Malay, Tamil and Mandarin Chinese.
“In the Singaporean context, we wanted to ensure that minority languages like Malay and Tamil could still be taught to our children,” said Chen, adding that their system allowed students to practise even outside the classroom setting and without direct teacher supervision.
The AI system evaluated and scored students’ reading abilities based on three dimensions: pronunciation, fluency and intonation. As with the automated audio highlighter, the tutor provided personalised feedback to help students self-correct. “The use of self-supervised representation learning and controllable neural generation techniques helped enhance the model’s accuracy and robustness,” added Sun.
English versions of this technology were deployed at the Singapore Examination and Assessment Board in 2022 upon completion of a TRANS Grant. They also led to an A*STAR spinoff, Nomopai, which focuses on providing communication companion services.
NLP innovation through partnership
Knowledge transfer at A*STAR is largely supported and led by the Innovation & Enterprise (I&E) Division, which aims to create synergies between the research community and industry players. To this end, I&E works with stakeholders within and beyond the agency to coordinate activities across the entire innovation ecosystem, from R&D’s early stages through to commercialisation.
“Basically, we take charge of the commercial pathways to bring tech to market,” said Yee Chia Yeo, Assistant Chief Executive (Designate) of the A*STAR I&E Division. “In parallel, we are aligned with government efforts to build a healthy pipeline of Singapore-based startups, and to scale and launch their products into global markets.”
Through strategic public-private partnerships, A*STAR is accelerating the translation of NLP technologies into new products and services. These include the Sentiment Analysis for Public Transport (SAPT) project, a collaboration between IHPC and Singapore’s Public Transport Council, to develop a web-based tool to rapidly gauge commuter sentiments on social media—even when layered with sarcasm or colloquial English (‘Singlish').
“SAPT utilised IHPC’s Resonance Social and CrystalFeel technologies, which played a key role in the project’s success,” said Yeo, adding that SAPT won the 2022 Minister’s Innovation Award (Merit) from the Ministry of Transport.
I&E also works closely with A*STAR research institutes to support startups with incubation and portfolio management. A case in point is Sentient.io, a 2017 A*STAR spinoff which provides an AI-as-a-service platform based on NLP tech licensed from I2R. With ready-to-use AI models, the platform helps software developers and enterprises quickly build smart and innovative applications.
“Sentient.io was the first spinoff formed via I&E’s Venture Co-Creation model in partnership with Origgin, a local investment firm,” said Yeo, adding that it raised over S$9 million in funding to date from international investors such as Asahi Broadcasting Group and Daido Steel.
Growing pains
Despite major advances in NLP models, several challenges remain both in their research and broader real-world adoption as ‘reliable, robust and fair systems’, according to Joey Tianyi Zhou, a Principal Investigator, Principal Scientist and Group Manager at A*STAR’s Centre for Frontier AI Research (CFAR).
“Firstly, NLP models often need vast amounts of carefully-labelled data to learn from, which costs time and resources,” said Zhou. “To overcome this, we’re exploring techniques like transfer learning and domain adaptation, which involve pre-training models on general knowledge and teaching them to apply their learning to a task or domain through smaller, more specific datasets.”
Such techniques reduce training time for new tasks and the need for extensive data labelling; they help models adapt faster and more efficiently to new tasks or domains, making them more cost-effective to deploy in low-resource scenarios, Zhou added.
Other concerns include data privacy and security, particularly for enterprises trying to integrate NLP and operationally sensitive data. “Currently, we’re researching techniques to make NLP models more resilient against adversarial attacks, which can come as carefully-crafted input modifications meant to deceive the model,” said Zhou. “This could lead to more secure NLP systems that can maintain consistent performance in various scenarios.”
Biases inherent in training data are also a significant challenge due to their potential for unintended harm. However, this can be mitigated through the thoughtful curation of training data and the development of debiasing techniques that foster diversity and inclusivity in NLP research and application, said Zhou.
For common meaning
NLP has emerged as a transformative force for how we interact with machines and each other. Moving forward, NLP is likely to more closely integrate with speech, vision and language into a single model driving the goal of artificial general intelligence (AGI), said Sun, adding, “Addressing security concerns on data leakage and protecting sensitive data will be of utmost importance for successful AGI model development.”
As the field advances and becomes more integrated into our world, resolving its associated ethical and safety concerns will become even more crucial. By fostering a collaborative approach among researchers, policymakers and stakeholders, A*STAR aims to ensure that NLP not only fulfils its potential but also prioritises the well-being of our societies.
“By harnessing their multidisciplinary capabilities, our leading institutes in this area can play key roles in addressing the ethical dimensions of AI; mitigating the disparity and biases in the use of NLP models; and enabling our collaborators with a greater capacity to innovate with equity and trust,” said Yeo.
