Highlights

In brief

The new findings could make cyber-physical systems (such as self-driving cars) safer and more reliable by improving how they identify errors.

© Unsplash

What is the best way to detect errors in intelligent systems?

19 Jul 2022

A systematic comparison of different existing methods reveals best practices for anomaly detection and diagnosis in cyber-physical systems.

The self-driving car is one of the most iconic technological breakthroughs of the 20th century. By getting passengers to their destinations with minimal human input, self-driving cars have the potential to revolutionise urban transport, but they are not without risks.

Self-driving cars are a form of cyber-physical systems: computer algorithms that control and learn from mechanical components. Hence, system anomalies need to be continuously monitored, diagnosed and rectified in real-time to avoid component failures, or worse, crashes.

There are several proposed methods of anomaly detection; however, until now, researchers have yet to perform a systematic, head-to-head comparison of these various techniques. Consequently, identifying the best anomaly detection models for specific applications remains an uphill task.

Astha Garg, who was at A*STAR’s Institute for Infocomm Research (I2R), led a team of researchers, which included Zhang Wenyu, currently a Research Scientist at the institute, to fill this gap. The researchers tested 45 unique anomaly detection methods using data inputs from seven publicly available datasets. They also performed anomaly diagnosis using 29 techniques on four datasets, representing the largest and most comprehensive review of anomaly assessment algorithms to date.

The team took a fresh approach to their analysis, breaking down the detection and diagnosis methods into three distinct modules. The first was a reconstruction model that predicts upcoming algorithmic errors, next, a function that aggregates errors into an overall anomaly score, and finally, a thresholding function that determines a pass or a fail based on the score.

“By decomposing methods into the modular framework, we can investigate the effect of independent choices of each module,” said Zhang, adding that this approach also allowed them to propose optimal design choices for anomaly detection and diagnosis.

Some modules proved to be more impactful than others. For example, in four of the seven tested datasets for anomaly detection, the scoring function had a bigger effect on detection performance than the reconstruction model. In particular, dynamic scoring functions, which continually adapt to unforeseen events, performed better than static ones.

The researchers found that a simple model, called the univariate fully connected autoencoder (UAE), had the best predictive performance overall, outperforming other models in five of the seven anomaly detection datasets.

“We find that existing evaluation metrics for event-wise anomaly detection can be misleading and propose a new metric that accounts for event-wise recall and point-wise precision,” said Zhang, adding that their work sets the stage for the safe and reliable cyber-physical systems of tomorrow.

The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research (I2R).

Want to stay up to date with breakthroughs from A*STAR? Follow us on Twitter and LinkedIn!

References

Garg, A., Zhang, W., Samaran, J., Savitha, R., and Foo, C.S. An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series. IEEE Transactions on Neural Networks and Learning Systems 33, 2508-2517 (2022) | article

About the Researcher

Wenyu is currently a Research Scientist at the Institute for Infocomm Research, working on robustness in deep learning, and time series prediction and anomaly detection. She is broadly interested in model robustness, and developing statistical and machine learning methodology for sequential data.

This article was made for A*STAR Research by Wildtype Media Group