Anomaly Detection

Anomaly detection is identifying rare events or observations which do not conform with the general pattern of distribution of the population of events in the data. An anomaly raises the suspicion that there is something wrong with the underlying business process, which is generating this data.

Applications of Anomaly Detection in Financial Services

Due to recent advances towards industry 4.0 and machine learning, most business processes are going through automation to lower the cost of doing business, optimize capacity, and enhance efficiency. An automated process can be real-time or near real-time. It may be processing hundreds or thousands of events during a specific timeframe.

In the financial services industry, we can build anomaly detection models around business processes such as;

For the sake of reputation and risk management, a business must keep monitoring any anomalous events which a business process may encounter.

3 Types of Anomalies

In business settings, we typically have to deal with three types of anomaly detection use cases:

1. Point or Global Anomaly

Point or global anomaly refers to an individual event that is rare and stands out concerning the rest of the data. In the banking and finance industry, a significant transaction by an 80 years old retired person or by a college student can be a global anomaly.

2. Contextual Anomaly

Contextual anomaly is a rare event in a specific context. For instance, a spike in the number of bank transactions over the weekend, or 80 years old person applying for (undergraduate) student loan.

3. Collective Anomaly

Collective anomaly occurs when a set of events or observations are collectively causing significant deviating from the remainder of the data. For example, a drop in stock prices amid the COVID-19 crisis.

Methods of Anomaly Detection

There are two main types of anomaly detection methods:

1. Unsupervised Anomaly Detection

In these methods, we use unsupervised machine learning algorithms. Unsupervised machine learning models, for example, K-Means, are trained on unlabeled data sets. In the unlabeled data set, the events or observations do not have 'normal' or 'anomaly' labels.

Though majority events in a typical data set are usually normal events, learning whether an (unlabeled) event belongs to normal or anomaly is a difficult problem. Simple clustering algorithms work well in low dimensional spaces. In machine learning, high dimensional space problems often require advance unsupervised learning methods.

2. Supervised Anomaly Detection

Supervised anomaly detection relies on supervised machine learning algorithms such as logistic regression, KNN, decision trees, neural networks. These algorithms get trained on labeled training data and deployed to predict an anomaly (around business process) in the future. Often the challenge with the training of these algorithms is unbalanced data set of anomaly detection problems. As we know, the anomalies are rare events; data sets for anomaly detection often have an unbalanced distribution with a majority of negative examples.

How Can Datalya Help with Implementation of Anomaly Detection?

Using machine learning, at Datalya, we have implemented a handful of anomaly detection solutions in banks, insurance companies, credit card institutions, energy, and telecommunication businesses. If you have any questions or need help, please do not hesitate to contact us through email or phone.