Anomaly detection

Part of a series on
Machine learning and data mining
hide Problems Classification Clustering Regression Anomaly detection Data Cleaning AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
show Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
show Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean shift
show Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE
show Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
show Anomaly detection k-NN Local outlier factor
show Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Multilayer perceptron RNN LSTM GRU ESN Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net Transformer Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
show Reinforcement learning Q-learning SARSA Temporal difference (TD)
show Theory Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
show Machine-learning venues NeurIPS ICML ML JMLR ArXiv:cs.LG
show Related articles Glossary of artificial intelligence List of datasets for machine-learning research Outline of machine learning
v t

Part of a series on
Information security
Related security categories
Automotive security Cybercrime Cybersex trafficking Computer fraud Cybergeddon Cyberterrorism Cyberwarfare Electronic warfare Information warfare Internet security Mobile security Network security Copy protection Digital rights management
Threats
Adware Advanced persistent threat Arbitrary code execution Backdoors Hardware backdoors Code injection Crimeware Cross-site scripting Cryptojacking malware Botnets Data breach Drive-by download browser helper objects Computer crime Viruses Data scraping Denial of service Eavesdropping Email fraud Email spoofing Exploits Keyloggers Logic bombs Time bombs Fork bombs Zip bombs Fraudulent dialers Malware Payload Phishing Polymorphic engine Privilege escalation Ransomware Rootkits Bootkits Scareware Shellcode Spamming Social engineering (security) Screen scraping Spyware Software bugs Trojan horses Hardware Trojans Remote access trojans Vulnerability Web shells Wiper Worms SQL injection Rogue security software Zombie
Defenses
Application security Secure coding Secure by default Secure by design Misuse case Computer access control Authentication Multi-factor authentication Authorization Computer security software Antivirus software Security-focused operating system Data-centric security Code obfuscation Encryption Firewall Intrusion detection system Host-based intrusion detection system (HIDS) Anomaly detection Security information and event management (SIEM) Mobile secure gateway Runtime application self-protection
v t

In data analysis, anomaly detection (also outlier detection)^[1] is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.^[1] Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions.^[2]

In particular, in the context of abuse and network intrusion detection, the interesting objects are often not rare objects, but unexpected bursts in activity. This pattern does not adhere to the common statistical definition of an outlier as a rare object, and many outlier detection methods (in particular unsupervised methods) will fail on such data, unless it has been aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro clusters formed by these patterns.^[3]

Three broad categories of anomaly detection techniques exist.^[4] Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal by looking for instances that seem to fit least to the remainder of the data set. Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier (the key difference to many other statistical classification problems is the inherent unbalanced nature of outlier detection). Semi-supervised anomaly detection techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance to be generated by the utilized model.

Applications[]

Anomaly detection is applicable in a variety of domains, such as intrusion detection, fraud detection, fault detection, system health monitoring, event detection in sensor networks, detecting ecosystem disturbances, and defect detection in images using machine vision.^[5] It is often used in preprocessing to remove anomalous data from the dataset. In supervised learning, removing the anomalous data from the dataset often results in a statistically significant increase in accuracy.^[6][7]

Popular techniques[]

Several anomaly detection techniques have been proposed in literature.^[8] Some of the popular techniques are:

• Density-based techniques (k-nearest neighbor,^[9][10][11] local outlier factor,^[12] isolation forests,^[13][14] and many more variations of this concept^[15]).
• Subspace-,^[16] correlation-based^[17] and tensor-based ^[18] outlier detection for high-dimensional data.^[19]
• One-class support vector machines.^[20]
• Replicator neural networks.,^[21] autoencoders, variational autoencoders,^[22] long short-term memory neural networks^[23]
• Bayesian networks.^[21]
• Hidden Markov models (HMMs).^[21]
• Cluster analysis-based outlier detection.^[24][25]
• Deviations from association rules and frequent itemsets.
• Fuzzy logic-based outlier detection.
• Ensemble techniques, using feature bagging,^[26][27] score normalization^[28][29] and different sources of diversity.^[30][31]

The performance of different methods depends a lot on the data set and parameters, and methods have little systematic advantages over another when compared across many data sets and parameters.^[32][33]

Application to data security[]

Anomaly detection was proposed for intrusion detection systems (IDS) by Dorothy Denning in 1986.^[34] Anomaly detection for IDS is normally accomplished with thresholds and statistics, but can also be done with soft computing, and inductive learning.^[35] Types of statistics proposed by 1999 included profiles of users, workstations, networks, remote hosts, groups of users, and programs based on frequencies, means, variances, covariances, and standard deviations.^[36] The counterpart of anomaly detection in intrusion detection is misuse detection.

In data pre-processing[]

In supervised learning, anomaly detection is often an important step in data pre-processing to provide the learning algorithm a proper dataset to learn on. This is also known as Data cleansing. After detecting anomalous samples classifiers remove them, however, at times corrupted data can still provide useful samples for learning. A common method for finding appropriate samples to use is identifying Noisy data. One approach to find noisy values is to create a probabilistic model from data using models of uncorrupted data and corrupted data.^[37]

Below is an example of the Iris flower data set with an anomaly added. With an anomaly included, classification algorithm may have difficulties properly finding patterns, or run into errors.

By removing the anomaly, training will be enabled to find patterns in classifications more easily.

In data mining, high-dimensional data will also propose high computing challenges with intensely large sets of data. By removing numerous samples that can find itself irrelevant to a classifier or detection algorithm, runtime can be significantly reduced on even the largest sets of data.

Software[]

• ELKI is an open-source Java data mining toolkit that contains several anomaly detection algorithms, as well as index acceleration for them.
• Scikit-Learn is an open-source Python library that has built functionality to provide unsupervised anomaly detection.

Datasets[]

• Anomaly detection benchmark data repository of the Ludwig-Maximilians-Universität München; Mirror at University of São Paulo.
• ODDS – ODDS: A large collection of publicly available outlier detection datasets with ground truth in different domains.
• Unsupervised Anomaly Detection Benchmark at Harvard Dataverse: Datasets for Unsupervised Anomaly Detection with ground truth.
• KMASH Data Repository at Research Data Australia having more than 12,000 anomaly detection datasets with ground truth.

See also[]

• Change detection
• Statistical process control
• Novelty detection
• Hierarchical temporal memory

References[]