Feature engineering

Part of a series on
Machine learning and data mining
hide Problems Classification Clustering Regression Anomaly detection Data Cleaning AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
show Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
show Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean shift
show Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE
show Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
show Anomaly detection k-NN Local outlier factor
show Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Multilayer perceptron RNN LSTM GRU ESN Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net Transformer Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
show Reinforcement learning Q-learning SARSA Temporal difference (TD)
show Theory Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
show Machine-learning venues NeurIPS ICML ML JMLR ArXiv:cs.LG
show Related articles Glossary of artificial intelligence List of datasets for machine-learning research Outline of machine learning
v t

Feature engineering is the process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data.^[1]

A feature is a property shared by independent units on which analysis or prediction is to be done.^[2]

Features are used by predictive models and influence results.^[3]

Feature engineering has been employed in Kaggle competitions^[4] and machine learning projects.^[5]

Process[]

The feature engineering process is:^[6]

• Brainstorming or testing features;^[7]
• Deciding what features to create;
• Creating features;
• Testing the impact of the identified features on the task;
• Improving your features if needed;
• Repeat.

Relevance[]

Features vary in significance.^[8] Even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent a model from becoming too specific to the training data set (overfitting).^[9]

Explosion[]

Feature explosion occurs when the number of identified features grows inappropriately. Common causes include:

• Feature templates - implementing feature templates instead of coding new features
• Feature combinations - combinations that cannot be represented by a linear system

Feature explosion can be limited via techniques such as: regularization, kernel method, and feature selection.^[10]

Automation[]

Automation of feature engineering is a research topic that dates back to the 1990s.^[11] Machine learning software that incorporates automated feature engineering has been commercially available since 2016.^[12] Related academic literature can be roughly separated into two types:

• Multi-relational decision tree learning (MRDTL) uses a supervised algorithm that is similar to a decision tree.
• Deep Feature Synthesis uses simpler methods.^{[citation needed]}

MRDTL generates features in the form of SQL queries by successively adding clauses to the queries.^[13] For instance, the algorithm might start out with

SELECT COUNT(*) FROM ATOM t1 LEFT JOIN MOLECULE t2 ON t1.mol_id = t2.mol_id GROUP BY t1.mol_id

The query can then successively be refined by adding conditions, such as "WHERE t1.charge <= -0.392".^[14]

However, most MRDTL studies base implementations on relational databases, which results in many redundant operations. These redundancies can be reduced by using tricks such as tuple id propagation.^[15][16] Efficiency can be increased by using incremental updates, which eliminates redundancies.^[17]

Deep Feature Synthesis[]

The Deep Feature Synthesis algorithm beat 615 of 906 human teams in a competition.^[18][19]

Libraries:

• Featuretools.^[20]
• OneBM^[21]
• ExploreKit.^[22]

[OneBM] helps data scientists reduce data exploration time allowing them to try and error many ideas in short time. On the other hand, it enables non-experts, who are not familiar with data science, to quickly extract value from their data with a little effort, time, and cost.^[23]

Feature stores[]

This section contains content that is written like a disguised advert; Feast is maintained by former Uber engineers that where working on Michelangelo; they are effectively advertising themselves. Please help by removing promotional content and inappropriate external links, and by adding encyclopedic content written from a neutral point of view. (June 2021) (Learn how and when to remove this template message)

A feature store includes the ability to store code used to generate features, apply the code to raw data, and serve those features to models upon request. Useful capabilities include feature versioning and policies governing the circumstances under which features can be used.^[24]

Feature stores can be standalone software tools or built into machine learning platforms. For example, Feast^[25] is an open source feature store, while platforms like Uber's Michelangelo use feature stores as a component.^[26]

See also[]

• Covariate
• Data transformation
• Feature extraction
• Feature learning
• Hashing trick
• Kernel method
• List of datasets for machine learning research
• Space mapping

References[]

Further reading[]

• Boehmke, Bradley; Greenwell, Brandon (2019). "Feature & Target Engineering". Hands-On Machine Learning with R. Chapman & Hall. pp. 41–75. ISBN 978-1-138-49568-5.
• Zheng, Alice; Casari, Amanda (2018). Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O'Reilly. ISBN 978-1-4919-5324-2.
• Zumel, Nina; Mount, John (2020). "Data Engineering and Data Shaping". Practical Data Science with R (2nd ed.). Manning. pp. 113–160. ISBN 978-1-61729-587-4.