Awesome-AutoML-Papers

Awesome-AutoML-Papers is a curated list of automated machine learning papers, articles, tutorials, slides and projects.

What is AutoML?

Automated Machine Learning (AutoML) provides methods and processes to make Machine Learning available for non-Machine Learning experts, to improve efficiency of Machine Learning and to accelerate research on Machine Learning.

Machine Learning (ML) has achieved considerable successes in recent years and an ever-growing number of disciplines rely on it. However, this success crucially relies on human machine learning experts to perform the following tasks:

Preprocess the data,
Select appropriate features,
Select an appropriate model family,
Optimize model hyperparameters,
Postprocess machine learning models,
Critically analyze the results obtained.

As the complexity of these tasks is often beyond non-ML-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML. As a new sub-area in machine learning, AutoML has got more attention not only in machine learning but also in computer vision, natural language processing and graph computing.

There are no formal definition of AutoML. From the descriptions of most papers，the basic procedure of AutoML can be shown as the following.

AutoML approaches are already mature enough to rival and sometimes even outperform human machine learning experts. Put simply, AutoML can lead to improved performance while saving substantial amounts of time and money, as machine learning experts are both hard to find and expensive. As a result, commercial interest in AutoML has grown dramatically in recent years, and several major tech companies and start-up companies are now developing their own AutoML systems. An overview comparison of some of them can be summarized to the following table.

Company	AutoFE	HPO	NAS
4paradigm	√	√	×
Alibaba	×	√	×
Baidu	×	×	√
Google	√	√	√
H2O.ai	√	√	×
Microsoft	×	√	√
RapidMiner	√	√	×
Tencent	×	√	×
Transwarp	√	√	√

Awesome-AutoML-Papers includes very up-to-date overviews of the bread-and-butter techniques we need in AutoML:

Automated Data Clean (Auto Clean)
Automated Feature Enginnering (Auto FE)
Hyperparameter Optimization (HPO)
Meta-Learning
Neural Architecture Search (NAS)

Papers
Tutorials
- Bayesian Optimization
- Meta Learning
Articles
- Bayesian Optimization
- Meta Learning
Slides
- Bayesian Optimization
Books
- Meta Learning
Projects
Prominent Researchers

Papers

Surveys

2019 AutoML: A Survey of the State-of-the-Art Xin He, et al. arXiv PDF
2019 Survey on Automated Machine Learning Marc Zoeller, Marco F. Huber arXiv PDF
2019 Automated Machine Learning: State-of-The-Art and Open Challenges Radwa Elshawi, et al. arXiv PDF

2018 Taking Human out of Learning Applications: A Survey on Automated Machine Learning Quanming Yao, et al. arXiv PDF

Automated Feature Engineering

Expand Reduce

2017 AutoLearn — Automated Feature Generation and Selection Ambika Kaul, et al. ICDM PDF

2017 One button machine for automating feature engineering in relational databases Hoang Thanh Lam, et al. arXiv PDF

2016 Automating Feature Engineering Udayan Khurana, et al. NIPS PDF

2016 ExploreKit: Automatic Feature Generation and Selection Gilad Katz, et al. ICDM PDF

2015 Deep Feature Synthesis: Towards Automating Data Science Endeavors James Max Kanter, Kalyan Veeramachaneni DSAA PDF

Hierarchical Organization of Transformations

2016 Cognito: Automated Feature Engineering for Supervised Learning Udayan Khurana, et al. ICDMW PDF

Meta Learning

2017 Learning Feature Engineering for Classification Fatemeh Nargesian, et al. IJCAI PDF

Reinforcement Learning

2017 Feature Engineering for Predictive Modeling using Reinforcement Learning Udayan Khurana, et al. arXiv PDF

2010 | Feature Selection as a One-Player Game | Romaric Gaudel, Michele Sebag | ICML | PDF
Architecture Search

Evolutionary Algorithms

2019 Evolutionary Neural AutoML for Deep Learning Jason Liang, et al. arXiv PDF

2017 Large-Scale Evolution of Image Classifiers Esteban Real, et al. PMLR PDF

2002 Evolving Neural Networks through Augmenting Topologies Kenneth O.Stanley, Risto Miikkulainen Evolutionary Computation PDF

Local Search

2017 Simple and Efficient Architecture Search for Convolutional Neural Networks Thomoas Elsken, et al. ICLR PDF

Meta Learning
- 2016 Learning to Optimize Ke Li, Jitendra Malik arXiv PDF

Reinforcement Learning

2018 AMC: AutoML for Model Compression and Acceleration on Mobile Devices Yihui He, et al. ECCV PDF

2018 Efficient Neural Architecture Search via Parameter Sharing Hieu Pham, et al. arXiv PDF

2017 Neural Architecture Search with Reinforcement Learning Barret Zoph, Quoc V. Le ICLR PDF

Transfer Learning

2017 Learning Transferable Architectures for Scalable Image Recognition Barret Zoph, et al. arXiv PDF

Network Morphism

2018 Efficient Neural Architecture Search with Network Morphism Haifeng Jin, et al. arXiv PDF

Continuous Optimization

2018 Neural Architecture Optimization Renqian Luo, et al. arXiv PDF

2019 DARTS: Differentiable Architecture Search Hanxiao Liu, et al. ICLR PDF

Frameworks

2019 Towards modular and programmable architecture search Renato Negrinho, et al. NeurIPS PDF
2019 Evolutionary Neural AutoML for Deep Learning Jason Liang, et al. arXiv PDF

2017 ATM: A Distributed, Collaborative, Scalable System for Automated Machine Learning T. Swearingen, et al. IEEE PDF

2017 Google Vizier: A Service for Black-Box Optimization Daniel Golovin, et al. KDD PDF
2015 AutoCompete: A Framework for Machine Learning Competitions Abhishek Thakur, et al. ICML PDF

Hyperparameter Optimization

Bayesian Optimization

2018 A Tutorial on Bayesian Optimization. PDF

2018 Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features Mojmír Mutný, et al. NeurIPS PDF

2018 High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups. PMLR PDF

2016 Bayesian Optimization with Robust Bayesian Neural Networks Jost Tobias Springenberg， et al. NIPS PDF

2016 Scalable Hyperparameter Optimization with Products of Gaussian Process Experts Nicolas Schilling, et al. PKDD PDF

2016 Taking the Human Out of the Loop: A Review of Bayesian Optimization Bobak Shahriari, et al. IEEE PDF

2016 Towards Automatically-Tuned Neural Networks Hector Mendoza, et al. JMLR PDF

2016 Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization Martin Wistuba, et al. PKDD PDF

2015 Efficient and Robust Automated Machine Learning PDF

2015 Hyperparameter Optimization with Factorized Multilayer Perceptrons Nicolas Schilling, et al. PKDD PDF

2015 Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization Martin Wistua, et al. PDF

2015 Joint Model Choice and Hyperparameter Optimization with Factorized Multilayer Perceptrons Nicolas Schilling, et al. ICTAI PDF

2015 Learning Hyperparameter Optimization Initializations Martin Wistuba, et al. DSAA PDF

2015 Scalable Bayesian optimization using deep neural networks Jasper Snoek, et al. ACM PDF

2015 Sequential Model-free Hyperparameter Tuning Martin Wistuba, et al. ICDM PDF

2013 Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms PDF

2013 Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures J. Bergstra JMLR PDF

2012 Practical Bayesian Optimization of Machine Learning Algorithms PDF
2011 Sequential Model-Based Optimization for General Algorithm Configuration(extended version) PDF

Evolutionary Algorithms

2018 Autostacker: A Compositional Evolutionary Learning System Boyuan Chen, et al. arXiv PDF

2017 Large-Scale Evolution of Image Classifiers Esteban Real, et al. PMLR PDF

2016 Automating biomedical data science through tree-based pipeline optimization Randal S. Olson, et al. ECAL PDF

2016 Evaluation of a tree-based pipeline optimization tool for automating data science Randal S. Olson, et al. GECCO PDF

Lipschitz Functions

2017 Global Optimization of Lipschitz functions C´edric Malherbe, Nicolas Vayatis arXiv PDF

Local Search

2009 ParamILS: An Automatic Algorithm Configuration Framework Frank Hutter, et al. JAIR PDF

Meta Learning

2008 Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection PDF

2019 SMARTML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning for Machine Learning Algorithms PDF

Particle Swarm Optimization

2017 Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks Pablo Ribalta Lorenzo, et al. GECCO PDF

2008 Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines Shih-Wei Lin, et al. Expert Systems with Applications PDF

Random Search

2016 Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Lisha Li, et al. arXiv PDF

2012 Random Search for Hyper-Parameter Optimization James Bergstra, Yoshua Bengio JMLR PDF

2011 Algorithms for Hyper-parameter Optimization James Bergstra, et al. NIPS PDF

Transfer Learning

2016 Efficient Transfer Learning Method for Automatic Hyperparameter Tuning Dani Yogatama, Gideon Mann JMLR PDF

2016 Flexible Transfer Learning Framework for Bayesian Optimisation Tinu Theckel Joy, et al. PAKDD PDF

2016 Hyperparameter Optimization Machines Martin Wistuba, et al. DSAA PDF

2013 Collaborative Hyperparameter Tuning R´emi Bardenet, et al. ICML PDF

Miscellaneous

2018 Accelerating Neural Architecture Search using Performance Prediction Bowen Baker, et al. ICLR PDF
2017 Automatic Frankensteining: Creating Complex Ensembles Autonomously Martin Wistuba, et al. SIAM PDF

Tutorials

Bayesian Optimization

2010 | A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning | PDF
Meta Learning
2008 Metalearning - A Tutorial PDF

Blog

Books

Projects

Slides

AutoML领域的最新论文合集

Awesome-AutoML-Papers

What is AutoML?

Table of Contents

Papers

Surveys

Automated Feature Engineering

Expand Reduce

Hierarchical Organization of Transformations

Meta Learning

Reinforcement Learning

Architecture Search

Evolutionary Algorithms

Local Search

Meta Learning

Reinforcement Learning

Transfer Learning

Network Morphism

Continuous Optimization

Frameworks

Hyperparameter Optimization

Bayesian Optimization

Evolutionary Algorithms

Lipschitz Functions

Local Search

Meta Learning

Particle Swarm Optimization

Random Search

Transfer Learning

Miscellaneous

Tutorials

Bayesian Optimization

Meta Learning

Blog

Books

Projects

Slides

FEATURED TAGS

FRIENDS