How Do AutoML Solutions Find The Optimal Machine Learning Models?

AutoML refers to techniques and tools which automate parts of the machine learning process, ranging from data collection and cleaning, to model development and testing, to production deployment and scaling. While the umbrella term does refer to a wide array of functionality, it’s most commonly used to refer to automated model selection and / or hyperparameter optimization.

As part of our enterprise AI education for business and technology leaders, we’ve previously written a quick overview of different AutoML solutions on the market, both free and paid. We’ve also featured an in-depth expert article from Rachel Thomas, co-founder of fast.ai, on how hyperparameter optimization works in neural architecture search, which consistitutes model selection for deep learning approaches.

In this article, we describe how one of the solution providers, OptiML from the company BigML, handles model selection and optimization on your data. You should also note that while AutoML can handle many components of the machine learning model development process for you, there are many key areas which it does not handle and an experienced data scientist or machine learning engineer will need to step in.

If our educational series on enterprise AI and AutoML is useful for you, you can subscribe to receive more industry updates and resources.

How Does OptiML Approach the Problem of Model Selection?

Everything starts with the understanding that there are so many models and parameters that we don’t have time to try even close to everything. But what if we could predict which parameters can be good and then try only this set of “promising” parameters?

Machine learning can help us to predict the quality of a set of modeling parameters even before we train a model on them. OptiML uses Bayesian parameter optimization for predicting the model’s performance on the given dataset:

• OptiML assumes that the performance of a machine learning algorithm with associated parameters is data dependent.
• After trying few models OptiML learns a regression model to predict performance of other models that have not been tested yet.
• Then, OptiML tests only “promising” models.

In contrast to random or grid search, Bayesian-based optimization approaches are more efficient because they choose the hyperparameters in an informed manner. Specifically, Bayesian methods for parameter optimization keep track of past evaluation results and use them to predict, which hyperparameters are going to work better.

Issues That Are Not Solved with AutoML

There are few things that a user has to take care of even when selecting the model with AutoML. These are things that the algorithm has no way of knowing:

• Metric selection. Depending on the specific dataset, different metrics will be the best fit for evaluating the models.
• Validation. Cross-validation is a default technique of validating the models in OptiML. However, in a number of cases, cross-validation performs very bad, for example, when data comes in batches. In such scenarios, the user may choose to evaluate models on a holdout dataset built from the whole batches of data instead of random data points.
• Evaluation beyond performance. Performance is not the only issue when selecting the “best” model. Other issues that should be considered include model stability, prediction speed, ability to handle different amounts of data. The user has to take care of several modeling tradeoffs:
  • weak vs. slow;
  • interpretability vs. representability;
  • confidence vs. performance;
  • biased vs. data-hungry.

How Can You Learn More About OptiML?

If you want to get more details on how OptiML can help you to select the best model for your specific dataset, check the video below which this summary article was based on. You can also read BigML’s “nitty gritty” article on how their Bayesian Parameter Optimization works.

Enjoy this article? Sign up for more enterprise AI education.

We’ll let you know when we release more educational articles covering different enterprise AI topics.