Opinions on State of the Art of Automated Machine Learning

Very useful piece here,  the individual comments at the link are most interesting, the key takeaways are mostly obvious. 

State of the Art in Automated Machine Learning in InfoQ

Anthony Alford, Francesca Lazzeri

Key Takeaways:

-Automated Machine Learning (AutoML) is important because it allows data scientists to save time and resources, delivering business value faster and more efficiently

-AutoML is not likely to remove the need for a "human in the loop" for industry-specific knowledge and translating the business problem into a machine learning problem

-Some important research topics in the area are feature engineering, model transparency, and addressing bias

-There are several commercial and open-source AutoML solutions available now for automating different parts of the machine learning process

-Some limitations of AutoML are the amount of computational resources required and the needs of domain-specific applications  ... "  

(Below at the link lots of individual, often technical comments on the progress and state of automation.  Well worth scanning at least)  ...

Replacing Data Scientists With AutoML?

Excerpt from a current KDNuggets article, which links further to a poll that asks the questions of practitioners.  My answer is yes. AutoML will replace the current needs for data science analysis.  Within a decade.  Of course the needs are likely to expand as well, so there will always be research and new requirements emerging.  And interpretation for specific context needs.  Just as there are needs for statisticians and analytics specialists for the same purposes.

When Will AutoML (Automated Machine Learning) Replace Data Scientists (if ever)?

Soon after tech giants Google and Microsoft introduced their AutoML services to the world, the popularity and interest in these services skyrocketed. We first review AutoML, compare the platforms available, and then test them out against real data scientists to answer the question: will AutoML replace us?

Introduction of AutoML:
One cannot introduce AutoML without mentioning the machine learning project’s life cycle, which includes data cleaning, feature selection/engineering, model selection, parameter optimization, and finally, model validation. As advanced as technology has become, the traditional data science project still incorporates a lot of manual processes and remains time-consuming and repetitive. ... "

More on MIT Open Source AutoML: ATMSeer

More on the topic, and some additional background information and links.  Very powerful concept that that should continue to expand.  Automation is the word,   See also Google AutoML, at tag below.

MIT Researchers Open-Source AutoML Visualization Tool ATMSeer    by  Anthony Alford   ... 

A research team from MIT, Hong Kong University, and Zhejiang University has open-sourced ATMSeer, a tool for visualizing and controlling automated machine-learning processes.
Solving a problem with machine learning (ML) requires more than just a dataset and training. For any given ML tasks, there are a variety of algorithms that could be used, and for each algorithm there can be many hyperparameters that can be tweaked. Because different values of hyperparameters will produce models with different accuracies, ML practitioners usually try out several sets of hyperparameter values on a given dataset to try to find hyperparameters that produce the best model. 

This can be time-consuming, as a separate training job and model evaluation process must be conducted for each set. Of course, they can be run in parallel, but the jobs must be setup and triggered, and the results recorded. Furthermore, choosing the particular values for hyperparameters can involve a bit of guesswork, especially for ones that can take on any numeric value: if 2.5 and 2.6 produce good results, maybe 2.55 would be even better? What about 2.56 or 2.54?

Enter automated machine learning, or AutoML. These are techniques and tools for automating the selection and evaluation of hyperparameters (as well as other common ML tasks such as data cleanup and feature engineering). Both Google Cloud Platform and Microsoft Azure provide commercial AutoML solutions, and there are several open-source packages such as auto-sklearn and Auto-Keras.  ...."

Advances in Automated Machine Learning

Automated machine learning is inevitable.  How good will it be, and how much human oversight needs to be applied to ensure confidence in their results is important.  This article is a  good overview of work underway.  Its no only about searching for the right model,  in context of needed goals, its also about maintaining the interaction between data and solutions.   Not too unlike the use of any kind of analytics optimization, which has been studied for years.

Cracking open the black box of automated machine learning
Interactive tool lets users see and control how automated model searches work.
 By Rob Matheson | MIT News Office 

Researchers from MIT and elsewhere have developed an interactive tool that, for the first time, lets users see and control how automated machine-learning systems work. The aim is to build confidence in these systems and find ways to improve them.

Designing a machine-learning model for a certain task — such as image classification, disease diagnoses, and stock market prediction — is an arduous, time-consuming process. Experts first choose from among many different algorithms to build the model around. Then, they manually tweak “hyperparameters” — which determine the model’s overall structure — before the model starts training.

Recently developed automated machine-learning (AutoML) systems iteratively test and modify algorithms and those hyperparameters, and select the best-suited models. But the systems operate as “black boxes,” meaning their selection techniques are hidden from users. Therefore, users may not trust the results and can find it difficult to tailor the systems to their search needs.

In a paper presented at the ACM CHI Conference on Human Factors in Computing Systems, researchers from MIT, the Hong Kong University of Science and Technology (HKUST), and Zhejiang University describe a tool that puts the analyses and control of AutoML methods into users’ hands. Called ATMSeer, the tool takes as input an AutoML system, a dataset, and some information about a user’s task. Then, it visualizes the search process in a user-friendly interface, which presents in-depth information on the models’ performance.

“We let users pick and see how the AutoML systems works,” says co-author Kalyan Veeramachaneni, a principal research scientist in the MIT Laboratory for Information and Decision Systems (LIDS), who leads the Data to AI group. “You might simply choose the top-performing model, or you might have other considerations or use domain expertise to guide the system to search for some models over others.”

In case studies with science graduate students, who were AutoML novices, the researchers found about 85 percent of participants who used ATMSeer were confident in the models selected by the system. Nearly all participants said using the tool made them comfortable enough to use AutoML systems in the future.  ... " 

Also discusses Auto-Tuned Models ATMs ... "

Figure Eight and Google for AutoML Training

Yes, this is the Grail ... good training data, initially and ongoing, is essential.  Also sufficient contextual metadata.   We had projects fail because we did not consider these issues.  Was much impressed with a Figure Eight webinar I attended, they appear to understand the issues more than others.  Ultimately in autonomous learning,  you have to plan for re-learning,   Will be following up.

Figure Eight partners with Google to give AutoML developers better training data   By Frederic Lardinois in TechCrunch

Figure Eight, a platform that helps developers train, test and fine-tune their machine learning models, today announced a major new collaboration with Google that essentially turns Figure Eight into the de facto standard for creating and annotating machine learning data for Google Cloud’s AutoML service.

As Figure Eight’s CEO Robin Bordoli told me, Google had long been a customer, but the two companies decided to work closer together now that AutoML is launching in beta and expanding its product portfolio, too. As Bordoli argues, training data remains one of the biggest bottlenecks for developers who want to build their own machine learning models — and Google recognized this, too. “It’s their recognition that the lack of training data is a fundamental bottleneck to the adoption of AutoML,” he told me.

Since AutoML’s first product focuses on machine vision, it’s maybe no surprise that Figure Eight’s partnership with Google is also currently mostly about this kind of visual training data. Its service is meant to help relatively inexperienced developers collect data, prepare it for use in AutoML and then experiment with the results. ... " 

Google AutoML Learns

Its long been suggested that the best AI is one that is the best learner.  Then it can learn from other humans, other AIs and the masses of data we are gathering.   Leading to an ultimate intelligence.  This is not quite that,  so the suggestion is a push, but points in the right direction.    From Weforum.org:

This Google AI created a ‘child’ AI to help it solve problems

Google's AutoML project, designed to make AI build other AIs, has now developed a computer vision system that vastly outperforms state-of-the-art-models. The project could improve how autonomous vehicles and next-generation AI robots "see."

An AI That Can Build AI

In May 2017, researchers at Google Brain announced the creation of AutoML, an artificial intelligence (AI) that’s capable of generating its own AIs. More recently, they decided to present AutoML with its biggest challenge to date, and the AI that can build AI created a “child” that outperformed all of its human-made counterparts.

The Google researchers automated the design of machine learning models using an approach called reinforcement learning. AutoML acts as a controller neural network that develops a child AI network for a specific task. For this particular child AI, which the researchers called NASNet, the task was recognizing objects — people, cars, traffic lights, handbags, backpacks, etc. — in a video in real-time.  ... " 

Google ML Automation Service

We will continue to see more automation in driving machine learning problems.   Here the training aspects for some kinds of problems are addressed, like image recognition. 

Cloud AutoML Alpha   by Google
Train high quality custom machine learning models with minimum effort and machine learning expertise

Train Custom Machine Learning Models

Cloud AutoML is a suite of Machine Learning products that enables developers with limited machine learning expertise to train high quality models by leveraging Google’s state of the art transfer learning, and Neural Architecture Search technology.

AutoML Vision is the first product to be released. It is a simple, secure and flexible ML service that lets you train custom vision models for your own use cases. Soon, Cloud AutoML will release other services for all other major fields of AI.   ....  "

Google’s AutoML lets you train custom machine learning models without having to code   by Frederic Lardinois (@fredericl)  in TechCrunch

Google today announced the alpha launch of AutoML Vision, a new service that helps developers — including those with no machine learning (ML) expertise — build custom image recognition models. While Google plans to expand this custom ML model builder under the AutoML brand to other areas, the service for now only supports computer vision models, but you can expect the company to launch similar versions of AutoML for all the standard ML building blocks in its repertoire (think speech, translation, video, natural language recognition, etc.).  .... "