Data Labeling

We worked with both the crowdsourcing methods for this and automated methods. 

AI's New Workforce: The Data-Labeling Industry Spreads Globally
in Financial Times
By Madhumita Murgia

There is a growing international data-labeling industry that employs hundreds of thousands of workers in lower-income countries. Many small companies are forming parts of a so-called "artificial intelligence (AI) supply chain" helping to train algorithms that can interpret material including driving footage, search results, and photos for some of the world's largest multinational corporations. Companies are embracing AI as a way to automate decision-making and help drive new business opportunities, but first they must train their algorithms on millions of labelled examples. Said Leila Janah, founder and chief executive of data labeling vendor Samasource, "We work with a population usually coming from informal settlements, rural villages, so the chance to have a job that pays well, and gives you computer skills and exposes you to AI, it means people treat this very seriously." ....."

Training Data for Autonomous Driving

Getting, saving, and usefully tagging data and metadata for key purposes is a powerful idea.

Training Data for Autonomous Driving   By Karlsruhe Institute of Technology 

Using processed images, algorithms learn to recognize the real environment for autonomous driving.
Philip Kessler at the Karlsruhe Institute of Technology (KIT) in Germany has launched understand.ai, a startup that improves and accelerates the labeling of image elements for autonomous driving algorithms.

These labels, also called annotations, must agree with the real environment with pixel accuracy. The better the quality of the processed image data, the better the algorithm will be at using the data for training.

Traditionally, objects in images are labeled manually by human staff, but the process is troublesome and time-consuming; artificial intelligence makes the labeling process up to 10 times quicker and more precise.

Said Kessler, "As training images cannot be supplied for all situations, such as accidents, we now also offer simulations based on real data."    ....

From Karlsruhe Institute of Technology (More  article and data at the link)

Figure 8 Talks Active Learning

Have now been a 'guest' in many machine learning efforts.  My experience that the pre consideration of how you effectively learn and continue to learn is rare.   Leads back to the truth we discovered early on, that maintenance of your effort is crucial.  Most of our failures ended there. Also that data is an asset that needs to be labeled, measured, secured.   Its often a too-late afterthought.  Here Figure Eight offers a free E-book on the topic of 'Active Learning', at the link.

An Introduction to Active Learning

The increased availability of computer resources and the prevalence of high-quality training data combined with smart learning schemas, have resulted in a rise in successful AI deployments. However, many organizations simply have too much data, posing a challenge for data scientists: unless at least some of that data is labeled, it's essentially useless for any ML approach that relies on supervised or semi-supervised learning. So, which data needs to be labeled? How much of a dataset needs to be labeled for an ML application to be viable? How can we solve the problem of having more data than we can reasonably analyze?

One promising answer is active learning. Active learning is unique in that it can both solve this data labeling crisis and train models to be more accurate with less data overall. Download this eBook to learn:

The pros and cons of active learning as an approach
The three major categories of active learning
How your active learner should decide which rows need labeling
How to tell if active learning is appropriate for your ML project  ... 

Labeled Datasets for Use AND Value

Note how this links to other goals, like understanding the value of datasets as an asset.  Why not use labeling as a means to attach to value analyses as well?   Labels are usually assigned for business purposes that do not work when linking to specific analytic approach. 

The Data Scientist’s Holy Grail — Labeled Data Sets  #ODSC  in Medium

The Holy Grail for data scientists is the ability to obtain labeled data sets for the purpose of training a supervised machine learning algorithm. An algorithm’s ability to “learn” is based on training it using a labeled training set — having known response variable values that correspond to a number of predictor variable values.

There are a number of common and maybe not-so-common methods for labeling a data set. In this article, we’ll run down a short list of such methods and then you can choose the best for your specific circumstances.

Readily Available Labeled Data Sets: 

Sometimes, labeled datasets are readily available as a byproduct of on-going business operations. For example, if a company is trying to predict customer churn (a very common classification problem), the company’s data assets will likely contain the label values: “churned,” or “not-churned” based on the customer’s account history. The company knows when the customer canceled their account, thus generating a churn transaction.

Sometimes, the label is not readily available and must be acquired or derived. For example, in a real estate application that wishes to predict the monthly rental value of a residential apartment building, the desired label may only come from a laborious process conducted by problem domain experts who can determine the value based on their industry knowledge. Sometimes finding label values can be time-consuming and labor-intensive, especially if a large amount of labeled data is needed for the project. ... "