Thinking Humans in the Computing Loop

An idea we played with for years.   Why not integrate humans deeply into processes that need judgement and intelligence?   Even if only temporarily until we figure our how to do it otherwise. The question is how do we closely integrate people and machines, each with their particular skills?   Heard about Crowdflower only this year, great start.   Understand too, that humans and machines can both be wrong in their own particular ways.  Consider modeling the business process involved in understand how they will interact.  Track early results to adapt the model to reality.  This turns out to be an excellent way to put decision process in the loop.

Why human-in-the-loop computing is the future of machine learning
Now that machine learning is becoming more and more mainstream, some design patterns are starting to emerge. As the CEO of CrowdFlower, I’ve worked with many companies building machine learning algorithms and I’ve noticed a best practice in nearly every successful deployment of machine learning on tough business problems. That practice is called “human-in-the-loop” computing. Here’s how it works:

First, a machine learning model takes a first pass on the data, or every video, image or document that needs labeling. That model also assigns a confidence score, or how sure the algorithm is that it’s making the right judgment. If the confidence score is below a certain value, it sends the data to a human annotator to make a judgment. That new human judgment is used both for the business process and is fed back into the machine learning algorithm to make it smarter. In other words, when the machine isn’t sure what the answer is, it relies on a human, then adds that human judgment to its model.  .... "