Formerly Crowdflower has been renamed Figure Eight, with a new web presence. Impressed by what I have seen there so far. Putting humans in the loop of machine learning ...
" .... Figure Eight is the essential Human-in-the-Loop AI platform for data science and machine learning teams. The Figure Eight software platform trains, tests, and tunes machine learning models to make AI work in the real world.
At Figure Eight, we believe that AI’s three essential ingredients are training data, machine learning, and human-in-the-loop technology. We pride ourselfvesin providing every component necessary to make AI work in the real world.
Training data is the fuel for machine learning, where humans guide the algorithms by labeling data for the algorithms to build their knowledge. You can read about how we approach training data in our Definitive Guide to Training Data. Human-in-the-loop processing allows the feedback loop between humans and machines to be as optimal as possible, whether that’s finding the right raw data to annotate through Active Learning, or selecting the right annotation interfaces and quality controls for accurate and efficient human input.
The most important problem we face in technology today is how humans and machines and work together to solve tasks. These breakthroughs will be vital in many of the use cases we power today, whether it’s the obvious cases today like autonomous vehicles and AI-powered assistants, to the future technologies that we are supporting in areas like healthcare and agriculture. .... "
Showing posts with label Crowdflower. Show all posts
Showing posts with label Crowdflower. Show all posts
Friday, June 01, 2018
Thursday, March 15, 2018
Human in the Loop Machine Learning
Attended a very good webinar today in the DSC series. Strongly recommend joining DSC and taking advantage of their free resources.
This Webinar answers the question you will have as a data scientist. Where will I get the data to train my models, when its mostly held by people?
Now renamed: Figure Eight https://www.figure-eight.com/
Robert Munro, CTO of Figure Eight answers in this recorded Webinar:
" ... Curious about what human-in-the-loop machine learning actually looks like? Join CrowdFlower and learn how to effectively incorporate Active Learning, Transfer Learning, and Annotation Quality in your ML projects to achieve better results.
Join us in this latest Data Science Central webinar, where we will cover the following topics:
When to use the human-in-the-loop as an effective strategy for machine learning projects
How to set up an effective interface to get the most out of human intelligence
How to ensure high-quality, accurate training data sets
How to use ML models from different domains to improve your own labeling
This webinar will include an end-to-end look at setting up and running a job that generates high-quality training data, and shows how to incorporate that training data into human-in-the-loop machine learning systems that you can run in your own environment.
Speaker: Robert Munro, Chief Technology Officer -- CrowdFlower
Hosted by: Bill Vorhies, Editorial Director -- Data Science Central .... "
This Webinar answers the question you will have as a data scientist. Where will I get the data to train my models, when its mostly held by people?
Now renamed: Figure Eight https://www.figure-eight.com/
Robert Munro, CTO of Figure Eight answers in this recorded Webinar:
" ... Curious about what human-in-the-loop machine learning actually looks like? Join CrowdFlower and learn how to effectively incorporate Active Learning, Transfer Learning, and Annotation Quality in your ML projects to achieve better results.
Join us in this latest Data Science Central webinar, where we will cover the following topics:
When to use the human-in-the-loop as an effective strategy for machine learning projects
How to set up an effective interface to get the most out of human intelligence
How to ensure high-quality, accurate training data sets
How to use ML models from different domains to improve your own labeling
This webinar will include an end-to-end look at setting up and running a job that generates high-quality training data, and shows how to incorporate that training data into human-in-the-loop machine learning systems that you can run in your own environment.
Speaker: Robert Munro, Chief Technology Officer -- CrowdFlower
Hosted by: Bill Vorhies, Editorial Director -- Data Science Central .... "
Tuesday, March 06, 2018
Human in the Loop Machine Learning
Planning to Attend ....
Practical Human-in-the-Loop Machine Learning
Join us for this latest DSC Webinar on March 15th, 2018
Register Now!
Curious about what human-in-the-loop machine learning actually looks like? Join CrowdFlower and learn how to effectively incorporate Active Learning, Transfer Learning, and Annotation Quality in your ML projects to achieve better results.
Join us in this latest Data Science Central webinar, where we will cover the following topics:
When to use the human-in-the-loop as an effective strategy for machine learning projects
How to set up an effective interface to get the most out of human intelligence
How to ensure high-quality, accurate training data sets
How to use ML models from different domains to improve your own labeling
This webinar will include an end-to-end look at setting up and running a job that generates high-quality training data, and shows how to incorporate that training data into human-in-the-loop machine learning systems that you can run in your own environment.
Speaker: Robert Munro, Chief Technology Officer -- CrowdFlower
Hosted by: Bill Vorhies, Editorial Director -- Data Science Central
Title: Practical Human-in-the-Loop Machine Learning
Date: Thursday, March 15th, 2018, Time: 9:00 AM - 10:00 AM PT
Practical Human-in-the-Loop Machine Learning
Join us for this latest DSC Webinar on March 15th, 2018
Register Now!
Curious about what human-in-the-loop machine learning actually looks like? Join CrowdFlower and learn how to effectively incorporate Active Learning, Transfer Learning, and Annotation Quality in your ML projects to achieve better results.
Join us in this latest Data Science Central webinar, where we will cover the following topics:
When to use the human-in-the-loop as an effective strategy for machine learning projects
How to set up an effective interface to get the most out of human intelligence
How to ensure high-quality, accurate training data sets
How to use ML models from different domains to improve your own labeling
This webinar will include an end-to-end look at setting up and running a job that generates high-quality training data, and shows how to incorporate that training data into human-in-the-loop machine learning systems that you can run in your own environment.
Speaker: Robert Munro, Chief Technology Officer -- CrowdFlower
Hosted by: Bill Vorhies, Editorial Director -- Data Science Central
Title: Practical Human-in-the-Loop Machine Learning
Date: Thursday, March 15th, 2018, Time: 9:00 AM - 10:00 AM PT
Tuesday, June 20, 2017
Figure Eight (Formerly Crowdflower) with AI Augmentation
We spent considerable time using crowd sourcing techniques, and now follow some of the methods of Crowdflower. Could have used the methods in the enterprise. In SiliconAngle:
Crowdflower is Now renamed: Figure Eight https://www.figure-eight.com/
AI-augmented crowdsourcing company CrowdFlower raises $20M for enterprise push by Kyt Dotson
" .... People power and machine learning go hand in hand at San Francisco-based CrowdFlower, which uses data and training from large groups of people – a practice known as crowdsourcing – to train machine learning algorithms to do tedious data science work.
Robin Bordoli, chief executive at CrowdFlower, believes that AI applications within the enterprise is on the verge of a “Cambrian explosion.” This is a reference to a point in the biological history of life on Earth when a huge variety of different body designs begin to appear in the fossil record. In short, “living thing” applications began to try out a lot of different ideas.
With CrowdFlower’s approach to crowdsourcing AI training, Bordoli said, enterprise data science could find its killer app. “The bottleneck for the large-scale adoption of machine learning still remains the availability of high-quality training data and human-in-the-loop workflows to handle the failure states,” he said.
The crowdsourced labor works by applying simple tasks to individual workers, such as transcribing text seen in an image, determining the sentiment of a sentence, statement or forum post, annotating images and other work that humans do well. These are processes that can be broken down into thousands of small tasks, and each individual task is executed by a small group of people. .... "
Crowdflower is Now renamed: Figure Eight https://www.figure-eight.com/
AI-augmented crowdsourcing company CrowdFlower raises $20M for enterprise push by Kyt Dotson
" .... People power and machine learning go hand in hand at San Francisco-based CrowdFlower, which uses data and training from large groups of people – a practice known as crowdsourcing – to train machine learning algorithms to do tedious data science work.
Robin Bordoli, chief executive at CrowdFlower, believes that AI applications within the enterprise is on the verge of a “Cambrian explosion.” This is a reference to a point in the biological history of life on Earth when a huge variety of different body designs begin to appear in the fossil record. In short, “living thing” applications began to try out a lot of different ideas.
With CrowdFlower’s approach to crowdsourcing AI training, Bordoli said, enterprise data science could find its killer app. “The bottleneck for the large-scale adoption of machine learning still remains the availability of high-quality training data and human-in-the-loop workflows to handle the failure states,” he said.
The crowdsourced labor works by applying simple tasks to individual workers, such as transcribing text seen in an image, determining the sentiment of a sentence, statement or forum post, annotating images and other work that humans do well. These are processes that can be broken down into thousands of small tasks, and each individual task is executed by a small group of people. .... "
Friday, November 13, 2015
Thinking Humans in the Computing Loop
An idea we played with for years. Why not integrate humans deeply into processes that need judgement and intelligence? Even if only temporarily until we figure our how to do it otherwise. The question is how do we closely integrate people and machines, each with their particular skills? Heard about Crowdflower only this year, great start. Understand too, that humans and machines can both be wrong in their own particular ways. Consider modeling the business process involved in understand how they will interact. Track early results to adapt the model to reality. This turns out to be an excellent way to put decision process in the loop.
Why human-in-the-loop computing is the future of machine learning
Now that machine learning is becoming more and more mainstream, some design patterns are starting to emerge. As the CEO of CrowdFlower, I’ve worked with many companies building machine learning algorithms and I’ve noticed a best practice in nearly every successful deployment of machine learning on tough business problems. That practice is called “human-in-the-loop” computing. Here’s how it works:
First, a machine learning model takes a first pass on the data, or every video, image or document that needs labeling. That model also assigns a confidence score, or how sure the algorithm is that it’s making the right judgment. If the confidence score is below a certain value, it sends the data to a human annotator to make a judgment. That new human judgment is used both for the business process and is fed back into the machine learning algorithm to make it smarter. In other words, when the machine isn’t sure what the answer is, it relies on a human, then adds that human judgment to its model. .... "
Why human-in-the-loop computing is the future of machine learning
Now that machine learning is becoming more and more mainstream, some design patterns are starting to emerge. As the CEO of CrowdFlower, I’ve worked with many companies building machine learning algorithms and I’ve noticed a best practice in nearly every successful deployment of machine learning on tough business problems. That practice is called “human-in-the-loop” computing. Here’s how it works:
First, a machine learning model takes a first pass on the data, or every video, image or document that needs labeling. That model also assigns a confidence score, or how sure the algorithm is that it’s making the right judgment. If the confidence score is below a certain value, it sends the data to a human annotator to make a judgment. That new human judgment is used both for the business process and is fed back into the machine learning algorithm to make it smarter. In other words, when the machine isn’t sure what the answer is, it relies on a human, then adds that human judgment to its model. .... "
Sunday, August 16, 2015
Figure Eight Cognitive Systems with CrowdFlower
Learning and adapting knowledge to train a cognitive system is key. While we experimented with corwdsourcing methods, they were difficult to manage for AI systems.
Now renamed: Figure Eight https://www.figure-eight.com/
Figure Eight:
" .... This is especially true in the interpretation of health data from a seemingly infinite array of sources -- and a big reason why we’re so excited to see CrowdTruth.org leveraging CrowdFlower to train IBM Watson.
Watson represents the state of the art in computational linguistics and computer vision put into action. It uses its never-before-seen understanding of language and imagery to comb through volumous datasets to unearth helpful information and make predictions (such as tips for disease diagnosis). According to Lora Aroyo, Principal Investigator of CrowdTruth, Watson acts as "a cognitive prosthetic to extend the decision making capabilities of an expert," such as a doctor, who will utilize it as a tool for recommendations on how best to analyze a patient's condition.
In parallel, data enrichment platforms have become a valuable resource for data scientists looking to automate and scale the cleaning, labeling, and enrichment of data using human intelligence for machine learning -- ie., training data creation. While Watson continuously engages in active learning, its intelligence is strengthened by the quality of the training data it takes in from crowd contributors on data enrichment platforms such as CrowdFlower. ... "
Now renamed: Figure Eight https://www.figure-eight.com/
Figure Eight:
" .... This is especially true in the interpretation of health data from a seemingly infinite array of sources -- and a big reason why we’re so excited to see CrowdTruth.org leveraging CrowdFlower to train IBM Watson.
Watson represents the state of the art in computational linguistics and computer vision put into action. It uses its never-before-seen understanding of language and imagery to comb through volumous datasets to unearth helpful information and make predictions (such as tips for disease diagnosis). According to Lora Aroyo, Principal Investigator of CrowdTruth, Watson acts as "a cognitive prosthetic to extend the decision making capabilities of an expert," such as a doctor, who will utilize it as a tool for recommendations on how best to analyze a patient's condition.
In parallel, data enrichment platforms have become a valuable resource for data scientists looking to automate and scale the cleaning, labeling, and enrichment of data using human intelligence for machine learning -- ie., training data creation. While Watson continuously engages in active learning, its intelligence is strengthened by the quality of the training data it takes in from crowd contributors on data enrichment platforms such as CrowdFlower. ... "
Wednesday, June 24, 2015
CrowdTruth Collects Gold Standard Data
Nice idea, examining further. Experiences? We experimented with similar ideas using Mechanical Turk. More will follow here.
CrowdTruth via #CrowdTruth by Anca Dumitrache
collecting gold standard data for training and evaluation of cognitive computing systems
Welcome to the CrowdTruth blog!
The CrowdTruth Framework implements an approach to machine-human computing for collecting annotation data on text, images and videos. The approach is focussed specifically on collecting gold standard data for training and evaluation of cognitive computing systems. The original framework was inspired by the IBM Watson project for providing improved (multi-perspective) gold standard (medical) text annotation data for the training and evaluation of various IBM Watson components, such as Medical Relation Extraction, Medical Factor Extraction and Question-Answer passage alignment.
The CrowdTruth framework supports the composition of CrowdTruth gathering workflows, where a sequence of micro-annotation tasks can be configured and sent out to a number of crowdsourcing platforms (e.g. CrowdFlower and Amazon Mechanical Turk) and applications (e.g. Expert annotation game Dr. Detective). The CrowdTruth framework has a special focus on micro-tasks for knowledge extraction in medical text (e.g. medical documents, from various sources such as Wikipedia articles or patient case reports). The main steps involved in the CrowdTruth workflow are: (1) exploring & processing of input data, (2) collecting of annotation data, and (3) applying disagreement analytics on the results. These steps are realised in an automatic end-to-end workflow, that can support a continuous collection of high quality gold standard data with feedback loop to all steps of the process. Have a look at our presentations and papers for more details on the research. ... "
CrowdTruth via #CrowdTruth by Anca Dumitrache
collecting gold standard data for training and evaluation of cognitive computing systems
Welcome to the CrowdTruth blog!
The CrowdTruth Framework implements an approach to machine-human computing for collecting annotation data on text, images and videos. The approach is focussed specifically on collecting gold standard data for training and evaluation of cognitive computing systems. The original framework was inspired by the IBM Watson project for providing improved (multi-perspective) gold standard (medical) text annotation data for the training and evaluation of various IBM Watson components, such as Medical Relation Extraction, Medical Factor Extraction and Question-Answer passage alignment.
The CrowdTruth framework supports the composition of CrowdTruth gathering workflows, where a sequence of micro-annotation tasks can be configured and sent out to a number of crowdsourcing platforms (e.g. CrowdFlower and Amazon Mechanical Turk) and applications (e.g. Expert annotation game Dr. Detective). The CrowdTruth framework has a special focus on micro-tasks for knowledge extraction in medical text (e.g. medical documents, from various sources such as Wikipedia articles or patient case reports). The main steps involved in the CrowdTruth workflow are: (1) exploring & processing of input data, (2) collecting of annotation data, and (3) applying disagreement analytics on the results. These steps are realised in an automatic end-to-end workflow, that can support a continuous collection of high quality gold standard data with feedback loop to all steps of the process. Have a look at our presentations and papers for more details on the research. ... "
Thursday, January 29, 2015
Crowdflower Crowdsourcing
I was asked to take a look at Crowdflower (Now called FigureEight ) in order to construct a database of HR data. Previously we had used Mechanical Turk for a related need. To provide a crowdsourced way to construct a derived database. Background. Comments on its use?
Subscribe to:
Comments (Atom)