Crowd Sourcing vs Algorithm

We actively used Mechanical Turk, so the comparison is interesting, but depends much on the details of the set up involved.

Mechanical Turkers may have out-predicted the most popular crime-predicting algorithm   The Verge by Russell Brandom

Our most sophisticated crime-predicting algorithms may not be as good as we thought. A study published today in Science Advances takes a look at the popular COMPAS algorithm — used to assess the likelihood that a given defendant will reoffend — and finds the algorithm is no more accurate than the average person’s guess. If the findings hold, they would be a black eye for sentencing algorithms in general, indicating we may simply not have the tools to accurately predict whether a defendant will commit further crimes.

Developed by Equivant (formerly Northpointe), the COMPAS algorithm examines a defendant’s criminal record alongside a series of other factors to assess how likely they are to be arrested again in the next two years. COMPAS’ risk assessment can then inform a judge’s decisions about bail or even sentencing. If the algorithm is inaccurate, the result could be a longer sentence for an otherwise low-risk defendant, a significant harm for anyone impacted.

Reached by The Verge, Equivant contested the accuracy of the paper in a lengthy statement, calling the work “highly misleading.”  .... "