Self Reporting Data

Have been involved in several projects where this considerably effected the accuracy of results of the model constructed.   Personal data is often misreported.    Hardly hidden, but rather behavioral expected misstatements. 

Everybody Lies - Hidden Information    By: Peter Bruce in Statistics.com

Each person checking in for a Cape Air flight is asked for his or her weight, so the pilot can determine flight parameters.  It can be an awkward moment, particularly if there are other passengers crowding in behind you. For most aircraft, individual weights are not needed - the airline can apply a formula based on the number of passengers, the sample size (passenger capacity) being large enough to minimize uncertainty.  But Cape Air, a commuter airline, connects Boston to small communities in New England with 9-seat aircraft, and cannot use the “large sample average.” One group of 9 passengers may weigh considerably more or less than another group. In 2003, a 19-seat US Airways flight crashed due to overloading, its actual load being 580 pounds over what the formula showed.

Do people truthfully report their weight?  Airlines typically apply a buffer to allow for lying.  Bush pilots in Alaska will ask passengers to actually step on a scale.  People’s true weight is hidden information, though probably less hidden in a life or death matter such as a flight on a small aircraft than in a casual conversation.

Much of big data analytics relates to uncovering this hidden information. This week’s book review is of Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are,   (by) Seth Stephens-Davidowitz’s fascinating book about how Google knows all sorts of things about us that we barely know ourselves.  ... "