AI versus Human Perception Performance

Interesting challenge because we always emphasize that we need to be able to measure something to use/improve it.   Which leads to our design of the measurement system. My response is that you can build measurement systems for particular use contexts.    Reading the noted paper which emphasises " .. compare deep neural networks and the human vision system ... "   which is a very broad statement for the problem.  Like the discussion here. 

Why AI and human perception are too complex to be compared   By Ben Dickson in Tnw
Human-level performance. Human-level accuracy. Those are terms you hear a lot from companies developing artificial intelligence systems, whether it’s facial recognition, object detection, or question answering. And to their credit, the recent years have seen many great products powered by AI algorithms, mostly thanks to advances in machine learning and deep learning.

But many of these comparisons only take into account the end-result of testing the deep learning algorithms on limited data sets. This approach can create false expectations about AI systems and yield dangerous results when they are entrusted with critical tasks.

In a recent study, a group of researchers from various German organizations and universities has highlighted the challenges of evaluating the performance of deep learning in processing visual data. In their paper, titled, “The Notorious Difficulty of Comparing Human and Machine Perception,” the researchers highlight the problems in current methods that compare deep neural networks and the human vision system.  ... "

In their research, the scientist conducted a series of experiments that dig beneath the surface of deep learning results and compare them to the workings of the human visual system. Their findings are a reminder that we must be cautious when comparing AI to humans, even if it shows equal or better performance on the same task.  ... "