OpenAI Monitors Machine Learning Efficiency

For now,  the most measurable and definable kind of AI is machine learning,   It uses much data and compute cycles.   Useful to measure the advances in efficiency.  That has been improving. according to OpenAI.  Notably that ' ... algorithmic progress has yielded more gains than classical hardware efficiency ...'.    At least for recent investments.  That is notable for future understanding of where to address our efforts.

AI and Efficiency  by OpenAI

We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet1 classification has been decreasing by a factor of 2 every 16 months. Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet2 (by contrast, Moore’s Law3 would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.

READ PAPER:  https://arxiv.org/abs/2005.04305  (technical)

Algorithmic improvement is a key factor driving the advance of AI. It’s important to search for measures that shed light on overall algorithmic progress, even though it’s harder than measuring such trends in compute ... '