Deep Learning and Optimization

Technical point about deep learning, diagrams at the link:

The hard thing about deep learning
Deeper neural nets often yield harder optimization problems.  By Reza Zadeh

At the heart of deep learning lies a hard optimization problem. So hard that for several decades after the introduction of neural networks, the difficulty of optimization on deep neural networks was a barrier to their mainstream usage and contributed to their decline in the 1990s and 2000s. Since then, we have overcome this issue. In this post, I explore the “hardness” in optimizing neural networks and see what the theory has to say. In a nutshell: the deeper the network becomes, the harder the optimization problem becomes. ... "