|
Download: |
Observations on deep puzzlesProf. Tomaso Poggio In recent years, artificial intelligence researchers have built impressive systems. Two of my former postdocs - Demis Hassabis and Amnon Shashua - are behind two main recent success stories of AI: AlphaFold and Mobileye, based on two key algorithms, both originally suggested by discoveries in neuroscience: deep learning and reinforcement learning. To create artifacts that are as intelligent as we are, we need several additional breakthroughs. I will describe why a good bet is that several of them will come from interdisciplinary research between the natural science and the engineering of Intelligence. I will then outline recent progress in the mathematics of deep learbing networks. Very recently, square loss has been observed to perform well in classification tasks with deep networks. I will discuss several observations on the dynamics of gradient flow under the square loss in overparametrized ReLU networks that fit the training data. In particular, I will show how convergence to a local minimum norm solution is expected when normalization techniques such as Batch Normalization (BN) or Weight Normalization (WN) are used. The main property of the minimizer that bounds its expected classification error is its norm: among all the interpolating solutions, the ones associated with smaller Frobenius norms of the weight matrices have better margin and better bounds on the expected classification error. |
 |
