Spotting expressivity bottlenecks in neural networks and fixing them by optimal architecture growth

Abstract

We aim at adapting a neural network architecture during training, by detecting expressivity bottlenecks. This would allow to start from tiny architectures and make them grow as necessary, which would gain considerable training time and computational ressources compared to the standard approach where over-sized models are trained and then reduced. A promising mathematical formulation of the problem yields a way to directly precisely locate expressivity bottlenecks, at low computational cost, to the opposite of standard Auto-Deep-Learning techniques which proceed by trial and error via random architecture modifications (which is very costly).

Jury

• Rémi Gribonval, Directeur de recherche Inria, ENS Lyon, rapporteur
• Hervé Luga, Professeur Université Toulouse Jean Jaurès, rapporteur
• David Picard, Directeur de recherche École des Ponts ParisTech
• Aurélie Névéol, Directrice de recherche, CNRS, examinatrice
• Guillaume Lecué, Professeur ESSEC, CY Cergy Paris Université, examinateur

Keywords

Neural networks,expressive power,low-rank approximation,optimization,