Spectral Tools for Neural Networks

Deep Feedforward Neural Networks (FFNNs) play a central role in the Machine Learning field. They are usually trained in the space of nodes, by adjusting the weights of existing links via suitable optimization protocols. Recently a radically new approach has been proposed (1). By anchoring the learning process to reciprocal space, the new targets of the optimization process are eigenvectors and eigenvalues of the transfer operators between layers.

Shifting the focus on such fundamental mathematical structures we have been able to understand their pivotal role in training and analyzing NNs. Indeed, while seeking for a small subset of trainable parameters capable of carrying out the training procedure, eigenvalues are what to look for. Choosing them as trainable parameters allows the optimizer to exploit the parallel adjustment of several weights, the ones underlined by the corresponding eigenvector, and therefore made their after-training interpretation possible.

Firstly, eigenvalues magnitude after the training procedure has occurred has been empirically and heuristically proven being a proxy their relevance in the optimization process. Indeed, a precise correspondence between nodes and eigenvalues can be established, leading to a novel pruning procedure. The nodes related with low magnitude eigenvalues can be removed leading to a fast and easy implemented network compression algorithm.

Secondly, accounting for eigenvalues in the optimization process, it is possible to dynamically train sparse network. Sparsity constrains in the direct space implies that certain weights got filtered under a mask, leading to a gradient equal to zero during the training procedure. Working in the reciprocal space, however, allows masked weights to still be modified, due to the non-local effect of the eigenvalues. Such approach leads to sparse networks whose topology is not fixed to the starting one, resulting in a much more efficient training.