Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton; Alena Kopanicakova; Philippe TOINT

doi:10.1137/23M1553455

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton
, Alena Kopanicakova
, Philippe TOINT

Namur Institute for Complex Systems

Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

82 Téléchargements (Pure)

Résumé

A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

langue originale	Anglais
Pages (de - à)	2772-2800
Nombre de pages	29
journal	SIAM Journal on Optimization
Volume	33
Numéro de publication	4
Les DOIs	https://doi.org/10.1137/23M1553455
Etat de la publication	Publié - 15 févr. 2023

Financement

\ast Received by the editors February 15, 2023; accepted for publication (in revised form) July 11, 2023; published electronically October 13, 2023. https://doi.org/10.1137/23M1553455 Funding: The work of the first author was partially supported by 3IA Artificial and Natural Intelligence Toulouse Institute (ANITI), French ``Investing for the Future --PIA3"" program (grant agreement ANR-19-PI3A-0004). The work of the second author was supported by the Swiss National Science Foundation through the project ``Multilevel training of DeepONets --multiscale and multiphysics applications"" (grant 206745), and by the Platform for Advanced Scientific Computing under the project EXATRAIN. The third author acknowledges the continued and friendly partial support of ANITI. \dagger Universit\e' de Toulouse, INP, IRIT, Toulouse, 31000, France ([email protected]).

Bailleurs de fonds	Numéro du bailleur de fonds
Artificial and Natural Intelligence Toulouse Institute	ANR-19-PI3A-0004
Artificial and Natural Intelligence Toulouse Institute
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung	206745
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Accès au document

10.1137/23M1553455

arXiv-2302.07049manuscrit soumis, 1,32 MBLicense: autre

Autres fichiers et liens

Lien vers la publication sur Scopus

5 Citations
5 Article
1 Livre
1 Papier de travail

Complexity and performance for two classes of noise-tolerant first-order algorithms
Gratton, S., Jerad, S. & Toint, P., 2025, (Accepté/sous presse) Dans: Optimization Methods and Software.
Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

File
112 Téléchargements (Pure)
A block-coordinate approach of multi-level optimization with an application to physics-informed neural networks
Gratton, S., Mercier, V., Riccietti, E. & Toint, P. L., nov. 2024, Dans: Computational Optimization and Applications. 89, 2, p. 385-417 33 p.
Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

File
3 Téléchargements (Pure)
Convergence properties of an Objective-Function-Free Optimization regularization algorithm, including an 0(epsilon^{-3/2}) complexity bound
Gratton, S., Jerad, S. & TOINT, P., févr. 2023, Dans: SIAM Journal on Optimization. 33, 3, p. 1621-1646
Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

File

1 Terminé
1 Actif

ADALGOPT: ADALGOPT - Algorithmes avancés en optimisation non-linéaire
Sartenaer, A. (Co-investigateur) & Toint, P. (Co-investigateur)
1/01/87 → …
Projet: Axe de recherche
Optimisation multi-échelle non-linéaire
Sartenaer, A. (Responsable du Projet), Toint, P. (Responsable du Projet), Malmedy, V. (Chercheur), Tomanos, D. (Chercheur) & Weber Mendonca, M. (Chercheur)
1/07/04 → 31/07/11
Projet: Recherche

3 Recherche/Enseignement dans une institution externe
1 Participation à une conférence, un congrès
1 Participation à un atelier/workshop, un séminaire, un cours

ALGOPT2024 workshop on Algorithmic Optimization
Toint, P. (Orateur)
27 août 2024 → 30 août 2024
Activité: Participation ou organisation d'un événement › Participation à un atelier/workshop, un séminaire, un cours
Institut National Polytechnique de Toulouse
Toint, P. (Chercheur visiteur)
19 janv. 2024 → 26 janv. 2024
Activité: Visite d'une organisation externe › Recherche/Enseignement dans une institution externe
Institut National Polytechnique de Toulouse
Toint, P. (Chercheur visiteur)
4 juil. 2023 → 8 juil. 2023
Activité: Visite d'une organisation externe › Recherche/Enseignement dans une institution externe

Contient cette citation

@article{c0c1d1fddfbb48768f5dba1e75ae1e52,

title = "Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training",

abstract = "A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.",

keywords = "complexity, deep learning, multilevel methods, neural networks, nonlinear optimization, objective-function-free optimization (OFFO)",

author = "Serge Gratton and Alena Kopanicakova and Philippe TOINT",

year = "2023",

month = feb,

day = "15",

doi = "10.1137/23M1553455",

language = "English",

volume = "33",

pages = "2772--2800",

journal = "SIAM Journal on Optimization",

issn = "1052-6234",

publisher = "Society for Industrial and Applied Mathematics Publications",

number = "4",

}

TY - JOUR

T1 - Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

AU - Gratton, Serge

AU - Kopanicakova, Alena

AU - TOINT, Philippe

PY - 2023/2/15

Y1 - 2023/2/15

N2 - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

AB - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

KW - complexity

KW - deep learning

KW - multilevel methods

KW - neural networks

KW - nonlinear optimization

KW - objective-function-free optimization (OFFO)

UR - https://www.scopus.com/pages/publications/85175637452

U2 - 10.1137/23M1553455

DO - 10.1137/23M1553455

M3 - Article

SN - 1052-6234

VL - 33

SP - 2772

EP - 2800

JO - SIAM Journal on Optimization

JF - SIAM Journal on Optimization

IS - 4

ER -

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Résumé

Financement

Accès au document

Autres fichiers et liens

Empreinte digitale

Résultat de recherche

Projets

Activités

Contient cette citation