Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton; Alena Kopanicakova; Philippe TOINT

doi:10.1137/23M1553455

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton, Alena Kopanicakova, Philippe TOINT

Namur Institute for Complex Systems

Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

27 Téléchargements (Pure)

Résumé

A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

langue originale	Anglais
Pages (de - à)	2772-2800
Nombre de pages	29
journal	SIAM Journal on Optimization
Volume	33
Numéro de publication	4
Les DOIs	https://doi.org/10.1137/23M1553455
Etat de la publication	Publié - 15 févr. 2023

Accès au document

10.1137/23M1553455

arXiv-2302.07049manuscrit soumis, 1,32 MBLicense: autre

Autres fichiers et liens

Lien vers la publication sur Scopus

3 Article
1 Livre
1 Préprint
1 Papier de travail

An optimally fast objective-function-free minimization algorithm using random subspaces
Bellavia, S., Gratton, S., Morini, B. & TOINT, P., 25 oct. 2023, Arxiv, 23 p.
Résultats de recherche: Papier de travail › Préprint

File
Convergence properties of an Objective-Function-Free Optimization regularization algorithm, including an 0(epsilon^{-3/2}) complexity bound
Gratton, S., Jerad, S. & TOINT, P., févr. 2023, Dans: SIAM Journal on Optimization. 33, 3, p. 1621-1646
Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

File
OFFO minimization algorithms for second-order optimality and their complexity
Gratton, S. & TOINT, P., 15 févr. 2023, Dans: Computational Optimization and Applications. 84, 2, p. 573-607 35 p.
Résultats de recherche: Contribution à un journal/une revue › Article › Revue par des pairs

File
19 Téléchargements (Pure)

1 Terminé
1 Actif

ADALGOPT: ADALGOPT - Algorithmes avancés en optimisation non-linéaire
Sartenaer, A. (Co-investigateur) & Toint, P. (Co-investigateur)
1/01/87 → …
Projet: Axe de recherche
Optimisation multi-échelle non-linéaire
Sartenaer, A. (Responsable du Projet), Toint, P. (Responsable du Projet), Malmedy, V. (Chercheur), Tomanos, D. (Chercheur) & Weber Mendonca, M. (Chercheur)
1/07/04 → 31/07/11
Projet: Recherche

3 Recherche/Enseignement dans une institution externe
1 Participation à une conférence, un congrès
1 Participation à un atelier/workshop, un séminaire, un cours

ALGOPT2024 workshop on Algorithmic Optimization
Toint, P. (Orateur)
27 août 2024 → 30 août 2024
Activité: Participation ou organisation d'un événement › Participation à un atelier/workshop, un séminaire, un cours
Institut National Polytechnique de Toulouse
Toint, P. (Chercheur visiteur)
19 janv. 2024 → 26 janv. 2024
Activité: Visite d'une organisation externe › Recherche/Enseignement dans une institution externe
Institut National Polytechnique de Toulouse
Toint, P. (Chercheur visiteur)
4 juil. 2023 → 8 juil. 2023
Activité: Visite d'une organisation externe › Recherche/Enseignement dans une institution externe

Contient cette citation

@article{c0c1d1fddfbb48768f5dba1e75ae1e52,

title = "Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training",

abstract = "A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.",

keywords = "complexity, deep learning, multilevel methods, neural networks, nonlinear optimization, objective-function-free optimization (OFFO)",

author = "Serge Gratton and Alena Kopanicakova and Philippe TOINT",

year = "2023",

month = feb,

day = "15",

doi = "10.1137/23M1553455",

language = "English",

volume = "33",

pages = "2772--2800",

journal = "SIAM Journal on Optimization",

issn = "1052-6234",

publisher = "Society for Industrial and Applied Mathematics Publications",

number = "4",

}

TY - JOUR

T1 - Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

AU - Gratton, Serge

AU - Kopanicakova, Alena

AU - TOINT, Philippe

PY - 2023/2/15

Y1 - 2023/2/15

N2 - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

AB - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

KW - complexity

KW - deep learning

KW - multilevel methods

KW - neural networks

KW - nonlinear optimization

KW - objective-function-free optimization (OFFO)

UR - http://www.scopus.com/inward/record.url?scp=85175637452&partnerID=8YFLogxK

U2 - 10.1137/23M1553455

DO - 10.1137/23M1553455

M3 - Article

SN - 1052-6234

VL - 33

SP - 2772

EP - 2800

JO - SIAM Journal on Optimization

JF - SIAM Journal on Optimization

IS - 4

ER -

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Résumé

Accès au document

Autres fichiers et liens

Empreinte digitale

Résultat de recherche

Projets

Activités

Contient cette citation