Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton; Alena Kopanicakova; Philippe TOINT

doi:10.1137/23M1553455

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton, Alena Kopanicakova, Philippe TOINT

Namur Institute for Complex Systems

Research output: Contribution to journal › Article › peer-review

27 Downloads (Pure)

Abstract

A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

Original language	English
Pages (from-to)	2772-2800
Number of pages	29
Journal	SIAM Journal on Optimization
Volume	33
Issue number	4
DOIs	https://doi.org/10.1137/23M1553455
Publication status	Published - 15 Feb 2023

Keywords

complexity
deep learning
multilevel methods
neural networks
nonlinear optimization
objective-function-free optimization (OFFO)

Access to Document

10.1137/23M1553455

arXiv-2302.07049Submitted manuscript, 1.32 MBLicence: Other

Cite this

@article{c0c1d1fddfbb48768f5dba1e75ae1e52,

title = "Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training",

abstract = "A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.",

keywords = "complexity, deep learning, multilevel methods, neural networks, nonlinear optimization, objective-function-free optimization (OFFO)",

author = "Serge Gratton and Alena Kopanicakova and Philippe TOINT",

year = "2023",

month = feb,

day = "15",

doi = "10.1137/23M1553455",

language = "English",

volume = "33",

pages = "2772--2800",

journal = "SIAM Journal on Optimization",

issn = "1052-6234",

publisher = "Society for Industrial and Applied Mathematics Publications",

number = "4",

}

TY - JOUR

T1 - Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

AU - Gratton, Serge

AU - Kopanicakova, Alena

AU - TOINT, Philippe

PY - 2023/2/15

Y1 - 2023/2/15

N2 - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

AB - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

KW - complexity

KW - deep learning

KW - multilevel methods

KW - neural networks

KW - nonlinear optimization

KW - objective-function-free optimization (OFFO)

UR - http://www.scopus.com/inward/record.url?scp=85175637452&partnerID=8YFLogxK

U2 - 10.1137/23M1553455

DO - 10.1137/23M1553455

M3 - Article

SN - 1052-6234

VL - 33

SP - 2772

EP - 2800

JO - SIAM Journal on Optimization

JF - SIAM Journal on Optimization

IS - 4

ER -

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

An optimally fast objective-function-free minimization algorithm using random subspaces

Convergence properties of an Objective-Function-Free Optimization regularization algorithm, including an 0(epsilon^{-3/2}) complexity bound

OFFO minimization algorithms for second-order optimality and their complexity

ADALGOPT: ADALGOPT - Advanced algorithms in nonlinear optimization

Multiscale nonlinear optimization

ALGOPT2024 workshop on Algorithmic Optimization

Institut National Polytechnique de Toulouse

Institut National Polytechnique de Toulouse

Cite this

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Research output

Projects

Activities

Cite this