Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton; Alena Kopanicakova; Philippe TOINT

doi:10.1137/23M1553455

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Serge Gratton
, Alena Kopanicakova
, Philippe TOINT

Namur Institute for Complex Systems

Research output: Contribution to journal › Article › peer-review

77 Downloads (Pure)

Abstract

A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

Original language	English
Pages (from-to)	2772-2800
Number of pages	29
Journal	SIAM Journal on Optimization
Volume	33
Issue number	4
DOIs	https://doi.org/10.1137/23M1553455
Publication status	Published - 15 Feb 2023

Funding

\ast Received by the editors February 15, 2023; accepted for publication (in revised form) July 11, 2023; published electronically October 13, 2023. https://doi.org/10.1137/23M1553455 Funding: The work of the first author was partially supported by 3IA Artificial and Natural Intelligence Toulouse Institute (ANITI), French ``Investing for the Future --PIA3"" program (grant agreement ANR-19-PI3A-0004). The work of the second author was supported by the Swiss National Science Foundation through the project ``Multilevel training of DeepONets --multiscale and multiphysics applications"" (grant 206745), and by the Platform for Advanced Scientific Computing under the project EXATRAIN. The third author acknowledges the continued and friendly partial support of ANITI. \dagger Universit\e' de Toulouse, INP, IRIT, Toulouse, 31000, France ([email protected]).

Funders	Funder number
Artificial and Natural Intelligence Toulouse Institute	ANR-19-PI3A-0004
Artificial and Natural Intelligence Toulouse Institute
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung	206745
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Keywords

complexity
deep learning
multilevel methods
neural networks
nonlinear optimization
objective-function-free optimization (OFFO)

Access to Document

10.1137/23M1553455

arXiv-2302.07049Submitted manuscript, 1.32 MBLicence: Other

Cite this

@article{c0c1d1fddfbb48768f5dba1e75ae1e52,

title = "Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training",

abstract = "A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.",

keywords = "complexity, deep learning, multilevel methods, neural networks, nonlinear optimization, objective-function-free optimization (OFFO)",

author = "Serge Gratton and Alena Kopanicakova and Philippe TOINT",

year = "2023",

month = feb,

day = "15",

doi = "10.1137/23M1553455",

language = "English",

volume = "33",

pages = "2772--2800",

journal = "SIAM Journal on Optimization",

issn = "1052-6234",

publisher = "Society for Industrial and Applied Mathematics Publications",

number = "4",

}

TY - JOUR

T1 - Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

AU - Gratton, Serge

AU - Kopanicakova, Alena

AU - TOINT, Philippe

PY - 2023/2/15

Y1 - 2023/2/15

N2 - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

AB - A class of multilevel algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multilevel feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behavior in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.

KW - complexity

KW - deep learning

KW - multilevel methods

KW - neural networks

KW - nonlinear optimization

KW - objective-function-free optimization (OFFO)

UR - https://www.scopus.com/pages/publications/85175637452

U2 - 10.1137/23M1553455

DO - 10.1137/23M1553455

M3 - Article

SN - 1052-6234

VL - 33

SP - 2772

EP - 2800

JO - SIAM Journal on Optimization

JF - SIAM Journal on Optimization

IS - 4

ER -

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Complexity and performance for two classes of noise-tolerant first-order algorithms

A block-coordinate approach of multi-level optimization with an application to physics-informed neural networks

Convergence properties of an Objective-Function-Free Optimization regularization algorithm, including an 0(epsilon^{-3/2}) complexity bound

ADALGOPT: ADALGOPT - Advanced algorithms in nonlinear optimization

Multiscale nonlinear optimization

ALGOPT2024 workshop on Algorithmic Optimization

Institut National Polytechnique de Toulouse

Institut National Polytechnique de Toulouse

Cite this

Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Research output

Projects

Activities

Cite this