Accelerating backpropagation using effective parameters at each step and an experimental evaluation

Abstract

An acceleration of backpropagation algorithm with momentum (BPM) is introduced. At every stage of the learning process, local quadratic approximation of the error function is performed and the Hessian matrix of the quadratic function is approximated. Effective learning rate and momentum factor are determined by means of maximum and minimum eigenvalues of the approximated Hessian matrix at each step. BPM algorithm is modified so as to work automatically with these effective parameters. Performance of this new approach is demonstrated in comparison with well-known training algorithms on conventional problems by an experimental evaluation.