Levenberg-Marquardt Algorithm will always compute the approximate Hessian matrix, which has dimensions n-by-n. More...

#include <levenberg_marquardt_algorithm.h>

Inheritance diagram for LevenbergMarquardtAlgorithm:

Public Member Functions
	LevenbergMarquardtAlgorithm ()

	LevenbergMarquardtAlgorithm (LossIndex *)

virtual	~LevenbergMarquardtAlgorithm ()

const type &	get_minimum_loss_decrease () const
	Returns the minimum loss improvement during training. More...

const type &	get_loss_goal () const

const Index &	get_maximum_selection_failures () const
	Returns the maximum number of selection failures during the training process. More...

const Index &	get_maximum_epochs_number () const
	Returns the maximum number of iterations for training. More...

const type &	get_maximum_time () const
	Returns the maximum training time. More...

const type &	get_damping_parameter () const
	Returns the damping parameter for the hessian approximation. More...

const type &	get_damping_parameter_factor () const
	Returns the damping parameter factor(beta in the User's Guide) for the hessian approximation. More...

const type &	get_minimum_damping_parameter () const
	Returns the minimum damping parameter allowed in the algorithm. More...

const type &	get_maximum_damping_parameter () const
	Returns the maximum damping parameter allowed in the algorithm. More...

void	set_default ()

void	set_damping_parameter (const type &)

void	set_damping_parameter_factor (const type &)

void	set_minimum_damping_parameter (const type &)

void	set_maximum_damping_parameter (const type &)

void	set_minimum_loss_decrease (const type &)

void	set_loss_goal (const type &)

void	set_maximum_selection_failures (const Index &)

void	set_maximum_epochs_number (const Index &)

void	set_maximum_time (const type &)

void	check () const

TrainingResults	perform_training ()

void	update_parameters (const DataSetBatch &, NeuralNetworkForwardPropagation &, LossIndexBackPropagationLM &, LevenbergMarquardtAlgorithmData &)
	LevenbergMarquardtAlgorithm::update_parameters. More...

string	write_optimization_algorithm_type () const
	Writes the optimization algorithm type. More...

Tensor< string, 2 >	to_string_matrix () const
	Writes as matrix of strings the most representative atributes. More...

void	from_XML (const tinyxml2::XMLDocument &)

void	write_XML (tinyxml2::XMLPrinter &) const

Public Member Functions inherited from OptimizationAlgorithm
	OptimizationAlgorithm ()

	OptimizationAlgorithm (LossIndex *)

virtual	~OptimizationAlgorithm ()
	Destructor. More...

LossIndex *	get_loss_index_pointer () const

string	get_hardware_use () const
	Hardware use. More...

void	set_hardware_use (const string &)
	Set hardware to use. Default: Multi-core. More...

bool	has_loss_index () const

const bool &	get_display () const

const Index &	get_display_period () const
	Returns the number of iterations between the training showing progress. More...

const Index &	get_save_period () const
	Returns the number of iterations between the training saving progress. More...

const string &	get_neural_network_file_name () const
	Returns the file name where the neural network will be saved. More...

const string	write_time (const type &) const
	Writes the time from seconds in format HH:mm:ss. More...

void	set ()

virtual void	set_threads_number (const int &)

virtual void	set_loss_index_pointer (LossIndex *)

virtual void	set_display (const bool &)

void	set_display_period (const Index &)

void	set_save_period (const Index &)

void	set_neural_network_file_name (const string &)

virtual void	print () const
	Prints to the screen the XML-type representation of the optimization algorithm object. More...

void	save (const string &) const

void	load (const string &)

Private Attributes
type	damping_parameter
	Initial Levenberg-Marquardt parameter. More...

type	minimum_damping_parameter
	Minimum Levenberg-Marquardt parameter. More...

type	maximum_damping_parameter
	Maximum Levenberg-Marquardt parameter. More...

type	damping_parameter_factor
	Damping parameter increase/decrease factor. More...

type	minimum_loss_decrease
	Minimum loss improvement between two successive iterations. It is used as a stopping criterion. More...

type	training_loss_goal
	Goal value for the loss. It is used as a stopping criterion. More...

Index	maximum_selection_failures

Index	maximum_epochs_number
	Maximum number of epoch to perform_training. It is used as a stopping criterion. More...

type	maximum_time
	Maximum training time. It is used as a stopping criterion. More...

Additional Inherited Members
Public Types inherited from OptimizationAlgorithm
enum class	StoppingCondition { MinimumLossDecrease , LossGoal , MaximumSelectionErrorIncreases , MaximumEpochsNumber , MaximumTime }
	Enumeration of all possibles condition of stop for the algorithms. More...

Protected Attributes inherited from OptimizationAlgorithm
NonBlockingThreadPool *	non_blocking_thread_pool = nullptr

ThreadPoolDevice *	thread_pool_device

LossIndex *	loss_index_pointer = nullptr
	Pointer to a loss index for a neural network object. More...

Index	epochs_number = 10000
	Number of training epochs in the neural network. More...

string	hardware_use = "Multi-core"
	Hardware use. More...

Index	display_period = 10
	Number of iterations between the training showing progress. More...

Index	save_period = numeric_limits<Index>::max()
	Number of iterations between the training saving progress. More...

string	neural_network_file_name = "neural_network.xml"
	Path where the neural network is saved. More...

bool	display = true
	Display messages to screen. More...

const Eigen::array< IndexPair< Index >, 1 >	AT_B = {IndexPair<Index>(0, 0)}

const Eigen::array< IndexPair< Index >, 1 >	product_vector_matrix = {IndexPair<Index>(0, 1)}

const Eigen::array< IndexPair< Index >, 1 >	A_B = {IndexPair<Index>(1, 0)}

Detailed Description

Levenberg-Marquardt Algorithm will always compute the approximate Hessian matrix, which has dimensions n-by-n.

This concrete class represents a Levenberg-Marquardt Algorithm training algorithm[1], use to minimize loss function.

[1] Neural Designer "5 Algorithms to Train a Neural Network." https://www.neuraldesigner.com/blog/5_algorithms_to_train_a_neural_network

Definition at line 47 of file levenberg_marquardt_algorithm.h.

Constructor & Destructor Documentation

◆ LevenbergMarquardtAlgorithm() [1/2]

LevenbergMarquardtAlgorithm ( )

explicit

Default constructor. It creates a Levenberg-Marquardt optimization algorithm object not associated to any loss index object. It also initializes the class members to their default values.

Definition at line 18 of file levenberg_marquardt_algorithm.cpp.

◆ LevenbergMarquardtAlgorithm() [2/2]

LevenbergMarquardtAlgorithm ( LossIndex * new_loss_index_pointer )

explicit

Loss index constructor. It creates a Levenberg-Marquardt optimization algorithm object associated associated with a given loss index object. It also initializes the class members to their default values.

Parameters

new_loss_index_pointer Pointer to an external loss index object.

Definition at line 30 of file levenberg_marquardt_algorithm.cpp.

◆ ~LevenbergMarquardtAlgorithm()

~LevenbergMarquardtAlgorithm ( )

virtual

Destructor. This destructor does not delete any object.

Definition at line 40 of file levenberg_marquardt_algorithm.cpp.

Member Function Documentation

◆ check()

void check ( ) const

virtual

Checks that the Levenberg-Marquard object is ok for training. In particular, it checks that:

The loss index pointer associated to the optimization algorithm is not nullptr,
The neural network associated to that loss index is neither nullptr.
The data set associated to that loss index is neither nullptr.

If that checkings are not hold, an exception is thrown.

Reimplemented from OptimizationAlgorithm.

Definition at line 324 of file levenberg_marquardt_algorithm.cpp.

◆ from_XML()

void from_XML ( const tinyxml2::XMLDocument & document )

virtual

Loads a Levenberg-Marquardt method object from a XML document. Please mind about the format, wich is specified in the OpenNN manual.

Parameters

document TinyXML document containint the object data.

Reimplemented from OptimizationAlgorithm.

Definition at line 845 of file levenberg_marquardt_algorithm.cpp.

◆ get_damping_parameter()

const type & get_damping_parameter ( ) const

Returns the damping parameter for the hessian approximation.

Definition at line 88 of file levenberg_marquardt_algorithm.cpp.

◆ get_damping_parameter_factor()

const type & get_damping_parameter_factor ( ) const

Returns the damping parameter factor(beta in the User's Guide) for the hessian approximation.

Definition at line 96 of file levenberg_marquardt_algorithm.cpp.

◆ get_loss_goal()

const type & get_loss_goal ( ) const

Returns the goal value for the loss. This is used as a stopping criterion when training a neural network.

Definition at line 56 of file levenberg_marquardt_algorithm.cpp.

◆ get_maximum_damping_parameter()

const type & get_maximum_damping_parameter ( ) const

Returns the maximum damping parameter allowed in the algorithm.

Definition at line 112 of file levenberg_marquardt_algorithm.cpp.

◆ get_maximum_epochs_number()

const Index & get_maximum_epochs_number ( ) const

Returns the maximum number of iterations for training.

Definition at line 72 of file levenberg_marquardt_algorithm.cpp.

◆ get_maximum_selection_failures()

const Index & get_maximum_selection_failures ( ) const

Returns the maximum number of selection failures during the training process.

Definition at line 64 of file levenberg_marquardt_algorithm.cpp.

◆ get_maximum_time()

const type & get_maximum_time ( ) const

Returns the maximum training time.

Definition at line 80 of file levenberg_marquardt_algorithm.cpp.

◆ get_minimum_damping_parameter()

const type & get_minimum_damping_parameter ( ) const

Returns the minimum damping parameter allowed in the algorithm.

Definition at line 104 of file levenberg_marquardt_algorithm.cpp.

◆ get_minimum_loss_decrease()

const type & get_minimum_loss_decrease ( ) const

Returns the minimum loss improvement during training.

Definition at line 47 of file levenberg_marquardt_algorithm.cpp.

◆ perform_training()

TrainingResults perform_training ( )

virtual

Trains a neural network with an associated loss index according to the Levenberg-Marquardt algorithm. Training occurs according to the training parameters.

Implements OptimizationAlgorithm.

Definition at line 364 of file levenberg_marquardt_algorithm.cpp.

◆ set_damping_parameter()

void set_damping_parameter ( const type & new_damping_parameter )

Sets a new damping parameter(lambda in the User's Guide) for the hessian approximation.

Parameters

new_damping_parameter Damping parameter value.

Definition at line 163 of file levenberg_marquardt_algorithm.cpp.

◆ set_damping_parameter_factor()

void set_damping_parameter_factor ( const type & new_damping_parameter_factor )

Sets a new damping parameter factor(beta in the User's Guide) for the hessian approximation.

Parameters

new_damping_parameter_factor Damping parameter factor value.

Definition at line 183 of file levenberg_marquardt_algorithm.cpp.

◆ set_default()

void set_default ( )

virtual

Sets the following default values for the Levenberg-Marquardt algorithm: Training parameters:

Levenberg-Marquardt parameter: 0.001.

Stopping criteria:

Loss goal: 1.0e-6.
Maximum training time: 1000 secondata_set.
Maximum number of epochs: 1000.

User stuff:

Iterations between showing progress: 10.

Reimplemented from OptimizationAlgorithm.

Definition at line 134 of file levenberg_marquardt_algorithm.cpp.

◆ set_loss_goal()

void set_loss_goal ( const type & new_loss_goal )

Sets a new goal value for the loss. This is used as a stopping criterion when training a neural network.

Parameters

new_loss_goal Goal value for the loss.

Definition at line 265 of file levenberg_marquardt_algorithm.cpp.

◆ set_maximum_damping_parameter()

void set_maximum_damping_parameter ( const type & new_maximum_damping_parameter )

Sets a new maximum damping parameter allowed in the algorithm.

Parameters

new_maximum_damping_parameter Maximum damping parameter value.

Definition at line 231 of file levenberg_marquardt_algorithm.cpp.

◆ set_maximum_epochs_number()

void set_maximum_epochs_number ( const Index & new_maximum_epochs_number )

Sets a maximum number of iterations for training.

Parameters

new_maximum_epochs_number Maximum number of epochs for training.

Definition at line 285 of file levenberg_marquardt_algorithm.cpp.

◆ set_maximum_selection_failures()

void set_maximum_selection_failures ( const Index & new_maximum_selection_failures )

Sets a new maximum number of selection error increases.

Parameters

new_maximum_selection_failures Maximum number of epochs in which the selection evalutation increases.

Definition at line 275 of file levenberg_marquardt_algorithm.cpp.

◆ set_maximum_time()

void set_maximum_time ( const type & new_maximum_time )

Sets a new maximum training time.

Parameters

new_maximum_time Maximum training time.

Definition at line 294 of file levenberg_marquardt_algorithm.cpp.

◆ set_minimum_damping_parameter()

void set_minimum_damping_parameter ( const type & new_minimum_damping_parameter )

Sets a new minimum damping parameter allowed in the algorithm.

Parameters

new_minimum_damping_parameter Minimum damping parameter value.

Definition at line 207 of file levenberg_marquardt_algorithm.cpp.

◆ set_minimum_loss_decrease()

void set_minimum_loss_decrease ( const type & new_minimum_loss_decrease )

Sets a new minimum loss improvement during training.

Parameters

new_minimum_loss_decrease Minimum improvement in the loss between two iterations.

Definition at line 255 of file levenberg_marquardt_algorithm.cpp.

◆ to_string_matrix()

Tensor< string, 2 > to_string_matrix ( ) const

virtual

Writes as matrix of strings the most representative atributes.

Reimplemented from OptimizationAlgorithm.

Definition at line 712 of file levenberg_marquardt_algorithm.cpp.

◆ update_parameters()

void update_parameters	(	const DataSetBatch &	batch,
		NeuralNetworkForwardPropagation &	forward_propagation,
		LossIndexBackPropagationLM &	back_propagation_lm,
		LevenbergMarquardtAlgorithmData &	optimization_data
	)

LevenbergMarquardtAlgorithm::update_parameters.

Parameters

batch
forward_propagation
back_propagation
loss_index_back_propagation_lm
optimization_data

Definition at line 617 of file levenberg_marquardt_algorithm.cpp.

◆ write_optimization_algorithm_type()

string write_optimization_algorithm_type ( ) const

virtual

Writes the optimization algorithm type.

Reimplemented from OptimizationAlgorithm.

Definition at line 704 of file levenberg_marquardt_algorithm.cpp.

◆ write_XML()

void write_XML ( tinyxml2::XMLPrinter & file_stream ) const

virtual

Serializes the Levenberg Marquardt algorithm object into a XML document of the TinyXML library without keep the DOM tree in memory. See the OpenNN manual for more information about the format of this document.

Reimplemented from OptimizationAlgorithm.

Definition at line 754 of file levenberg_marquardt_algorithm.cpp.

Member Data Documentation

◆ damping_parameter

type damping_parameter

private

Initial Levenberg-Marquardt parameter.

Definition at line 132 of file levenberg_marquardt_algorithm.h.

◆ damping_parameter_factor

type damping_parameter_factor

private

Damping parameter increase/decrease factor.

Definition at line 144 of file levenberg_marquardt_algorithm.h.

◆ maximum_damping_parameter

type maximum_damping_parameter

private

Maximum Levenberg-Marquardt parameter.

Definition at line 140 of file levenberg_marquardt_algorithm.h.

◆ maximum_epochs_number

Index maximum_epochs_number

private

Maximum number of epoch to perform_training. It is used as a stopping criterion.

Definition at line 163 of file levenberg_marquardt_algorithm.h.

◆ maximum_selection_failures

Index maximum_selection_failures

private

Maximum number of epochs at which the selection error increases. This is an early stopping method for improving selection.

Definition at line 159 of file levenberg_marquardt_algorithm.h.

◆ maximum_time

type maximum_time

private

Maximum training time. It is used as a stopping criterion.

Definition at line 167 of file levenberg_marquardt_algorithm.h.

◆ minimum_damping_parameter

type minimum_damping_parameter

private

Minimum Levenberg-Marquardt parameter.

Definition at line 136 of file levenberg_marquardt_algorithm.h.

◆ minimum_loss_decrease

type minimum_loss_decrease

private

Minimum loss improvement between two successive iterations. It is used as a stopping criterion.

Definition at line 150 of file levenberg_marquardt_algorithm.h.

◆ training_loss_goal

type training_loss_goal

private

Goal value for the loss. It is used as a stopping criterion.

Definition at line 154 of file levenberg_marquardt_algorithm.h.

The documentation for this class was generated from the following files:

Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ LevenbergMarquardtAlgorithm() [1/2]

◆ LevenbergMarquardtAlgorithm() [2/2]

◆ ~LevenbergMarquardtAlgorithm()

Member Function Documentation

◆ check()

◆ from_XML()

◆ get_damping_parameter()

◆ get_damping_parameter_factor()

◆ get_loss_goal()

◆ get_maximum_damping_parameter()

◆ get_maximum_epochs_number()

◆ get_maximum_selection_failures()

◆ get_maximum_time()

◆ get_minimum_damping_parameter()

◆ get_minimum_loss_decrease()

◆ perform_training()

◆ set_damping_parameter()

◆ set_damping_parameter_factor()

◆ set_default()

◆ set_loss_goal()

◆ set_maximum_damping_parameter()

◆ set_maximum_epochs_number()

◆ set_maximum_selection_failures()

◆ set_maximum_time()

◆ set_minimum_damping_parameter()

◆ set_minimum_loss_decrease()

◆ to_string_matrix()

◆ update_parameters()

◆ write_optimization_algorithm_type()

◆ write_XML()

Member Data Documentation

◆ damping_parameter

◆ damping_parameter_factor

◆ maximum_damping_parameter

◆ maximum_epochs_number

◆ maximum_selection_failures

◆ maximum_time

◆ minimum_damping_parameter

◆ minimum_loss_decrease

◆ training_loss_goal