AdaptiveMomentEstimation Class Reference

This concrete class represents the adaptive moment estimation (Adam) optimization algorithm, based on adaptive estimates of lower-order moments. More...

#include <adaptive_moment_estimation.h>

Inheritance diagram for AdaptiveMomentEstimation:
OptimizationAlgorithm

Public Member Functions

 AdaptiveMomentEstimation ()
 
 AdaptiveMomentEstimation (LossIndex *)
 
virtual ~AdaptiveMomentEstimation ()
 Destructor. More...
 
const type & get_initial_learning_rate () const
 Returns the initial learning rate. More...
 
const type & get_beta_1 () const
 Returns beta 1. More...
 
const type & get_beta_2 () const
 Returns beta 2. More...
 
const type & get_epsilon () const
 Returns epsilon. More...
 
const type & get_loss_goal () const
 
const type & get_maximum_time () const
 Returns the maximum training time. More...
 
void set_loss_index_pointer (LossIndex *)
 
void set_batch_samples_number (const Index &new_batch_samples_number)
 Set number of samples in each batch. Default 1000. More...
 
void set_default ()
 Sets the members of the optimization algorithm object to their default values. More...
 
Index get_batch_samples_number () const
 
void set_initial_learning_rate (const type &)
 
void set_beta_1 (const type &)
 
void set_beta_2 (const type &)
 
void set_epsilon (const type &)
 
void set_maximum_epochs_number (const Index &)
 
void set_loss_goal (const type &)
 
void set_maximum_time (const type &)
 
TrainingResults perform_training ()
 
string write_optimization_algorithm_type () const
 Return the algorithm optimum for your model. More...
 
Tensor< string, 2 > to_string_matrix () const
 Writes as matrix of strings the most representative atributes. More...
 
void from_XML (const tinyxml2::XMLDocument &)
 
void write_XML (tinyxml2::XMLPrinter &) const
 
void update_parameters (LossIndexBackPropagation &, AdaptiveMomentEstimationData &)
 Update iteration parameters. More...
 
- Public Member Functions inherited from OptimizationAlgorithm
 OptimizationAlgorithm ()
 
 OptimizationAlgorithm (LossIndex *)
 
virtual ~OptimizationAlgorithm ()
 Destructor. More...
 
LossIndexget_loss_index_pointer () const
 
string get_hardware_use () const
 Hardware use. More...
 
void set_hardware_use (const string &)
 Set hardware to use. Default: Multi-core. More...
 
bool has_loss_index () const
 
const bool & get_display () const
 
const Index & get_display_period () const
 Returns the number of iterations between the training showing progress. More...
 
const Index & get_save_period () const
 Returns the number of iterations between the training saving progress. More...
 
const string & get_neural_network_file_name () const
 Returns the file name where the neural network will be saved. More...
 
const string write_time (const type &) const
 Writes the time from seconds in format HH:mm:ss. More...
 
void set ()
 
virtual void set_threads_number (const int &)
 
virtual void set_display (const bool &)
 
void set_display_period (const Index &)
 
void set_save_period (const Index &)
 
void set_neural_network_file_name (const string &)
 
virtual void check () const
 
virtual void print () const
 Prints to the screen the XML-type representation of the optimization algorithm object. More...
 
void save (const string &) const
 
void load (const string &)
 

Private Attributes

type initial_learning_rate = static_cast<type>(0.001)
 Initial learning rate. More...
 
type initial_decay = type(0)
 Learning rate decay over each update. More...
 
type beta_1 = static_cast<type>(0.9)
 Exponential decay over gradient estimates. More...
 
type beta_2 = static_cast<type>(0.999)
 Exponential decay over square gradient estimates. More...
 
type epsilon =static_cast<type>(1.e-7)
 Small number to prevent any division by zero. More...
 
type training_loss_goal = type(0)
 Goal value for the loss. It is used as a stopping criterion. More...
 
Index maximum_epochs_number = 10000
 Maximum epochs number. More...
 
Index maximum_selection_failures = numeric_limits<Index>::max()
 Maximum number of times when selection error increases. More...
 
type maximum_time = type(3600)
 Maximum training time. It is used as a stopping criterion. More...
 
Index batch_samples_number = 1000
 Training and selection batch size. More...
 

Additional Inherited Members

- Public Types inherited from OptimizationAlgorithm
enum class  StoppingCondition {
  MinimumLossDecrease , LossGoal , MaximumSelectionErrorIncreases , MaximumEpochsNumber ,
  MaximumTime
}
 Enumeration of all possibles condition of stop for the algorithms. More...
 
- Protected Attributes inherited from OptimizationAlgorithm
ThreadPool * thread_pool = nullptr
 
ThreadPoolDevice * thread_pool_device
 
LossIndexloss_index_pointer = nullptr
 Pointer to a loss index for a neural network object. More...
 
Index epochs_number = 10000
 Number of training epochs in the neural network. More...
 
string hardware_use = "Multi-core"
 Hardware use. More...
 
Index display_period = 10
 Number of iterations between the training showing progress. More...
 
Index save_period = numeric_limits<Index>::max()
 Number of iterations between the training saving progress. More...
 
string neural_network_file_name = "neural_network.xml"
 Path where the neural network is saved. More...
 
bool display = true
 Display messages to screen. More...
 
const Eigen::array< IndexPair< Index >, 1 > AT_B = {IndexPair<Index>(0, 0)}
 
const Eigen::array< IndexPair< Index >, 1 > product_vector_matrix = {IndexPair<Index>(0, 1)}
 
const Eigen::array< IndexPair< Index >, 1 > A_B = {IndexPair<Index>(1, 0)}
 

Detailed Description

This concrete class represents the adaptive moment estimation (Adam) optimization algorithm, based on adaptive estimates of lower-order moments.

For more information visit:

[1] C. Barranquero "High performance optimization algorithms for neural networks." https://www.opennn.net/files/high_performance_optimization_algorithms_for_neural_networks.pdf .

[2] D. P. Kingma and J. L. Ba, "Adam: A Method for Stochastic Optimization." arXiv preprint arXiv:1412.6980v8 (2014).

Definition at line 51 of file adaptive_moment_estimation.h.

Constructor & Destructor Documentation

◆ AdaptiveMomentEstimation() [1/2]

Default constructor. It creates a adaptive moment estimation optimization algorithm not associated to any loss index object. It also initializes the class members to their default values.

Definition at line 18 of file adaptive_moment_estimation.cpp.

◆ AdaptiveMomentEstimation() [2/2]

AdaptiveMomentEstimation ( LossIndex new_loss_index_pointer)
explicit

Loss index constructor. It creates a adaptive moment estimation optimization algorithm associated to a loss index. It also initializes the class members to their default values.

Parameters
new_loss_index_pointerPointer to a loss index object.

Definition at line 30 of file adaptive_moment_estimation.cpp.

◆ ~AdaptiveMomentEstimation()

Destructor.

Definition at line 39 of file adaptive_moment_estimation.cpp.

Member Function Documentation

◆ from_XML()

void from_XML ( const tinyxml2::XMLDocument document)
virtual

Loads a default optimization algorithm from a XML document.

Parameters
documentTinyXML document containing the error term members.

Reimplemented from OptimizationAlgorithm.

Definition at line 595 of file adaptive_moment_estimation.cpp.

◆ get_batch_samples_number()

Index get_batch_samples_number ( ) const

Definition at line 720 of file adaptive_moment_estimation.cpp.

◆ get_beta_1()

const type & get_beta_1 ( ) const

Returns beta 1.

Definition at line 54 of file adaptive_moment_estimation.cpp.

◆ get_beta_2()

const type & get_beta_2 ( ) const

Returns beta 2.

Definition at line 62 of file adaptive_moment_estimation.cpp.

◆ get_epsilon()

const type & get_epsilon ( ) const

Returns epsilon.

Definition at line 70 of file adaptive_moment_estimation.cpp.

◆ get_initial_learning_rate()

const type & get_initial_learning_rate ( ) const

Returns the initial learning rate.

Definition at line 46 of file adaptive_moment_estimation.cpp.

◆ get_loss_goal()

const type & get_loss_goal ( ) const

Returns the goal value for the loss. This is used as a stopping criterion when training a neural network

Definition at line 79 of file adaptive_moment_estimation.cpp.

◆ get_maximum_time()

const type & get_maximum_time ( ) const

Returns the maximum training time.

Definition at line 87 of file adaptive_moment_estimation.cpp.

◆ perform_training()

TrainingResults perform_training ( )
virtual

Trains a neural network with an associated loss index, according to the gradient descent method. Training occurs according to the training parameters and stopping criteria. It returns a results structure with the history and the final values of the reserved variables.

Implements OptimizationAlgorithm.

Definition at line 206 of file adaptive_moment_estimation.cpp.

◆ set_batch_samples_number()

void set_batch_samples_number ( const Index &  new_batch_samples_number)

Set number of samples in each batch. Default 1000.

Definition at line 708 of file adaptive_moment_estimation.cpp.

◆ set_beta_1()

void set_beta_1 ( const type &  new_beta_1)

Sets beta 1 generally close to 1.

Parameters
new_beta_1Type to set.

Definition at line 115 of file adaptive_moment_estimation.cpp.

◆ set_beta_2()

void set_beta_2 ( const type &  new_beta_2)

Sets beta 2 generally close to 1.

Parameters
new_beta_2Type to set.

Definition at line 124 of file adaptive_moment_estimation.cpp.

◆ set_default()

void set_default ( )
virtual

Sets the members of the optimization algorithm object to their default values.

Reimplemented from OptimizationAlgorithm.

Definition at line 714 of file adaptive_moment_estimation.cpp.

◆ set_epsilon()

void set_epsilon ( const type &  new_epsilon)

Sets epsilon.

Parameters
new_epsilonType to set.

Definition at line 133 of file adaptive_moment_estimation.cpp.

◆ set_initial_learning_rate()

void set_initial_learning_rate ( const type &  new_learning_rate)

Sets a new learning rate.

Parameters
new_learning_rateType to set.

Definition at line 106 of file adaptive_moment_estimation.cpp.

◆ set_loss_goal()

void set_loss_goal ( const type &  new_loss_goal)

Sets a new goal value for the loss. This is used as a stopping criterion when training a neural network

Parameters
new_loss_goalGoal value for the loss.

Definition at line 169 of file adaptive_moment_estimation.cpp.

◆ set_loss_index_pointer()

void set_loss_index_pointer ( LossIndex new_loss_index_pointer)
virtual

Sets a pointer to a loss index object to be associated to the gradient descent object. It also sets that loss index to the learning rate algorithm.

Parameters
new_loss_index_pointerPointer to a loss index object.

Reimplemented from OptimizationAlgorithm.

Definition at line 97 of file adaptive_moment_estimation.cpp.

◆ set_maximum_epochs_number()

void set_maximum_epochs_number ( const Index &  new_maximum_epochs_number)

Set the a new maximum for the epochs number.

Parameters
new_maximum_epochs_numberNew maximum epochs number.

Definition at line 142 of file adaptive_moment_estimation.cpp.

◆ set_maximum_time()

void set_maximum_time ( const type &  new_maximum_time)

Sets a new maximum training time.

Parameters
new_maximum_timeMaximum training time.

Definition at line 178 of file adaptive_moment_estimation.cpp.

◆ to_string_matrix()

Tensor< string, 2 > to_string_matrix ( ) const
virtual

Writes as matrix of strings the most representative atributes.

Reimplemented from OptimizationAlgorithm.

Definition at line 472 of file adaptive_moment_estimation.cpp.

◆ update_parameters()

void update_parameters ( LossIndexBackPropagation back_propagation,
AdaptiveMomentEstimationData optimization_data 
)

Update iteration parameters.

Definition at line 728 of file adaptive_moment_estimation.cpp.

◆ write_optimization_algorithm_type()

string write_optimization_algorithm_type ( ) const
virtual

Return the algorithm optimum for your model.

Reimplemented from OptimizationAlgorithm.

Definition at line 464 of file adaptive_moment_estimation.cpp.

◆ write_XML()

void write_XML ( tinyxml2::XMLPrinter file_stream) const
virtual

Serializes the gradient descent object into a XML document of the TinyXML library without keep the DOM tree in memory. See the OpenNN manual for more information about the format of this document.

Reimplemented from OptimizationAlgorithm.

Definition at line 528 of file adaptive_moment_estimation.cpp.

Member Data Documentation

◆ batch_samples_number

Index batch_samples_number = 1000
private

Training and selection batch size.

Definition at line 167 of file adaptive_moment_estimation.h.

◆ beta_1

type beta_1 = static_cast<type>(0.9)
private

Exponential decay over gradient estimates.

Definition at line 137 of file adaptive_moment_estimation.h.

◆ beta_2

type beta_2 = static_cast<type>(0.999)
private

Exponential decay over square gradient estimates.

Definition at line 141 of file adaptive_moment_estimation.h.

◆ epsilon

type epsilon =static_cast<type>(1.e-7)
private

Small number to prevent any division by zero.

Definition at line 145 of file adaptive_moment_estimation.h.

◆ initial_decay

type initial_decay = type(0)
private

Learning rate decay over each update.

Definition at line 133 of file adaptive_moment_estimation.h.

◆ initial_learning_rate

type initial_learning_rate = static_cast<type>(0.001)
private

Initial learning rate.

Definition at line 129 of file adaptive_moment_estimation.h.

◆ maximum_epochs_number

Index maximum_epochs_number = 10000
private

Maximum epochs number.

Definition at line 155 of file adaptive_moment_estimation.h.

◆ maximum_selection_failures

Index maximum_selection_failures = numeric_limits<Index>::max()
private

Maximum number of times when selection error increases.

Definition at line 159 of file adaptive_moment_estimation.h.

◆ maximum_time

type maximum_time = type(3600)
private

Maximum training time. It is used as a stopping criterion.

Definition at line 163 of file adaptive_moment_estimation.h.

◆ training_loss_goal

type training_loss_goal = type(0)
private

Goal value for the loss. It is used as a stopping criterion.

Definition at line 151 of file adaptive_moment_estimation.h.


The documentation for this class was generated from the following files: