opennn::Optimizer Class Reference

Abstract base class for every training algorithm in OpenNN. More...

#include <optimizer.h>

Inheritance diagram for opennn::Optimizer:

Public Types
enum class	StoppingCondition {   None , MinimumLossDecrease , LossGoal , MaximumSelectionErrorIncreases ,   MaximumEpochsNumber , MaximumTime }
	Reasons that can terminate a training run. More...

Public Member Functions
	Optimizer (Loss *loss=nullptr)
	Constructs an optimizer bound to a loss function.
virtual	~Optimizer ()=default
	Virtual destructor.
const Loss *	get_loss () const
	Read-only access to the loss being optimized.
bool	get_display () const
	Whether progress should be printed to stdout during training.
void	set (Loss *new_loss)
	Re-initializes the optimizer by setting its loss pointer.
virtual void	set_loss (Loss *new_loss)
	Updates the loss pointer; subclasses may override to refresh cached state derived from the loss.
virtual void	set_display (bool new_display)
	Toggles per-epoch progress printing.
void	set_display_period (const Index new_display_period)
	Sets how often progress is printed.
void	set_maximum_epochs (const Index new_maximum_epochs)
	Sets the maximum number of epochs.
void	set_maximum_time (const float new_maximum_time)
	Sets the maximum wall-clock training time.
void	set_loss_goal (const float new_loss_goal)
	Sets the training-loss goal.
void	set_maximum_validation_failures (const Index new_maximum_validation_failures)
	Sets the maximum number of consecutive validation-error increases tolerated.
virtual TrainingResults	train ()=0
	Runs the optimization to completion.
const string &	get_name () const
	Canonical name of the optimizer (set by subclasses).
virtual void	print () const
	Prints a human-readable summary of the optimizer to stdout.
virtual void	from_JSON (const JsonDocument &)
	Loads optimizer hyperparameters from a parsed JSON document.
virtual void	to_JSON (JsonWriter &) const
	Writes optimizer hyperparameters to a streaming JSON writer.
void	save (const filesystem::path &) const
	Saves the optimizer state to a file.
void	load (const filesystem::path &)
	Loads the optimizer state from a file.

Static Public Member Functions
static float	get_elapsed_time (const time_t &beginning_time)
	Computes the elapsed wall-clock time since a reference instant.

Protected Member Functions
void	set_names ()
	Subclass hook to refresh layer name caches after a loss change.
void	set_scaling ()
	Subclass hook to install the dataset-derived input scalers.
void	set_unscaling ()
	Subclass hook to install the dataset-derived output unscalers.
bool	check_stopping_condition (TrainingResults &results, Index epoch, float elapsed_time, float training_error, Index validation_failures) const
	Evaluates every stopping criterion and updates the result accordingly.
void	write_common_xml (JsonWriter &) const
	Writes the common Optimizer fields to JSON.
void	read_common_xml (const Json *)
	Reads the common Optimizer fields from JSON.
void	setup_device_training ()
	Allocates the CUDA stream and events used for batch prefetching.
void	teardown_device_training ()
	Releases the CUDA stream and events allocated by setup_device_training().
void	prefetch_batch (Batch &batch, Index sample_count, int slot)
	Asynchronously prefetches the next training batch into a slot.
void	wait_prefetch (int slot)
	Waits for the prefetch into a given slot to finish.
void	sync_device ()
	Synchronizes the device on the optimizer's CUDA stream.
bool	should_display (Index epoch) const
	Whether the current epoch should print progress.
EpochStats	train_epoch (bool is_classification, ForwardPropagation &forward_propagation, BackPropagation &back_propagation, ThreadSafeQueue< Batch * > &empty_queue, ThreadSafeQueue< Batch * > &ready_queue, const vector< vector< Index > > &batches, const vector< Index > &input_feature_indices, const vector< Index > &decoder_feature_indices, const vector< Index > &target_feature_indices, const std::function< void(BackPropagation &)> &update)
	Runs a single training epoch over all batches.
EpochStats	evaluate_epoch (bool is_classification, ForwardPropagation &forward_propagation, ThreadSafeQueue< Batch * > &empty_queue, ThreadSafeQueue< Batch * > &ready_queue, const vector< vector< Index > > &batches, const vector< Index > &input_feature_indices, const vector< Index > &decoder_feature_indices, const vector< Index > &target_feature_indices)
	Runs a single evaluation pass over all batches without updating parameters.

Static Protected Member Functions
static void	clip_gradient_norm (Buffer &gradient, float max_norm)
	In-place gradient norm clipping.

Protected Attributes
Loss *	loss = nullptr
	Loss being optimized; not owned.
float	training_loss_goal = 0.0f
	Training stops when the training loss reaches this value.
Index	maximum_validation_failures = numeric_limits<Index>::max()
	Maximum number of consecutive validation-error increases tolerated.
Index	maximum_epochs = 10000
	Maximum number of training epochs.
float	maximum_time = 360000.0f
	Maximum wall-clock training time in seconds.
Index	display_period = 10
	Number of epochs between progress prints.
bool	display = true
	Whether progress should be printed to stdout during training.
string	name
	Canonical name of the optimizer (set by subclasses).
cudaStream_t	memory_stream = nullptr
	CUDA stream used to prefetch batches into device memory.
cudaEvent_t	batch_ready_event [2] = {nullptr, nullptr}
	CUDA events signaling when each prefetched batch is ready.

Detailed Description

Abstract base class for every training algorithm in OpenNN.

Holds the loss function pointer, the training stopping criteria (epoch budget, time budget, loss goal, validation failure budget) and the device-side machinery used to overlap data transfer with compute (CUDA streams, batch prefetching queues).

Subclasses implement train(); the base class provides the helpers train_epoch() and evaluate_epoch() that drive a complete pass over the dataset.

Member Enumeration Documentation

StoppingCondition

enum class opennn::Optimizer::StoppingCondition

strong

Reasons that can terminate a training run.

Enumerator
None	No stopping condition was hit (still running).
MinimumLossDecrease	Loss decrease between epochs fell below the threshold.
LossGoal	Training loss reached the configured goal.
MaximumSelectionErrorIncreases	Validation error increased too many times.
MaximumEpochsNumber	Configured epoch budget exhausted.
MaximumTime

Constructor & Destructor Documentation

Optimizer()

opennn::Optimizer::Optimizer

(

Loss *

loss = nullptr

)

Constructs an optimizer bound to a loss function.

Parameters

loss	Loss to optimize; may be nullptr if set later via set_loss().

~Optimizer()

virtual opennn::Optimizer::~Optimizer ( )

virtualdefault

Virtual destructor.

Member Function Documentation

check_stopping_condition()

bool opennn::Optimizer::check_stopping_condition ( TrainingResults & results, Index epoch, float elapsed_time, float training_error, Index validation_failures ) const

protected

Evaluates every stopping criterion and updates the result accordingly.

Parameters

results	Mutable training results being built incrementally.
epoch	Current epoch index.
elapsed_time	Elapsed time in seconds since training started.
training_error	Current training error.
validation_failures	Consecutive validation-error increases observed.

Returns

True if training should stop.

clip_gradient_norm()

static void opennn::Optimizer::clip_gradient_norm ( Buffer & gradient, float max_norm )

staticprotected

In-place gradient norm clipping.

Parameters

gradient	Gradient buffer to clip.
max_norm	Maximum allowed L2 norm.

evaluate_epoch()

EpochStats opennn::Optimizer::evaluate_epoch ( bool is_classification, ForwardPropagation & forward_propagation, ThreadSafeQueue< Batch * > & empty_queue, ThreadSafeQueue< Batch * > & ready_queue, const vector< vector< Index > > & batches, const vector< Index > & input_feature_indices, const vector< Index > & decoder_feature_indices, const vector< Index > & target_feature_indices )

protected

Runs a single evaluation pass over all batches without updating parameters.

Parameters

is_classification	Whether the model produces class probabilities.
forward_propagation	Forward buffer reused across batches.
empty_queue	Queue feeding the prefetcher with empty batches.
ready_queue	Queue receiving batches that are ready to evaluate.
batches	Per-batch sample index lists.
input_feature_indices	Indices of dataset input columns.
decoder_feature_indices	Indices of dataset decoder columns.
target_feature_indices	Indices of dataset target columns.

Returns

Mean error and accuracy over the epoch.

from_JSON()

virtual void opennn::Optimizer::from_JSON ( const JsonDocument & )

virtual

Loads optimizer hyperparameters from a parsed JSON document.

Reimplemented in opennn::AdaptiveMomentEstimation, opennn::LevenbergMarquardtAlgorithm, opennn::QuasiNewtonMethod, and opennn::StochasticGradientDescent.

get_display()

bool opennn::Optimizer::get_display ( ) const

inline

Whether progress should be printed to stdout during training.

get_elapsed_time()

static float opennn::Optimizer::get_elapsed_time ( const time_t & beginning_time )

static

Computes the elapsed wall-clock time since a reference instant.

Parameters

beginning_time

Reference instant produced by std::time().

Returns

Elapsed time in seconds.

get_loss()

const Loss * opennn::Optimizer::get_loss ( ) const

inline

Read-only access to the loss being optimized.

get_name()

const string & opennn::Optimizer::get_name ( ) const

inline

Canonical name of the optimizer (set by subclasses).

load()

void opennn::Optimizer::load

(

const filesystem::path &

)

Loads the optimizer state from a file.

Receives the source path.

prefetch_batch()

void opennn::Optimizer::prefetch_batch ( Batch & batch, Index sample_count, int slot )

protected

Asynchronously prefetches the next training batch into a slot.

Parameters

batch	Batch buffer that will be filled.
sample_count	Number of samples in this batch.
slot	Double-buffer slot index (0 or 1).

print()

virtual void opennn::Optimizer::print ( ) const

inlinevirtual

Prints a human-readable summary of the optimizer to stdout.

read_common_xml()

void opennn::Optimizer::read_common_xml ( const Json * )

protected

Reads the common Optimizer fields from JSON.

Receives the JSON node; called by subclasses' from_JSON().

save()

void opennn::Optimizer::save

(

const filesystem::path &

)

const

Saves the optimizer state to a file.

Receives the destination path.

set()

void opennn::Optimizer::set ( Loss * new_loss )

inline

Re-initializes the optimizer by setting its loss pointer.

Parameters

new_loss

Loss to optimize.

set_display()

virtual void opennn::Optimizer::set_display ( bool new_display )

inlinevirtual

Toggles per-epoch progress printing.

Parameters

new_display

True to print progress to stdout.

set_display_period()

void opennn::Optimizer::set_display_period ( const Index new_display_period )

inline

Sets how often progress is printed.

Parameters

new_display_period

Number of epochs between progress prints.

set_loss()

virtual void opennn::Optimizer::set_loss ( Loss * new_loss )

inlinevirtual

Updates the loss pointer; subclasses may override to refresh cached state derived from the loss.

Parameters

new_loss

Loss to optimize.

set_loss_goal()

void opennn::Optimizer::set_loss_goal ( const float new_loss_goal )

inline

Sets the training-loss goal.

Parameters

new_loss_goal

Training stops when the loss reaches this value.

set_maximum_epochs()

void opennn::Optimizer::set_maximum_epochs ( const Index new_maximum_epochs )

inline

Sets the maximum number of epochs.

Parameters

new_maximum_epochs

Epoch budget; training stops after that many epochs.

set_maximum_time()

void opennn::Optimizer::set_maximum_time ( const float new_maximum_time )

inline

Sets the maximum wall-clock training time.

Parameters

new_maximum_time

Time budget in seconds.

set_maximum_validation_failures()

void opennn::Optimizer::set_maximum_validation_failures ( const Index new_maximum_validation_failures )

inline

Sets the maximum number of consecutive validation-error increases tolerated.

Parameters

new_maximum_validation_failures

Failure budget for early stopping.

set_names()

void opennn::Optimizer::set_names ( )

protected

Subclass hook to refresh layer name caches after a loss change.

set_scaling()

void opennn::Optimizer::set_scaling ( )

protected

Subclass hook to install the dataset-derived input scalers.

set_unscaling()

void opennn::Optimizer::set_unscaling ( )

protected

Subclass hook to install the dataset-derived output unscalers.

setup_device_training()

void opennn::Optimizer::setup_device_training ( )

protected

Allocates the CUDA stream and events used for batch prefetching.

should_display()

bool opennn::Optimizer::should_display ( Index epoch ) const

inlineprotected

Whether the current epoch should print progress.

Parameters

epoch

Current epoch index.

Returns

True if display is enabled and the epoch is on the period.

sync_device()

void opennn::Optimizer::sync_device ( )

protected

Synchronizes the device on the optimizer's CUDA stream.

teardown_device_training()

void opennn::Optimizer::teardown_device_training ( )

protected

Releases the CUDA stream and events allocated by setup_device_training().

to_JSON()

virtual void opennn::Optimizer::to_JSON ( JsonWriter & ) const

virtual

Writes optimizer hyperparameters to a streaming JSON writer.

Reimplemented in opennn::AdaptiveMomentEstimation, opennn::LevenbergMarquardtAlgorithm, opennn::QuasiNewtonMethod, and opennn::StochasticGradientDescent.

train()

virtual TrainingResults opennn::Optimizer::train ( )

pure virtual

Runs the optimization to completion.

Returns

Per-epoch error history and the stopping condition that fired.

Implemented in opennn::AdaptiveMomentEstimation, opennn::LevenbergMarquardtAlgorithm, opennn::QuasiNewtonMethod, and opennn::StochasticGradientDescent.

train_epoch()

EpochStats opennn::Optimizer::train_epoch ( bool is_classification, ForwardPropagation & forward_propagation, BackPropagation & back_propagation, ThreadSafeQueue< Batch * > & empty_queue, ThreadSafeQueue< Batch * > & ready_queue, const vector< vector< Index > > & batches, const vector< Index > & input_feature_indices, const vector< Index > & decoder_feature_indices, const vector< Index > & target_feature_indices, const std::function< void(BackPropagation &)> & update )

protected

Runs a single training epoch over all batches.

Parameters

is_classification	Whether the model produces class probabilities.
forward_propagation	Forward buffer reused across batches.
back_propagation	Back buffer reused across batches.
empty_queue	Queue feeding the prefetcher with empty batches.
ready_queue	Queue receiving batches that are ready to train on.
batches	Per-batch sample index lists.
input_feature_indices	Indices of dataset input columns.
decoder_feature_indices	Indices of dataset decoder columns (empty when the model has no decoder).
target_feature_indices	Indices of dataset target columns.
update	Subclass-supplied parameter update functor.

Returns

Mean error and accuracy over the epoch.

wait_prefetch()

void opennn::Optimizer::wait_prefetch ( int slot )

protected

Waits for the prefetch into a given slot to finish.

Parameters

slot	Double-buffer slot index (0 or 1).

write_common_xml()

void opennn::Optimizer::write_common_xml ( JsonWriter & ) const

protected

Writes the common Optimizer fields to JSON.

Receives the streaming JSON writer; called by subclasses' to_JSON().

Member Data Documentation

batch_ready_event

cudaEvent_t opennn::Optimizer::batch_ready_event[2] = {nullptr, nullptr}

protected

CUDA events signaling when each prefetched batch is ready.

display

bool opennn::Optimizer::display = true

protected

Whether progress should be printed to stdout during training.

display_period

Index opennn::Optimizer::display_period = 10

protected

Number of epochs between progress prints.

loss

Loss* opennn::Optimizer::loss = nullptr

protected

Loss being optimized; not owned.

maximum_epochs

Index opennn::Optimizer::maximum_epochs = 10000

protected

Maximum number of training epochs.

maximum_time

float opennn::Optimizer::maximum_time = 360000.0f

protected

Maximum wall-clock training time in seconds.

maximum_validation_failures

Index opennn::Optimizer::maximum_validation_failures = numeric_limits<Index>::max()

protected

Maximum number of consecutive validation-error increases tolerated.

memory_stream

cudaStream_t opennn::Optimizer::memory_stream = nullptr

protected

CUDA stream used to prefetch batches into device memory.

name

string opennn::Optimizer::name

protected

Canonical name of the optimizer (set by subclasses).

training_loss_goal

float opennn::Optimizer::training_loss_goal = 0.0f

protected

Training stops when the training loss reaches this value.