‹ Back to Tutorials

The training strategy class

TrainingStrategy connects a neural network and a data set with the loss function and optimizer used to estimate the model parameters.

It selects suitable defaults from the network task, while exposing the loss, regularization, optimizer, stopping criteria, batching and hardware controls needed for a custom training run.

Contents:

  1. Create a training strategy
  2. Choose the loss function
  3. Configure regularization
  4. Choose an optimizer
  5. Set stopping and batching
  6. Train the model
  7. Use callbacks and CUDA
  8. Save the strategy

1. Create a training strategy

Construct the object with pointers to the network and data set. Both objects must remain alive during training:

TrainingStrategy training_strategy(
    &neural_network,
    &dataset);

set_default chooses a loss and optimizer from the current NetworkTask. Approximation and forecasting default to mean squared error and Adam. Classification chooses weighted squared error for a single output or cross-entropy for multiple outputs, while image, object-detection, text and language tasks use their corresponding cross-entropy or detection losses.

2. Choose the loss function

Replace the default by its registered name:

training_strategy.set_loss("MeanSquaredError");

Loss* loss = training_strategy.get_loss();

The maintained loss names are:

  • MeanSquaredError and MeanAbsoluteError for general regression.
  • NormalizedSquaredError when error should be relative to target variability.
  • WeightedSquaredError for imbalanced binary targets.
  • CrossEntropy for classification and CrossEntropyError3d for token sequences.
  • MinkowskiError for a robust compromise between absolute and squared error.
  • YoloError for object-detection networks.

3. Configure regularization

Regularization is configured on the active Loss. Use L1 to encourage sparse parameters, L2 to shrink large parameters, or None to disable the penalty:

Loss* loss = training_strategy.get_loss();

loss->set_regularization("L2");
loss->set_regularization_weight(0.001f);

The regularization term is added to the training objective. Validation and testing errors continue to measure predictive performance without the penalty.

4. Choose an optimizer

Select an optimizer by name, then cast only when algorithm-specific settings are required:

training_strategy.set_optimization_algorithm(
    "AdaptiveMomentEstimation");

auto* adam = dynamic_cast<AdaptiveMomentEstimation*>(
    training_strategy.get_optimization_algorithm());

if(adam)
{
    adam->set_learning_rate(0.001f);
    adam->set_batch_size(32);
}

The current optimizers are:

  • AdaptiveMomentEstimation (Adam) for mini-batch training and large data sets.
  • StochasticGradientDescent for mini-batch training with learning-rate decay, momentum and optional Nesterov acceleration.
  • QuasiNewtonMethod for fast full-batch training on small and medium tabular data.
  • LevenbergMarquardt for high-accuracy full-batch least-squares problems.

5. Set stopping and batching

All optimizers share the main stopping, validation and batching controls:

Optimizer* optimizer =
    training_strategy.get_optimization_algorithm();

optimizer->set_maximum_epochs(1000);
optimizer->set_maximum_time(3600.0f);
optimizer->set_loss_goal(1.0e-4f);
optimizer->set_maximum_validation_failures(20);
optimizer->set_validation_period(1);
optimizer->set_restore_best(true);
optimizer->set_gradient_clip_norm(5.0f);
optimizer->set_batch_size(32);

Training stops when one configured criterion is met. With restore_best enabled, the parameters with the lowest validation error are restored before the result is returned.

6. Train the model

train prepares scaling from the training subset, iterates over training batches, evaluates validation data and updates the network in place:

TrainingResult result = training_strategy.train();

result.print();

const float training_error =
    result.get_training_error();
const float validation_error =
    result.get_validation_error();

The result contains the training and validation histories, elapsed time, stopping condition, final loss and—when used—the epoch whose best parameters were restored.

7. Use callbacks and CUDA

Callbacks can inspect the model after an epoch or batch, or react when a new best validation value is found:

Optimizer* optimizer =
    training_strategy.get_optimization_algorithm();

optimizer->post_epoch_callback =
    [](Index epoch, float training_error,
       float validation_error, NeuralNetwork*)
    {
        // Record custom metrics or checkpoints here.
    };

When the network is configured for CUDA, supported losses and mini-batch optimizers run on the GPU. Adam and SGD can also enable CUDA graph execution to reduce launch overhead:

optimizer->set_workers_number(4);
optimizer->set_cuda_graph(true);

8. Save the strategy

Persist the selected loss, regularization, optimizer and training settings as JSON:

training_strategy.save("training_strategy.json");

TrainingStrategy restored_strategy(
    &neural_network,
    &dataset);
restored_strategy.load("training_strategy.json");

The trained parameters belong to NeuralNetwork and must be saved with the network itself.

References

Continue with the model selection class tutorial.