Software model of OpenNN

In this tutorial, we present the software model of OpenNN. The whole process uses the Unified Modeling Language (UML), a general-purpose visual modeling language used to specify, visualize, construct, and document the artifacts of a software system. To construct the model for OpenNN, we follow a top-down approach. We start at the highest conceptual level and work down to the details, iteratively building:
  1. Classes
  2. Associations
  3. Compositions
  4. Derived classes
  5. Members and methods

1. Classes

In object-oriented modeling, concepts are represented by classes. The main classes that build the conceptual model of OpenNN are shown below. 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    class Dataset
    class NeuralNetwork
    class TrainingStrategy
    class ModelSelection
    class TestingAnalysis
  • Dataset: holds the data and the utilities for loading, splitting, scaling and preprocessing it.
  • NeuralNetwork: represents a neural network as a stack of layers.
  • TrainingStrategy: glues together a loss function and an optimizer to train a network on a dataset.
  • ModelSelection: searches for the network architecture with the best generalization (input and neurons selection).
  • TestingAnalysis: evaluates a trained network against the testing samples of a dataset.

2. Associations

An association is a connection between two classes that highlights a meaningful relationship. The associations among the main OpenNN classes are: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction LR
    TrainingStrategy --> NeuralNetwork : trains
    TrainingStrategy --> Dataset : uses
    ModelSelection --> TrainingStrategy : drives
    TestingAnalysis --> NeuralNetwork : evaluates
    TestingAnalysis --> Dataset : evaluates on
  • TrainingStrategy → NeuralNetwork & Dataset: TrainingStrategy takes a network and a dataset and runs the training loop.
  • ModelSelection → TrainingStrategy: ModelSelection uses TrainingStrategy to score candidate architectures.
  • TestingAnalysis → NeuralNetwork & Dataset: TestingAnalysis measures generalization of a network on the dataset’s testing samples.

3. Compositions

The high-level classes are themselves built from smaller classes. OpenNN encapsulates each basic concept (a layer, a loss, an optimizer, a selection algorithm) in its own class and aggregates them into the larger ones.

Dataset

Dataset is the base data container. It owns a tensor of values plus per-variable metadata (roles, scalers, types). For specialized data formats, OpenNN provides four derived classes: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class Dataset
    class TabularDataset
    class ImageDataset
    class LanguageDataset
    class TimeSeriesDataset
    Dataset <|-- TabularDataset
    Dataset <|-- ImageDataset
    Dataset <|-- LanguageDataset
    Dataset <|-- TimeSeriesDataset
  • TabularDataset: CSV-like data with named columns.
  • ImageDataset: folder-of-images datasets for computer vision.
  • LanguageDataset: tokenized text data for NLP.
  • TimeSeriesDataset: sequence data with lag/lookback windows.

NeuralNetwork

A NeuralNetwork is a stack of Layer instances. The current layer catalogue is: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class Layer
    class Scaling
    class Unscaling
    class Bounding
    class Dense
    class DenseRelu
    class Recurrent
    class Convolutional
    class ConvolutionalRelu
    class Pooling
    class Pooling3d
    class Flatten
    class Embedding
    class MultiHeadAttention
    class Normalization3d
    class Addition
    Layer <|-- Scaling
    Layer <|-- Unscaling
    Layer <|-- Bounding
    Layer <|-- Dense
    Layer <|-- DenseRelu
    Layer <|-- Recurrent
    Layer <|-- Convolutional
    Layer <|-- ConvolutionalRelu
    Layer <|-- Pooling
    Layer <|-- Pooling3d
    Layer <|-- Flatten
    Layer <|-- Embedding
    Layer <|-- MultiHeadAttention
    Layer <|-- Normalization3d
    Layer <|-- Addition
  • Scaling / Unscaling: input/output normalization layers.
  • Bounding: clamps outputs to a [min, max] range.
  • Dense / DenseRelu: fully-connected layers (generic activation, or fused ReLU).
  • Recurrent: recurrent (RNN/LSTM-style) layer.
  • Convolutional / ConvolutionalRelu: 2D convolution layers (generic activation, or fused ReLU).
  • Pooling / Pooling3d: spatial pooling.
  • Flatten: rank reduction for the conv-to-dense transition.
  • Embedding: token embedding layer.
  • MultiHeadAttention / Normalization3d / Addition: transformer building blocks.
For convenience, OpenNN ships standard networks — ready-made NeuralNetwork subclasses that wire common architectures for you: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class NeuralNetwork
    class ApproximationNetwork
    class ClassificationNetwork
    class ForecastingNetwork
    class AutoAssociationNetwork
    class ImageClassificationNetwork
    class SimpleResNet
    class VGG16
    class TextClassificationNetwork
    class Transformer
    NeuralNetwork <|-- ApproximationNetwork
    NeuralNetwork <|-- ClassificationNetwork
    NeuralNetwork <|-- ForecastingNetwork
    NeuralNetwork <|-- AutoAssociationNetwork
    NeuralNetwork <|-- ImageClassificationNetwork
    NeuralNetwork <|-- SimpleResNet
    NeuralNetwork <|-- VGG16
    NeuralNetwork <|-- TextClassificationNetwork
    NeuralNetwork <|-- Transformer
  • ApproximationNetwork: regression model (Scaling → Dense* → Unscaling).
  • ClassificationNetwork: classifier (Scaling → Dense* → Dense+Softmax/Sigmoid).
  • ForecastingNetwork: time-series forecaster built around Recurrent.
  • AutoAssociationNetwork: autoencoder for anomaly detection.
  • ImageClassificationNetwork / SimpleResNet / VGG16: convolutional image classifiers.
  • TextClassificationNetwork: embedding-based text classifier.
  • Transformer: attention-based encoder-decoder for sequence tasks.

TrainingStrategy

A TrainingStrategy is built from a Loss and an Optimizer, plus pointers to the network and dataset it operates on: 
classDiagram
    direction LR
    class TrainingStrategy {
        +Loss* loss
        +Optimizer* optimizer
        +NeuralNetwork* neural_network
        +Dataset* dataset
        +train() TrainingResults
    }
    class Loss
    class Optimizer
    TrainingStrategy *-- Loss
    TrainingStrategy *-- Optimizer
    TrainingStrategy --> NeuralNetwork
    TrainingStrategy --> Dataset
  • Loss: error term computed during training. The error type (MSE, cross-entropy, etc.) is selected through an enum, not a subclass — see section 4.
  • Optimizer: parameter-update rule. Optimizer is abstract; concrete optimizers are described in section 4.

ModelSelection

ModelSelection aggregates two abstract algorithms, one for picking the input variables and one for picking the number of neurons: 
classDiagram
    direction LR
    class ModelSelection {
        +InputsSelection* inputs_selection
        +NeuronSelection* neurons_selection
        +perform_input_selection()
        +perform_neurons_selection()
    }
    class InputsSelection
    class NeuronSelection
    ModelSelection *-- InputsSelection
    ModelSelection *-- NeuronSelection
  • InputsSelection: abstract input-feature selection algorithm.
  • NeuronSelection: abstract algorithm for choosing the size of the hidden layers.

4. Derived classes

Some of the building blocks above are abstract: they exist only to be specialized by concrete subclasses. The derived class inherits the base class interface and supplies an implementation. Associations between a base and a derived class are of the form is-a.

Loss

Unlike previous OpenNN releases, Loss is now a single concrete class. The error term is selected by an enum and configured at runtime via TrainingStrategy::set_loss(name). The available error types are: 
classDiagram
    class Loss {
        +enum Error
        +MeanSquaredError
        +NormalizedSquaredError
        +WeightedSquaredError
        +CrossEntropy
        +CrossEntropy3d
        +MinkowskiError
    }

Optimizer

Optimizer is abstract and has four concrete implementations: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class Optimizer
    class AdaptiveMomentEstimation
    class QuasiNewtonMethod
    class StochasticGradientDescent
    class LevenbergMarquardtAlgorithm
    Optimizer <|-- AdaptiveMomentEstimation
    Optimizer <|-- QuasiNewtonMethod
    Optimizer <|-- StochasticGradientDescent
    Optimizer <|-- LevenbergMarquardtAlgorithm

InputsSelection

InputsSelection has two concrete strategies for choosing which features feed the network: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class InputsSelection
    class GrowingInputs
    class GeneticAlgorithm
    InputsSelection <|-- GrowingInputs
    InputsSelection <|-- GeneticAlgorithm

NeuronSelection

NeuronSelection has one concrete strategy: 
%%{init: {"class": {"hideEmptyMembersBox": true}} }%%
classDiagram
    direction TB
    class NeuronSelection
    class GrowingNeurons
    NeuronSelection <|-- GrowingNeurons

5. Members and methods

A member is a named value held by the class; a method is a procedure associated with it. The most important members and methods of the five core classes are summarized below. Full listings live in the API reference. 
classDiagram
    class Dataset {
        +set(path, separator, has_header, has_ids)
        +split_samples_random(train, sel, test)
        +scale_features(role) vector~Descriptives~
        +unscale_features(role, descriptives)
        +get_features_number(role) Index
        +get_feature_names(role) vector~string~
        +get_input_shape() Shape
        +get_target_shape() Shape
    }
classDiagram
    class NeuralNetwork {
        +add_layer(unique_ptr~Layer~)
        +get_first(name) Layer*
        +set_input_names(names)
        +set_output_names(names)
        +calculate_outputs(MatrixR) MatrixR
        +save(path)
        +load(path)
    }
classDiagram
    class TrainingStrategy {
        +TrainingStrategy(NeuralNetwork*, Dataset*)
        +set_loss(name)
        +set_optimization_algorithm(name)
        +get_loss() Loss*
        +get_optimization_algorithm() Optimizer*
        +train() TrainingResults
    }
classDiagram
    class ModelSelection {
        +ModelSelection(TrainingStrategy*)
        +set_inputs_selection(name)
        +set_neurons_selection(name)
        +perform_input_selection() InputsSelectionResults
        +perform_neurons_selection() NeuronsSelectionResults
    }
classDiagram
    class TestingAnalysis {
        +TestingAnalysis(NeuralNetwork*, Dataset*)
        +calculate_confusion(threshold) MatrixI
        +calculate_errors() MatrixR
        +perform_roc_analysis() RocAnalysis
        +print_binary_classification_tests()
    }

Bibliography

  • C. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
  • H. Demuth, M. Beale, and M. Hagan. Neural Network Toolbox User's Guide. The MathWorks, Inc., 2009.
  • S. Haykin. Neural Networks: A Comprehensive Foundation. Prentice Hall.
  • R. Lopez. Neural Networks for Variational Problems in Engineering. Ph.D. Thesis, Technical University of Catalonia, 2008.