‹ Back to Tutorials

The data set class

Dataset is the common interface for the data consumed by OpenNN. It stores variables, expanded model features, sample roles, shapes and the data path, while concrete subclasses load tabular, time-series, image and language data.

This tutorial uses TabularDataset, the concrete class for CSV files, and points out the specialized classes available for other data types.

Contents:

  1. Choose a data set class
  2. Load tabular data
  3. Inspect variables and features
  4. Set roles and split samples
  5. Handle missing values
  6. Configure scaling
  7. Analyze and clean data
  8. Save the configuration

1. Choose a data set class

Dataset is an abstract base class. Create the subclass that matches the source data:

  • TabularDataset reads CSV files and supports numeric, integer, binary, categorical, date-time and constant variables.
  • TimeSeriesDataset extends tabular data with past and future time steps, sequential splitting and auto/cross-correlations.
  • ImageDataset reads class-labelled image folders and supports resizing, caching and data augmentation.
  • LanguageDataset reads paired text for text classification and sequence-to-sequence models.
  • TextGenerationDataset converts a corpus into token blocks for language modelling.
  • BertDataset prepares token, segment and attention data from a text file and a fixed vocabulary.
  • YoloDataset reads image and label directories for object-detection training.

Every subclass exposes the common sample, variable, feature, shape, batching and persistence operations defined by Dataset.

2. Load tabular data

The current Iris example is a semicolon-separated CSV file with a header row and no sample-index column:

TabularDataset dataset(
    "../data/iris_plant/iris_plant_original.csv",
    ";",
    true,
    false);

The constructor reads the file immediately. The equivalent step-by-step form is useful when the separator or other options are decided at runtime:

TabularDataset dataset;

dataset.set_data_path(
    "../data/iris_plant/iris_plant_original.csv");
dataset.set_separator(Dataset::Separator::Semicolon);
dataset.set_has_header(true);
dataset.set_has_ids(false);
dataset.read_csv();

OpenNN detects column types, uses the last variable as the target by default and randomly assigns samples to training, validation and testing subsets using 60%, 20% and 20%.

3. Inspect variables and features

A variable is a source column. A feature is a numeric value presented to the neural network. Numeric variables normally contribute one feature, while a categorical variable contributes one feature per category.

const Index samples_number = dataset.get_samples_number();
const Index variables_number = dataset.get_variables_number();

const Index input_features =
    dataset.get_features_number(VariableRole::Input);
const Index target_features =
    dataset.get_features_number(VariableRole::Target);

const vector<string> input_names =
    dataset.get_feature_names("Input");
const vector<string> target_names =
    dataset.get_feature_names("Target");

For Iris there are five source variables, but seven model features: four numeric inputs and three one-hot target features.

Retrieve a subset as a matrix when direct inspection is needed:

const MatrixR training_inputs =
    dataset.get_data("Training", "Input");

const MatrixR testing_targets =
    dataset.get_data("Testing", "Target");

4. Set roles and split samples

Variable roles are Input, Target, Decoder, InputTarget, Time and None. Assign them by index or name:

dataset.set_variable_role(
    "iris_class",
    VariableRole::Target);

dataset.set_variable_role(
    "sepal_width",
    VariableRole::Input);

Sample roles are Training, Validation, Testing and None. Recreate the default random split, choose different ratios or preserve order with a sequential split:

dataset.split_samples(0.6f, 0.2f, 0.2f, true);

// Appropriate for ordered observations and time series.
dataset.split_samples_sequential(0.7f, 0.15f, 0.15f);

// Individual samples can also be assigned explicitly.
dataset.set_sample_role(0, SampleRole::Testing);

Training estimates the model parameters, validation controls stopping and model selection, and testing is reserved for the final independent evaluation.

5. Handle missing values

TabularDataset recognizes a configurable missing-value label. Missing inputs can be excluded or imputed with the mean, median or interpolation:

dataset.set_missing_values_label("NA");
dataset.set_missing_values_method(
    TabularDataset::MissingValuesMethod::Median);

dataset.scrub_missing_values();

The available methods are Unuse, Mean, Median and Interpolation. Rows with missing targets are not used for supervised training.

6. Configure scaling

Each variable stores its scaler. The default is mean-standard-deviation for numeric/integer variables and minimum-maximum for other supported variable types:

dataset.set_default_variable_scalers();

Normal training does not require changing the matrix manually. The optimizer calculates statistics from the training subset and configures the network scaling and unscaling layers automatically.

For an explicit preprocessing workflow, scale and later restore a feature group as follows:

const vector<Descriptives> input_descriptives =
    dataset.scale_features("Input");

dataset.unscale_features(
    "Input",
    input_descriptives);

7. Analyze and clean data

TabularDataset provides descriptive statistics, distributions, box plots, correlations, outlier detection and feature filtering:

const vector<Descriptives> descriptives =
    dataset.calculate_feature_descriptives("Input");

const auto correlations =
    dataset.calculate_input_target_variable_pearson_correlations();

const auto outliers =
    dataset.calculate_Tukey_outliers();

const vector<string> removed_inputs =
    dataset.unuse_uncorrelated_variables(0.25f);

unuse_collinear_variables removes strongly redundant inputs, while unuse_least_correlated_variables keeps a requested number of inputs. These operations change variable roles; they do not delete the original columns.

8. Save the configuration

Save the data-set configuration, roles, scaling choices and source path as JSON:

dataset.save("iris_dataset.json");

TabularDataset restored_dataset;
restored_dataset.load("iris_dataset.json");

For large tabular, image and language data, OpenNN can use binary caches and device-resident storage to avoid repeatedly parsing or transferring the source data.

References

Continue with the neural network class tutorial.