‹ Back to Tutorials

The testing analysis class

TestingAnalysis evaluates a trained neural network against a data set. It calculates regression, classification and reconstruction metrics without changing the model.

Use the testing subset for the final independent assessment. The same methods can inspect training or validation data when diagnosing underfitting, overfitting or data drift.

Contents:

  1. Create the analysis
  2. Retrieve targets and outputs
  3. Evaluate regression
  4. Analyze error distributions
  5. Evaluate classification
  6. Analyze ROC and lift
  7. Detect reconstruction anomalies
  8. Interpret results safely

1. Create the analysis

Pass pointers to a trained network and the data set that defines the sample roles and targets:

TestingAnalysis testing_analysis(
    &neural_network,
    &dataset);

testing_analysis.set_batch_size(256);
testing_analysis.check();

check verifies that the objects are present and compatible. A batch size of zero lets OpenNN choose an appropriate value.

2. Retrieve targets and outputs

Obtain aligned target and prediction matrices for any sample role:

const auto [targets, outputs] =
    testing_analysis.get_targets_and_outputs(
        "Testing");

Valid role names include Training, Validation and Testing. The returned outputs already include the network’s scaling, activation, unscaling and clamping stages.

3. Evaluate regression

For an approximation or forecasting model, calculate the standard error vector on the testing subset:

const VectorR errors =
    testing_analysis.calculate_errors("Testing");

const auto goodness_of_fit =
    testing_analysis.perform_goodness_of_fit_analysis();

The five error positions are sum squared error, OpenNN mean squared error, its square root, normalized squared error and Minkowski error with exponent 1.5. perform_goodness_of_fit_analysis returns the coefficient of determination and the target/output pairs for every output.

OpenNN’s reported mean squared error follows the training-loss convention SSE/(2N); it is therefore half of the textbook SSE/N.

4. Analyze error distributions

Aggregate metrics can hide systematic failures. Inspect sample-level errors, percentages, descriptive statistics and histograms:

const Tensor3 error_data =
    testing_analysis.calculate_error_data();

const MatrixR percentage_errors =
    testing_analysis.calculate_percentage_error_data();

const auto descriptives =
    testing_analysis.calculate_error_data_descriptives();

const vector<Histogram> histograms =
    testing_analysis.calculate_error_data_histograms(10);

Use these results to find biased predictions, heavy tails and outputs whose errors depend on magnitude. Percentage errors need special care around targets close to zero.

5. Evaluate classification

The confusion matrix works for binary and multiclass outputs. Its last row and column contain totals:

const MatrixI confusion =
    testing_analysis.calculate_confusion(0.50f);

const VectorR tests =
    testing_analysis
        .calculate_binary_classification_tests(0.50f);

The 15 binary tests are accuracy, error rate, sensitivity, specificity, precision, positive and negative likelihood ratios, F1 score, false-positive, false-discovery and false-negative rates, negative predictive value, Matthews correlation coefficient, informedness and markedness.

calculate_binary_classification_rates returns the sample indices for true positives, false positives, false negatives and true negatives. calculate_multiple_classification_rates provides the equivalent class-by-class groups for multiclass models.

6. Analyze ROC and lift

For a binary classifier with one output, ROC analysis evaluates every decision threshold:

const TestingAnalysis::RocAnalysis roc =
    testing_analysis.perform_roc_analysis();

cout << roc.area_under_curve << '\n';
cout << roc.confidence_limit << '\n';
cout << roc.optimal_threshold << '\n';

const MatrixR lift =
    testing_analysis.perform_lift_chart_analysis();

The result contains the ROC curve, area under the curve, confidence limit and optimal threshold. Lift and cumulative gain show how effectively the model concentrates positive cases when only part of the population can be selected.

7. Detect reconstruction anomalies

Auto-associative networks are assessed with one reconstruction error per sample:

const VectorR reconstruction_errors =
    testing_analysis.calculate_reconstruction_errors(
        "Testing");

const auto statistics =
    testing_analysis
        .calculate_reconstruction_error_statistics(
            reconstruction_errors);

const float threshold =
    testing_analysis.calculate_anomaly_threshold(
        statistics, 3.0f);

const VectorI anomalies =
    testing_analysis.calculate_anomaly_predictions(
        reconstruction_errors, threshold);

The threshold is the mean plus a chosen number of population standard deviations. Calibrate that multiplier on representative normal data and confirm the operating point with labelled anomalies when available.

8. Interpret results safely

  • Do not use testing results to choose architecture, inputs, regularization or a decision threshold; that turns the testing subset into validation data.
  • Report the sample count and class balance together with metrics.
  • Inspect per-output and per-group errors, not only one global average.
  • Compare training, validation and testing results when diagnosing generalization, but reserve final claims for untouched testing data.

References

Continue with the response optimization class tutorial.