‹ Back to Tutorials

Text Classification: Amazon reviews classification

This example trains an attention-based text classification network to label short Amazon product reviews as positive or negative.

It follows the current OpenNN Amazon reviews example, including tokenization, embedding and multi-head attention.

Contents:

  1. Identify the application type
  2. Load and tokenize the reviews
  3. Build the text classification network
  4. Configure and train the model
  5. Evaluate classification performance
  6. Classify new text and save the model
  7. Complete example

1. Identify the application type

Each document belongs to one of two sentiment classes. The objective is to estimate the positive-versus-negative class from the words in the review, making this a binary text classification problem.

2. Load and tokenize the reviews

The current amazon_cells_labelled.txt file contains 1,000 tab-separated review and label pairs: 500 positive and 500 negative samples.

LanguageDataset language_dataset(
    "../data/amazon_reviews/amazon_cells_labelled.txt");

const Index vocabulary_size =
    language_dataset.get_input_vocabulary_size();
const Index sequence_length =
    language_dataset.get_maximum_input_sequence_length();
const Index targets_number =
    language_dataset.get_features_number("Target");

LanguageDataset reads the tab-separated text, builds a word-level vocabulary, encodes each review and creates random training, validation and testing subsets using the default 60%, 20% and 20% split.

The tokenizer reserves padding, unknown, start and end tokens. The maximum sequence length is inferred from the loaded documents.

3. Build the text classification network

The maintained architecture uses a tokenizer layer, trainable embeddings with positional encoding, multi-head attention, 3D pooling and dense classification layers:

const Index embedding_dimension = 64;
const Index heads_number = 4;

TextClassificationNetwork text_network(
    {vocabulary_size, sequence_length, embedding_dimension},
    {heads_number},
    {targets_number});

text_network.set_tokenizer(
    language_dataset.get_input_tokenizer().clone());

The embedding dimension is divisible by the four attention heads. Copying the trained data-set tokenizer into the network is essential so raw text is encoded with the same vocabulary during inference.

4. Configure and train the model

Use cross-entropy with L2 regularization and Adam:

TrainingStrategy training_strategy(
    &text_network,
    &language_dataset);

training_strategy.set_loss("CrossEntropy");
training_strategy.get_loss()->set_regularization("L2");

auto* adam = dynamic_cast<AdaptiveMomentEstimation*>(
    training_strategy.get_optimization_algorithm());

if(!adam)
    throw runtime_error("Adam configuration failed.");

adam->set_maximum_epochs(50);
adam->set_display_period(10);

training_strategy.train();

TrainingStrategy selects Adam by default for the text classification task, so the cast accesses its current optimizer settings.

5. Evaluate classification performance

Calculate the confusion matrix on held-out testing reviews:

TestingAnalysis testing_analysis(
    &text_network,
    &language_dataset);

cout << "Confusion matrix:\n"
     << testing_analysis.calculate_confusion()
     << '\n';

Because this data set is balanced, the confusion matrix shows how errors are distributed between positive and negative reviews without a large class-frequency bias.

6. Classify new text and save the model

calculate_text_outputs accepts raw documents and runs the network’s tokenizer before inference:

Tensor<string, 1> documents(2);
documents(0) = "This product is amazing and I love it!";
documents(1) = "It stopped working after one day.";

const MatrixR outputs =
    text_network.calculate_text_outputs(documents);

cout << "First positive score: "
     << outputs(0, 0) << '\n';

Save the complete network, including its tokenizer configuration, for later use:

text_network.save("amazon_reviews_model.json");

7. Complete example

#include <iostream>
#include <stdexcept>

#include "opennn/core/configuration.h"
#include "opennn/dataset/language_dataset.h"
#include "opennn/models/models.h"
#include "opennn/testing_analysis/testing_analysis.h"
#include "opennn/training_strategy/adaptive_moment_estimation.h"
#include "opennn/training_strategy/training_strategy.h"

using namespace opennn;

int main()
{
    try
    {
        Configuration::instance().set(Device::Auto, Type::Auto);

        const Index embedding_dimension = 64;
        const Index heads_number = 4;

        LanguageDataset language_dataset(
            "../data/amazon_reviews/amazon_cells_labelled.txt");

        const Index vocabulary_size =
            language_dataset.get_input_vocabulary_size();
        const Index sequence_length =
            language_dataset.get_maximum_input_sequence_length();
        const Index targets_number =
            language_dataset.get_features_number("Target");

        TextClassificationNetwork text_network(
            {vocabulary_size, sequence_length, embedding_dimension},
            {heads_number},
            {targets_number});

        text_network.set_tokenizer(
            language_dataset.get_input_tokenizer().clone());

        TrainingStrategy training_strategy(
            &text_network,
            &language_dataset);

        training_strategy.set_loss("CrossEntropy");
        training_strategy.get_loss()->set_regularization("L2");

        auto* adam = dynamic_cast<AdaptiveMomentEstimation*>(
            training_strategy.get_optimization_algorithm());

        if(!adam)
            throw runtime_error("Adam configuration failed.");

        adam->set_maximum_epochs(50);
        adam->set_display_period(10);
        training_strategy.train();

        TestingAnalysis testing_analysis(
            &text_network,
            &language_dataset);
        cout << "Confusion matrix:\n"
             << testing_analysis.calculate_confusion()
             << '\n';

        Tensor<string, 1> documents(2);
        documents(0) =
            "This product is amazing and I love it!";
        documents(1) =
            "It stopped working after one day.";

        const MatrixR outputs =
            text_network.calculate_text_outputs(documents);
        cout << "First positive score: "
             << outputs(0, 0) << '\n';

        text_network.save("amazon_reviews_model.json");

        return 0;
    }
    catch(const exception& error)
    {
        cerr << error.what() << '\n';
        return 1;
    }
}

References