‹ Back to Tutorials

Configuring the Device and Precision

OpenNN uses a process-wide Configuration object to select the execution device and numeric type before a neural network is compiled.

The default automatic configuration chooses the fastest supported option available on the current machine.

Contents:

  1. Introduction
  2. The two configuration axes
  3. Use the configuration API
  4. Understand automatic resolution
  5. Choose a common configuration
  6. Handle validation and errors
  7. Query the current state
  8. Conclusions

1. Introduction

Device selection and precision affect performance, memory use and hardware compatibility. OpenNN centralizes these choices so that data buffers, layers and optimizers compile with one consistent configuration.

Call Configuration::set before constructing or compiling networks. A later configuration change is applied when a network is compiled again.

2. The two configuration axes

  • Device: Device::Auto, Device::CPU or Device::CUDA.
  • Numeric type: Type::Auto, Type::FP32, Type::BF16 or Type::INT8.

FP32 and BF16 support training. INT8 uses quantized weights with BF16 activations and is an inference-only CUDA path.

3. Use the configuration API

Include the current configuration header and set both axes in one call:

#include "opennn/core/configuration.h"

using namespace opennn;

int main()
{
    Configuration::instance().set(
        Device::CUDA,
        Type::BF16);

    // Build or compile the neural network here.
}

Both arguments default to Auto, so Configuration::instance().set() restores automatic selection.

4. Understand automatic resolution

  • Device::Auto selects CUDA when a compatible GPU is visible; otherwise it selects CPU.
  • Type::Auto selects BF16 on a CUDA GPU with compute capability 8.0 or newer.
  • Type::Auto selects FP32 on CPU and on older CUDA devices.
  • resolve_for(Device::CPU) always produces CPU FP32 without changing the global configuration.

5. Choose a common configuration

// Automatic device and precision.
Configuration::instance().set();

// CPU execution.
Configuration::instance().set(Device::CPU, Type::FP32);

// Full-precision CUDA execution.
Configuration::instance().set(Device::CUDA, Type::FP32);

// Mixed-precision CUDA training.
Configuration::instance().set(Device::CUDA, Type::BF16);

// Quantized CUDA inference.
Configuration::instance().set(Device::CUDA, Type::INT8);

INT8 networks cannot be trained because back propagation requires FP32 or BF16. Compile and use INT8 only for supported inference workflows.

6. Handle validation and errors

Explicit requests are validated when the configuration is resolved:

  • CUDA without a visible GPU throws an exception.
  • BF16 or INT8 on CPU throws an exception.
  • BF16 or INT8 on a CUDA GPU below compute capability 8.0 throws an exception.
  • Training an INT8 network throws because INT8 is inference-only.

This makes unsupported combinations visible instead of silently changing the requested device or precision.

7. Query the current state

Read the effective process-wide configuration with resolve:

const EffectiveConfig config =
    Configuration::instance().resolve();

const bool uses_cuda =
    config.device == Device::CUDA;
const bool uses_bf16 =
    config.training_type == Type::BF16;

const unsigned generation = config.generation;

The generation increases after every call to set. A compiled network also exposes get_device(), get_training_type() and is_gpu().

8. Conclusions

  • Configure OpenNN once before compiling a network.
  • Use automatic selection for portable programs.
  • Use explicit values when reproducible hardware behavior is required.
  • Choose FP32 or BF16 for training and reserve INT8 for supported CUDA inference.

References