Skip to content

‹ Back to Benchmarks

Inference performance: OpenNN vs PyTorch

OpenNN made predictions faster than PyTorch, with less peak memory and less energy, in all 6 inference configurations we measured: 18 comparisons in total.

Measured on 6 September 2026 on one reference computer: Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux. CPU runs use FP32 with Intel MKL. GPU runs use BF16 mixed precision. The benchmark code is on GitHub.

Contents

Workloads

We tested four common network types. Both libraries ran the same network on the same data, with the same batch size and the same number of parameters. CNN and Transformer were measured on the GPU only.

Model Data Network Batch Throughput unit
Dense HIGGS particle-physics events, 28 features 28 → 1024 → 1024 → 1 classifier, 1,080,321 parameters 4,096 (CPU), 8,192 (GPU) samples/s
LSTM Beijing PM2.5 hourly air-quality records 128 LSTM units over 24 steps × 15 features, 73,857 parameters 256 windows/s
CNN ImageNet subset, 1,000 classes × 50 images, 224 × 224 pixels ResNet-50 v1.5, 25,557,032 parameters 128 (GPU only) images/s
Transformer WMT14 English–German text Encoder–decoder, 6 + 6 layers, width 512, 8 heads, 20,000-token vocabulary, 74,878,496 parameters 32 (GPU only) sequences/s

Speed

Inference throughput is the amount of data a trained network processes per second after a warm-up. Each value is the median of three launches. The speedup is OpenNN’s throughput divided by PyTorch’s, so a value above 1 means OpenNN was faster.

Inference throughput
Higher is better
Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux · median of three launches
CPU · Dense · 1.29× (samples/s)
CPU · LSTM · 1.16× (windows/s)
GPU · Dense · 1.02× (samples/s)
GPU · LSTM · 5.29× (windows/s)
GPU · CNN · 1.25× (images/s)
GPU · Transformer · 1.15× (seq/s)
Bars compare the two libraries within each configuration; units differ between models.
Configuration Unit OpenNN PyTorch Speedup
CPU · Dense samples/s 220,512 170,329 1.29×
CPU · LSTM windows/s 80,285 69,304 1.16×
GPU · Dense samples/s 39,387,890 38,689,107 1.02×
GPU · LSTM windows/s 2,716,548 513,813 5.29×
GPU · CNN images/s 7,060 5,634 1.25×
GPU · Transformer seq/s 5,413 4,694 1.15×

Faster inference lets an application serve more predictions with the same hardware.

Memory

Peak memory is the highest value of three launches. On the CPU it is the process’s peak private (anonymous) resident memory. On the GPU it is the device memory in use minus the idle baseline, which includes runtime and allocator overhead. 1 MiB = 1,048,576 bytes.

Peak memory reduction with OpenNN, inference
Higher is better
Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux · bar length = reduction from 0% to 100%
Reduction = 100 × (1 − OpenNN ÷ PyTorch). PyTorch would be 0%.
Configuration OpenNN (MiB) PyTorch (MiB) Reduction
CPU · Dense 280.4 568.7 50.69%
CPU · LSTM 156.5 458.8 65.89%
GPU · Dense 371.3 411.3 9.73%
GPU · LSTM 293.6 441.6 33.51%
GPU · CNN 845.8 1,267.2 33.25%
GPU · Transformer 627.2 1,161.2 45.99%

Lower memory use leaves room for larger models, larger batches or other applications on the same device. These figures do not set a maximum model or dataset size.

Energy

Energy is the median of three launches for the work each configuration performs. CPU readings cover the processor package (Intel RAPL). GPU readings cover the graphics board (NVIDIA NVML power samples). Neither is energy at the wall socket. Each configuration does a different amount of work, so compare OpenNN with PyTorch within a row, not one row with another.

Energy reduction with OpenNN, inference
Higher is better
Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux · bar length = reduction from 0% to 100%
Reduction = 100 × (1 − OpenNN ÷ PyTorch). PyTorch would be 0%.
Configuration OpenNN (mWh) PyTorch (mWh) Reduction
CPU · Dense 97.91 106.30 7.89%
CPU · LSTM 21.56 23.32 7.56%
GPU · Dense 173.13 183.01 5.40%
GPU · LSTM 257.53 647.79 60.24%
GPU · CNN 2,397.58 3,132.69 23.47%
GPU · Transformer 11,450.14 14,999.95 23.67%

How we measured

  • Before timing, the runner checked that both libraries reported the same number of samples, input shapes and parameters.
  • Each configuration ran three launches per library. The order of the libraries rotated between launches, and the warm-up was excluded from the timing. GPU work was synchronized at both ends of the timed region.
  • GPU clocks were locked at 2,692 MHz, CPU turbo was off, and background activity was sampled before, during and after every launch.
  • Software: PyTorch 2.13.0 built for CUDA 13.0, Python 3.12, NVIDIA driver 610.43.02, Linux. OpenNN was built from development commits e76425bd3 (dense and LSTM) and 520f0f2f8 (CNN and Transformer), with clean working trees.

The programs for both libraries, the data preparation scripts and the complete measurement rules are in the benchmarks folder of the OpenNN repository. For example, to repeat the GPU dense inference run:

python benchmarks/prepare.py dense
python benchmarks/run.py --family dense --mode infer --device cuda --precision bf16 --batch 8192 --rounds 3

Limits

  • These results come from one computer and one software stack. Other hardware, drivers or library versions can change the ratios.
  • CPU results for the CNN and Transformer were not measured in this set.
  • In the GPU LSTM runs, PyTorch executed its recurrent kernels in FP16 while OpenNN used BF16, and PyTorch’s inference program kept the whole dataset on the GPU, which adds to its memory use. Treat the GPU LSTM ratios as indicative.
  • The two programs for each model are not identical in every detail: the CNN programs handle image input differently and the Transformer programs differ in masking. The input files were not content-hashed.
  • PyTorch’s CPU LSTM inference throughput varied by 4.2% between launches, above our 3% stability limit.

We will repeat these measurements with fully matched programs and add more models, datasets and hardware. See also training performance and prediction quality.