GPU ResNet-50 inference: OpenNN vs PyTorch vs TensorFlow

OpenNN reaches a five-run median of 107,229 samples/s in fp32 and 176,990 samples/s in bf16 on an NVIDIA GeForce RTX 4080. In the same forward-only ResNet-50 workload, that is 1.232x and 1.418x PyTorch throughput, respectively.

Contents

• Introduction
• Benchmark application
• Reference computer
• Methodology
• Results
• Discussion
• Conclusions
• Reproducing
• References

Introduction

ResNet-50 remains a useful test of convolution, normalization, residual connections, and repeated GPU kernel execution. The CIFAR-10 adaptation keeps the ResNet-50 v1.5 bottleneck structure while using the smaller spatial geometry required by 32×32 images.

This benchmark compares sustained fp32 and bf16 forward-pass throughput in OpenNN, PyTorch, and TensorFlow. It measures the optimized inference path of each framework rather than preprocessing, model construction, or cold start.

Benchmark application

Reference computer

Methodology

All three frameworks use the same CIFAR-10 input geometry, ResNet-50 v1.5 bottleneck topology, batch size, precision, and forward-only timing scope. Five successful runs are recorded for each framework and precision. The tables report the median throughput and batch time, plus the standard deviation of throughput across runs.

• OpenNN keeps inputs on the GPU and uses its captured resident-output inference path.
• PyTorch runs in evaluation mode under no_grad with PT_FAST=1, channels-last tensors, TF32 for fp32, and torch.compile(mode="reduce-overhead"). This is PyTorch’s compile-plus-CUDA-Graphs inference path.
• TensorFlow runs with training=False under XLA and uses its bf16 policy for the bf16 cell.

Dataset preparation, process startup, model construction, host-to-device setup, and file loading are outside the measured region. GPU synchronization is included before timing is finalized.

Results

OpenNN leads PyTorch by 23.2% in fp32 and 41.8% in bf16. Against TensorFlow, the corresponding leads are 103.1% and 119.1%.

Discussion

The five-run result separates the fp32 paths clearly: OpenNN records 107.2k samples/s with a standard deviation of 374 samples/s, while PyTorch reaches 87.0k samples/s and TensorFlow processes 52.8k samples/s in the same cell.

The bf16 path raises OpenNN throughput to 177.0k samples/s, a 1.65x increase over its own fp32 result. OpenNN also retains the highest throughput in this precision, reaching 1.418x PyTorch and 2.191x TensorFlow.

Relative dispersion is lowest for OpenNN in both cells: 0.35% in fp32 and 0.34% in bf16. PyTorch records 0.57% and 1.39%, while TensorFlow records 1.45% and 4.11%. These values describe repeatability in this five-run sample; they do not by themselves identify which runtime mechanism causes the difference.

The small CIFAR spatial geometry makes launch scheduling and framework overhead more visible than an ImageNet-resolution workload. The results therefore support this exact CIFAR-10 deployment shape and should not be generalized to 224×224 ResNet-50 without a separate measurement.

Conclusions

• OpenNN reaches 107,229 ± 374 samples/s in fp32 and 176,990 ± 598 samples/s in bf16.
• OpenNN is 1.232x PyTorch in fp32 and 1.418x PyTorch in bf16.
• OpenNN is 2.031x TensorFlow in fp32 and 2.191x TensorFlow in bf16.
• Five-run dispersion remains below 0.4% for OpenNN in both precision cells.
• The result applies to ResNet-50 v1.5 with CIFAR-10 geometry and batch size 128.

Reproducing

The canonical runner is docs/benchmarks/throughput/resnet50/run_resnet50_infer.py:

python docs/benchmarks/throughput/resnet50/run_resnet50_infer.py \
  --dataset cifar10 --batch 128 --runs 5 --precision both

The immutable result artifact is docs/benchmarks/results/gpu-resnet50-inference-speed-cifar10-20260714T124333Z.json.

References

• CIFAR-10 dataset.
• Deep Residual Learning for Image Recognition.
• PyTorch.
• TensorFlow.
• NVIDIA CUDA.