GPU HIGGS dense inference: OpenNN vs PyTorch vs TensorFlow

OpenNN leads HIGGS dense inference in both fp32 and bf16 on an NVIDIA GeForce RTX 4080, reaching 15.34 million samples/s in fp32 and 34.84 million samples/s in bf16.

Contents

• Introduction
• Benchmark application
• Reference computer
• Methodology
• Results
• Discussion
• Conclusions
• Reproducing
• References

Introduction

Inference removes gradient and optimizer work from the HIGGS dense network and measures the forward path alone. This exposes device residency, kernel launch overhead, dense GEMM efficiency, activation fusion, and the cost of the selected precision.

The comparison uses the same 28-1024-1024-1 ReLU network in OpenNN, PyTorch, and TensorFlow. OpenNN leads both precision cells, with a larger relative advantage in fp32 and a narrower field in bf16.

Benchmark application

Reference computer

Methodology

Each engine processes the same held-out rows with the same network, batch, activation, parameter count, and precision. Labels are ignored by the timed inference path.

• OpenNN uses calculate_outputs_resident, keeps parameters and activations on the GPU, and replays a captured CUDA Graph.
• PyTorch uses inference mode, bf16 autocast for the bf16 cell, and a manually captured torch.cuda.CUDAGraph. The eager path remains available with PT_NOGRAPH=1 for diagnostic comparisons.
• OpenNN and PyTorch copy each resident batch into a fixed capture buffer with one device-to-device copy before graph replay, so both graph paths use stable pointers under the same staging contract.
• TensorFlow uses compiled graph execution and mixed bf16 for the bf16 cell.
• Framework warmup and TensorFlow XLA compilation occur before the timed passes.
• Samples per second and milliseconds per batch are reported from the same measured pass.
• The artifact contains five successful runs per engine and precision; the tables report medians and standard deviations across those runs.

Dataset loading, process startup, model construction, the initial host-to-device upload, graph capture, and warmup are outside the measured region. The per-batch device-to-device staging copy is included.

Results

Discussion

In fp32, OpenNN is 41.7% faster than PyTorch and 35.0% faster than TensorFlow. In bf16, all three frameworks are closer: OpenNN remains first, with a 9.8% lead over PyTorch and a 9.9% lead over TensorFlow.

OpenNN’s own bf16 path is 2.27x faster than its fp32 path for this exact model and batch. PyTorch and TensorFlow also gain substantially, which explains why the competitive bf16 margin is smaller than the fp32 margin.

These are steady-state, device-resident figures. The five-run sample also shows that the lead is not an isolated pass: all 30 framework-and-precision executions completed successfully, and the reported result for each cell is their median.

Conclusions

• OpenNN leads both fp32 and bf16 HIGGS dense inference cells.
• OpenNN reaches 15.34 million samples/s in fp32 and 34.84 million samples/s in bf16.
• The fp32 lead is substantial; the bf16 lead is real but comparatively narrow.
• OpenNN is 1.417x PyTorch and 1.350x TensorFlow in fp32.
• OpenNN is approximately 1.10x both frameworks in bf16.
• CUDA Graphs and symmetric fixed-buffer staging are part of the optimized OpenNN and PyTorch benchmark contract.

Reproducing

The canonical runner is docs/benchmarks/throughput/higgs-gpu/run_higgs_infer.py:

python run_higgs_infer.py \
  --test "$OPENNN_BENCH_DATA/higgs/higgs_test.csv" \
  --batch 8192 --hidden 1024 --hidden-layers 2 \
  --activation relu --precision both --runs 5

The result artifact is docs/benchmarks/results/gpu-higgs-dense-inference-speed-20260714T125416Z.json.

References

• HIGGS dataset, UCI Machine Learning Repository.
• Searching for exotic particles in high-energy physics with deep learning.
• OpenNN source repository.
• PyTorch.
• TensorFlow.