CPU HIGGS dense inference: OpenNN vs PyTorch vs TensorFlow

OpenNN runs the canonical HIGGS dense classifier at 441,467 samples/s on eight Intel Core i9-12900K P-core threads. It delivers 1.212x PyTorch and 1.217x TensorFlow throughput in the same CPU-only inference workload.

Contents

• Introduction
• Benchmark application
• Reference computer
• Methodology
• Results
• Discussion
• Conclusions
• Reproducing
• References

Introduction

CPU inference remains important for tabular models deployed in services, local applications, and environments where a dedicated accelerator is unnecessary. The HIGGS dataset provides a large, real input matrix for comparing framework overhead and dense linear-algebra performance under sustained load.

This benchmark uses the same canonical HIGGS contract as the current training comparison: 28 normalized inputs and a 28-1024-1024-1 ReLU binary classifier. OpenNN, PyTorch, and TensorFlow process the same prepared held-out rows with the same batch size and eight-thread limit.

Benchmark application

Reference computer

Methodology

The three implementations use the same topology, activation, input file, batch size, and number of inference passes. GPUs are hidden from every engine. OpenNN runs through an executable verified to link Intel oneMKL, and each framework receives an eight-thread limit corresponding to the i9-12900K’s eight performance cores.

OpenNN and PyTorch use close core placement in the harness. TensorFlow uses the same eight-thread intra-op pool without explicit affinity because forced affinity regresses this workload. The result reported for each engine is the throughput calculated from its median pass time across three passes.

Only forward inference is timed. Dataset preparation, process startup, model construction, and file loading are outside the measured pass.

Results

OpenNN processes about 77,200 more samples per second than PyTorch and 78,600 more than TensorFlow. PyTorch and TensorFlow remain nearly level with each other in this workload.

Discussion

The current result means the OpenNN pass completes in about 82% of the time required by either competitor. Because the input contains almost half a million examples, the comparison reflects sustained dense inference rather than a single small request.

This result is specific to a large batch of 1,024 and a wide two-hidden-layer network. It does not predict single-row latency, where dispatch and framework overhead have a different weight. It also does not compare end-to-end application startup; only the forward pass is included.

The three-pass median reduces sensitivity to a slow individual pass within each process. The artifact nevertheless has one process-level run per engine, so repeated process-level runs would still be required for a formal uncertainty estimate. OpenNN’s current lead over both competitors is about 21-22%, while PyTorch and TensorFlow differ by less than 0.4%.

Conclusions

• OpenNN reaches 441,467 samples/s on eight P-core threads.
• OpenNN is 1.212x PyTorch and 1.217x TensorFlow.
• The corresponding median OpenNN pass is 1.132 seconds over 499,712 rows.
• The benchmark measures sustained batched inference, not single-sample latency or cold start.

Reproducing

The canonical runner is docs/benchmarks/throughput/higgs/run_higgs_cpu.py:

python docs/benchmarks/throughput/higgs/run_higgs_cpu.py infer \
  --engines opennn,pytorch,tensorflow \
  --test "$OPENNN_BENCH_DATA/higgs/higgs_test.csv" \
  --reps 3 --batch 1024 --hidden 1024 \
  --hidden-layers 2 --activation relu \
  --threads 8 --runs 1

The runner searches build-mkl/bin first and warns when the selected OpenNN binary does not link Intel oneMKL. The immutable result artifact is docs/benchmarks/results/cpu-dense-higgs-infer-20260714T092522Z.json.

References

• HIGGS dataset, UCI Machine Learning Repository.
• Searching for exotic particles in high-energy physics with deep learning.
• OpenNN source repository.
• Intel oneAPI Math Kernel Library.
• PyTorch.
• TensorFlow.