‹ Back to Benchmarks

GPU fp32 vs bf16 precision sweep in OpenNN

Across four matched OpenNN GPU workloads on an NVIDIA GeForce RTX 4080, bf16 improves throughput by 1.26x to 2.37x over fp32. Dense inference records the largest gain at 2.371x, followed by Transformer inference at 2.094x.

This is an OpenNN precision comparison, not a cross-framework benchmark. The artifact contains one successful run per workload and precision, so the figures are controlled snapshots without repeated-run uncertainty.

Contents

Introduction

Precision is one of the most consequential GPU performance choices. fp32 provides a familiar numerical format, while bf16 reduces storage and bandwidth requirements and maps efficiently to modern tensor-core hardware while retaining the same exponent width as fp32.

This sweep answers a narrow question: with the workload shape held fixed, how much additional OpenNN throughput does bf16 provide over fp32? It covers both Transformer and dense models, in inference and training, so the answer does not depend on a single execution path.

What the sweep measures

Workload Fixed configuration Timed iterations Throughput unit
Transformer inference seq256, d512, 8 heads, ff2048, 6 layers, batch 32 80 tokens/s
Transformer training d512, 8 heads, ff2048, 6 layers, batch 32 12 samples/s
Dense inference 28-4096-4096-1, batch/tile 8,000 100 samples/s
Dense training 28-4096-4096-1, batch/tile 2,000 100 samples/s

The dense driver uses synthetic data with the same 28-input binary-classifier shape as HIGGS. It is a precision-path measurement rather than a dataset quality benchmark.

Reference computer

Component Value
GPU NVIDIA GeForce RTX 4080, 16 GB
NVIDIA driver 595.71.05
Python 3.12.3
OpenNN 9.0.0
Run ID 20260710T085143Z

Methodology

Each pair uses the same OpenNN executable, model shape, batch, iteration count, and timing metric. The OPENNN_BF16 environment selector changes only the requested precision path: unset for fp32 and set for bf16.

The Transformer inference cell reports GPU-bound tokens per second; Transformer training and both dense cells report samples per second. Since those units and model shapes differ, absolute rates should only be compared within a row. The cross-workload summary uses the dimensionless bf16/fp32 speedup.

The dense cells use 100 timed steps so fixed setup, cuBLASLt heuristic selection, parameter mirrors, and GPU clock ramp do not dominate a short measurement. One run was recorded per cell; the stored standard deviation is therefore zero by construction and is not an uncertainty estimate.

Results

OpenNN bf16 speedup over fp32
Higher is better
Matched workload shape ยท dimensionless speedup
Transformer infer
2.094x
Transformer train
1.853x
Dense infer
2.371x
Dense train
1.261x
Workload fp32 bf16 bf16 / fp32
Transformer inference 309,882 tokens/s 648,812 tokens/s 2.094x
Transformer training 345.4 samples/s 640.1 samples/s 1.853x
Dense inference 1,159,110 samples/s 2,748,510 samples/s 2.371x
Dense training 243,286 samples/s 306,717 samples/s 1.261x

bf16 is faster in all four matched cells. Rounded to two decimals, the gains are 2.09x for Transformer inference, 1.85x for Transformer training, 2.37x for dense inference, and 1.26x for dense training.

Why bf16 is faster

On this OpenNN GPU path, bf16 lets dense, feed-forward, and projection matrix multiplications use bf16 tensor-core execution. The fp32 configuration performs more expensive arithmetic: dense GEMMs use the configured TF32 compute path, while attention includes conversions around its bf16 flash-attention implementation.

bf16 also halves the bytes required for each scalar relative to fp32. That can reduce memory traffic and improve effective cache capacity. The realized gain depends on whether a workload is dominated by matrix multiplication, memory movement, launch overhead, or setup work, which is why the four speedups are not identical.

Discussion

Inference benefits most in this sweep. Dense inference is 2.371x faster and Transformer inference is 2.094x faster. Transformer training still gains 1.853x, while dense training improves by a more modest 1.261x because its timed path includes work that does not scale in direct proportion to matrix precision.

These figures are performance ratios, not a claim that bf16 and fp32 are numerically interchangeable for every model. bf16 preserves fp32’s exponent range but has fewer significand bits. Model quality, stability, loss scaling, and any sensitive reductions should be validated for the target application before precision is selected only from throughput.

The ratios also belong to these exact shapes on an RTX 4080. Smaller models may be launch-bound, and different GPUs can have different tensor-core, bandwidth, and software characteristics.

Conclusions

  • bf16 improves all four OpenNN workloads in the sweep.
  • The largest gain is dense inference at 2.371x.
  • Transformer inference and training gain 2.094x and 1.853x.
  • Dense training gains 1.261x.
  • The result quantifies speed; application-specific numerical quality still needs separate validation.

Reproducing

Build the Transformer and dense benchmark drivers, create the Transformer corpus, and run:

cd docs/benchmarks/throughput/precision-sweep
python run_precision_sweep.py \
  --runs 1 \
  --workloads transformer,dense \
  --modes inference,training

For a stronger uncertainty estimate, use at least five runs. The immutable source artifact for the published table is docs/benchmarks/results/gpu-precision-sweep-20260710T085143Z.json.

References