GPU Transformer inference: OpenNN vs PyTorch vs TensorFlow

OpenNN leads all six Transformer inference configurations on an NVIDIA GeForce RTX 4080, reaching 680,361 tokens/s in bf16 and up to 1.519x PyTorch throughput.

Contents

• Introduction
• Benchmark application
• Reference computer
• Methodology
• Results
• Discussion
• Conclusions
• Reproducing
• References

Introduction

Transformer inference stresses several parts of a GPU stack at once: token embeddings, multi-head attention, feed-forward blocks, layer normalization, and the vocabulary projection. Sequence length changes both the amount of work and the memory pressure, so a useful comparison must report more than one sequence and must keep the model, batch, precision, and timing boundary fixed across frameworks.

This benchmark compares OpenNN with PyTorch and TensorFlow in fp32 and bf16. OpenNN is fastest at every tested sequence length in both precisions, including the short-sequence fp32 configuration.

Benchmark application

The workload is an encoder-decoder Transformer based on the architecture from Attention Is All You Need.

Reference computer

Methodology

All engines execute the same shape, batch, sequence length, and precision. Framework initialization and warmup are outside the steady-state throughput measurement.

• OpenNN uses its device-resident inference path, uploads parameters once, reuses the activation workspace, runs fused attention, and replays a captured CUDA Graph.
• PyTorch uses nn.Transformer, bf16 autocast for the bf16 cells, and a manually captured torch.cuda.CUDAGraph. The eager path can be selected with PT_NOGRAPH=1 for diagnostic comparisons.
• TensorFlow uses a compiled tf.function/XLA path and mixed bf16 for the bf16 cells.
• Tokens per second are calculated from the fixed batch and sequence length over the timed iterations.
• The artifact stores five successful runs per engine and cell; the tables report their median and standard deviation.

The OpenNN fp32 attention path uses the fused attention implementation through its fp32-via-bf16 attention route, while the surrounding network remains fp32. The benchmark therefore measures the production fast path rather than an unfused reference implementation.

Results

fp32 throughput

bf16 throughput

Discussion

The strongest relative results are bf16 at the longer sequences. At sequence 256, OpenNN processes 680,361 tokens/s, 51.9% more than PyTorch and 58.6% more than TensorFlow. At sequence 512, OpenNN reaches 646,737 tokens/s, 51.7% more than PyTorch and 110.2% more than TensorFlow.

OpenNN also leads all three fp32 cells. The margin is 3.7% over PyTorch at sequence 128, then grows to 15.5% at sequence 256 and 30.2% at sequence 512. The short-sequence result is the narrowest lead, but it remains clearly larger than the observed five-run dispersion.

Conclusions

• OpenNN is fastest in all six tested Transformer inference configurations.
• OpenNN wins every bf16 cell, reaching a maximum of 680,361 tokens/s.
• OpenNN reaches up to 1.519x PyTorch throughput and 2.102x TensorFlow throughput.
• OpenNN leads fp32 at sequence lengths 128, 256, and 512.
• Every displayed cell is a five-run median with its standard deviation.
• The result demonstrates throughput only. A separate quality or convergence benchmark is required for claims about time to a fixed model quality.
• Together with the current ResNet-50 and HIGGS GPU inference results, OpenNN leads every tested cell in the three-model inference suite.

Reproducing

The canonical runner is docs/benchmarks/throughput/attention-speed/run_transformer.py:

python run_transformer.py --seqs 128,256,512 --batch 32 --iters 30 --runs 5 --precision both

The complete result artifact is docs/benchmarks/results/gpu-transformer-inference-20260714T140427Z.json. It records the configuration, framework versions, GPU, absolute throughputs, dispersion, and ratios.

References

• Attention Is All You Need.
• OpenNN source repository.
• PyTorch Transformer documentation.
• TensorFlow mixed precision guide.