Startup time: OpenNN vs PyTorch
From process launch to the first prediction, OpenNN applications took 63% to 99.8% less time than the same applications written with PyTorch’s Python API.
Measured on 15 September 2026 on the reference computer (Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux): 660 timed launches, 15 for each configuration and library. Values are medians.
Results
Lower is better
Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux · median of 15 launches · GPU at FP32 with an empty tuning cache
CPU Eigen · Dense · 99.8% less time
OpenNN
PyTorch
CPU MKL · Dense · 98.2% less time
OpenNN
PyTorch
GPU · Dense · 69.6% less time
OpenNN
PyTorch
GPU · LSTM · 68.3% less time
OpenNN
PyTorch
GPU · CNN · 62.8% less time
OpenNN
PyTorch
GPU · Transformer · 72.3% less time
OpenNN
PyTorch
Each launch starts a new process. No trained model file is loaded.
| Configuration | OpenNN (ms) | PyTorch (ms) | Reduction |
|---|---|---|---|
| CPU Eigen · Dense | 1.11 | 497.86 | 99.78% |
| CPU MKL · Dense | 8.96 | 497.86 | 98.20% |
| GPU · Dense | 238.09 | 784.30 | 69.64% |
| GPU · LSTM | 253.70 | 800.68 | 68.31% |
| GPU · CNN | 316.38 | 849.48 | 62.76% |
| GPU · Transformer | 243.28 | 877.87 | 72.29% |
A shorter startup means an application, command-line tool, or serverless function can deliver its first prediction sooner after launch.
All configurations
The table above shows the CPU Dense applications and the four GPU applications at FP32 with an empty tuning cache. These are all the measured configurations:
| Backend | Application | Precision | GPU tuning cache | OpenNN (ms) | PyTorch (ms) | Reduction |
|---|---|---|---|---|---|---|
| CPU Eigen | Dense | FP32 | – | 1.11 | 497.86 | 99.78% |
| CPU Eigen | LSTM | FP32 | – | 3.58 | 497.31 | 99.28% |
| CPU Eigen | CNN | FP32 | – | 2.47 | 496.38 | 99.50% |
| CPU Eigen | Transformer | FP32 | – | 1.76 | 500.77 | 99.65% |
| CPU MKL | Dense | FP32 | – | 8.96 | 497.86 | 98.20% |
| CPU MKL | LSTM | FP32 | – | 11.82 | 497.31 | 97.62% |
| CPU MKL | CNN | FP32 | – | 10.30 | 496.38 | 97.93% |
| CPU MKL | Transformer | FP32 | – | 9.58 | 500.77 | 98.09% |
| GPU | Dense | FP32 | Empty | 238.09 | 784.30 | 69.64% |
| GPU | Dense | FP32 | Reused | 237.15 | 783.68 | 69.74% |
| GPU | Dense | BF16 | Empty | 254.30 | 812.14 | 68.69% |
| GPU | Dense | BF16 | Reused | 253.00 | 812.69 | 68.87% |
| GPU | LSTM | FP32 | Empty | 253.70 | 800.68 | 68.31% |
| GPU | LSTM | FP32 | Reused | 249.07 | 794.70 | 68.66% |
| GPU | LSTM | BF16 | Empty | 262.14 | 816.94 | 67.91% |
| GPU | LSTM | BF16 | Reused | 258.15 | 809.43 | 68.11% |
| GPU | CNN | FP32 | Empty | 316.38 | 849.48 | 62.76% |
| GPU | CNN | FP32 | Reused | 312.75 | 845.22 | 63.00% |
| GPU | CNN | BF16 | Empty | 341.29 | 916.75 | 62.77% |
| GPU | CNN | BF16 | Reused | 330.69 | 909.97 | 63.66% |
| GPU | Transformer | FP32 | Empty | 243.28 | 877.87 | 72.29% |
| GPU | Transformer | FP32 | Reused | 242.28 | 877.89 | 72.40% |
| GPU | Transformer | BF16 | Empty | 260.92 | 897.16 | 70.92% |
| GPU | Transformer | BF16 | Reused | 255.88 | 899.40 | 71.55% |
What we measured
- Each sample starts a new process and stops when its first prediction reaches host memory.
- The time includes starting the Python interpreter and importing PyTorch, loading native libraries, creating the GPU context, building the network and its input, and the first forward pass.
- No trained model file is loaded. File-system caches were warm. “Empty” and “reused” refer to each library’s GPU tuning cache.
- PyTorch runs in its default eager mode, without a compilation warm-up.
- The applications are the same small Dense, LSTM, CNN, and Transformer networks used in the deployment size test.
Limits
- GPU clocks were not locked, and other desktop activity was present on the GPU. In 11 of the 44 engine configurations, the variation between launches exceeded 3%, so treat the exact ratios as indicative.
- The figures describe this computer and these software versions.
The benchmark protocol describes the startup procedure. See also inference performance.