Startup time: OpenNN vs PyTorch
From process launch to the first prediction, OpenNN applications took 63% to 99.8% less time than the same applications written with PyTorch’s Python API.
Measured on 15 September 2026 on the reference computer (Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux): 660 timed launches, 15 for each configuration and library. Values are medians.
Results
| Configuration | OpenNN (ms) | PyTorch (ms) | Reduction |
|---|---|---|---|
| CPU Eigen · Dense | 1.11 | 497.86 | 99.78% |
| CPU MKL · Dense | 8.96 | 497.86 | 98.20% |
| GPU · Dense | 238.09 | 784.30 | 69.64% |
| GPU · LSTM | 253.70 | 800.68 | 68.31% |
| GPU · CNN | 316.38 | 849.48 | 62.76% |
| GPU · Transformer | 243.28 | 877.87 | 72.29% |
A shorter startup means an application, a command-line tool or a serverless function can deliver its first prediction sooner after it is launched.
All configurations
The table above shows the CPU Dense applications and the four GPU applications at FP32 with an empty tuning cache. These are all the measured configurations:
| Backend | Application | Precision | GPU tuning cache | OpenNN (ms) | PyTorch (ms) | Reduction |
|---|---|---|---|---|---|---|
| CPU Eigen | Dense | FP32 | – | 1.11 | 497.86 | 99.78% |
| CPU Eigen | LSTM | FP32 | – | 3.58 | 497.31 | 99.28% |
| CPU Eigen | CNN | FP32 | – | 2.47 | 496.38 | 99.50% |
| CPU Eigen | Transformer | FP32 | – | 1.76 | 500.77 | 99.65% |
| CPU MKL | Dense | FP32 | – | 8.96 | 497.86 | 98.20% |
| CPU MKL | LSTM | FP32 | – | 11.82 | 497.31 | 97.62% |
| CPU MKL | CNN | FP32 | – | 10.30 | 496.38 | 97.93% |
| CPU MKL | Transformer | FP32 | – | 9.58 | 500.77 | 98.09% |
| GPU | Dense | FP32 | Empty | 238.09 | 784.30 | 69.64% |
| GPU | Dense | FP32 | Reused | 237.15 | 783.68 | 69.74% |
| GPU | Dense | BF16 | Empty | 254.30 | 812.14 | 68.69% |
| GPU | Dense | BF16 | Reused | 253.00 | 812.69 | 68.87% |
| GPU | LSTM | FP32 | Empty | 253.70 | 800.68 | 68.31% |
| GPU | LSTM | FP32 | Reused | 249.07 | 794.70 | 68.66% |
| GPU | LSTM | BF16 | Empty | 262.14 | 816.94 | 67.91% |
| GPU | LSTM | BF16 | Reused | 258.15 | 809.43 | 68.11% |
| GPU | CNN | FP32 | Empty | 316.38 | 849.48 | 62.76% |
| GPU | CNN | FP32 | Reused | 312.75 | 845.22 | 63.00% |
| GPU | CNN | BF16 | Empty | 341.29 | 916.75 | 62.77% |
| GPU | CNN | BF16 | Reused | 330.69 | 909.97 | 63.66% |
| GPU | Transformer | FP32 | Empty | 243.28 | 877.87 | 72.29% |
| GPU | Transformer | FP32 | Reused | 242.28 | 877.89 | 72.40% |
| GPU | Transformer | BF16 | Empty | 260.92 | 897.16 | 70.92% |
| GPU | Transformer | BF16 | Reused | 255.88 | 899.40 | 71.55% |
What we measured
- Each sample starts a new process and stops when its first prediction reaches host memory.
- The time includes starting the Python interpreter and importing PyTorch, loading native libraries, creating the GPU context, building the network and its input, and the first forward pass.
- No trained model file is loaded. File-system caches were warm. “Empty” and “reused” refer to each library’s GPU tuning cache.
- PyTorch runs in its default eager mode, without a compilation warm-up.
- The applications are the same small Dense, LSTM, CNN and Transformer networks used in the deployment size test.
Limits
- GPU clocks were not locked and other desktop activity was present on the GPU. In 11 of the 44 engine configurations the variation between launches exceeded 3%, so treat the exact ratios as indicative.
- The figures describe this computer and these software versions.
The startup procedure is described in the benchmark protocol. See also inference performance.