Prediction quality: OpenNN vs PyTorch

OpenNN and PyTorch reached similar mean test scores on all four network types we tested. Faster training only helps if the trained model is as good, so we checked prediction quality alongside speed.

Measured in September 2026 on the reference computer (Intel Core i7-14700F, 32 GB RAM, NVIDIA GeForce RTX 5070 Ti, Linux) with CUDA FP32 training. Scores are means over repeated training runs, evaluated on held-out data.

Results

Model	Data	Metric	OpenNN	PyTorch	Training runs
Dense	HIGGS	Test accuracy (higher is better)	68.72%	68.83%	20
LSTM	Beijing PM2.5	Test RMSE (lower is better)	23.11	23.11	20
CNN	CIFAR-10	Test accuracy (higher is better)	72.60%	72.80%	5
Transformer	News Commentary English–German	BLEU (higher is better)	8.32	8.32	5

Each metric belongs to its own task. We do not average accuracy, RMSE and BLEU into a single quality score.

How we tested

Both libraries trained the same network on the same prepared data, with the same batch size, number of epochs, learning rate and sample order.
Both started from the same initial weights: for each run, the weights were generated once and loaded into both libraries.
Both used the Adam optimizer (β1 = 0.9, β2 = 0.999) with no gradient clipping and no early stopping.
One neutral Python program scored the saved predictions of both libraries on the held-out data. The Transformer is scored with BLEU on translations generated word by word, not on teacher-forced guesses.

Model	Network	Training / test data	Training
Dense	28 → 1024 → 1024 → 1 classifier, 1,080,321 parameters	250,000 / 500,000 HIGGS events	100 epochs, batch 256, learning rate 0.001
LSTM	128 LSTM units over 24 hourly steps × 15 features, 73,857 parameters	35,035 / 8,741 windows, chronological 80/20 split	50 epochs, batch 256, learning rate 0.001
CNN	ResNet-50-style network for 32 × 32 images, 23,528,522 parameters	50,000 / 10,000 images, official CIFAR-10 split, no augmentation	10 epochs, batch 32, learning rate 0.001, weight decay 0.0005, batch-normalization statistics recalibrated on the training data
Transformer	Encoder–decoder, 6 layers, width 512, 8 heads, 20,000-token vocabulary, 74,878,496 parameters	180,102 sentence pairs / 1,000 test sentences	10 epochs, batch 32, learning rate 0.0001

How similar is similar?

For each model we set the largest difference we would accept; for the dense, LSTM and CNN tests it was fixed before the runs. For every run we computed the difference between the two libraries and then a 90% confidence interval for the mean difference. When the whole interval lies inside the margin, the two libraries are equivalent for that task.

Model	Mean difference (OpenNN − PyTorch)	90% confidence interval	Accepted margin	Result
Dense	−0.11 points	−0.21 to −0.004 points	±0.5 points	Within margin
LSTM	−0.0005 RMSE	−0.28 to +0.28 RMSE	±0.90 RMSE (1% of the target’s scale)	Within margin
CNN	−0.20 points	−0.67 to +0.27 points	±2 points	Within the ±2-point margin, not the stricter ±0.5
Transformer	−0.0003 BLEU	−0.43 to +0.43 BLEU	±0.5 BLEU	Within margin; review in progress

None of the four tests shows that either library is better. For the CNN, five runs are enough to place the difference within 2 percentage points but not within 0.5.

Limits

The results cover these datasets, this computer and CUDA FP32 training. They do not show identical training trajectories or equal quality on other tasks.
These quality tests use different data from the speed tests: CIFAR-10 instead of the ImageNet subset for the CNN, and News Commentary instead of WMT14 text for the Transformer.
The general procedure is in the benchmark protocol. The scripts for these specific runs will be added to the OpenNN repository.