‹ Back to Tutorials

CUDA Graphs Explained

A neural-network training step can issue many short CUDA kernels. CUDA Graphs record a stable sequence once and replay it with a single launch, reducing repeated CPU launch overhead.

OpenNN can use CUDA Graphs with its Adam and stochastic gradient descent optimizers when the network, loss and batch execution path are compatible.

Contents:

  1. Introduction
  2. The three-phase model
  3. Enable CUDA Graphs in OpenNN
  4. When CUDA Graphs help
  5. When they do not
  6. Capture and fallback
  7. Graph-friendly execution
  8. Conclusions

1. Introduction

Each CUDA kernel launch crosses the runtime and driver before work reaches the GPU. The overhead is small for a few large kernels, but it can become significant when a training loop launches many short operations.

CUDA Graphs replace repeated submission of the same sequence with a reusable graph executable. The optimization is most useful when tensor shapes and execution flow remain stable across many steps.

2. The three-phase model

  1. Capture: the CUDA runtime records the operations submitted to a stream.
  2. Instantiate: the recorded graph becomes an executable graph with resolved dependencies.
  3. Replay: one graph launch submits the recorded sequence again.

The initial capture has a cost, so the graph must be replayed repeatedly for the optimization to pay off.

3. Enable CUDA Graphs in OpenNN

Configure CUDA, select Adam or SGD, and enable graph execution on the optimizer:

Configuration::instance().set(
    Device::CUDA,
    Type::BF16);

TrainingStrategy training_strategy(
    &neural_network,
    &dataset);

training_strategy.set_optimization_algorithm(
    "AdaptiveMomentEstimation");

Optimizer* optimizer =
    training_strategy.get_optimization_algorithm();

optimizer->set_cuda_graph(true);

TrainingResult result = training_strategy.train();

The same setting is available for StochasticGradientDescent. Full-batch optimizers do not currently expose a capturable epoch path.

4. When CUDA Graphs help

  • Repeated training steps with stable layer topology and tensor shapes.
  • Many short kernels whose launch overhead is significant relative to their execution time.
  • Fixed mini-batches that reuse preallocated buffers over many epochs.

OpenNN performs warm-up work before capture and reuses graph executables for compatible training batches.

5. When they do not

  • Dynamic shapes: changed batch or tensor shapes may require another execution path.
  • Dynamic host control: runtime decisions that change submitted operations cannot remain in one fixed graph.
  • Large kernels: when computation dominates the step, reduced launch overhead may have little effect.
  • Active dropout: OpenNN currently disables optimizer graph capture for networks with active dropout.
  • Unsupported losses: the active loss must provide device-side epoch metrics.

6. Capture and fallback

OpenNN captures a graph only when the CUDA build, optimizer, loss and network are compatible. If capture fails at runtime, training continues without graphs and the optimizer records the failure.

TrainingResult result = training_strategy.train();

if(optimizer->get_cuda_graph_capture_failed())
{
    // Training completed through the normal CUDA path.
}

This fallback keeps graph execution an optional performance optimization rather than a requirement for successful training.

7. Graph-friendly execution

Capturable execution favors stable operations and reusable memory:

  • Preallocate buffers instead of allocating inside the captured path.
  • Avoid host synchronization and callbacks during capture.
  • Keep shapes and operation order stable across replayed batches.
  • Fuse operations where doing so reduces launch and memory overhead.

8. Conclusions

  • CUDA Graphs reduce repeated launch overhead for stable GPU workloads.
  • OpenNN supports optimizer graph execution with Adam and SGD.
  • Compatibility is checked automatically and a failed capture falls back to normal CUDA execution.
  • Measure the complete training workload because gains depend on model and batch characteristics.

References