Skip to content

Back to Tutorials

Multi-GPU Data Parallelization

Data parallelism replicates a model across several GPUs, gives each replica a different mini-batch shard and synchronizes gradients before the parameter update.

OpenNN currently compiles one network for one device per process. It does not provide a built-in NCCL distributed trainer, so the strategy described here requires external process and collective-communication orchestration.

Contents:

  1. Introduction
  2. The training step across N GPUs
  3. All-reduce: the critical primitive
  4. Effective batch size
  5. When data parallelism fits
  6. When it stops scaling
  7. Conclusions

1. Introduction

When a model fits on one GPU but training throughput is insufficient, multiple replicas can process different samples concurrently. A synchronization step makes the combined update equivalent to training with a larger batch.

The model architecture does not need to change, but the launcher must assign devices, partition data, synchronize gradients and coordinate failures across processes.

2. The training step across N GPUs

  1. Split a global mini-batch of size B into N shards.
  2. Run forward and backward propagation independently on each GPU.
  3. Reduce the local gradients across all replicas.
  4. Apply an equivalent optimizer update to every replica.

Replicas remain synchronized when they start from identical parameters and use the same reduced gradient and optimizer state.

3. All-reduce: the critical primitive

All-reduce combines every replica’s gradient and returns the result to every process. GPU communication libraries such as NCCL can perform this operation over NVLink, PCIe or network interconnects.

A ring all-reduce partitions the gradient buffer and transfers chunks between neighboring devices in reduce-scatter and all-gather phases. Its performance depends on gradient volume, latency, bandwidth and topology.

4. Effective batch size

With N replicas and a local batch size of b, the effective global batch size is N × b. This can increase throughput, but it also changes optimization behavior.

  • Learning rate: larger global batches often require retuning the schedule.
  • Reduction: gradients must be summed or averaged consistently with the loss convention.
  • Validation: compare convergence and final accuracy, not only samples per second.

5. When data parallelism fits

  • The complete model, activations and optimizer state fit on every GPU.
  • The data set supports a larger global batch.
  • Compute time is sufficient to amortize gradient synchronization.
  • The available interconnect can sustain the collective communication.

Under these conditions, data parallelism is usually the simplest multi-GPU scaling strategy.

6. When it stops scaling

  • Communication saturation: all-reduce consumes a growing part of each step.
  • Batch-size ceiling: a larger effective batch stops improving useful throughput or degrades convergence.
  • Load imbalance: uneven work causes faster replicas to wait.
  • Model size: if one replica no longer fits on one GPU, model partitioning is also required.

7. Conclusions

  • Every GPU owns a complete model and processes a different data shard.
  • A gradient collective keeps the replicas synchronized.
  • Scaling depends on global batch behavior and communication cost.
  • Current OpenNN applications must supply the distributed launcher and all-reduce layer externally.

References