Communication Backends[1]

Backend Comm. Functions Optimized for Float32 Float16
MPI All CPU, GPU Yes No
GLOO All (on CPU), broadcast & all-reduce (on GPU) CPU Yes Yes
NCCL broadcast, all reduce, reduce and all gather (on GPU) GPU only Yes Yes

Overview [1]

分布式的通信有两大类：

  1. 集合通信（Collective communication，CC）：在一组节点内进行通信
  2. 点对点通信（Point to point communication，P2P）：在两个节点之间进行通信

目前机器学习中主要由两种分布式架构：

Collective Communication[2][5]

一对多