17
Chapter 1: Introduction
Figure 1-8. NVLink Achitecture: Cube Mesh
GPU Add-on Module
The system supports eight Tesla SXM2 V100 GPUs installed on the X10DGO-SXMV add-on
module which is connected to the motherboard by a midplane.
The X10DGO-SXMV is connected through Nvidia’s Cube Mesh NVLINK architecture. Direct
connection between all GPUs are single NVlink (25 GB/s).
A direct connection from the GPUs to the network can be provided from the PCI-E PLX switch
to the PCI-E3.0 x16 up front. Each PCI-E PLX switch is root to two SXM2 modules and one
PCI-E3.0 x16 slot. A fast network (expansion) card installed on the riser card affords very
high speeds. Data can go from GPU to the PLX to the NIC without going through the CPU.
For the NIC, the system can support both FDR and EDR (in x16 slots).
GPU5
GPU4
GPU6
GPU7
CPU1
GPU1
GPU0
GPU2
GPU3
CPU0
INTEL
INTEL
0
1
3
2
3
3
3
0
1
2
2
2 1
1
0
0
1 2
3
0
3
2
3
2
1
1
3
2 0
1
0
0
PEX9765
PCIE SLOT X16
PEX9765
PEX9765
PEX9765
6
1
X
T
O
L
S
E
I
C
P
6
1
X
T
O
L
S
E
I
C
P
6
1
X
T
O
L
S
E
I
C
P
FROM MB
FROM MB
PCIE x16
NVLINK x8 (Pascal)
4
4
5
4
5
5
4
5
5
5
4
4
4
5
5
4
NVLINK x8 (Volta)
and
For Volta V100, NVLink connections are represented above by solid and dashed green lines.
