4
InfiniBand technology
InfiniBand is an industry-standard, channel-based architecture in which congestion-management
capabilities and zero-copy data transfers using remote direct memory access (RDMA) are core
capabilities—resulting in high-speed, low-latency interconnects for scale-out compute infrastructures.
InfiniBand uses a multi-layer architecture to transfer data from one node to another. In the InfiniBand
layer model (Figure 3), separate layers perform different tasks in the message passing process.
The upper layer protocols (ULPs) work closest to the operating system and application; they define the
services and affect how much software overhead the data transfer will require. The InfiniBand
transport layer is responsible for communication between applications. The transport layer splits the
messages into data payloads and encapsulates each data payload and an identifier of the
destination node into one or more packets. Packets can contain data payloads of up to four kilobytes.
The packets are passed to the network layer, which selects a route to the destination node and, if
necessary, attaches the route information to the packets. The data link layer attaches a local identifier
(LID) to the packet for communication at the subnet level. The physical layer transforms the packet into
an electromagnetic signal based on the type of network media—copper or fiber.
Figure 3.
Distributed computing using InfiniBand architecture
InfiniBand has these important characteristics:
Very high bandwidth—up to 40Gbps Quad Data Rate (QDR)
Low latency end-to-end communication—MPI ping-pong latency approaching 1 microsecond
Hardware-based protocol handling, resulting in faster throughput and low CPU overhead due to
efficient OS bypass and RDMA
Native support for fat-tree and other common mesh topologies in fabric design
InfiniBand Link
