5
InfiniBand components
InfiniBand architecture involves four key components:
Host channel adapter
Subnet manager
Target channel adapter
InfiniBand switch
A host node or server requires a host channel adapter (HCA) to connect to an InfiniBand
infrastructure. An HCA can be a card installed in an expansion slot or integrated onto the host’s
system board. An HCA can communicate directly with another HCA, with a target channel adapter,
or with an InfiniBand switch.
InfiniBand uses subnet manager (SM) software to manage the InfiniBand fabric and to monitor
interconnect performance and health at the fabric level. A fabric can be as simple as a point-to-point
connection or multiple connections through one or more switches. The SM software resides on a node
or switch within the fabric, and provides switching and configuration information to all of the switches
in the fabric. Additional backup SMs may be located within the fabric for failover should the primary
SM fail. All other nodes in the fabric will contain an SM agent that processes management data.
Managers and agents communicate using management datagrams (MADs).
A target channel adapter (TCA) is used to connect an external storage unit or I/O interface to an
InfiniBand infrastructure. The TCA includes an I/O controller specific to the device’s protocol (SCSI,
Fibre Channel, Ethernet, etc.) and can communicate with an HCA or an InfiniBand switch.
An InfiniBand switch provides scalability by allowing a number of HCAs, TCAs, and other IB switches
to connect to an InfiniBand infrastructure. The switch handles network traffic by checking the local link
header of each data packet received and forwarding the packet to the proper destination.
The most basic InfiniBand infrastructure will consist of host nodes or servers equipped with HCAs, an
InfiniBand switch, and subnet manager software. More expansive networks will include multiple
switches.
InfiniBand software architecture
InfiniBand, like Ethernet, uses a multi-layer processing stack to transfer data between nodes.
InfiniBand architecture, however, provides OS-bypass features such as the communication processing
duties and RDMA operations as core capabilities and offers greater adaptability through a variety of
services and protocols.
While the majority of existing InfiniBand clusters operate on the Linux platform, drivers and HCA
stacks are also available for Microsoft® Windows®, HP-UX, Solaris, and other operating systems
from various InfiniBand hardware and software vendors.
The layered software architecture of the HCA allows writing code without specific hardware in mind.
The functionality of an HCA is defined by its verb set, which is a table of commands used by the
application programming interface (API) of the operating system being run. A number of services and
software protocols are available (Figure 4) and, depending on type, can be implemented from user
space or from the kernel.
