Silicon Graphics Altix ICE 8000 Series, Quick Reference Manual

Page: 61 / 84

Chapter 3
System Fabric Management
The InfiniBand network on SGI Altix ICE 8000 series systems uses Open Fabrics
Enterprise Distribution (OFED) 1.2 software. This section describes the InfiniBand
fabric and how to manage it. For background information on OFED, see
http://www.openfabrics.org.
Fabric management on SGI Altix ICE 8000 series systems uses the OFED 1.2 OpenSM
software package. The InfiniBand fabric connects the service nodes, rack leader
controllers (leader nodes), and the compute nodes. It does not connect to the system
admin controller (admin node) or the chassis management control (CMC) blades. The
InfiniBand network has two separate network fabrics,
ib0
and
ib1
(see "InfiniBand
Fabric" on page 21) with the following characteristics:
• Each network fabric has its own subnet manager (SM).
• For a system with two racks or more, one rack leader controller (leader node) runs
an instance of SM to manage the
ib0
fabric and a second leader node runs an
instance of SM to manage the
ib1
fabric.
• On a system with a single rack, both instances of
opensm
run on the same rack
leader node.
• Each instance of SM on the rack leader controller is controlled by the
/etc/opensm-ib0.conf
or
/etc/opensm-ib1.conf
configuration file.
• Rack leader controllers run the
opensm
daemon for each fabric over separate
HCA ports (see Figure 1-9 on page 22).
Note: After a system reboot, you need to manually restart the
opensm
daemons
running on the InfiniBand fabric. If the
opensm
daemons are allowed to start
automatically, as the leader nodes boot, you will not know which leader is the
Master
and it is highly likely that the fabric will be routed incorrectly. To start
the InfiniBand fabric, you can use the following command:
scalimanage-cli restartaltixiceopensm
• Each fabric is addressed by a global unique identifier (GUID) and unique HCA
port.
The GUID and HCA port is set in the configuration file.
007–5450–001 43