IBM RS/6000 SP, Problem Determination Manual

Page: 153 / 316

This soft copy for use by IBM employees only.
IP address aliasing will be dealt with in detail later in the chapter, but it is
sufficient to know at this stage that each partition is identified by having a
different alias. The topology file for the second partition is shown in Figure 61
on page 133 for completeness.
# Node connections in frame L01 to switch 1 in L01
s 16 0 tb0 2 0
E01-S17-BH-J20 to E01-N3
s 16 1 tb0 3 0 E01-S17-BH-J22 to E01-N4
s 16 2 tb0 6 0 E01-S17-BH-J24 to E01-N7
s 16 3 tb0 7 0 E01-S17-BH-J26 to E01-N8
s 17 0 tb0 10 0 E01-S17-BH-J28 to E01-N11
s 17 1 tb0 11 0 E01-S17-BH-J30 to Exx-Nxx
s 17 2 tb0 14 0 E01-S17-BH-J32 to E01-N15
s 17 3 tb0 15 0 E01-S17-BH-J34 to Exx-Nxx
# On board connections between switch chips on switch 1 in Frame L01
s 16 5
s 11 6 E01-S17-SC
s 16 4 s 10 6 E01-S17-SC
s 17 5 s 11 7 E01-S17-SC
s 17 4 s 10 7 E01-S17-SC
Figure 61. Insert of Topology File for the Second Partition
This example is based on the Jack cable (Jxx) numbering system used on the
High Performance Switch, rather than on the newer SP Switch. The example is
based also on a system with eight thin nodes (1-8) and four thin nodes (nodes 9,
11, 13, and 15). The right side of the topology file shows “Nxx” to represent
where a wrap plug is inserted (rather than a Jack Cable that would normally be
present to connect to a node). In this case, the wrap plug is required due to a
wide node being present in that particular drawer.
There is another issue that should be considered in relation to performance
when setting up System Partitioning on a multi-switch system. Take an example
of a
5nsb.0isb configuration that is not currently partitioned. If you have a
resource pool of 20 nodes set up, for example, for running parallel jobs across
the switch, then the best placement of the nodes is likely to be with four nodes
on each switch to provide the greatest bandwidth possible. In this case, where
you are not using Intermediate Switch Boards (ISBs), there are only four cables
available to connect to each of the other switch boards, given that there are only
16 external Jack sockets available for external connections (rather than node
connections) and there are four other switches to connect to.
By placing the nodes evenly across the switches, each of the four nodes can
communicate with any of the other four nodes over four data cables. If you
placed ten nodes each on two switches only, then these ten nodes would only be
able to communicate over four cables to the other ten nodes and the bandwidth
is reduced considerably.
If a large amount of data is flowing across the switch during these parallel jobs,
then the chance of packet contention over the routes is increased by
concentrating the nodes in the resource pool on fewer switches. Of course, if
you can place all the nodes on one switch, you will resolve the issue; but in this
case, it is not possible because the pool consists of 20 nodes.
It is worth taking this issue of bisectional bandwidth into account when
partitioning the system, and when you have this type of resource pool where
Chapter 5. System Partitioning
133