Qlogic OFED+ Host, Software User'S Manual, Page: 115 / 284

6 – SHMEM Description and Configuration
QLogic SHMEM Relationship with MPI
IB0054606-02 A 6-7
QLogic SHMEM Relationship with MPI
QLogic SHMEM requires the QLogic PSM layer to provide the network transport
function and this runs exclusively on QLogic IB HCAs. It also requires a
compatible MPI implementation (also running over PSM) to provide program start
up and other miscellaneous services. The one-sided operations in QLogic
SHMEM are not layered on top of MPI, however, and go directly to PSM to give
low-latency, high-performance access to the HCA architecture.
Typical SHMEM programs are written using calls to the SHMEM API and do not
use MPI calls. In this case the program binary generated by
shmemcc
contains
references to the SHMEM dynamic library and no references at all to MPI
libraries. These binaries are portable across all MPI implementations supported
by QLogic SHMEM. This is true of the
get/put
micro-benchmarks provided by
QLogic SHMEM. The desired MPI can be selected at run time simply by placing
the desired
mpirun
on
$PATH
, or by using the
$SHMEM_MPIRUN
environment
variable.
Alternatively, it is possible to write hybrid SHMEM/MPI programs that use features
from both the SHMEM and MPI libraries. These programs must call
shmem_init()
to initialize the SHMEM library state. They may also use
MPI_Init()
and
MPI_Finalize()
if needed. There will be a direct one-to-one
correspondence between the SHMEM and
MPI_COMM_WORLD
rank assignments:
shmem_my_pe()
will match
MPI_Comm_rank()
on
MPI_COMM_WORLD
shmem_n_pes() will match MPI_Comm_size() on MPI_COMM_WORLD
Hybrid SHMEM/MPI programs must be linked against SHMEM libraries and the
correct MPI libraries. It is recommended that the implementation of the MPI
wrapper script(s) (
mpicc
) is used for compilation and that additional options are
specified to find the SHMEM include and library files. One approach is to set up
the
shmemcc
wrapper script to use
mpicc
as its compiler using the environment
variable setting:
export SHMEM_CC=mpicc
This setting needs to be adjusted if
mpicc
is not already on the $PATH. The
generated binary has references to both SHMEM and MPI libraries and is specific
to that MPI implementation. QLogic recommends that
shmemrun
is used to run
the program. The user must ensure that the correct
mpirun
is picked up from
$PATH
or using the
$SHMEM_MPIRUN
environment variable.