37
Each MPI process is allowed to lock down up to
VIADEV_MAX_RENDEZVOUS
amount of
application buffer memory. Once this limit is reached, IO buffers are evicted and
unregistered on a “least recently used” basis.
The default value for this parameter is 209715200 bytes (200MB).
4.7.2 Tuning suggestions
Based on the default values, one can determine how much memory is being consumed by
each MPI process in a run. Then the desired amount of memory should be determined based
on how much memory your MPI application needs for doing computation. Given you can
actually determine that, you can then adjust these parameters to reduce the amount of
memory used by the MPI RDMA driver.
A few guidelines are offered here:
•
Try reducing
VIADEV_NUM_RDMA_BUFFERS
first. The default for this parameter
doesn’t particularly scale well or evenly as your NP value increases.
•
Try reducing
VIADEV_MAX_RENDEZVOUS
as a next step.
•
Keep
VIADEV_SQ_DEPTH
>=
VIADEV_RQ_DEPTH
•
Try to avoid lowering
VIADEV_RQ_DEPTH
. It is probably safe to drop this to
around 64, but avoid going lower if possible.