IPU-POD128 build and test guide
Step 21: Refresh overlay files on lrack1
Use
rack_tool
to refresh the overlay files.
rack_tool.py update-root-overlay
Step 22: Check IPU-M2000s logging to lrack1
You need to check whether all 32 IPU-M2000s are logging to the lrack1
syslog
. This will either be located in
/var/log/syslog
or in a specified location if you have filters in place for IPU-M2000 logs. A common
syslog
filter
is
/etc/rsyslog.d/99_ipum.conf
:
#
Graphcore.ai - Config Management
#
This file is managed by puppet, so any manual changes will be overwritten automatically!
$
template tplremote,
"%timegenerated% %HOSTNAME% %fromhost-ip% %syslogtag%%msg:::drop-last-lf%\n"
$
template bmclogdir,
"/var/log/ipumlogs/bmclogs/%fromhost-ip%.log"
$
template gwlogdir,
"/var/log/ipumlogs/gwlogs/%fromhost-ip%.log"
if $fromhost-ip startswith '10.1' then ?bmclogdir;tplremote
if $fromhost-ip startswith '10.2' then ?gwlogdir;tplremote
& ~
Run this command to check if the
syslog
server is running on lrack1’s management server (server1):
systemctl | grep syslog
If the
syslog
server is not running then you need to start it.
Step 23: Check IPU-M2000s have NTP date and time
You need to check whether all 32 IPU-M2000s are taking their date and time from the NTP server on lrack1.
This can be checked by ssh to itadmin @ IPUM Gateway and checking the system time/date.
Step 24: Run ML application
Run a machine learning application on the IPU-POD
128
to check that is it correctly using both lrack1 and lrack2
resources. This will verify that the merge to an IPU-POD
128
has been successful. You can find examples of ML
applications to run in
Graphcore’s GihHub examples repo
8.5 IPU-M2000 setup files
8.5.1 Syslog and chrony on the IPU-Gateway
1. Update the
chrony.conf
file in the
root_overlay
files.
Below is a truncated example of a
chrony.conf
file.
(venv) ipuuser@lr23-poplar1:~/IPU_M_SW-2.3.0$
more ~/.rack_tool/root-overlay/etc/chrony/chrony.conf
#
Welcome to the chrony configuration file. See chrony.conf
(
5
)
for
more
#
information about usuable directives.
pool 10.2.7.150 iburst
2. Update the
rsyslog.d
configuration file in the
root_overlay
files.
(venv) ipuuser@lr23-poplar1:~/IPU_M_SW-2.3.0$
more ~/.rack_tool/root-overlay/etc/rsyslog.d/99-loghost.conf
*.* @10.2.7.150:514
3. Update the
root_overlay
files with
rack_tool
:
Version: latest (2021-11-25)
97