manualshive.com logo in svg

Graphcore IPU-POD128, Build And Test Manual, Page: 105 / 113

Share
Download
Graphcore IPU-POD128 Build And Test Manual Download Page 105

IPU-POD128 build and test guide

(continued from previous page)

host lr8-ipum7bmc { hardware ethernet 70:69:79:20:14:D4; fixed-address 10.1.8.7; }

host lr8-ipum8bmc { hardware ethernet 70:69:79:20:14:BC; fixed-address 10.1.8.8; }

host lr8-ipum9bmc { hardware ethernet 70:69:79:20:14:38; fixed-address 10.1.8.9; }

host lr8-ipum10bmc { hardware ethernet 70:69:79:20:14:94; fixed-address 10.1.8.10; }

host lr8-ipum11bmc { hardware ethernet 70:69:79:20:13:F4; fixed-address 10.1.8.11; }

host lr8-ipum12bmc { hardware ethernet 70:69:79:20:12:74; fixed-address 10.1.8.12; }

host lr8-ipum13bmc { hardware ethernet 70:69:79:20:12:6C; fixed-address 10.1.8.13; }

host lr8-ipum14bmc { hardware ethernet 70:69:79:20:14:84; fixed-address 10.1.8.14; }

host lr8-ipum15bmc { hardware ethernet 70:69:79:20:16:BC; fixed-address 10.1.8.15; }

host lr8-ipum16bmc { hardware ethernet 70:69:79:20:17:A4; fixed-address 10.1.8.16; }

ipum-gw.conf

root@gbnwp-pod7-1:/etc/dhcp/dhcpd.d/lrack2#

cat ipum-gw.conf

host lr8-ipum1gw {

hardware ethernet 70:69:79:20:24:D1; fixed-address 10.2.8.1; }

host lr8-ipum2gw {

hardware ethernet 70:69:79:20:14:49; fixed-address 10.2.8.2; }

host lr8-ipum3gw {

hardware ethernet 70:69:79:20:12:BD; fixed-address 10.2.8.3; }

host lr8-ipum4gw {

hardware ethernet 70:69:79:20:13:2D; fixed-address 10.2.8.4; }

host lr8-ipum5gw {

hardware ethernet 70:69:79:20:13:05; fixed-address 10.2.8.5; }

host lr8-ipum6gw {

hardware ethernet 70:69:79:20:14:69; fixed-address 10.2.8.6; }

host lr8-ipum7gw {

hardware ethernet 70:69:79:20:14:D5; fixed-address 10.2.8.7; }

host lr8-ipum8gw {

hardware ethernet 70:69:79:20:14:BD; fixed-address 10.2.8.8; }

host lr8-ipum9gw {

hardware ethernet 70:69:79:20:14:39; fixed-address 10.2.8.9; }

host lr8-ipum10gw { hardware ethernet 70:69:79:20:14:95; fixed-address 10.2.8.10; }

host lr8-ipum11gw { hardware ethernet 70:69:79:20:13:F5; fixed-address 10.2.8.11; }

host lr8-ipum12gw { hardware ethernet 70:69:79:20:12:75; fixed-address 10.2.8.12; }

host lr8-ipum13gw { hardware ethernet 70:69:79:20:12:6D; fixed-address 10.2.8.13; }

host lr8-ipum14gw { hardware ethernet 70:69:79:20:14:85; fixed-address 10.2.8.14; }

host lr8-ipum15gw { hardware ethernet 70:69:79:20:16:BD; fixed-address 10.2.8.15; }

host lr8-ipum16gw { hardware ethernet 70:69:79:20:17:A5; fixed-address 10.2.8.16; }

8.6.7 Lrack2: /etc/dhcp/dhcpd.d/lrack2 files

Note:

This setup is ONLY required on lrack2 if there is no spine switch. In this situation you will also need a

DHCP server to run on lrack2.

ipum-rnic.conf

root@gbnwp-pod8-1:/etc/dhcp/dhcpd.d/lrack2#

cat ipum-rnic.conf

host lr8-ipum1mx { hardware ethernet 0c:42:a1:78:90:35; fixed-address 10.5.8.1; }

host lr8-ipum2mx { hardware ethernet 0c:42:a1:78:90:d5; fixed-address 10.5.8.2; }

host lr8-ipum3mx { hardware ethernet 0c:42:a1:78:7e:55; fixed-address 10.5.8.3; }

host lr8-ipum4mx { hardware ethernet 0c:42:a1:78:82:65; fixed-address 10.5.8.4; }

host lr8-ipum5mx { hardware ethernet 0c:42:a1:84:ec:3d; fixed-address 10.5.8.5; }

host lr8-ipum6mx { hardware ethernet 0c:42:a1:78:95:65; fixed-address 10.5.8.6; }

host lr8-ipum7mx { hardware ethernet 0c:42:a1:78:96:c5; fixed-address 10.5.8.7; }

host lr8-ipum8mx { hardware ethernet 0c:42:a1:84:f5:85; fixed-address 10.5.8.8; }

host lr8-ipum9mx { hardware ethernet 0c:42:a1:84:ea:85; fixed-address 10.5.8.9; }

host lr8-ipum10mx { hardware ethernet 0c:42:a1:78:7e:65; fixed-address 10.5.8.10; }

host lr8-ipum11mx { hardware ethernet 0c:42:a1:78:8a:0d; fixed-address 10.5.8.11; }

host lr8-ipum12mx { hardware ethernet 0c:42:a1:78:85:55; fixed-address 10.5.8.12; }

host lr8-ipum13mx { hardware ethernet 0c:42:a1:78:89:f5; fixed-address 10.5.8.13; }

host lr8-ipum14mx { hardware ethernet 0c:42:a1:78:8a:4d; fixed-address 10.5.8.14; }

host lr8-ipum15mx { hardware ethernet 0c:42:a1:84:e7:bd; fixed-address 10.5.8.15; }

host lr8-ipum16mx { hardware ethernet 0c:42:a1:78:87:25; fixed-address 10.5.8.16; }

Version: latest (2021-11-25)

101

Summary of Contents for IPU-POD128

Page 1: ...IPU POD128 build and test guide Version latest Graphcore Ltd Nov 25 2021...

Page 2: ...10 3 2 Document reproduction 11 3 3 Required tools 11 3 4 Preparing the rack 11 3 4 1 Rail distance 11 3 4 2 Unpacking the rack 12 3 4 3 Removing the side panels and doors 12 3 4 4 Removing the verti...

Page 3: ...ion recommendations 58 4 1 3 Memory con guration recommendations 58 4 1 4 BIOS con guration 58 4 1 5 Operating system installation 60 4 1 6 User accounts and groups 62 4 1 7 DHCP Service Dynamic Host...

Page 4: ...rack2 etc dhcp dhcpd conf 98 8 6 2 Lrack1 etc dhcp dhcpd d ipum dhcp conf 98 8 6 3 Lrack2 etc dhcp dhcpd d ipum dhcp conf 99 8 6 4 Lrack1 and lrack2 etc dhcp dhcpd d les 99 8 6 5 Lrack1 etc dhcp dhcpd...

Page 5: ...0 For more information on IPU POD systems available from Graphcore see https www graphcore ai products Warning This guide is for properly trained service personnel and technicians who are required to...

Page 6: ...four IPUs in the IPU M2000 across a RoCE network provides external IPU Exchange Memory and enables IPU scaleout across 100GbE GW Links for rack to rack connectivity GW Link High speed communication li...

Page 7: ...rted 2 Switches one 1GbE management switch and one 100GbE ToR switch 2 Power distribution units 1 Rack Supplementary mounting components Cables 2 2 IPU M2000s 2 2 1 Overview There are 32 IPU M2000s in...

Page 8: ...rt number BMC Ethernet MAC address GW Ethernet MAC address URL for Graphcore support portal 2 2 3 LED indicators The IPU M2000 has LED indicators on both sides of the chassis Rear side LEDs The rear s...

Page 9: ...me interrupts are not serviced for over 2 seconds Steady green light System operational 2 Amber Attention or Service action required 3 White Here I am This is the item being sought or Unit ID 2 3 Serv...

Page 10: ...s an Arista DCS 7010T 48 F Contact Graphcore sales for details of other supported switch types This document describes the default switch 7010T installation only Other switches may have di erent insta...

Page 11: ...ty 2 8 3 QSFP cables IPU M2000 to ToR switch connectivity For server to ToR switch connectivity All IPU POD64 cable connections are described in Section 3 IPU POD64 rack assembly 2 9 Connecting cables...

Page 12: ...face of the IPU M2000 units connectivity ports should be matched with the front door of the rack cold aisle The rear interface of the server and switches power and fans should be matched with the rear...

Page 13: ...default build servers 2 to 4 only have two blue RJ45 cables See Section 3 6 7 Dell R6525 server s cabling for more information about server cabling Note also that Fig 3 2 shows a four server version o...

Page 14: ...00 16 16 IPU M2000 slider kits 16 16 Dell R6525 server 1 4 Arista DCS 7010T 48 F switch 1 1 Arista DCS 7060CX 32S F switch 1 1 2m purple Ethernet 2 2 1 5m blue Ethernet 11 17 1m blue Ethernet 9 9 1m y...

Page 15: ...ing at least 32cm between forks and at least 750KG SWL Knife suitable for cutting plastic tape No 2 Phillips screwdriver Torx TX30 screwdriver Scissors suitable for cutting Velcro If you will be re pa...

Page 16: ...Do not remove the rack from the shock pallet Remove the white bag Fig 3 3 from the rack This contains screws and cage nuts to be used in the assembly of the components into the rack Fig 3 3 White bag...

Page 17: ...e the top and bottom side panels The vertical accessory channels should be positioned at the very front and very rear of the rack If necessary move these from their shipping positions Fig 3 5 Fig 3 5...

Page 18: ...he rack Fig 3 6 Fig 3 6 Accessory channel removal 3 4 5 Adjusting the rear accessory channels Set the rear accessory channel to the furthest position in the rack Tighten up the screws ensuring the tee...

Page 19: ...t 3 4 7 Adjusting the front vertical rails Using a Torx TX30 screwdriver make both front vertical rack rails loose and freely movable Install the accessory channels in the front of the rack one on the...

Page 20: ...ing the rack rails Fig 3 10 M2000 rack rail kit unboxed The IPU M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws The inner rail a xes to the body of...

Page 21: ...eparated rails which has a captive thumb screw at one end to the body of the IPU M2000 Note that the inner rails are mirrored and are not handed As such the procedure for inner rail xing is the same f...

Page 22: ...s behind the head of a xing pin Fig 3 15 Fig 3 15 M2000 rack rail kit xing pins locked 3 Ensure all xing pins are correctly engaged with their respective retention channels 4 Locate the four at head x...

Page 23: ...IPU M2000 body Place the outer rails to one side for later use Fig 3 18 Fig 3 18 M2000 rack rail kit outer rails 3 4 9 Installing PDU brackets Install four cage nuts on the outside at the top and bott...

Page 24: ...et The PDU brackets should be installed at the rear of the rack one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack Fig 3...

Page 25: ...ack The outer rail is also embossed with the text FRONT at the front end of the rail The outer rail large metal latch end is to be installed at the rear of the rack as shown in Fig 3 22 Fig 3 22 Attac...

Page 26: ...the vertical rack rail Fig 3 24 Fig 3 24 IPU M2000 outer rail secured to vertical rack rail 4 Included in the rack rail accessory bag are two screws and two washers One screw with one washer should be...

Page 27: ...d washers attached Next to install an IPU M2000 unit into the rack rails work your way through the following steps in order 1 Pull the sliding rail located within the outer rack rail completely forwar...

Page 28: ...nto the receiving channel of the extended outer rails Fig 3 29 Fig 3 29 Slide IPU M2000 inner rails into outer rails 4 Whilst the server lift is supporting the full weight of the IPU M2000 slide the I...

Page 29: ...de Fig 3 30 Sliding IPU M2000 into rack Fig 3 31 Blue tab release mechanism 6 Screw the captive thumb screw into the inner rack rail Fig 3 32 Fig 3 32 Screwing captive thumb screw into inner rack rail...

Page 30: ...position using four M6 screws Fig 3 33 Fig 3 33 Installing management switch with four M6 screws 3 5 3 Installing the ToR switch Fit the sliders for the ToR switch into position 18 on the rack ensurin...

Page 31: ...the two PDUs vertically at the rear of the rack one on the left side and one on the right side Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown i...

Page 32: ...ull out the rail and t the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail Ensure that the power supplies on the server face the rear of the rack Fig 3...

Page 33: ...mall plastic tab on the left front side of the server bezel and clip the bezel in place on the front of the server ensuring the connection pins on the right hand side of the bezel line up with the con...

Page 34: ...ack The following sections detail the cabling of the rack and the dressing of the cables within the rack For reference the IPU M2000s and server s are numbered as shown in Fig 3 39 Fig 3 39 Server and...

Page 35: ...of the top most IPU M2000 16 and the bottom row 1 4 of the bottom most IPU M2000 1 which are connected together using the 1m OSFP cables Fig 3 40 IPU M2000 IPU Link port numbering and IPU Link cables...

Page 36: ...00 11 IPU Link ports 1 2 3 4 OSFP 0 3m IPU M2000 9 IPU Link ports 5 6 7 8 IPU M2000 10 IPU Link ports 1 2 3 4 OSFP 0 3m IPU M2000 8 IPU Link ports 5 6 7 8 IPU M2000 9 IPU Link ports 1 2 3 4 OSFP 0 3m...

Page 37: ...IPU POD128 build and test guide Fig 3 43 Final IPU M2000 to IPU M2000 IPU Link cabling Version latest 2021 11 25 33...

Page 38: ...6 of IPU M2000 1 into port 2 of IPU M2000 2 and the other side of the cable from port 7 of IPU M2000 1 into port 3 of IPU M2000 2 Fig 3 44 IPU M2000 Sync Link ports Continue the cabling for all IPU M2...

Page 39: ...15 red IPU M2000 9 Sync Link ports 6 7 IPU M2000 10 Sync Link ports 2 3 RJ45 0 15 red IPU M2000 8 Sync Link ports 6 7 IPU M2000 9 Sync Link ports 2 3 RJ45 0 15 red IPU M2000 7 Sync Link ports 6 7 IPU...

Page 40: ...ted from each IPU M2000 to the management switch Fig 3 47 with RJ45 cables The cables required are 12x RJ45 Yellow 1 0m and 4x RJ45 Yellow 1 5m BMC 8x RJ45 Blue 1 0m and 8x RJ45 Blue 1 5m BMC GW SoC T...

Page 41: ...for BMC only future update 3 Customer management interface 5 6 PDU management 7 Server 1 PDU management switch 8 ToR to management switch 9 12 1GbE server management iDRAC 13 16 Server data ports 17 3...

Page 42: ...BMC port on IPU M2000 16 top IPU M2000 in the rack Fig 3 48 Management switch BMC cabling Repeat the process for ports 47 to 37 using 1 0m yellow cables and ports 33 to 36 using 1 5m cables When all c...

Page 43: ...yellow IPU M2000 10 BMC port Management switch port 42 RJ45 1 0m yellow IPU M2000 9 BMC port Management switch port 41 RJ45 1 0m yellow IPU M2000 8 BMC port Management switch port 40 RJ45 1 0m yellow...

Page 44: ...process for ports 31 to 25 using 1 0m blue cables and ports 24 to 17 using 1 5m blue cables When all cables have been connected dress the loom down the side of the cabinet and secure the bundle with a...

Page 45: ...PU M2000 10 BMC GW Soc port Management switch port 26 RJ45 1 0m blue IPU M2000 9 BMC GW Soc port Management switch port 25 RJ45 1 0m blue IPU M2000 8 BMC GW Soc port Management switch port 24 RJ45 1 5...

Page 46: ...to ToR switch ports 16 to 9 8x QSFP1 5m from IPU M2000 1 8 to ToR switch ports 17 to 24 Start from port 9 on the ToR switch and connect the cable to port 2 of IPU M2000 16 Continue cabling from ports...

Page 47: ...port 14 QSFP 1 0m IPU M2000 10 port 2 ToR switch port 15 QSFP 1 0m IPU M2000 9 port 2 ToR switch port 16 QSFP 1 0m IPU M2000 8 port 2 ToR switch port 17 QSFP 1 5m IPU M2000 7 port 2 ToR switch port 18...

Page 48: ...IPU POD128 build and test guide Fig 3 53 Final IPU M2000 to ToR switch cabling Version latest 2021 11 25 44...

Page 49: ...nagement switch to Dell server s iDRAC and Section 3 6 10 Management switch to Dell server s network connector Fig 3 54 Server cabling 4 server version Note that Fig 3 54 shows the four server version...

Page 50: ...Server 4 RNIC port 1 ToR switch port 1 QSFP 1 5m Server 4 RNIC port 2 ToR switch port 2 QSFP 1 5m Server 3 RNIC port 1 ToR switch port 3 QSFP 1 5m Server 3 RNIC port 2 ToR switch port 4 QSFP 1 5m Ser...

Page 51: ...IPU POD128 build and test guide Fig 3 56 ToR switch to server cabling cable connections 4 server version Version latest 2021 11 25 47...

Page 52: ...witch is used Fig 3 57 Management switch to server cabling iDRAC for 4 server version 3 6 10 Management switch to Dell server s network connector Using four of the 1 5m blue RJ45 cables connect the ma...

Page 53: ...ch to server 1 as follows Port 8 from the management server is connected to the lowest server server 1 the correct port to connect to on the server is shown in Fig 3 59 This is used for control of the...

Page 54: ...net port Management switch port Ca bles Ethernet port on left side PDU Management switch port 5 RJ45 2m Ethernet port on right side PDU Management switch port 6 RJ45 2m Allow all the cables going to t...

Page 55: ...Power cabling The designated sockets on the PDUs for servers and IPU M2000s are shown in Fig 3 61 Fig 3 61 Designated PDU sockets for server s and IPU M2000s Fig 3 62 and Fig 3 63 show the nal power c...

Page 56: ...IPU POD128 build and test guide Fig 3 62 Final power cabling direct view Fig 3 63 Final power cabling angled view Version latest 2021 11 25 52...

Page 57: ...3 10 de nes the power cable lengths for each IPU M2000 Table 3 10 Power cables for IPU M2000s Cable colour Cable length IPU M2000 Blue 1m IPU M2000 13 to IPU M2000 16 Red 1m IPU M2000 13 to IPU M2000...

Page 58: ...66 Server power cabling Note The photo above shows the four server version The default reference design has one server the one in the lowest position 3 7 3 Switch power cabling Connect the mains cabl...

Page 59: ...nstall the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack Fig 3 67 1U blanking panels 3 8 2 Front and rear doors Re install the front and rear doors Ensure the...

Page 60: ...eps taken when you unpacked the rack It is important to ensure that the large clear plastic bag has no tears or holes and is re tted correctly as this bag provides protection from water ingress It is...

Page 61: ...nstall the appropriate OS and packages Operating system installation Setup default user accounts and permissions User accounts and groups Setup a DHCP server DHCP Service Dynamic Host Con guration Pro...

Page 62: ...system Data storage 7x 1TB NVME SSD units as a logical RAID 6 managed with MDADM Partitioned to use xfs le system 4 1 3 Memory con guration recommendations The DIMMs should be installed in a fully sym...

Page 63: ...abled L2StreamHwPrefetcher Enabled Enabled MadtCoreEnumeration Linear Linear NumaNodesPerSocket 4 4 CcxAsNumaDomain Disabled Disabled CpuMinSevAsid 1 1 ProcCcds All All CcdCores All All EmbSata AhciMo...

Page 64: ...the Aptitude package manager tool Other packages may be required for local requirements Table 4 2 Required Ubuntu packages autoconf golang go minicom rsyslog automake htop moreutils screen bc ibverbs...

Page 65: ...cmake json c devel ntp sshpass containerd io json c doc parallel sysfsutils devtoolset 7 libaio devel pciutils devel tmux dhcp libibverbs utils php cli vim dkms libuser protobuf devel wdi eigen3 lldpa...

Page 66: ...D64 Table 4 6 Default usernames Log in to Username IPU M2000 BMC OS root IPU M2000 GW OS itadmin Server as Poplar SDK user poplaruser Server as IPU POD admin user ipuuser Server as IT admin user itadm...

Page 67: ...etc syscon g dhcpd DHCPDARGS em4 p1p2 etc dhcp dhcpd d vlan 11 conf le Example of content Add all RNICs below using this format host ipum1mx hardware ethernet aa bb cc dd ee ff fixed address 10 1 5 1...

Page 68: ...ernet a1 b1 c1 d1 e1 f1 fixed address 10 1 2 8 host ipum9bmc hardware ethernet a0 b0 c0 d0 e0 f0 fixed address 10 1 1 9 host ipum9gw hardware ethernet a1 b1 c1 d1 e1 f1 fixed address 10 1 2 9 host ipu...

Page 69: ...10 1 5 101 include etc dhcp dhcpd d vlan 11 conf subnet 10 1 0 0 netmask 255 255 252 0 option subnet mask 255 255 252 0 range 10 1 3 200 10 1 3 254 option ntp servers 10 1 5 101 include etc dhcp dhcp...

Page 70: ...input type imudp port 514 module load imklog permitnonkernelfacility on ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat RepeatedMsgReduction on FileOwner ipuuser FileGroup ipugroup FileCreateM...

Page 71: ...ool con guration le root root 0744 disable monitor driftfile var lib ntp drift fudge 127 127 1 0 stratum 10 includefile etc ntp crypto pw keys etc ntp keys restrict 1 restrict 127 0 0 1 restrict defau...

Page 72: ...ot root 0744 soft memlock unlimited hard memlock unlimited Create directories required for logging Table 4 10 Logging directories Directory User Group Access localdata root ipupodgroup 0775 localdata...

Page 73: ...IPU POD128 build and test guide 4 2 Network con guration 4 2 1 Overview Fig 4 2 gives a logical overview of the network setup within the IPU POD64 Version latest 2021 11 25 69...

Page 74: ...provided as an upgrade for customers that want an isolated BMC network This separates the BMC and GW tra c inside the IPU M2000 onto the two ports BMC on the lower port and GW on the upper port With t...

Page 75: ...CLI Switch BMC man agement 1GbE 10 1 6 2 22 Manual setup 99 32x 100GbE 4x 10G switch management port CLI Switch BMC man agement 1GbE 10 1 6 3 22 Manual setup 99 PDU Power dist unit 1GbE 10 1 6 8 11 Dy...

Page 76: ...itchport default mode access Spanning tree mode mstp No aaa root No ip routing 1 Datacentre BMC 13 no snmp trap link status 2 Datacentre GW 14 no snmp trap link status 3 Datacentre management 99 no sn...

Page 77: ...con guration for the ToR switch is given in Table 4 13 Table 4 13 ToR switch con guration Port Description Vlan Options Hostname MGMT Switch Switchport default mode access Spanning tree mode mstp No...

Page 78: ...tch is con gured independently based on the number of interfaces needed for the IPU POD size in this case IPU POD64 This section describes the interfaces used in an IPU POD64 with gures showing interf...

Page 79: ...switch A xed IP address is required Subnets for each interface should be capable to contain the number of devices of the installation Example netplan con guration le The default location for this le...

Page 80: ...contains the latest IPU M2000 resident software It also includes the server resident rack_tool which is required for doing operations from the management server related to all IPU M2000s Note For larg...

Page 81: ...ains the tool rack_tool which is used for the software upgrade and other rack related tasks targeting the IPU M2000s The rack_tool upgrade command performs the software upgrade See details later for h...

Page 82: ...this le is HOME ipuuser rack_tool rack_config json This le can be edited by a site administrator who integrates the IPU POD64 into the site speci c network in cases where the default IPU POD64 IP addr...

Page 83: ...o use it can be found on the rack_tool man page A suggested naming scheme for IPU M2000s is lr1_ipum n logical rack 1 IPU M2000 n rack_tool is used for the following 1 Querying the version of all IPU...

Page 84: ...s This is especially important if the tests are to be run on a system that has active users rack_tool bist performs BMC chassis hardware testing rack_tool vipu test performs V IPU connectivity related...

Page 85: ...ink 0 00s 0 0 GW Link test skipped The output above shows a successful test with no errors reported As the test results show ve test types were executed on cluster1 The results for each test type are...

Page 86: ...failed are displayed This will give you a hint to which cables to physically inspect and correct Very often a loose cable is the root cause of problems Below is an example of a test run when the 4 OS...

Page 87: ...ster1 Test Type Duration Passed Summary Sync Link 0 90s x y Failed Sync Links ipum1 6 2 ipum2 ipum1 7 3 ipum2 test cluster failed Some tests failed IPU Link training test IPU Link training test veri e...

Page 88: ...Summary Traffic 92 23s 3 4 Traffic test failed Errors encountered in traffic test 1 corrected link errors 460 error counter IPU Link 1 in ipum1 IPU 1 is 250 error counter IPU Link 1 in ipum4 IPU 1 is...

Page 89: ...7 1 BOM for an IPU POD128 Item Product number Description Quantity 1 900 0020 IPU POD64 Logical Rack 2 2 SM ULC DX PVC 1 5M 1 5m LC UPC to LC UPC Duplex OS2 Optical Cable for GW Links 32 3 TR ZC13H N...

Page 90: ...guide Fig 7 1 IPU POD128 topology with cross connected GW links Fig 7 2 shows the location of the GW Link ports on the front of each IPU M2000 Fig 7 2 IPU M2000 front panel showing GW Link ports Vers...

Page 91: ...IPU M2000 14 GW Link ports 1 2 IPU M2000 14 GW Link ports 2 1 IPU M2000 13 GW Link ports 1 2 IPU M2000 13 GW Link ports 2 1 IPU M2000 12 GW Link ports 1 2 IPU M2000 12 GW Link ports 2 1 IPU M2000 11 G...

Page 92: ...arger IPU PODs With this updated IP address scheme multiple IPU POD64 racks can be connected together and can also form part of a larger IPU POD later on without having to change their IP addresses Th...

Page 93: ...r s management network interfaces OS not BMC not used for interface type 1 or 2 networks but closed management server network 4 Not used 5 net 10 5 0 0 16 RDMA interfaces on IPU M2000s and IPU POD ser...

Page 94: ...hased merge to a single IPU POD128 since the migration steps require the IPU POD64 racks to be fully isolated during the upgrade to the new IP address scheme If your IPU POD64 1GbE switches have the V...

Page 95: ...e lrack2 rack_tool json le instead Note There is no inter rack RDMA connectivity data plane between the two IPU POD64 racks unless there is a RoCE spine switch that brings the leaf switches together T...

Page 96: ...using the new management network address 10 5 7 150 4d lrack2 RNIC interface Note This part of the step is only required if there is no spine switch enabling full RDMA connectivity between the two IPU...

Page 97: ...con guration to combine the two IPU POD64 racks into an IPU POD128 While this is ongoing users will not be able to use either of the two IPU POD64 racks Warning It is important that you do NOT apply t...

Page 98: ...ig json le so no network setup change should have been activated before this The IPU M2000s will reboot and request new IP as enabled above Step 10 netplan con guration on lrack1 You need to change to...

Page 99: ...interface access Note this step is only required if the IPU POD128 is NOT fully connected with spine switches Run the following rack_tool command on lrack1 to verify that there is no access to the IP...

Page 100: ...me out for client calls in seconds Optional Default 200 More details about how to use these environment variables are given in the V IPU user guide Use gc info on the management server to check that t...

Page 101: ...unning then you need to start it Step 23 Check IPU M2000s have NTP date and time You need to check whether all 32 IPU M2000s are taking their date and time from the NTP server on lrack1 This can be ch...

Page 102: ...cp dhcpd d ipum dhcp conf shared network L2 IPU M BMC subnet subnet 10 1 0 0 netmask 255 255 0 0 option subnet mask 255 255 0 0 default lease time 600 max lease time 7200 option ntp servers 10 1 7 150...

Page 103: ...0 10 5 8 244 list all lrack numbers that is served by this host here include etc dhcp dhcpd d lrack2 ipum rnic conf 8 6 4 Lrack1 and lrack2 etc dhcp dhcpd d les root gbnwp pod013 1 etc dhcp dhcpd d ll...

Page 104: ...10 2 7 14 host lr7 ipum15gw hardware ethernet 70 69 79 20 16 BD fixed address 10 2 7 15 host lr7 ipum16gw hardware ethernet 70 69 79 20 17 A5 fixed address 10 2 7 16 ipum rnic conf root gbnwp pod7 1...

Page 105: ...5 fixed address 10 2 8 11 host lr8 ipum12gw hardware ethernet 70 69 79 20 12 75 fixed address 10 2 8 12 host lr8 ipum13gw hardware ethernet 70 69 79 20 12 6D fixed address 10 2 8 13 host lr8 ipum14gw...

Page 106: ...described in this section If there are multiple servers on lrack1 up to four the management server lrack1 server 1 will control the remaining servers lrack1 servers 2 3 and 4 with DHCP lrack1 server...

Page 107: ...ing entries for lrack2 IPU M2000s will be required for RNIC connectivity checks on lrack2 IPU M2000s root gbnwp pod7 1 etc dhcp dhcpd d lrack13 cat home ipuuser rack_tool rack_config json gw_root_over...

Page 108: ...10 2 7 11 rnic_ip 10 5 7 11 name lr7_ipum12 bmc_ip 10 1 7 12 gw_ip 10 2 7 12 rnic_ip 10 5 7 12 name lr7_ipum13 bmc_ip 10 1 7 13 gw_ip 10 2 7 13 rnic_ip 10 5 7 13 name lr7_ipum14 bmc_ip 10 1 7 14 gw_ip...

Page 109: ...0 1 8 7 gw_ip 10 2 8 7 rnic_ip 10 5 8 7 name lr8_ipum8 bmc_ip 10 1 8 8 gw_ip 10 2 8 8 rnic_ip 10 5 8 8 name lr8_ipum9 bmc_ip 10 1 8 9 gw_ip 10 2 8 9 rnic_ip 10 5 8 9 name lr8_ipum10 bmc_ip 10 1 8 10 g...

Page 110: ...test guide continued from previous page rnic_ip 10 5 8 14 name lr8_ipum15 bmc_ip 10 1 8 15 gw_ip 10 2 8 15 rnic_ip 10 5 8 15 name lr8_ipum16 bmc_ip 10 1 8 16 gw_ip 10 2 8 16 rnic_ip 10 5 8 16 Version...

Page 111: ...t are Cabling topology verify by reading cable IDs that cables are all connected and consistent with supported topology Sync Links verify external sync cabling connecting IPU M2000s including across d...

Page 112: ...CHAPTER TEN REVISION HISTORY This document s revision history is as follows Version Date Notes 1 0 10th of October 2021 First release Version latest 2021 11 25 108...

Page 113: ...Processor Memory IPU Core IPU Exchange IPU Fabric IPU Link IPU M2000 IPU Machine IPU POD IPU Tile PopART PopLibs PopVision PopTorch Streaming Memory and Virtual IPU are trademarks of Graphcore Ltd Al...

Reviews:

No comments