Graphcore IPU-POD128 Скачать руководство пользователя страница 95

Страница: 95 / 113

IPU-POD128 build and test guide

Step 1: For lrack1 and lrack2

1a)

Copy

rack_config.json

rack_config.json_pod128

in the same directory:

cp /home/ipuuser/.rack_tool/rack_config.json /home/ipuuser/.rack_tool/rack_config.json_pod128

You then need to rewrite the

rack_config.json_pod128

conﬁg ﬁle to match the IPU-POD

128

IP addresses as

shown in

Section 8.7.3, Lrack1: rack_conﬁg.json ﬁle

Doing this means that you will be able to run

rack_tool

commands from the lrack1 management server for all the

IPU-M2000s, in both IPU-POD

racks (32 IPU-M2000s). Upgrading IPU-M2000 software for all 32 IPU-M2000s

can then be run as a single operation.

1b)

RNIC entries in lrack1 and lrack2

rack_tool.json

ﬁles

System fully connected with spine switches

If your IPU-POD

128

is using spine switches then you need to keep all the entries for the RNIC interfaces on both

lrack1 and lrack2

rnic_ip

in the lrack1

rack_tool.json

ﬁle. This ﬁle will also contain all the IPU-Gateway and

BMC interfaces for both lrack1 and lrack2. The lrack2

rack_tool.json

ﬁle is not required as lrack1 will be used

for all

rack_tool

operations.

System NOT fully connected with spine switches

If you are not using spine switches then you need to keep the entries for all the IPU-Gateway and BMC interfaces

for both lrack1 and lrack2 in the lrack1

rack_tool.json

ﬁle, but only the lrack1 RNIC interfaces

rnic_ip

. You

need to list the lrack2 RNIC interfaces

rnic_ip

in the lrack2

rack_tool.json

ﬁle instead.

Note:

There is no inter-rack RDMA connectivity (data plane) between the two IPU-POD

racks unless there is a

RoCE spine switch that brings the leaf switches together. This means that, on the data plane, the lrack1 server(s)

cannot access the lrack2 IPU-M2000s, and the lrack2 server(s) cannot access the lrack1 IPU-M2000s. The lrack1

server(s) can reach the lrack2 IPU-M2000s using the management network (control plane) for software updates.

Note:

Kubernetes will not be supported on the IPU-POD

128

unless there are spine switches providing inter-rack

RDMA connectivity.

Step 2: DHCP conﬁg ﬁles for lrack1 and lrack2

The DHCP conﬁg ﬁle

vlan-13.conf

contains the ports for the IPU-M2000 IPU-Gateway and BMC ports. You

need to add a copy of this ﬁle into a directory called

/etc/dhcp/dhcpd.d/lrack#

, where # denotes the rack (so 7

or 8 in our example). You also need to copy over the

vlan-11.conf

ﬁle.

2a) Lrack1

sudo mkdir /etc/dhcp/dhcpd.d/lrack1

sudo mkdir /etc/dhcp/dhcpd.d/lrack2

sudo cp /etc/dhcp/dhcpd.d/vlan-13.conf /etc/dhcp/dhcpd.d/lrack1

sudo cp /etc/dhcp/dhcpd.dvlan-11.conf /etc/dhcp/dhcpd.d/lrack1

2b) Lrack2

sudo mkdir /etc/dhcp/dhcpd.d/lrack2

sudo cp /etc/dhcp/dhcpd.d/vlan-11.conf /etc/dhcp/dhcpd.d/lrack2

sudo scp /etc/dhcp/dhcpd.d/vlan-13.conf ipuuser@pod7:/etc/dhcp/dhcpd.d/lrack2

2c) Lrack1 (required for spine switches)

You will need to run the following command if you have spine switches. If you don’t have spine switches then it is

not necessary, however, if you might add spine switches in the future then it is a good idea to run it now so that

lrack1 is prepared.

Version: latest (2021-11-25)

Содержание IPU-POD128

Страница 1: ...IPU POD128 build and test guide Version latest Graphcore Ltd Nov 25 2021...

Страница 2: ...10 3 2 Document reproduction 11 3 3 Required tools 11 3 4 Preparing the rack 11 3 4 1 Rail distance 11 3 4 2 Unpacking the rack 12 3 4 3 Removing the side panels and doors 12 3 4 4 Removing the verti...

Страница 3: ...ion recommendations 58 4 1 3 Memory con guration recommendations 58 4 1 4 BIOS con guration 58 4 1 5 Operating system installation 60 4 1 6 User accounts and groups 62 4 1 7 DHCP Service Dynamic Host...

Страница 4: ...rack2 etc dhcp dhcpd conf 98 8 6 2 Lrack1 etc dhcp dhcpd d ipum dhcp conf 98 8 6 3 Lrack2 etc dhcp dhcpd d ipum dhcp conf 99 8 6 4 Lrack1 and lrack2 etc dhcp dhcpd d les 99 8 6 5 Lrack1 etc dhcp dhcpd...

Страница 5: ...0 For more information on IPU POD systems available from Graphcore see https www graphcore ai products Warning This guide is for properly trained service personnel and technicians who are required to...

Страница 6: ...four IPUs in the IPU M2000 across a RoCE network provides external IPU Exchange Memory and enables IPU scaleout across 100GbE GW Links for rack to rack connectivity GW Link High speed communication li...

Страница 7: ...rted 2 Switches one 1GbE management switch and one 100GbE ToR switch 2 Power distribution units 1 Rack Supplementary mounting components Cables 2 2 IPU M2000s 2 2 1 Overview There are 32 IPU M2000s in...

Страница 8: ...rt number BMC Ethernet MAC address GW Ethernet MAC address URL for Graphcore support portal 2 2 3 LED indicators The IPU M2000 has LED indicators on both sides of the chassis Rear side LEDs The rear s...

Страница 9: ...me interrupts are not serviced for over 2 seconds Steady green light System operational 2 Amber Attention or Service action required 3 White Here I am This is the item being sought or Unit ID 2 3 Serv...

Страница 10: ...s an Arista DCS 7010T 48 F Contact Graphcore sales for details of other supported switch types This document describes the default switch 7010T installation only Other switches may have di erent insta...

Страница 11: ...ty 2 8 3 QSFP cables IPU M2000 to ToR switch connectivity For server to ToR switch connectivity All IPU POD64 cable connections are described in Section 3 IPU POD64 rack assembly 2 9 Connecting cables...

Страница 12: ...face of the IPU M2000 units connectivity ports should be matched with the front door of the rack cold aisle The rear interface of the server and switches power and fans should be matched with the rear...

Страница 13: ...default build servers 2 to 4 only have two blue RJ45 cables See Section 3 6 7 Dell R6525 server s cabling for more information about server cabling Note also that Fig 3 2 shows a four server version o...

Страница 14: ...00 16 16 IPU M2000 slider kits 16 16 Dell R6525 server 1 4 Arista DCS 7010T 48 F switch 1 1 Arista DCS 7060CX 32S F switch 1 1 2m purple Ethernet 2 2 1 5m blue Ethernet 11 17 1m blue Ethernet 9 9 1m y...

Страница 15: ...ing at least 32cm between forks and at least 750KG SWL Knife suitable for cutting plastic tape No 2 Phillips screwdriver Torx TX30 screwdriver Scissors suitable for cutting Velcro If you will be re pa...

Страница 16: ...Do not remove the rack from the shock pallet Remove the white bag Fig 3 3 from the rack This contains screws and cage nuts to be used in the assembly of the components into the rack Fig 3 3 White bag...

Страница 17: ...e the top and bottom side panels The vertical accessory channels should be positioned at the very front and very rear of the rack If necessary move these from their shipping positions Fig 3 5 Fig 3 5...

Страница 18: ...he rack Fig 3 6 Fig 3 6 Accessory channel removal 3 4 5 Adjusting the rear accessory channels Set the rear accessory channel to the furthest position in the rack Tighten up the screws ensuring the tee...

Страница 19: ...t 3 4 7 Adjusting the front vertical rails Using a Torx TX30 screwdriver make both front vertical rack rails loose and freely movable Install the accessory channels in the front of the rack one on the...

Страница 20: ...ing the rack rails Fig 3 10 M2000 rack rail kit unboxed The IPU M2000 rail kit comprises two mated inner and outer rack rails and an accessory bag containing screws The inner rail a xes to the body of...

Страница 21: ...eparated rails which has a captive thumb screw at one end to the body of the IPU M2000 Note that the inner rails are mirrored and are not handed As such the procedure for inner rail xing is the same f...

Страница 22: ...s behind the head of a xing pin Fig 3 15 Fig 3 15 M2000 rack rail kit xing pins locked 3 Ensure all xing pins are correctly engaged with their respective retention channels 4 Locate the four at head x...

Страница 23: ...IPU M2000 body Place the outer rails to one side for later use Fig 3 18 Fig 3 18 M2000 rack rail kit outer rails 3 4 9 Installing PDU brackets Install four cage nuts on the outside at the top and bott...

Страница 24: ...et The PDU brackets should be installed at the rear of the rack one bracket on top with 9cm distance from the top of rack and one bracket on the bottom with 12cm distance from the bottom of rack Fig 3...

Страница 25: ...ack The outer rail is also embossed with the text FRONT at the front end of the rail The outer rail large metal latch end is to be installed at the rear of the rack as shown in Fig 3 22 Fig 3 22 Attac...

Страница 26: ...the vertical rack rail Fig 3 24 Fig 3 24 IPU M2000 outer rail secured to vertical rack rail 4 Included in the rack rail accessory bag are two screws and two washers One screw with one washer should be...

Страница 27: ...d washers attached Next to install an IPU M2000 unit into the rack rails work your way through the following steps in order 1 Pull the sliding rail located within the outer rack rail completely forwar...

Страница 28: ...nto the receiving channel of the extended outer rails Fig 3 29 Fig 3 29 Slide IPU M2000 inner rails into outer rails 4 Whilst the server lift is supporting the full weight of the IPU M2000 slide the I...

Страница 29: ...de Fig 3 30 Sliding IPU M2000 into rack Fig 3 31 Blue tab release mechanism 6 Screw the captive thumb screw into the inner rack rail Fig 3 32 Fig 3 32 Screwing captive thumb screw into inner rack rail...

Страница 30: ...position using four M6 screws Fig 3 33 Fig 3 33 Installing management switch with four M6 screws 3 5 3 Installing the ToR switch Fit the sliders for the ToR switch into position 18 on the rack ensurin...

Страница 31: ...the two PDUs vertically at the rear of the rack one on the left side and one on the right side Push the mains cable through the roof of the rack and then clip the PDUs onto the PDU bracket as shown i...

Страница 32: ...ull out the rail and t the server to the rail ensuring the T pins on the side of the server locate in the slots on the rail Ensure that the power supplies on the server face the rear of the rack Fig 3...

Страница 33: ...mall plastic tab on the left front side of the server bezel and clip the bezel in place on the front of the server ensuring the connection pins on the right hand side of the bezel line up with the con...

Страница 34: ...ack The following sections detail the cabling of the rack and the dressing of the cables within the rack For reference the IPU M2000s and server s are numbered as shown in Fig 3 39 Fig 3 39 Server and...

Страница 35: ...of the top most IPU M2000 16 and the bottom row 1 4 of the bottom most IPU M2000 1 which are connected together using the 1m OSFP cables Fig 3 40 IPU M2000 IPU Link port numbering and IPU Link cables...

Страница 36: ...00 11 IPU Link ports 1 2 3 4 OSFP 0 3m IPU M2000 9 IPU Link ports 5 6 7 8 IPU M2000 10 IPU Link ports 1 2 3 4 OSFP 0 3m IPU M2000 8 IPU Link ports 5 6 7 8 IPU M2000 9 IPU Link ports 1 2 3 4 OSFP 0 3m...

Страница 37: ...IPU POD128 build and test guide Fig 3 43 Final IPU M2000 to IPU M2000 IPU Link cabling Version latest 2021 11 25 33...

Страница 38: ...6 of IPU M2000 1 into port 2 of IPU M2000 2 and the other side of the cable from port 7 of IPU M2000 1 into port 3 of IPU M2000 2 Fig 3 44 IPU M2000 Sync Link ports Continue the cabling for all IPU M2...

Страница 39: ...15 red IPU M2000 9 Sync Link ports 6 7 IPU M2000 10 Sync Link ports 2 3 RJ45 0 15 red IPU M2000 8 Sync Link ports 6 7 IPU M2000 9 Sync Link ports 2 3 RJ45 0 15 red IPU M2000 7 Sync Link ports 6 7 IPU...

Страница 40: ...ted from each IPU M2000 to the management switch Fig 3 47 with RJ45 cables The cables required are 12x RJ45 Yellow 1 0m and 4x RJ45 Yellow 1 5m BMC 8x RJ45 Blue 1 0m and 8x RJ45 Blue 1 5m BMC GW SoC T...

Страница 41: ...for BMC only future update 3 Customer management interface 5 6 PDU management 7 Server 1 PDU management switch 8 ToR to management switch 9 12 1GbE server management iDRAC 13 16 Server data ports 17 3...

Страница 42: ...BMC port on IPU M2000 16 top IPU M2000 in the rack Fig 3 48 Management switch BMC cabling Repeat the process for ports 47 to 37 using 1 0m yellow cables and ports 33 to 36 using 1 5m cables When all c...

Страница 43: ...yellow IPU M2000 10 BMC port Management switch port 42 RJ45 1 0m yellow IPU M2000 9 BMC port Management switch port 41 RJ45 1 0m yellow IPU M2000 8 BMC port Management switch port 40 RJ45 1 0m yellow...

Страница 44: ...process for ports 31 to 25 using 1 0m blue cables and ports 24 to 17 using 1 5m blue cables When all cables have been connected dress the loom down the side of the cabinet and secure the bundle with a...

Страница 45: ...PU M2000 10 BMC GW Soc port Management switch port 26 RJ45 1 0m blue IPU M2000 9 BMC GW Soc port Management switch port 25 RJ45 1 0m blue IPU M2000 8 BMC GW Soc port Management switch port 24 RJ45 1 5...

Страница 46: ...to ToR switch ports 16 to 9 8x QSFP1 5m from IPU M2000 1 8 to ToR switch ports 17 to 24 Start from port 9 on the ToR switch and connect the cable to port 2 of IPU M2000 16 Continue cabling from ports...

Страница 47: ...port 14 QSFP 1 0m IPU M2000 10 port 2 ToR switch port 15 QSFP 1 0m IPU M2000 9 port 2 ToR switch port 16 QSFP 1 0m IPU M2000 8 port 2 ToR switch port 17 QSFP 1 5m IPU M2000 7 port 2 ToR switch port 18...

Страница 48: ...IPU POD128 build and test guide Fig 3 53 Final IPU M2000 to ToR switch cabling Version latest 2021 11 25 44...

Страница 49: ...nagement switch to Dell server s iDRAC and Section 3 6 10 Management switch to Dell server s network connector Fig 3 54 Server cabling 4 server version Note that Fig 3 54 shows the four server version...

Страница 50: ...Server 4 RNIC port 1 ToR switch port 1 QSFP 1 5m Server 4 RNIC port 2 ToR switch port 2 QSFP 1 5m Server 3 RNIC port 1 ToR switch port 3 QSFP 1 5m Server 3 RNIC port 2 ToR switch port 4 QSFP 1 5m Ser...

Страница 51: ...IPU POD128 build and test guide Fig 3 56 ToR switch to server cabling cable connections 4 server version Version latest 2021 11 25 47...

Страница 52: ...witch is used Fig 3 57 Management switch to server cabling iDRAC for 4 server version 3 6 10 Management switch to Dell server s network connector Using four of the 1 5m blue RJ45 cables connect the ma...

Страница 53: ...ch to server 1 as follows Port 8 from the management server is connected to the lowest server server 1 the correct port to connect to on the server is shown in Fig 3 59 This is used for control of the...

Страница 54: ...net port Management switch port Ca bles Ethernet port on left side PDU Management switch port 5 RJ45 2m Ethernet port on right side PDU Management switch port 6 RJ45 2m Allow all the cables going to t...

Страница 55: ...Power cabling The designated sockets on the PDUs for servers and IPU M2000s are shown in Fig 3 61 Fig 3 61 Designated PDU sockets for server s and IPU M2000s Fig 3 62 and Fig 3 63 show the nal power c...

Страница 56: ...IPU POD128 build and test guide Fig 3 62 Final power cabling direct view Fig 3 63 Final power cabling angled view Version latest 2021 11 25 52...

Страница 57: ...3 10 de nes the power cable lengths for each IPU M2000 Table 3 10 Power cables for IPU M2000s Cable colour Cable length IPU M2000 Blue 1m IPU M2000 13 to IPU M2000 16 Red 1m IPU M2000 13 to IPU M2000...

Страница 58: ...66 Server power cabling Note The photo above shows the four server version The default reference design has one server the one in the lowest position 3 7 3 Switch power cabling Connect the mains cabl...

Страница 59: ...nstall the supplied APC 1U blanking panels in every unoccupied rack slot at the front of the rack Fig 3 67 1U blanking panels 3 8 2 Front and rear doors Re install the front and rear doors Ensure the...

Страница 60: ...eps taken when you unpacked the rack It is important to ensure that the large clear plastic bag has no tears or holes and is re tted correctly as this bag provides protection from water ingress It is...

Страница 61: ...nstall the appropriate OS and packages Operating system installation Setup default user accounts and permissions User accounts and groups Setup a DHCP server DHCP Service Dynamic Host Con guration Pro...

Страница 62: ...system Data storage 7x 1TB NVME SSD units as a logical RAID 6 managed with MDADM Partitioned to use xfs le system 4 1 3 Memory con guration recommendations The DIMMs should be installed in a fully sym...

Страница 63: ...abled L2StreamHwPrefetcher Enabled Enabled MadtCoreEnumeration Linear Linear NumaNodesPerSocket 4 4 CcxAsNumaDomain Disabled Disabled CpuMinSevAsid 1 1 ProcCcds All All CcdCores All All EmbSata AhciMo...

Страница 64: ...the Aptitude package manager tool Other packages may be required for local requirements Table 4 2 Required Ubuntu packages autoconf golang go minicom rsyslog automake htop moreutils screen bc ibverbs...

Страница 65: ...cmake json c devel ntp sshpass containerd io json c doc parallel sysfsutils devtoolset 7 libaio devel pciutils devel tmux dhcp libibverbs utils php cli vim dkms libuser protobuf devel wdi eigen3 lldpa...

Страница 66: ...D64 Table 4 6 Default usernames Log in to Username IPU M2000 BMC OS root IPU M2000 GW OS itadmin Server as Poplar SDK user poplaruser Server as IPU POD admin user ipuuser Server as IT admin user itadm...

Страница 67: ...etc syscon g dhcpd DHCPDARGS em4 p1p2 etc dhcp dhcpd d vlan 11 conf le Example of content Add all RNICs below using this format host ipum1mx hardware ethernet aa bb cc dd ee ff fixed address 10 1 5 1...

Страница 68: ...ernet a1 b1 c1 d1 e1 f1 fixed address 10 1 2 8 host ipum9bmc hardware ethernet a0 b0 c0 d0 e0 f0 fixed address 10 1 1 9 host ipum9gw hardware ethernet a1 b1 c1 d1 e1 f1 fixed address 10 1 2 9 host ipu...

Страница 69: ...10 1 5 101 include etc dhcp dhcpd d vlan 11 conf subnet 10 1 0 0 netmask 255 255 252 0 option subnet mask 255 255 252 0 range 10 1 3 200 10 1 3 254 option ntp servers 10 1 5 101 include etc dhcp dhcp...

Страница 70: ...input type imudp port 514 module load imklog permitnonkernelfacility on ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat RepeatedMsgReduction on FileOwner ipuuser FileGroup ipugroup FileCreateM...

Страница 71: ...ool con guration le root root 0744 disable monitor driftfile var lib ntp drift fudge 127 127 1 0 stratum 10 includefile etc ntp crypto pw keys etc ntp keys restrict 1 restrict 127 0 0 1 restrict defau...

Страница 72: ...ot root 0744 soft memlock unlimited hard memlock unlimited Create directories required for logging Table 4 10 Logging directories Directory User Group Access localdata root ipupodgroup 0775 localdata...

Страница 73: ...IPU POD128 build and test guide 4 2 Network con guration 4 2 1 Overview Fig 4 2 gives a logical overview of the network setup within the IPU POD64 Version latest 2021 11 25 69...

Страница 74: ...provided as an upgrade for customers that want an isolated BMC network This separates the BMC and GW tra c inside the IPU M2000 onto the two ports BMC on the lower port and GW on the upper port With t...

Страница 75: ...CLI Switch BMC man agement 1GbE 10 1 6 2 22 Manual setup 99 32x 100GbE 4x 10G switch management port CLI Switch BMC man agement 1GbE 10 1 6 3 22 Manual setup 99 PDU Power dist unit 1GbE 10 1 6 8 11 Dy...

Страница 76: ...itchport default mode access Spanning tree mode mstp No aaa root No ip routing 1 Datacentre BMC 13 no snmp trap link status 2 Datacentre GW 14 no snmp trap link status 3 Datacentre management 99 no sn...

Страница 77: ...con guration for the ToR switch is given in Table 4 13 Table 4 13 ToR switch con guration Port Description Vlan Options Hostname MGMT Switch Switchport default mode access Spanning tree mode mstp No...

Страница 78: ...tch is con gured independently based on the number of interfaces needed for the IPU POD size in this case IPU POD64 This section describes the interfaces used in an IPU POD64 with gures showing interf...

Страница 79: ...switch A xed IP address is required Subnets for each interface should be capable to contain the number of devices of the installation Example netplan con guration le The default location for this le...

Страница 80: ...contains the latest IPU M2000 resident software It also includes the server resident rack_tool which is required for doing operations from the management server related to all IPU M2000s Note For larg...

Страница 81: ...ains the tool rack_tool which is used for the software upgrade and other rack related tasks targeting the IPU M2000s The rack_tool upgrade command performs the software upgrade See details later for h...

Страница 82: ...this le is HOME ipuuser rack_tool rack_config json This le can be edited by a site administrator who integrates the IPU POD64 into the site speci c network in cases where the default IPU POD64 IP addr...

Страница 83: ...o use it can be found on the rack_tool man page A suggested naming scheme for IPU M2000s is lr1_ipum n logical rack 1 IPU M2000 n rack_tool is used for the following 1 Querying the version of all IPU...

Страница 84: ...s This is especially important if the tests are to be run on a system that has active users rack_tool bist performs BMC chassis hardware testing rack_tool vipu test performs V IPU connectivity related...

Страница 85: ...ink 0 00s 0 0 GW Link test skipped The output above shows a successful test with no errors reported As the test results show ve test types were executed on cluster1 The results for each test type are...

Страница 86: ...failed are displayed This will give you a hint to which cables to physically inspect and correct Very often a loose cable is the root cause of problems Below is an example of a test run when the 4 OS...

Страница 87: ...ster1 Test Type Duration Passed Summary Sync Link 0 90s x y Failed Sync Links ipum1 6 2 ipum2 ipum1 7 3 ipum2 test cluster failed Some tests failed IPU Link training test IPU Link training test veri e...

Страница 88: ...Summary Traffic 92 23s 3 4 Traffic test failed Errors encountered in traffic test 1 corrected link errors 460 error counter IPU Link 1 in ipum1 IPU 1 is 250 error counter IPU Link 1 in ipum4 IPU 1 is...

Страница 89: ...7 1 BOM for an IPU POD128 Item Product number Description Quantity 1 900 0020 IPU POD64 Logical Rack 2 2 SM ULC DX PVC 1 5M 1 5m LC UPC to LC UPC Duplex OS2 Optical Cable for GW Links 32 3 TR ZC13H N...

Страница 90: ...guide Fig 7 1 IPU POD128 topology with cross connected GW links Fig 7 2 shows the location of the GW Link ports on the front of each IPU M2000 Fig 7 2 IPU M2000 front panel showing GW Link ports Vers...

Страница 91: ...IPU M2000 14 GW Link ports 1 2 IPU M2000 14 GW Link ports 2 1 IPU M2000 13 GW Link ports 1 2 IPU M2000 13 GW Link ports 2 1 IPU M2000 12 GW Link ports 1 2 IPU M2000 12 GW Link ports 2 1 IPU M2000 11 G...

Страница 92: ...arger IPU PODs With this updated IP address scheme multiple IPU POD64 racks can be connected together and can also form part of a larger IPU POD later on without having to change their IP addresses Th...

Страница 93: ...r s management network interfaces OS not BMC not used for interface type 1 or 2 networks but closed management server network 4 Not used 5 net 10 5 0 0 16 RDMA interfaces on IPU M2000s and IPU POD ser...

Страница 94: ...hased merge to a single IPU POD128 since the migration steps require the IPU POD64 racks to be fully isolated during the upgrade to the new IP address scheme If your IPU POD64 1GbE switches have the V...

Страница 95: ...e lrack2 rack_tool json le instead Note There is no inter rack RDMA connectivity data plane between the two IPU POD64 racks unless there is a RoCE spine switch that brings the leaf switches together T...

Страница 96: ...using the new management network address 10 5 7 150 4d lrack2 RNIC interface Note This part of the step is only required if there is no spine switch enabling full RDMA connectivity between the two IPU...

Страница 97: ...con guration to combine the two IPU POD64 racks into an IPU POD128 While this is ongoing users will not be able to use either of the two IPU POD64 racks Warning It is important that you do NOT apply t...

Страница 98: ...ig json le so no network setup change should have been activated before this The IPU M2000s will reboot and request new IP as enabled above Step 10 netplan con guration on lrack1 You need to change to...

Страница 99: ...interface access Note this step is only required if the IPU POD128 is NOT fully connected with spine switches Run the following rack_tool command on lrack1 to verify that there is no access to the IP...

Страница 100: ...me out for client calls in seconds Optional Default 200 More details about how to use these environment variables are given in the V IPU user guide Use gc info on the management server to check that t...

Страница 101: ...unning then you need to start it Step 23 Check IPU M2000s have NTP date and time You need to check whether all 32 IPU M2000s are taking their date and time from the NTP server on lrack1 This can be ch...

Страница 102: ...cp dhcpd d ipum dhcp conf shared network L2 IPU M BMC subnet subnet 10 1 0 0 netmask 255 255 0 0 option subnet mask 255 255 0 0 default lease time 600 max lease time 7200 option ntp servers 10 1 7 150...

Страница 103: ...0 10 5 8 244 list all lrack numbers that is served by this host here include etc dhcp dhcpd d lrack2 ipum rnic conf 8 6 4 Lrack1 and lrack2 etc dhcp dhcpd d les root gbnwp pod013 1 etc dhcp dhcpd d ll...

Страница 104: ...10 2 7 14 host lr7 ipum15gw hardware ethernet 70 69 79 20 16 BD fixed address 10 2 7 15 host lr7 ipum16gw hardware ethernet 70 69 79 20 17 A5 fixed address 10 2 7 16 ipum rnic conf root gbnwp pod7 1...

Страница 105: ...5 fixed address 10 2 8 11 host lr8 ipum12gw hardware ethernet 70 69 79 20 12 75 fixed address 10 2 8 12 host lr8 ipum13gw hardware ethernet 70 69 79 20 12 6D fixed address 10 2 8 13 host lr8 ipum14gw...

Страница 106: ...described in this section If there are multiple servers on lrack1 up to four the management server lrack1 server 1 will control the remaining servers lrack1 servers 2 3 and 4 with DHCP lrack1 server...

Страница 107: ...ing entries for lrack2 IPU M2000s will be required for RNIC connectivity checks on lrack2 IPU M2000s root gbnwp pod7 1 etc dhcp dhcpd d lrack13 cat home ipuuser rack_tool rack_config json gw_root_over...

Страница 108: ...10 2 7 11 rnic_ip 10 5 7 11 name lr7_ipum12 bmc_ip 10 1 7 12 gw_ip 10 2 7 12 rnic_ip 10 5 7 12 name lr7_ipum13 bmc_ip 10 1 7 13 gw_ip 10 2 7 13 rnic_ip 10 5 7 13 name lr7_ipum14 bmc_ip 10 1 7 14 gw_ip...

Страница 109: ...0 1 8 7 gw_ip 10 2 8 7 rnic_ip 10 5 8 7 name lr8_ipum8 bmc_ip 10 1 8 8 gw_ip 10 2 8 8 rnic_ip 10 5 8 8 name lr8_ipum9 bmc_ip 10 1 8 9 gw_ip 10 2 8 9 rnic_ip 10 5 8 9 name lr8_ipum10 bmc_ip 10 1 8 10 g...

Страница 110: ...test guide continued from previous page rnic_ip 10 5 8 14 name lr8_ipum15 bmc_ip 10 1 8 15 gw_ip 10 2 8 15 rnic_ip 10 5 8 15 name lr8_ipum16 bmc_ip 10 1 8 16 gw_ip 10 2 8 16 rnic_ip 10 5 8 16 Version...

Страница 111: ...t are Cabling topology verify by reading cable IDs that cables are all connected and consistent with supported topology Sync Links verify external sync cabling connecting IPU M2000s including across d...

Страница 112: ...CHAPTER TEN REVISION HISTORY This document s revision history is as follows Version Date Notes 1 0 10th of October 2021 First release Version latest 2021 11 25 108...

Страница 113: ...Processor Memory IPU Core IPU Exchange IPU Fabric IPU Link IPU M2000 IPU Machine IPU POD IPU Tile PopART PopLibs PopVision PopTorch Streaming Memory and Virtual IPU are trademarks of Graphcore Ltd Al...

Результаты поиска

Содержание IPU-POD128

Отзывы:

Бренды по названию

Популярные бренды

Graphcore IPU-POD128, Руководство по сборке и тестированию

Результаты поиска

Содержание IPU-POD128

Отзывы:

Бренды по названию

Популярные бренды