Huawei FusionServer Pro CH121 V5 V100R005 Maintenance And Service Manual Download

Page: 78 / 192

Figure 4-18

Removing mezzanine cards

Place the removed mezzanine card in an ESD bag.

----End

4.3.12 Installing a Mezzanine Card

Procedure

Step 1

Power off the compute node.

For details, see

3.1 Power-Off Procedure

Step 2

Remove the compute node.

For details, see

3.3 Removing a CH121 V5

Step 3

Place the compute node on the ESD workstation.

Step 4

Remove the cover.

For details, see

3.5 Removing the Cover

Step 5

Take the spare part out of its ESD bag.

Step 6

Determine the position of the mezzanine card.

FusionServer Pro CH121 V5 Compute Node
Maintenance and Service Guide

4 Parts Replacement

Issue 06 (2019-08-10)

Summary of Contents for FusionServer Pro CH121 V5 V100R005

Page 1: ...FusionServer Pro CH121 V5 Compute Node V100R005 Maintenance and Service Guide Issue 06 Date 2019 08 10 HUAWEI TECHNOLOGIES CO LTD ...

Page 2: ...scribed in this document may not be within the purchase scope or the usage scope Unless otherwise specified in the contract all statements information and recommendations in this document are provided AS IS without warranties guarantees or representations of any kind either express or implied The information in this document is subject to change without notice Every effort has been made in the pre...

Page 3: ...ndicates a potentially hazardous situation which if not avoided could result in death or serious injury Indicates a potentially hazardous situation which if not avoided could result in minor or moderate injury Indicates a potentially hazardous situation which if not avoided could result in equipment damage data loss performance deterioration or unanticipated results NOTICE is used to address pract...

Page 4: ...l Added the procedures for removing and installing a supercapacitor 03 2018 05 14 l Added the procedures for removing and installing an NVMe drive l Added the procedures for removing and installing a battery l Added information about the Avago SAS3408 and Avago SAS3004 iMR RAID controller cards 02 2017 12 28 l Updated the memory module installation guidelines 01 2017 10 12 l This issue is the firs...

Page 5: ...rd Panel 22 3 11 Removing a PCIe Card 23 3 12 Installing a PCIe Card 25 3 13 Removing a SAS SATA Drive 28 3 14 Installing a SAS SATA Drive 29 3 15 Removing an NVMe Drive 31 3 16 Installing an NVMe Drive 44 4 Parts Replacement 53 4 1 Obtaining Tools 53 4 2 Replacement Guidelines 53 4 2 1 System Reliability 53 4 3 Field Replaceable Units 54 4 3 1 Removing an M 2 FRU 54 4 3 2 Installing an M 2 FRU 55...

Page 6: ...lt in USB Flash Drive 94 4 3 20 Installing a USB Flash Drive 95 4 3 20 1 Installing an External USB Flash Drive 95 4 3 20 2 Installing a Built in USB Flash Drive 96 4 3 21 Removing the Battery 98 4 3 22 Installing the Battery 99 4 3 23 Removing the Mainboard and Compute Node Case 101 4 3 24 Installing the Mainboard and Compute Node Case 103 5 Software and Hardware Compatibility 106 6 Troubleshooti...

Page 7: ...40 11 1 Technical Specifications 140 11 2 Environmental Specifications 143 11 3 Physical Specifications 144 12 Common Operations 145 12 1 Querying the iBMC IP Address 145 12 2 Logging In to the iBMC WebUI 146 12 3 Logging In to the MM910 WebUI 150 12 4 Collecting Log Information on the MM910 WebUI 155 12 5 Logging In to a Compute Node Using MM910 SOL 156 12 6 Managing the E9000 Server Using the Lo...

Page 8: ...ment 174 12 9 1 Enabling VMD 174 12 9 2 Disabling VMD 175 12 10 Clearing Data from a Storage Device 175 A More Information 178 A 1 Technical Support 178 A 2 Product Information 179 A 3 Product Configuration Resources 179 A 4 Maintenance Tools 180 B Appendix 181 B 1 Product SN 181 B 2 RAS Features 182 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide Contents Issue 06 2019 08 10 ...

Page 9: ...y The CH121 V5 provides dense computing capability and an ultra large memory It is optimized for virtualization cloud computing high performance computing and compute intensive enterprise applications The CH121 V5 compute nodes are installed in an E9000 chassis and are centrally managed by the management module FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 1 Overview Issue 0...

Page 10: ...Figure 1 1 CH121 V5 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 1 Overview Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 2 ...

Page 11: ...Optional TPM 6 Screw in RAID controller card 7 Optional Supercapacitor 8 Heat sinks 9 Mainboard 10 Optional M 2 FRUsb 11 Drivesa 12 Optional M 2 adapterb 13 Drive cage 14 Processors 15 Mezzanine card 16 BIOS battery 17 Memory modules FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 1 Overview Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 3 ...

Page 12: ...ough two UPI links at a speed of up to 10 4 GT s l The mezzanine cards connect to the processors through PCIe buses to provide service ports l The Platform Controller Hub PCH has a built in MAC chip and provides two 10 Gbit s ports l The storage module consisting of a RAID controller card and a drive backplane connects to the CPUs through PCIe buses l The BMC provides device management functions s...

Page 13: ...RSPs are provided on demand however the lead time is not committed Stored in a country s warehouses based on the contract service type Non Spare Part NSP NSPs are not spare parts and do not have replaceable units at lower levels Not supplied or stored RSP SUB Sub Part It is classified as a type of RSP and has RSPs or NRSPs at lower levels Stored in the warehouses near sites based on the contract s...

Page 14: ... on the contract service type Lower level parts vary with actual demands NSP SUB It is not a spare part but has RSPs or NRSPs at lower levels This part is not stored Lower level parts vary with actual demands FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 2 Spare Parts Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 6 ...

Page 15: ... the Air Duct 3 9 Removing the PCIe Card Panel 3 10 Installing the PCIe Card Panel 3 11 Removing a PCIe Card 3 12 Installing a PCIe Card 3 13 Removing a SAS SATA Drive 3 14 Installing a SAS SATA Drive 3 15 Removing an NVMe Drive 3 16 Installing an NVMe Drive FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co ...

Page 16: ...th the chassis l Press the power button on the compute node front panel to power off the compute node gracefully For details see 10 1 2 Indicators and Buttons l Hold down the power button on the compute node front panel for six seconds to power off the compute node forcibly For details see 10 1 2 Indicators and Buttons l Use the MM910 WebUI For details see MM910 Management Module V100R001 User Gui...

Page 17: ... any of the following ways l If the compute node is installed but the chassis is not yet powered on power on the chassis The compute node will be powered on along with the chassis l If the chassis is powered on but the compute node is not installed install the compute node The compute node will be automatically powered on after it is installed l If the chassis is powered on but the compute node is...

Page 18: ...ing In to the MM910 WebUI Step 3 On the HMM WebUI choose PSUs FANs The PSUs Fans page is displayed Step 4 Choose PSU Management Power Control from the navigation tree Figure 3 2 Power Control Step 5 Select the compute node to be powered on and click Power On The selected compute node will be powered on NOTE To power on all compute nodes at a time select the check box in the header row and click Po...

Page 19: ...hold the bottom of the device with both hands when removing it 1 Press the ejector release button on the compute node See 1 in Figure 3 4 2 Fully open the ejector lever See 2 in Figure 3 4 3 Holding the ejector lever pull the compute node out of the chassis horizontally See 3 in Figure 3 4 4 Close the ejector lever See 4 in Figure 3 4 FusionServer Pro CH121 V5 Compute Node Maintenance and Service ...

Page 20: ...panels in vacant slots Otherwise the ventilation heat dissipation electromagnetic shielding and dustproof effects of the chassis will be affected Insert the filler panel into the chassis horizontally until it clicks into place FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 12 ...

Page 21: ... Installing a CH121 V5 Procedure Step 1 Determine the position of the compute node in the chassis FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 13 ...

Page 22: ...ler panels are installed 1 Press and hold the groove inward on the filler panel to release the filler panel See 1 in Figure 3 7 2 Remove the filler panel from the chassis See 1 in Figure 3 7 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 14 ...

Page 23: ...th compute nodes Perform this operation only when no partition board is installed between two half width compute nodes a Tilt the partition board to the chassis insert it into the installation holes and make it upright See 1 and 2 in Figure 3 8 b Pull the partition board outwards See 3 and 4 in Figure 3 8 The partition board is properly installed when you hear a click FusionServer Pro CH121 V5 Com...

Page 24: ...izontally See 3 in Figure 3 9 5 Close the ejector lever See 4 in Figure 3 9 Figure 3 9 Installing a compute node Step 4 Power on the compute node For details see 3 2 Power On Procedure Step 5 Check indicator status For details see 10 1 2 Indicators and Buttons End FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologi...

Page 25: ...n the cover and push the cover backwards as far as it will go See 1 and 2 in Figure 3 10 2 Lift the chassis cover See 3 in Figure 3 10 Figure 3 10 Removing the cover End 3 6 Installing the Cover Procedure Step 1 Place the compute node on the ESD workstation Step 2 Take the spare part out of its ESD bag Step 3 Install the cover FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 ...

Page 26: ...5 Step 5 Power on the compute node For details see 3 2 Power On Procedure End 3 7 Removing the Air Ducts Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Determine the pos...

Page 27: ...Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Take the spare part out of its ESD bag Step 6 Determine the position of the air duct Step 7 Check whether a PCIe card is installed l If yes go to Step 8 l If no go to Step 9 Step 8 Transform the air duct FusionServer Pro CH121 V5 Compute Node Maintenance an...

Page 28: ...Transforming the air duct Step 9 Align the right air duct with the right DIMMs align the left air duct with the left DIMMs and heat sinks and vertically place the air ducts downwards Figure 3 14 Installing the air duct FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 20 ...

Page 29: ...emove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Determine the position of the PCIe card panel Step 6 Loosen the two screws that secure the PCIe card panel and lift the PCIe card panel vertically See 1 and 2 in Figure 3 15 Figure 3 15 Removing a PCIe card panel F...

Page 30: ... yes go to Step 6 l If no go to Step 7 Step 6 Remove the PCIe card For details see 3 11 Removing a PCIe Card Step 7 Take the spare part out of its ESD bag Step 8 Determine the position of the PCIe card panel Step 9 Install the PCIe card panel 1 Vertically slide the PCIe card panel into the slot and align the screw holes on the PCIe card panel with those on the pane See 1 in Figure 3 16 2 Tighten t...

Page 31: ...Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Remove the air duct For details see 3 7 Removing the Air Ducts FusionServer Pro CH121 V5 Compute Node Maintenance and Serv...

Page 32: ...at secure the PCIe card bracket and vertically lift the PCIe card bracket See 1 and 2 in Figure 3 18 2 Remove the PCIe card from the PCIe card bracket in the arrow direction See 3 in Figure 3 18 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 24 ...

Page 33: ...ly End 3 12 Installing a PCIe Card Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue ...

Page 34: ...far as it will go and ensure that the edge of the PCIe card is flush with the PCIe card bracket See 1 in Figure 3 20 2 Install the PCIe card bracket Ensure that the PCIe card connectors align with the connectors on the mainboard and the screw holes on the PCIe card filler panel align with the screw holes on the front panel See 2 in Figure 3 20 3 Tighten the three screws to secure the PCIe card bra...

Page 35: ...r details see 3 6 Installing the Cover Step 12 Install the compute node For details see 3 4 Installing a CH121 V5 Step 13 Power on the compute node For details see 3 2 Power On Procedure End FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 27 ...

Page 36: ...drives support RAID configuration l If the drive to be removed is not in a RAID array or is in a RAID array without redundancy for example RAID 0 back up the data on the drive before removing it l If the drive to be removed is in a RAID array with redundancy you do not need to back up data before removing it Procedure Step 1 Determine the slot and position of the drive Figure 3 21 Positions of dri...

Page 37: ...p 5 Place the removed component in an ESD bag Step 6 Install a drive filler NOTE Perform this operation only when a drive is not installed immediately End 3 14 Installing a SAS SATA Drive l The SATA or SAS drives support hot swap After a faulty drive is disconnected from its backplane wait 30 seconds before removing it The alarm will not be cleared immediately after a new drive is installed It wil...

Page 38: ...tion only when a drive filler is installed Step 4 Install a drive 1 Fully open the ejector lever and push the drive into the slot as far as it will go See 1 in Figure 3 24 2 Ensure that the ejector lever is fastened to the chassis beam and then close it to completely insert the drive into the slot See 2 in Figure 3 24 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Ope...

Page 39: ...D function is disabled the NVMe drives support only orderly hot swap For details about how to disable the VMD function see 12 9 2 Disabling VMD Procedure VMD Enabled NOTE l Use the Intelligent Computing Compatibility Checker to obtain information about the operating systems supported by the VMD function l Before removing an NVMe drive l Stop accessing the NVMe drive l Back up date of the NVMe driv...

Page 40: ... ejector lever See 1 in Figure 3 26 2 Open the drive ejector lever completely See 2 in Figure 3 26 3 Holding the ejector lever of the drive pull out the drive for about 3 cm 1 18 in Wait for at least 30 seconds and remove the drive from the server See 3 in Figure 3 26 Do not insert or remove drives frequently If a drive is frequently removed and installed in an interval shorter than 30 seconds the...

Page 41: ...the NVMe drives one by one Remove the next NVMe drive only after the previous one is removed completely l Run command to uninstall the drive driver Windows OS NOTE l Use the Intelligent Computing Compatibility Checker to obtain information about the Windows operating systems supported l The following uses Windows Server 2012 R2 as an example Step 1 Determine the slot and position of the drive Fusi...

Page 42: ...Management Disk Management 2 Right click a drive and choose Properties from the shortcut menu In the displayed dialog box view the slot ID of the drive 3 Determine whether the drive is an NVMe drive based on the slot ID If yes record its drive letter as shown in Figure 3 28 Determine the physical slot of the drive based on the slot ID Table 3 1 lists the mapping between the slot IDs and physical s...

Page 43: ...ol NVMe Toolbox The NVMe Toolbox can be used to query NVMe drive information upgrade the firmware and perform hot swap NOTE After installing ES3000 V5 NVMe drives install the toolkit required by the OS to manage the drives The ES3000 V5 tool is incompatible with the built in NVMe driver of the Windows OS Therefore ensure that you have installed Huawei NVMe driver before using the tool You can obta...

Page 44: ...ucceed will be displayed Figure 3 30 Success message NOTE If a failure message is displayed indicating that the NVMe drive is being accessed stop the related services and click eject again FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 3 Basic Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 36 ...

Page 45: ...form this operation only when a drive is not installed immediately End Linux OS NOTE Use the Intelligent Computing Compatibility Checker to obtain information about the Linux operating systems supported Step 1 Determine the slot and position of the drive Figure 3 31 Positions of drives Step 2 Check the running status of the drive by observing its indicators For details see NVMe SSD Indicators Step...

Page 46: ...t grub cfg command In the command output find the sentence that is displayed after the cat proc cmdline command is executed in Step 4 a and add a space and then pciehp pciehp_force 1 pci pcie_bus_perf at the end of the sentence Do not add the content in a new line Figure 3 33 Configuring kernel parameters 1 Figure 3 34 Configuring kernel parameters 2 c Save the file and restart the compute node d ...

Page 47: ... the command output find the sentence that is displayed after the cat proc cmdline command is executed in Step 4 a and add a space and then pciehp pciehp_force 1 pci pcie_bus_perf at the end of the sentence Do not add the content in a new line Figure 3 37 Configuring kernel parameters 1 Figure 3 38 Configuring kernel parameters 2 c Save the file and restart the compute node FusionServer Pro CH121 ...

Page 48: ...3 10 0 514 26 2 e17 x86_64 rpm command to upload the upgrade package to the OS for installation d Restart the OS after the installation is complete e Run the uname r command If 3 10 0 514 26 2 el7 x86_64 is displayed the kernel is upgraded successfully Figure 3 41 Checking the kernel version l Example 2 SUSE 12 2 Upgrade the kernel of SUSE 12 2 to kernel default 4 4 74 92 32 1 x86_64 or later to s...

Page 49: ...nd sys name nvme0n1 to query the bus number of the SSD device Run lspci vvv xxx s B D F grep i slot to query the slot No Figure 3 44 Querying the slot No B D F and slot ID Table 3 2 Mapping between physical slot numbers B D F and slot IDs Configuratio n Physical Slot No Root Port B D F Device B D F Slot ID 2 x 2 5 NVMe drive configuration Slot0 85 02 0 86 00 0 80 Slot2 85 03 0 87 00 0 81 The Devic...

Page 50: ... contact Huawei technical support b Change the value of the a8 register to e1 setpci s B D F a8 B e1 B D F indicates the root port B D F of the NVMe drive For details about the B D F information see Table 3 2 c Run the following command to check the modification result lspci xxx s B D F B D F indicates the root port B D F of the NVMe drive For details about the B D F information see Table 3 2 Figu...

Page 51: ... command echo 0 sys bus pci slots 81 power 5 Observe the indicator status of the NVMe drive If the green indicator is off and the yellow indicator blinks at 0 5 Hz remove the NVMe drive slowly For details about the drive indicator status see NVMe SSD Indicators For details about how to manually remove an NVMe drive see 3 13 Removing a SAS SATA Drive 6 Place the removed component in an ESD bag Step...

Page 52: ...Enabled NOTE l Use the Intelligent Computing Compatibility Checker to obtain information about the operating systems supported by the VMD function l Install the NVMe drives one by one Install the next NVMe drive only after the previous one is installed completely l If the OS is RHEL 7 3 and the VMD function is enabled the OS kernel 3 10 0 514 el7 x86_64 must be used for surprise hot swap of NVMe d...

Page 53: ... download the NVMe driver from the official Intel website for such Linux OSs For details about how to install the driver on the Linux OS see Intel_Linux_NVMe_Guide_330602 002 pdf obtained after the driver package is decompressed Driver download link https downloadcenter intel com download 28718 v t NOTE Download the NVMe driver of the latest version If no such driver is available use the latest dr...

Page 54: ... detected and working properly For details see NVMe SSD Indicators c Check the status of the newly installed NVMe drive in the system n If the drive status is normal no further action is required n If the newly installed NVMe drive cannot be identified perform Step 5 d to Step 5 e d Go to Device Manager and scan for hardware changes e Check the NVMe drive status on the OS l Linux a Slowly insert t...

Page 55: ... download the NVMe driver from the official Intel website for such Linux OSs For details about how to install the driver on the Linux OS see Intel_Linux_NVMe_Guide_330602 002 pdf obtained after the driver package is decompressed Driver download link https downloadcenter intel com download 28718 v t NOTE Download the NVMe driver of the latest version If no such driver is available use the latest dr...

Page 56: ...etected and working properly For details see NVMe SSD Indicators c Check the status of the newly installed NVMe drive in the system n If the drive status is normal no further action is required n If the newly installed NVMe drive cannot be identified perform Step 5 d to Step 5 e d Go to Device Manager and scan for hardware changes e Check the NVMe drive status on the OS l Linux a Configure kernel ...

Page 57: ...ins pciehp pciehp_force 1 pci pcie_bus_perf the kernel is configured successfully Figure 3 53 Checking the configuration n If SUSE 12 2 is installed in UEFI 1 Run the cat proc cmdline command If the command output does not contain pciehp pciehp_force 1 pci pcie_bus_perf you need to configure kernel parameters Figure 3 54 Opening the configuration file FusionServer Pro CH121 V5 Compute Node Mainten...

Page 58: ...he file and restart the compute node 4 Run the cat proc cmdline command If the command output contains pciehp pciehp_force 1 pci pcie_bus_perf the kernel is configured successfully Figure 3 57 Checking the configuration b Upgrade the OS kernel n Example 1 RHEL 7 3 Upgrade the kernel of RHEL 7 3 to kernel 3 10 0 514 26 2 e17 x86_64 or later to support orderly hot swap of NVMe drives 1 Run the uname...

Page 59: ...n n Example 2 SUSE 12 2 Upgrade the kernel of SUSE 12 2 to kernel default 4 4 74 92 32 1 x86_64 or later to support orderly hot swap of NVMe drives 1 Run the uname r command to check the current kernel version Figure 3 60 Kernel version 2 Log in to the official SUSE website to download the kernel upgrade package kernel 3 10 0 514 26 2 e17 x86_64 for example 3 Run the rpm ivh kernel default 4 4 74 ...

Page 60: ...n installing a supported OS on an NVMe drive load the NVMe driver for the system to enable the NVMe drives to be identified by the OS Then proceed with the OS installation l Only the UEFI mode supports OS installation on an NVMe drive 1 Set the UEFI mode a Press Del upon server startup to access the BIOS b Open the boot configuration screen and set Boot Type to UEFI Boot Type c Press F10 to save t...

Page 61: ... heat dissipation ensure that the server is completely filled with components or filler panels l If the processor socket is not installed with a processor install a socket cover If the CH121 V5 is configured with only one processor the processor must be installed in socket CPU 1 and memory modules must be configured for CPU 1 l Install air ducts above memory slots l The CMOS battery has sufficient...

Page 62: ...rect installation 4 3 Field Replaceable Units 4 3 1 Removing an M 2 FRU Procedure Step 1 Determine the position of the M 2 FRU Determine the position of the M 2 FRU to be removed Do not remove the entire M 2 module that has the operating system installed Otherwise services on the compute node will be affected Figure 4 1 Positions of the M 2 FRUs Step 2 Release the handle of the M 2 FRU and pull ou...

Page 63: ...form this operation only when a M 2 FRU is not installed immediately End 4 3 2 Installing an M 2 FRU Procedure Step 1 Determine the position of the M 2 FRU Figure 4 3 Positions of the M 2 FRUs Step 2 Remove the filler panel FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 55 ...

Page 64: ...alling an M 2 FRU Step 5 Check the indicator status of the M 2 FRU For details see M 2 FRU Indicators End 4 3 3 Removing the Screw in RAID Controller Card Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details s...

Page 65: ...percapacitor is configured l If yes go to Step 9 l If no go to Step 10 Step 9 Remove the supercapacitor For details see 4 3 9 Removing the Supercapacitor Step 10 Remove the RAID controller card 1 Loosen the screws that secure the RAID controller card See 1 in Figure 4 6 2 Remove the RAID controller card from the connector on the mainboard See 2 in Figure 4 6 FusionServer Pro CH121 V5 Compute Node ...

Page 66: ...e compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Check whether a PCIe card is installed l If yes go to Step 6 l If no go to Step 7 Step 6 Remove the PCIe card For details see 3 11 Removing a PCIe Card Step 7 Take the spare part out of its ESD bag Step 8 Determine the position of the screw in RAID controller card FusionServer Pro CH121 V5 C...

Page 67: ...See 2 in Figure 4 8 Figure 4 8 Installing a screw in RAID controller card Step 10 Check whether a supercapacitor is configured l If yes go to Step 11 l If no go to Step 12 Step 11 Install the supercapacitor For details see 4 3 10 Installing the Supercapacitor Step 12 Check whether a PCIe card is installed FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issu...

Page 68: ...rd User Guide 2 Activate RAID For details see LSI SAS3008IR Common Tasks Legacy Dual Mode Activating a RAID Configuration in Huawei V5 Server RAID Controller Card User Guide End 4 3 5 Removing the Avago SAS3004iMR PCIe RAID Control Card Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 P...

Page 69: ...r Ducts Step 6 Take the spare part out of its ESD bag Step 7 Determine the position of the Avago SAS3004iMR RAID controller card Step 8 Install the Avago SAS3004iMR RAID controller card For details see 3 12 Installing a PCIe Card Step 9 Install the M 2 FRU on the Avago SAS3004iMR RAID controller card For details see 4 3 8 Installing an M 2 FRU on the Avago SAS3004iMR PCIe RAID Controller Card Step...

Page 70: ...e 4 9 3 Place the removed M 2 FRU into an ESD bag Figure 4 9 Removing an M 2 FRU Step 3 Place the removed component in an ESD bag End 4 3 8 Installing an M 2 FRU on the Avago SAS3004iMR PCIe RAID Controller Card The M 2 FRU on the Avago SAS3004iMR RAID controller card is hot swappable Procedure Step 1 Determine the position of the M 2 FRU on the Avago SAS3004iMR RAID controller card Step 2 Install...

Page 71: ...Supercapacitor Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Determine the position of the supercapacitor FusionServer Pro CH121 V5 Compute Node Maintenance and Service...

Page 72: ...lder horizontally See 3 in Figure 4 12 Figure 4 12 Removing the supercapacitor Step 7 Place the removed component in an ESD bag Step 8 Remove the supercapacitor holder 1 Loosen the screws that secure the supercapacitor holder See 1 in Figure 4 13 2 Lift the supercapacitor holder See 2 in Figure 4 13 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2...

Page 73: ...edure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Take the spare part out of its ESD bag Step 6 Determine the position of the supercapacitor FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 ...

Page 74: ... tighten the screws See 2 in Figure 4 15 Figure 4 15 Installing the supercapacitor holder Step 8 Install the supercapacitor 1 Push the supercapacitor into the capacitor holder horizontally until the supercapacitor is secured by the plastic latches See 1 in Figure 4 16 2 Connect the cable to the supercapacitor See 2 in Figure 4 16 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide...

Page 75: ...anine card with another model record all the following configurations l BIOS configuration For details see the user guide of the mezzanine card you use l OS configuration including IP addresses VLAN IDs and NIC bonding configuration The method used to query the IP address varies depending on the operating system n If Windows is used query the IP address in the NIC property window n If VMware ESXi ...

Page 76: ...ool to query the WWPN If VMware ESXi is used run the esxcli storage san fc list command on the CLI If Linux is used run the cat sys class fc_host port_name command l NIC bus information On the CLI run the lspci grep i eth command to query the NIC chip name and obtain the bus number based on the NIC chip name Step 2 Power off the compute node For details see 3 1 Power Off Procedure Step 3 Remove th...

Page 77: ... card from the connector See 1 in Figure 4 18 3 Place the removed mezzanine card in an ESD bag 4 Loosen the two captive screws c and d shown in Figure 4 18 that secure the lower mezzanine card 5 Remove the lower mezzanine card from the connector See 2 in Figure 4 18 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technol...

Page 78: ...ve the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Take the spare part out of its ESD bag Step 6 Determine the position of the mezzanine card FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei ...

Page 79: ... screws c and d shown in Figure 4 20 to secure the lower mezzanine card 3 Align the upper mezzanine card with the positioning holes and metallic studs on the mounting base and press the mezzanine card vertically downwards into the connector on the mainboard See 2 in Figure 4 20 4 Tighten the two captive screws a and b shown in Figure 4 20 to secure the upper mezzanine card FusionServer Pro CH121 V...

Page 80: ...e user guide of the mezzanine card you use l OS configuration For details see the user guide of the mezzanine card you use l Switch configuration For details see the configuration guide of the switch you use l FC configuration For details see the configuration guide of the FC disk array you use Step 12 Check whether the mezzanine firmware version is the latest l If yes no further action is require...

Page 81: ...er see section Installing Drivers and Firmware in Huawei Server OS Installation Guide End 4 3 13 Removing a Processor l Only Huawei technical support or personnel authorized by Huawei can remove the processors from Huawei servers l Do not wear ESD gloves during processor replacement The gloves may catch on pins on the processor socket and damage it l Intel heat sinks or Huawei developed heat sinks...

Page 82: ...e a T20 Torx screwdriver to loosen the two diagonal screws marked 2 on the label on the heat sink See 1 in Figure 4 22 b Use a T20 Torx screwdriver to loosen the two diagonal screws marked 1 on the label on the heat sink See 2 in Figure 4 22 c Lift the heat sink and place it upside down on the desk FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 20...

Page 83: ...Figure 4 23 c Lift the heat sink and place it upside down on the desk Figure 4 23 Removing the heat sink Step 10 Remove the processor carrier and processor 1 Insert a flat head screwdriver into the groove of the processor carrier and twist it to pry up one side of the carrier See 1 in Figure 4 24 NOTE Insert only the tip of the flat head screwdriver Do not apply excessive force FusionServer Pro CH...

Page 84: ...nt damage to the pins do not touch the pins of the processor socket or the corresponding area on the processor l Keep the processor away from the chassis to prevent thermal compound from falling off 1 Bend the edge of the carrier with the triangular hole to release the processor from the carrier Figure 4 25 Removing a processor 1 2 Hold the two sides of the processor and lift it FusionServer Pro C...

Page 85: ...processor socket cannot be exposed and must be covered by a processor or cover during transportation and storage 1 Aligning the processor socket cover with the guide pin on the processor socket place the cover on the processor socket See 1 in Figure 4 27 2 Press the positions shown by 2 in Figure 4 27 until you hear a click FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Par...

Page 86: ...essor the heat sink and the processor carrier If there is residual thermal compound use a tissue to clean it l Intel heat sinks or Huawei developed heat sinks are used depending on the processor cooling solution Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 FusionServer Pro CH121 V5 Compute...

Page 87: ...art out of its ESD bag Step 9 Determine the position of the processor Figure 4 28 Positions of processors Step 10 Install the processor on the carrier 1 Take a processor carrier out of an ESD bag 2 Aligning the notch in one edge of the processor with the protrusion on the processor carrier secure it Ensure that the processor corner with a triangle mark is in the corner of the processor carrier wit...

Page 88: ...cessor is tilted press the tilted part to fasten it Step 11 Apply thermal compound 1 Determine the area on the processor that will be in contact with the heat sink and evenly apply 0 4 ml of thermal compound on the area NOTE When applying thermal compound place the processor carrier on the desktop The thermal compound injector has volume marks The two line five dot s shape and X shape patterns are...

Page 89: ...und is evenly and fully coated Figure 4 32 Smeared thermal compound layer Step 12 Install the processor carrier to the heat sink 1 Aligning the processor corner with a triangle mark with the corner of the heat sink with a notched triangle buckle the processor carrier on the heat sink FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyri...

Page 90: ...ly secured If the processor carrier is not firmly buckled with the heat sink press the processor carrier to secure it Figure 4 34 Checking the assembly Step 13 Install the processor and heat sink 1 Remove the processor socket cover FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 82 ...

Page 91: ... the operation and contact Huawei technical support 3 Horizontally holding the processor and heat sink align the notched corner on the heat sink with the notched corner on the processor socket and place the processor and heat sink downwards on the socket along the guide sleeves Keep the processor and heat sink parallel with the mainboard to prevent damage to processor socket pins FusionServer Pro ...

Page 92: ...hten the two diagonal screws marked 2 in the label in the middle of the heat sink See 2 in Figure 4 37 Figure 4 37 Installing the heat sink Huawei developed heat sink i Use a Phillips screwdriver to tighten the two diagonal screws marked 1 on the label on the heat sink See 1 in Figure 4 38 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 ...

Page 93: ...6 Install the air duct For details see 3 8 Installing the Air Duct Step 17 Install the cover For details see 3 6 Installing the Cover Step 18 Install the compute node For details see 3 4 Installing a CH121 V5 Step 19 Power on the compute node For details see 3 2 Power On Procedure End 4 3 15 Removing a Memory Module Removing a memory module Step 1 Power off the compute node For details see 3 1 Pow...

Page 94: ...l If yes go to Step 7 l If no go to Step 8 Step 7 Remove the PCIe card For details see 3 11 Removing a PCIe Card Step 8 Determine the position of the memory module Figure 4 39 Memory slots Step 9 Remove the memory module 1 Open the memory module ejectors on the memory slot See 1 in Figure 4 40 2 Remove the memory module from the slot See 2 in Figure 4 40 FusionServer Pro CH121 V5 Compute Node Main...

Page 95: ...ure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Remove the air duct For details see 3 7 Removing the Air Ducts Step 6 Check whether a PCIe card is installed l If yes go to Step 7 l If no go to Step 8 Step 7 Remove the PCIe card FusionServer Pro CH121...

Page 96: ...emory module ejectors 2 Align the edge connector of the memory module with the alignment key of the memory slot and insert the memory module in the slot The two memory ejectors are closed automatically after the memory module is firmly seated NOTE Do not touch the edge connector on a memory module with bare hands Before installing a memory module ensure that the edge connector is not contaminated ...

Page 97: ...ails see 3 2 Power On Procedure Step 17 Check whether the memory module is replaced successfully 1 Log in to the iBMC WebUI For details see 12 2 Logging In to the iBMC WebUI 2 Choose Information System Info Memory and check the memory information If information about the new memory module is displayed the replacement is successful If information about the original memory module is displayed access...

Page 98: ...latform BIOS Parameter Reference l If a broken or disfigured screw is found on a TPM take appropriate measures to protect data Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover St...

Page 99: ...mpute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Take the spare part out of its ESD bag Step 6 Determine the position of the TPM FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Part...

Page 100: ...e 1 in Figure 4 47 2 Tighten the screw to secure the TPM See 2 in Figure 4 47 Figure 4 47 Installing the TPM Step 8 Install the cover For details see 3 6 Installing the Cover FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 92 ...

Page 101: ...le 5 Press F10 The system displays Exit Saving changes dialog box is displayed 6 Select Yes The settings are saved 7 Restart the server The TPM settings take effect after the server is restarted End 4 3 19 Removing a USB Flash Drive 4 3 19 1 Removing an External USB Flash Drive Procedure Step 1 Determine the position of the USB flash drive Figure 4 48 Positions of the USB flash drives FusionServer...

Page 102: ...2 Removing a Built in USB Flash Drive Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Determine the position of the built in USB flash drive FusionServer Pro CH121 V5 Com...

Page 103: ... component in an ESD bag End 4 3 20 Installing a USB Flash Drive 4 3 20 1 Installing an External USB Flash Drive Procedure Step 1 Take the spare part out of its ESD bag Step 2 Determine the position of the USB flash drive FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 95 ...

Page 104: ... 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co L...

Page 105: ...ort Figure 4 55 Installing a USB flash drive Step 8 Install the cover For details see 3 6 Installing the Cover Step 9 Install the compute node For details see 3 4 Installing a CH121 V5 Step 10 Power on the compute node For details see 3 2 Power On Procedure End FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies...

Page 106: ...ation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Determine the position of the battery Figure 4 56 Position of the battery Step 6 Press and hold the two latches outwards until the battery ejects and remove the battery vertically See 1 and 2 in Figure 4 57 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyrigh...

Page 107: ...ling the Battery Procedure Step 1 Power off the compute node For details see 3 1 Power Off Procedure Step 2 Remove the compute node For details see 3 3 Removing a CH121 V5 Step 3 Place the compute node on the ESD workstation Step 4 Remove the cover For details see 3 5 Removing the Cover Step 5 Take the spare part out of its ESD bag Step 6 Determine the position of the battery FusionServer Pro CH12...

Page 108: ...s down the whole battery into the slot Figure 4 59 Installing the battery Step 8 Install the cover For details see 3 6 Installing the Cover Step 9 Install the compute node For details see 3 4 Installing a CH121 V5 FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 4 Parts Replacement Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 100 ...

Page 109: ...e iBMC However the exported configuration files do not contain password information You need to manually configure the password for the iBMC user For details see Import Export in FusionServer Pro E9000 Server iBMC User Guide l The iBMC and BIOS configuration files exported from the iBMC contain the configuration list For details see Configuration File Description in FusionServer Pro E9000 Server i...

Page 110: ...em For details see 3 11 Removing a PCIe Card Step 11 Remove the supercapacitor if there is any For details see 4 3 9 Removing the Supercapacitor Step 12 Remove the screw in RAID controller card For details see 4 3 3 Removing the Screw in RAID Controller Card Step 13 Remove the TPM For details see 4 3 17 Removing the TPM Step 14 Remove the USB flash drive For details see 4 3 19 Removing a USB Flash...

Page 111: ...SD bags NOTE Before placing the mainboard and the compute node case into an ESD bag ensure that the processor sockets are installed with covers End 4 3 24 Installing the Mainboard and Compute Node Case Procedure Step 1 Take the spare part out of its ESD bag Step 2 Place the compute node on the ESD workstation Step 3 Determine the position of the mainboard and the compute node case Figure 4 61 Main...

Page 112: ...ontroller Card Step 11 Install the supercapacitor if it is configured For details see 4 3 10 Installing the Supercapacitor Step 12 If PCIe cards are configured install them For details see 3 12 Installing a PCIe Card Step 13 Install the air duct For details see 3 8 Installing the Air Duct Step 14 Install the cover For details see 3 6 Installing the Cover Step 15 Install drives or M 2 modules l Ins...

Page 113: ...files imported before the replacement of the mainboard you can import the configuration files after the replacement However the exported configuration files do not contain password information You need to manually configure the password for the iBMC user For details see iBMC WebUI Configuration Import Export in FusionServer Pro E9000 Server iBMC User Guide l The iBMC and BIOS configuration files e...

Page 114: ...re supported Do not use incompatible components Otherwise the server may fail to work properly The technical support and warranty do not cover faults caused by incompatible components FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 5 Software and Hardware Compatibility Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 106 ...

Page 115: ...ault occurs on a server collect logs for fault diagnosis l Fault diagnosis Fault diagnosis rules and tools help Huawei technical support engineers and maintenance engineers to analyze and rectify faults according to alarms and hardware fault symptoms l Software and firmware upgrade Software and firmware upgrade packages can be downloaded by server model and installed as needed l Preventive mainten...

Page 116: ...9000 Server V100R001 Upgrade Guide l FusionServer iDriver List l FusionServer Tools 2 0 uMate User Guide Upgrading Software or Firmware l Upgrade the iBMC BIOS CPLD or other firmware For details see E9000 Server V100R001 Upgrade Guide Installing or Updating the Driver If the existing driver versions on a server are inconsistent with the driver version mapping install the drivers of the required ve...

Page 117: ...r network management system NMS Starts resets and performs power control of the compute nodes in the chassis Records and queries logs Manages users including MM910 administrators and compute node iBMC users l KVM over IP The KVM over IP function allows users to remotely monitor and control devices using a local keyboard video and mouse KVM on the client l Virtual media Virtual media is used with t...

Page 118: ...oprietary intelligent system for remotely managing a server The iBMC complies with IPMI 2 0 and SNMP standards and supports various functions including KVM redirection text console redirection remote virtual media and hardware monitoring and management The iBMC provides the following features l Multiple management interfaces for system integration The iBMC provides IPMI command line interface CLI ...

Page 119: ...and the hardware to interact with the keyboard display and other input output I O devices The BIOS data is stored on the Serial Peripheral Interface SPI flash memory The BIOS performs a power on self test POST initializes CPU and memory checks the I O and boot device and finally boots the OS The BIOS also provides the advanced configuration and power interface ACPI and hot swap The Huawei Purley b...

Page 120: ...function helps users determine hardware health status based on collected log information Firmware upgrade function helps users upgrade server firmware Power control function helps users power on power off and restart servers BIOS configuration function helps users perform BIOS configuration for servers BMC configuration function helps users perform BMC configuration for servers HMM configuration f...

Page 121: ...re supported l User friendly interfaces uMate provides a GUI and CLI l Easy operations Users can perform operations as prompted and batch operations are supported FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 7 Software and Configuration Utilities Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 113 ...

Page 122: ...klift l Take protective measures if a Class A product is used in residential areas as it is likely to cause radio interference Personal Safety l Only personnel certified or authorized by Huawei are allowed to install the hardware l Stop any operation that may cause personal injury or equipment damage report the problem to a project supervisor immediately and take protective measures l Do not opera...

Page 123: ... 8 2 Conductive objects to be removed Figure 8 3 shows how to wear an ESD wrist strap a Secure the ESD wrist strap around your wrist b Fasten the strap buckle and ensure that the ESD wrist strap is in contact with your skin c Insert the ground terminal attached to the ESD wrist strap into the jack on the grounded rack or chassis FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide ...

Page 124: ...e PSUs fan modules drives and the mainboard Handle the equipment with care l Exercise caution when using tools that could cause personal injury l Connect the primary and secondary power cables to different power distribution units PDUs to ensure reliable system operation l Properly ground a device before powering it on Transportation Precautions Improper transportation may damage equipment Contact...

Page 125: ...on is permitted to carry Organization Weight kg lb European Committee for Standardization CEN 25 55 13 International Organization for Standardization ISO 25 55 13 National Institute for Occupational Safety and Health NIOSH 23 50 72 Health and Safety Executive HSE 25 55 13 General Administration of Quality Supervision Inspection and Quarantine of the People s Republic of China AQSIQ l Men 15 33 08 ...

Page 126: ... The heating system reduces indoor humidity and increases ESD l Place unused products into ESD bags to prevent direct contact with humans during transportation and storage l Before products arrive at an ESD area store them in their own packages l Place the product package on a grounded surface before taking out the product l Wear an ESD wrist strap when installing or removing a server component En...

Page 127: ...e skin l Use heel straps toe straps or boot straps in standing workplaces Wear the straps on both feet when standing on conductive floors or dissipating floor mats l Use conductive field service tools l Use a portable field service kit with a folding static dissipating work mat FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 9 ESD Issue 06 2019 08 10 Copyright Huawei Technolog...

Page 128: ...age 10 5 Network 10 6 I O Expansion 10 7 Boards 10 1 Front Panel 10 1 1 Appearance Figure 10 1 Front view with an M 2 module FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 120 ...

Page 129: ... configuration of them l b An M 2 module is a 2 5 inch drive module that consists of one M 2 adapter and two M 2 FRUs 10 1 2 Indicators and Buttons Positions Figure 10 2 Indicators and buttons on the front panel 1 Power button indicator 2 UID button indicator 3 Health status indicator FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 C...

Page 130: ...a device UID indicator l Off The device is not being located l Blinking blue The device has been located and is differentiated from other devices that have also been located l Steady blue The device is being located UID button l You can turn on or off the UID indicator by pressing the UID button or by using the iBMC or MM910 CLI or WebUI l You can press this button to turn on or off the UID indica...

Page 131: ...at the USB device functions properly The server may operate abnormally if an abnormal USB device is connected 10 1 4 Installation Positions The CH121 V5 is installed in a half width slot in the front of the E9000 chassis An E9000 chassis can house a maximum of 16 CH121 V5 compute nodes FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 ...

Page 132: ...U1 l The same model of processors must be used in a server l Contact your local Huawei sales representative or use the Intelligent Computing Compatibility Checker to determine the components to be used FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 124 ...

Page 133: ...can determine the memory module properties based on the label attached to the memory module Figure 10 6 Memory identifier FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 125 ...

Page 134: ...17 7 DIMM type l R RDIMM l L LRDIMM 10 3 2 Memory Subsystem Architecture The CH121 V5 provides 24 memory slots Each processor integrates six memory channels Install the memory modules in the primary memory channels first If the primary memory channel is not populated the memory modules in secondary memory channels cannot be used Table 10 2 Memory channels CPU Memory Channel Memory Slot CPU 1 1A pr...

Page 135: ...rimary DIMM110 2B1 2B DIMM111 2B2 2C primary DIMM120 2C1 2C DIMM121 2C2 2D primary DIMM130 2D1 2D DIMM131 2D2 2E primary DIMM140 2E1 2E DIMM141 2E2 2F primary DIMM150 2F1 2F DIMM151 2F2 10 3 3 Memory Compatibility Observe the following rules when configuring DDR4 memory modules FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyrigh...

Page 136: ...y capacity supported by the CPUs l Use the Intelligent Computing Compatibility Checker to determine the capacity type of a single memory module l The maximum number of memory modules supported by a server varies depending on the CPU type memory type rank quantity and operating voltage NOTE Each memory channel supports a maximum of 8 ranks The number of memory modules supported by each channel vari...

Page 137: ... features such as memory mirroring single device data correction SDDC SDDC 1 and double device data correction DDDC are not supported if x4 and x8 memory modules are used together Table 10 3 DDR4 memory specifications Parameter Specifications Rank Single rank Dual rank Quad rank Octal rank Rated speed MT s 2933a 2933a 2666 2666 Operating voltage V 1 2 1 2 1 2 1 2 Maximum number of DDR4 memory modu...

Page 138: ...es in vacant slots l Observe the following when configuring DDR4 memory modules in specific operating mode Memory sparing mode n Comply with the general installation guidelines n Each memory channel must have a valid online spare configuration n The channels can have different online spare configurations n Each populated channel must have a spare rank Memory mirroring mode n Comply with the genera...

Page 139: ...installation guidelines 1 processor Figure 10 9 DDR4 memory installation guidelines 2 processors FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 131 ...

Page 140: ...ion Maximum Drives Drive Management Mode Front drivesa 2b l Slots 0 and 2 support SAS SATA NVMe drives l M 2 FRUs can be installed only in slots 0 to 3 1 x RAID controller card l a Only 2 5 inch drives fit into the front slots l b Mixed configuration of M 2 modules and SAS SATA NVMe drives is supported l Contact your local Huawei sales representative or use the Intelligent Computing Compatibility ...

Page 141: ...Blinking at 4 Hz Off Data is being read or written normally or data on the primary drive is being rebuilt Steady on Blinking at 1 Hz The drive is being located Blinking at 1 Hz Blinking at 1 Hz The data on the secondary drive is being rebuilt Off Steady on A member drive in the RAID array is removed Steady on Steady on The drive is faulty FusionServer Pro CH121 V5 Compute Node Maintenance and Serv...

Page 142: ...e M 2 FRU is inactive NVMe SSD Indicators Figure 10 14 NVMe SSD indicators l If the VMD function is enabled and the latest Intel RSTe driver is installed NVMe SSDs support surprise hot swap Table 10 7 NVMe SSD indicators VMD enabled Green Indicator Yellow Indicator Description Off Off The NVMe SSD cannot be detected Steady green Off The NVMe SSD is working properly Blinking green at 2 Hz Off Data ...

Page 143: ...vable Steady green Off Steady yellow The NVMe SSD is faulty 10 4 4 RAID Controller Card The RAID controller card supports RAID configuration RAID level migration and drive roaming l Contact your local Huawei sales representative or use the Intelligent Computing Compatibility Checker to determine the components to be used Table 10 9 RAID levels supported by the CH121 V5 RAID Level Reliability Read ...

Page 144: ... LOM port the switch module connected to the LOM port must also have flow control enabled Table 10 10 I O modules supported by the LOM I O Module I O Module Slot LOM Remarks CX916 2X 3X 1E 4E The LOM cannot communicate with the I O modules in slots 1E and 4E CX920 2X 3X 1E 4E The LOM cannot communicate with the I O modules in slots 1E and 4E 10 6 I O Expansion 10 6 1 PCIe Cards l The server suppor...

Page 145: ... 18 0 0 00 LOM CPU 1 PCIe 3 0 x8 x8 Port1C 17 02 00 1c 00 0 0 Mezz1 CPU 1 PCIe 3 0 x16 or two x8 x16 or two x8 Port2A or Port2 A Port 2C 3a 00 0 3a 02 0 3b 00 0 3c 00 0 Mezz2 CPU 2 PCIe 3 0 x16 or two x8 x16 or two x8 Port2A or Port 2A Port2 C ae 00 0 ae 02 0 af 00 0 b0 0 0 0 PCIe Riser1 PCIe riser card 1 CPU 1 PCIe 3 0 x16 x16 Port3A 5d 00 0 5e 00 0 HHHL NVMe SSD0 disk 0 CPU 2 PCIe 3 0 x4 x4 Port...

Page 146: ...e processor l Device B D F indicates the B D F displayed on the OS of an onboard or extended PCIe device 10 7 Boards 10 7 1 Mainboard Figure 10 16 CH121 V5 mainboard 1 TPM connector 2 PCIe riser module connector PCIe 3 0 x16 3 PCIe riser module connector x8 4 RAID controller card connectors 5 Mezzanine card 2 connector 6 Mezzanine card 1 connector 7 BIOS battery 8 Positioning sleeve 9 Backplane si...

Page 147: ... connector 16 Drive cage 17 PCH 18 HLY indicator 19 UID button indicator 20 Power button indicator 21 USB 3 0 ports FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 10 Hardware Description Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 139 ...

Page 148: ...ontroller and six memory channels l Built in PCIe controller supporting PCIe 3 0 and 48 lanes per processor l Two UPI buses between processors providing up to 10 4GT s transmission per channel l Up to 28 cores 2 7 GHz l Max 3 8 GHz four cores l Min 1 375 MB L3 cache per core l Max 205 W TDP NOTE The preceding information is for reference only Use the Intelligent Computing Compatibility Checker to ...

Page 149: ...lity Checker to obtain information about the specific RAID controller cards supported The RAID controller card supports RAID configuration RAID level migration and drive roaming The RAID controller card does not occupy a standard PCIe slot For details about the RAID controller card see Huawei V5 Server RAID Controller Card User Guide NOTE If the BIOS is in legacy mode 4Kn drives 4 KB sectors in ph...

Page 150: ...The preceding information is for reference only Use the Intelligent Computing Compatibility Checker to obtain specific information Ports Supports a variety of ports l Ports on the front panel Two USB 3 0 ports l Built in port One USB 3 0 port NOTE The maximum dimensions H x W x D of the built in USB device are 33 9 mm x 14 5 mm x 7 12 mm 1 33 in x 0 57 in x 0 28 in Video card An SM750 video chip w...

Page 151: ...50 m 10006 44 ft When the server configuration complies with ASHRAE Class A2 standards and the altitude is above 900 m 2952 76 ft the highest operating temperature decreases by 1 C 1 8 F for every increase of 300 m 984 24 ft When the server configuration complies with ASHRAE Class A3 standards and the altitude is above 900 m 2952 76 ft the highest operating temperature decreases by 1 C 1 8 F for e...

Page 152: ...pecifications Table 11 3 Physical specifications Category Description Dimensions H x W x D 60 46 mm x 210 mm x 537 2 mm 2 38 in x 8 27 in x 21 15 in Weight in full configuration l Net weight 6 5 kg 14 33 lb l Packing materials 2 3 kg 5 07 lb Power consumption The power consumption parameters vary with the configuration Use the Huawei Server Power Calculator to obtain specific information FusionSer...

Page 153: ...narios Query the IP address of the iBMC management network port The following describes how to query the iBMC IP address on the BIOS You can use any of the following methods to query the IP address of the iBMC management network port l BIOS l iBMC WebUI For details see FusionServer Pro E9000 Server iBMC User Guide l iBMC CLI Run the ipmcget d ipinfo command For details see FusionServer Pro E9000 S...

Page 154: ...lly logged out of the WebUI l The system locks a user account if the user enters incorrect passwords for consecutive five times The user account is automatically unlocked five minutes later The system administrator can also unlock a user account using the command line l For security purposes change the initial password after the first login and change your password periodically Procedure Step 1 Ch...

Page 155: ...Windows Server 2012 64 bit Internet Explorer 10 0 11 0 JRE 1 7 U45 JRE 1 8 U45 JRE 1 8 U144 Mozilla Firefox 39 0 to 54 0 Google Chrome 21 0 to 44 0 RHEL 6 0 64 bit Mozilla Firefox 39 0 to 54 0 JRE 1 7 U45 JRE 1 8 U45 JRE 1 8 U144 MAC X v10 7 Safari 8 0 JRE 1 7 U45 JRE 1 8 U45 JRE 1 8 U144 Mozilla Firefox 39 0 to 54 0 Step 2 Connect the PC to the iBMC management network port using a network cable S...

Page 156: ...duce the Java security level This operation however poses security risks Exercise caution when performing this operation Figure 12 1 iBMC UI for login Step 5 Log in to the iBMC WebUI l Log in as a local user a Select the language to be used b Enter the user name and password NOTE n The system provides a default user of the administrator group The default user name is Administrator and the default ...

Page 157: ...es only the access function for LDAP users a Select the language to be used b Enter the LDAP user name and password NOTE n If an LDAP user name is entered in User Name Domain can be Automatic matching or a specified domain n If LDAP user name Domain name is entered Domain must be Automatic matching n In versions earlier than iBMC V294 the maximum password length for an LDAP user is 20 characters I...

Page 158: ... WebUI of the standby MM910 displayed as This is the standby MM does not display the component installation status After logging in to the WebUI of the standby MM910 you can view monitoring information about the active MM910 set the DHCP and static IP addresses and set and query the thresholds and hysteresis of the sensors of the standby MM910 To perform other operations log in to the WebUI of the...

Page 159: ...12 CX913 as the management port of the management module MGMT port on the MM910 panel For the MM910 U54 2 25 or earlier the management port is provided by the switch module by default For the MM910 U54 2 26 or later the management port is provided by the MM910 by default For details about how to query the MM910 version see MM910 Management Module V100R001 User Guide You can run the outportmode com...

Page 160: ...he STACK port is used to stack the MGMT port in another chassis connect the local PC to an idle STACK port on an active MM910 in a chassis cascaded with the E9000 chassis If the MGMT port on the active MM910 is already connected with a network cable do not remove the network cable from the MGMT port Otherwise an active standby switchover of the MM910s will be triggered which may cause network inte...

Page 161: ... Step 5 Choose Connections and click LAN Settings The Local Area Network LAN Settings dialog box is displayed Step 6 In the Proxy server group box deselect Use a proxy server for your LAN Step 7 Click OK The Local Area Network LAN Settings dialog box is closed Step 8 Click OK The Internet Options dialog box is closed Step 9 Restart Internet Explorer enter https MM910 floating IP address in the add...

Page 162: ... Sites 4 In the Trusted sites dialog box click Add to add the IP address to the trusted site list 5 Click Close 6 Refresh the web page of the added site 7 In the status bar of Internet Explorer check that the added site is in the Trusted sites zone Step 11 Click Continue to this website not recommended The HMM Web login page is displayed Step 12 Set the login parameters NOTE The login page varies ...

Page 163: ...ect MM for Collected from l Select One click full collection for Collected content Step 3 Click Start Collection Figure 12 7 Collecting Information Step 4 In the displayed dialog box click Save and save the log file to the local PC Step 5 Decompress the one_touch_info_all tar gz file The log information is in the bladeN folder NOTE N indicates the compute node slot number which ranges from 1 to 32...

Page 164: ...og in with SMM account and password user name Step 3 Enter the user name and password NOTE The default user name of compute nodes is Administrator and the default password is Admin 9000 The CLI for selecting a slot is displayed Log in Success please input the SOL Blade1 Blade16 1 16 Blade1A Blade16A 17 32 Swi1 Swi4 33 36 and COM n press Ctrl R to return Blade1 Blade16 1 16 Blade1A Blade16A 17 32 S...

Page 165: ... node or switch module using SOL The local KVM of the MM910 has the following limitations l The CLI does not provide the text editor l You can access the system serial port of a compute node over SOL of the KVM NOTE If you have accessed the CLI of the primary MM backup MM or SOL screen shown in Figure 12 8 and the login has not timed out you do not need to enter the user name and password again wh...

Page 166: ...tem of a Compute Node Scenarios Log in to the operating system of a compute node using the local KVM Procedure Step 1 Connect the KVM to the MM910 1 Connect the mouse and keyboard to the two USB ports on the MM910 2 Use a VGA cable to connect the KVM monitor to the VGA port on the MM910 The main screen of the KVM is displayed FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 ...

Page 167: ... a compute node using the local KVM Procedure Step 1 Connect the KVM to the MM910 1 Connect the mouse and keyboard to the two USB ports on the MM910 2 Use a VGA cable to connect the KVM monitor to the VGA port on the MM910 The main screen of the KVM is displayed FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operations Issue 06 2019 08 10 Copyright Huawei Technologi...

Page 168: ...DVD drive is mounted to the compute node l indicates that the DVD drive is open l indicates that data is being read End 12 6 4 Logging In to a Compute Node or a Switch Module over SOL Scenarios Use the local KVM to log in to a compute node or switch module over SOL FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operations Issue 06 2019 08 10 Copyright Huawei Technol...

Page 169: ... SOL 1 Press CTRL S on the KVM home page The following information is displayed Welcome to SMM SOL Server Please log in with SMM account and password user name NOTE You can press Ctrl Shift Alt to return to the previous screen or press CTRL R to return to the KVM main screen 2 Type the MM910 user name and password and press Enter The following information is displayed FusionServer Pro CH121 V5 Com...

Page 170: ... 34 1 BMCcom 2 fabriccom 3 baseccom 4 FCcom COM 1 4 4 Enter the number of the serial port to be connected for example type 1 and press Enter The serial port screen is displayed 5 Enter the user name and password for connecting to the serial port You can query information and configure data on this interface End 12 7 Logging In to the Desktop of a Server 12 7 1 Using the Remote Virtual Console 12 7...

Page 171: ...ions performed by the other user l HTML5 Integrated Remote Console Private allows only one local user or VNC user to access and perform operations on the server through the iBMC l HTML5 Integrated Remote Console Shared allows two local users or up to five VNC users to simultaneously access and perform operations on the server through the iBMC Each user can view the operations performed by the othe...

Page 172: ...Figure 12 13 Remote console Java FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 164 ...

Page 173: ...I For details see 12 3 Logging In to the MM910 WebUI Step 2 Click KVM via MM The Security Warning dialog box is displayed Step 3 Click Yes The Warning Security dialog box is displayed Step 4 Click Yes The remote KVM page is displayed FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 165 ...

Page 174: ...dependent Remote Console 12 7 2 1 Windows The following Windows versions are supported l Windows 7 32 bit or 64 bit l Windows 8 32 bit or 64 bit l Windows 10 32 bit or 64 bit l Windows Server 2008 R2 32 bit or 64 bit l Windows Server 2012 64 bit Procedure Step 1 Configure an IP address for the client PC to enable communication between the client and the iBMC The IP address configured and the iBMC ...

Page 175: ... the port number is the default port number 443 the port number can be left blank Step 4 Select a login mode l Shared Mode allows two users to access and manage a server at the same time Each user can view the operations performed by the other user l Private Mode allows only one user to access and manage a server at a time Step 5 Click Connect A security warning is displayed Figure 12 17 Security ...

Page 176: ...igure an IP address for the client PC to enable communication between the client and the iBMC The IP address configured and the iBMC management network port IP address must be on the same network segment Step 2 Open the console and set the folder where the IRC is stored as the working folder Step 3 Run the chmod 777 KVM sh command to set the permission for the Independent Remote Console Step 4 Ope...

Page 177: ...ess Port number l When the port number is the default port number 443 the port number can be left blank Step 6 Select a login mode l Shared Mode allows two users to access and manage a server at the same time Each user can view the operations performed by the other user l Private Mode allows only one user to access and manage a server at a time Step 7 Click Connect A security warning is displayed ...

Page 178: ...P address for the client PC to enable communication between the client and the iBMC The IP address configured and the iBMC management network port IP address must be on the same network segment Step 2 Open the console and set the folder where the IRC is stored as the working folder Step 3 Run the chmod 777 KVM sh command to set the permission for the Independent Remote Console Step 4 Open the Inde...

Page 179: ...ess Port number l When the port number is the default port number 443 the port number can be left blank Step 6 Select a login mode l Shared Mode allows two users to access and manage a server at the same time Each user can view the operations performed by the other user l Private Mode allows only one user to access and manage a server at a time Step 7 Click Connect A security warning is displayed ...

Page 180: ...TTY over a Network Port Scenarios Use PuTTY to access a server over a local area network LAN NOTE l You can obtain the PuTTY software from the chiark home page l You are advised to use PuTTY of the latest version PuTTY of an earlier version may cause login failures Procedure Step 1 Set an IP address and subnet mask or add route information for the PC to communicate with the server FusionServer Pro...

Page 181: ...fault value 22 l Connection type Retain the default value SSH Step 5 In the navigation tree choose Session Step 6 Select SSH under Connection type Figure 12 25 PuTTY Configuration Step 7 Set Close window on exit to Only on clean exit NOTE Set Saved Sessions and click Save You can double click the saved record in Saved Sessions to log in to the server next time Step 8 Click Open FusionServer Pro CH...

Page 182: ...does not support this setting l Use Intelligent Computing Compatibility Checker to obtain information about the operating systems supported by the VMD function l After the VMD function is enabled and the latest VMD driver is installed the server supports Surprise hot plug of NVMe drives 12 9 1 Enabling VMD Procedure Step 1 Access the BIOS interface Step 2 Choose Advanced Step 3 Select Socket Confi...

Page 183: ...e data on the storage device The following describes how to clear the data on one HDD SSD as an example This operation is for reference only You can also use other methods to clear data from storage media The cleared data cannot be restored Exercise caution when performing this operation Procedure NOTE Before performing this operation check that l The storage device is not in a RAID array with red...

Page 184: ... drive l Do not directly clear system drive data Before clearing system drive data clear data from other storage media Figure 12 27 Querying drive information Step 5 Write all 0s to the drive to be cleared Command badblocks swft 0 Drive letter Example badblocks swft 0 dev sdb Figure 12 28 Clearing data example FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operation...

Page 185: ...and fails to execute contact technical support Step 6 Remove the drive NOTE After the data is cleared do not restart or reinstall the server Otherwise the system will reload data to the drives during the startup of the server End FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide 12 Common Operations Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 177 ...

Page 186: ...n more about servers and communicate with experts at l HUAWEI Server Information Service Platform for specific server product documentation l iKnow for Q A about products l Huawei Enterprise Support Community Server for learning and discussion News For notices about product life cycles warnings and updates visit Support Bulletins Product Bulletins Cases To learn server applications visit Support K...

Page 187: ...n Server user guide which provides information about the structure specifications and installation of the server Visit Intelligent Servers select a product model and view the Documentation tab page Intelligent Computing Compatibility Checker A tool used to query the OSs parts and peripherals compatible with a server Intelligent Computing Compatibility Checker Maintenance Information Inquiry System...

Page 188: ... maintenance Tool Server Model and Software Version Description FusionServer Tools uMate For details see FusionServer Tools 2 0 uMate User Guide uMate performs routine maintenance tasks for servers The tasks include preventive maintenance batch firmware upgrades and batch power control Download link FusionServer Tools FusionServer Tools InfoCollect For details see FusionServer Tools 2 0 InfoCollec...

Page 189: ...able B 1 SN description Callout No Description 1 SN ID two characters which is 21 2 Material identification code 8 characters that is the processing code 3 Vendor code two characters 10 indicates Huawei and other values indicate outsourcing vendors FusionServer Pro CH121 V5 Compute Node Maintenance and Service Guide B Appendix Issue 06 2019 08 10 Copyright Huawei Technologies Co Ltd 181 ...

Page 190: ...me B 2 RAS Features Table B 2 describes the reliability availability and serviceability RAS features supported by the server You can configure these features for better RAS For details about how to configure RAS features see Huawei Server Purley Platform BIOS Parameter Reference Table B 2 RAS features Module Feature Description CPU Corrected Machine Check Interrupt Corrects error triggered interru...

Page 191: ...CRC protection for UPI packets to improve system reliability System Core Disable for Fault Resilient Boot FRB Isolates a faulty CPU core during startup to improve system reliability and availability Corrupt Data Containment Mode Identifies the memory storage unit that contains corrupted data to minimize the impact on the running programs and improve system reliability Socket disable for Fault Resi...

Page 192: ...to facilitate fault analysis and locating and improve system serviceability BIOS Abstraction Layer for Error Handling The BIOS processes errors and reports the error information to the OS and the server in compliance with specifications to improve system serviceability BIOS based Predictive Failure Analysis PFA The BIOS provides physical unit information for DIMM errors and the OS traces and predi...

Search results

Summary of Contents for FusionServer Pro CH121 V5 V100R005

Reviews:

Related manuals for FusionServer Pro CH121 V5 V100R005

Brands by name

Popular brands

Huawei FusionServer Pro CH121 V5 V100R005, Maintenance And Service Manual

Search results

Summary of Contents for FusionServer Pro CH121 V5 V100R005

Reviews:

Related manuals for FusionServer Pro CH121 V5 V100R005

Brands by name

Popular brands