manualshive.com logo in svg

Huawei FusionServer Pro XH321 V5, Maintenance And Service Manual

Share
Download
Huawei FusionServer Pro XH321 V5 Maintenance And Service Manual Download Page 7

3.2 Environmental Specifications............................................................................................................................................ 27
3.3 Physical Specifications.........................................................................................................................................................28

4 Software and Hardware Compatibility............................................................................ 29
5 Safety Instructions................................................................................................................ 30

5.1 Safety Instructions................................................................................................................................................................ 30
5.2 Maintenance and Warranty.............................................................................................................................................. 33

6 ESD............................................................................................................................................ 34

6.1 ESD Prevention...................................................................................................................................................................... 34
6.2 Grounding Methods for ESD Prevention.......................................................................................................................35

7 Internal Cabling.....................................................................................................................36

7.1 TPM Cabling........................................................................................................................................................................... 36
7.2 Screw-in RAID Controller Card Cabling......................................................................................................................... 37
7.3 Cabling of the SATA Controller Integrated in the Southbridge............................................................................ 38
7.4 PCIe Card Cabling................................................................................................................................................................. 39

8 Parts Replacement................................................................................................................ 40

8.1 Spare Parts Description.......................................................................................................................................................40
8.2 Obtaining Tools..................................................................................................................................................................... 41
8.3 Basic Operations....................................................................................................................................................................42
8.3.1 Power-On Procedure........................................................................................................................................................ 42
8.3.2 Power-Off Procedure........................................................................................................................................................43
8.3.3 Removing an XH321 V5.................................................................................................................................................. 44
8.3.4 Installing an XH321 V5....................................................................................................................................................47
8.4 Field Replaceable Units.......................................................................................................................................................50
8.4.1 Removing an M.2 SSD..................................................................................................................................................... 51
8.4.2 Installing an M.2 SSD....................................................................................................................................................... 52
8.4.3 Removing the Screw-in RAID Controller Card......................................................................................................... 54
8.4.4 Installing the Screw-in RAID Controller Card.......................................................................................................... 56
8.4.5 Removing the Supercapacitor (Server with a Screw-in RAID Controller Card)........................................... 58
8.4.6 Installing the Supercapacitor.........................................................................................................................................59
8.4.7 Removing the Avago SAS3004iMR PCIe RAID Controller Card......................................................................... 61
8.4.8 Installing the Avago SAS3004iMR PCIe RAID Control Card................................................................................61
8.4.9 Removing an M.2 FRU on the Avago SAS3004iMR PCIe RAID Controller Card.......................................... 62
8.4.10 Installing an M.2 FRU on the Avago SAS3004iMR PCIe RAID Controller Card......................................... 64
8.4.11 Removing a Processor................................................................................................................................................... 66
8.4.12 Installing a Processor..................................................................................................................................................... 69
8.4.13 Removing a Memory Module..................................................................................................................................... 75
8.4.14 Installing a Memory Module.......................................................................................................................................76
8.4.15 Removing the TPM......................................................................................................................................................... 79
8.4.16 Installing the TPM...........................................................................................................................................................81
8.4.17 Removing the Battery....................................................................................................................................................84

FusionServer Pro XH321 V5 Server Node

Maintenance and Service Guide

Contents

Issue 17 (2021-03-09)

Copyright © Huawei Technologies Co., Ltd.

vi

Summary of Contents for FusionServer Pro XH321 V5

Page 1: ...FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide Issue 17 Date 2021 03 09 HUAWEI TECHNOLOGIES CO LTD ...

Page 2: ...cribed in this document may not be within the purchase scope or the usage scope Unless otherwise specified in the contract all statements information and recommendations in this document are provided AS IS without warranties guarantees or representations of any kind either express or implied The information in this document is subject to change without notice Every effort has been made in the prep...

Page 3: ...with a high level of risk which if not avoided will result in death or serious injury Indicates a hazard with a medium level of risk which if not avoided could result in death or serious injury Indicates a hazard with a low level of risk which if not avoided could result in minor or moderate injury Indicates a potentially hazardous situation which if not avoided could result in equipment damage da...

Page 4: ... 20 This issue is the tenth official release 09 2020 03 18 This issue is the ninth official release 08 2020 03 05 This issue is the eighth official release 07 2019 12 20 This issue is the seventh official release 06 2019 08 10 Added the support for the Cascade Lake processors 05 2019 01 04 Added the description about X722 LOMs and PCIe slots Updated the technical specifications Added the descripti...

Page 5: ...escription 01 2017 12 29 This issue is the first official release FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide About This Document Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd iv ...

Page 6: ... 3 Memory Compatibility 14 2 3 1 4 Memory Installation Guidelines 16 2 3 1 5 Memory Installation Positions 17 2 3 1 6 Memory Protection Technologies 18 2 4 Storage 18 2 4 1 Drive Configurations 19 2 4 2 Drive Numbering 19 2 4 3 RAID Controller Card 19 2 5 I O Expansion 20 2 5 1 PCIe Cards 20 2 5 2 PCIe Slots 20 2 5 3 PCIe Slot Description 20 2 6 Boards 21 2 6 1 Mainboard 21 3 Product Specification...

Page 7: ...M 2 SSD 51 8 4 2 Installing an M 2 SSD 52 8 4 3 Removing the Screw in RAID Controller Card 54 8 4 4 Installing the Screw in RAID Controller Card 56 8 4 5 Removing the Supercapacitor Server with a Screw in RAID Controller Card 58 8 4 6 Installing the Supercapacitor 59 8 4 7 Removing the Avago SAS3004iMR PCIe RAID Controller Card 61 8 4 8 Installing the Avago SAS3004iMR PCIe RAID Control Card 61 8 4...

Page 8: ...5 10 3 1 1 1 Versions Earlier than V600 115 10 3 1 1 2 V600 and Later Versions 118 10 3 2 Logging In to the Server Using the Independent Remote Console 121 10 3 2 1 Versions Earlier than V600 121 10 3 2 1 1 Windows 121 10 3 2 1 2 Ubuntu 123 10 3 2 1 3 Mac 126 10 3 2 1 4 Red Hat 128 10 3 2 2 V600 and Later Versions 130 10 3 2 2 1 Windows 130 10 3 2 2 2 Ubuntu 132 10 3 2 2 3 Mac 135 10 3 2 2 4 Red H...

Page 9: ... 2 2 Operating Temperature Limitations Optical Modules Configured 159 A 2 3 Operating Temperature Limitations Avago SAS3004iMR RAID Controller Card M 2 FRU Configured 161 A 2 4 Operating Temperature Limitations Different Models of Processors 163 A 3 RAS Features 165 B Glossary 168 B 1 A E 168 B 2 F J 169 B 3 K O 169 B 4 P T 169 B 5 U Z 170 C Acronyms and Abbreviations 171 C 1 A E 171 C 2 F J 172 C...

Page 10: ...rmance and high storage density in limited space through innovative design It is easy to manage and maintain The XH321 V5 supports six 2 5 inch SAS SATA drives or NVMe drives up to 16 DDR4 memory modules and two PCIe cards The XH321 V5 is ideal for data center cloud computing Big Data and Internet applications FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 1 Overview Issue 17 ...

Page 11: ...1 XH321 V5 1 2 Physical Structure Figure 1 2 XH321 V5 physical structure FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 1 Overview Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 2 ...

Page 12: ...ble processors The server supports up to 16 memory modules The CPUs processors interconnect with each other through two UPI links at a speed of up to 10 4 GT s The PCIe cards connected to the processors provide service ports The RAID controller card connects to the mainboard using a SAS cable and connects to the system backplane through a connector The BMC chipset interconnects with the PCH throug...

Page 13: ...dia component to provide device management functions such as power control slot ID acquisition power supply detection and KVM over IP FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 1 Overview Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 4 ...

Page 14: ...2 1 Front Panel 2 1 1 Appearance Figure 2 1 Front view 1 PCIe card RAID controller card not configured 2 PCIe cards 3 Slide out label plate with an SN label FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 5 ...

Page 15: ...indicator 4 Health status indicator 5 UID button indicator 6 Data transmission status indicator for a GE electrical port 7 Connection status indicator for a GE electrical port 8 Data transmission status indicator for the management network port 9 Connection status indicator for the management network port FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Is...

Page 16: ... button to gracefully shut down the OS NOTE For different OSs you may need to shut down the OS as prompted When the device is powered on holding down this button for 6 seconds will forcibly power off the device When the power indicator is steady yellow you can press this button to power on the device Health status indicator Off The device is powered off or is faulty Blinking red at 1 Hz A major al...

Page 17: ...ady green The data transmission rate is 10 Gbit s Connection status indicator Data transmission status indicator for a 10GE optical port Off The network port is not connected Blinking green Data is being transmitted Steady green The network port is properly connected Data transmission status indicator for a GE electrical port Off No data is being transmitted on the port Steady yellow The network p...

Page 18: ...reen The network port is properly connected 2 1 3 Ports Port Positions Ports on the front panel Figure 2 3 Ports on the front panel 1 Universal connector port 2 LOM port 1 10GE optical port 3 LOM port 2 10GE optical port 4 LOM port 3 GE electrical port 5 LOM port 4 GE electrical port 6 Management network port Ports on the multi port cable FusionServer Pro XH321 V5 Server Node Maintenance and Servi...

Page 19: ... to connect a multi port cable which provides one DB 15 VGA port three USB 2 0 ports and one RJ45 serial port Management network port RJ45 1 Used for server management NOTE The management network port is a GE port that supports 100 Mbit s and 1000 Mbit s auto negotiation FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Issue 17 2021 03 09 Copyright Huawei ...

Page 20: ... or even damage the NIC The electrical LOM ports do not support SR IOV Forcibly powering off a server will cause intermittent NC SI disconnection and disable the Wake on LAN WOL function of the LOM ports To restore the NC SI connection refresh the iBMC WebUI 2 1 4 Installation Positions An X6000 can house a maximum of four XH321 V5 nodes in a 2U chassis Figure 2 5 Installation positions 2 2 Proces...

Page 21: ... must be used in a server Contact your local Huawei sales representative or use the Computing Product Compatibility Checker to determine the components to be used Figure 2 6 Processor positions 2 3 Memory 2 3 1 Memory 2 3 1 1 Memory Identifier You can determine the memory module properties based on the label attached to the memory module FusionServer Pro XH321 V5 Server Node Maintenance and Servic...

Page 22: ... 4 Type of the memory interface PC3 DDR3 PC4 DDR4 5 Maximum memory speed 2133 MT S 2400 MT S 2666 MT S 2933 MT S 6 Column Access Strobe CAS latency P 15 T 17 7 DIMM type R RDIMM L LRDIMM 2 3 1 2 Memory Subsystem Architecture The XH321 V5 provides 16 memory slots Each processor integrates six memory channels FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description ...

Page 23: ...emory Compatibility Observe the following rules when configuring DDR4 DIMMs NOTICE A server must use the same model of DDR4 DIMMs and all the DIMMs operate at the same speed which is the smallest value of Memory speed supported by a processor Maximum operating speed of a DIMM The DDR4 DIMMs of different types RDIMM and LRDIMM and specifications capacity bit width rank and height cannot be used tog...

Page 24: ...depending on the CPU type memory type rank quantity and operating voltage NOTE Each memory channel supports a maximum of 8 ranks The number of DIMMs supported by each channel varies depending on the number of ranks supported by each channel Number of DIMMs supported by each channel Number of ranks supported by each memory channel Number of ranks supported by each DIMM A memory channel supports mor...

Page 25: ...essors are installed Do not install LRDIMMs and RDIMMs in the same server Install filler memory modules in vacant slots Observe the following when configuring DDR4 memory modules in specific operating mode Memory sparing mode Comply with the general installation guidelines Each memory channel must have a valid online spare configuration The channels can have different online spare configurations E...

Page 26: ...een the installed processors and load the channels similarly whenever possible NOTICE At least one DDR4 DIMM must be installed in the memory slots corresponding to CPU 1 Figure 2 8 Memory slots Figure 2 9 DDR4 memory installation guidelines 1 processor FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd...

Page 27: ... SDDC 1 Rank sparing mode Static virtual lockstep Faulty DIMM isolation Memory thermal throttling Memory address parity protection Memory demand patrol scrubbing Device tagging Data scrambling Adaptive double device data correction ADDDC ADDDC 1 2 4 Storage FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Issue 17 2021 03 09 Copyright Huawei Technologies C...

Page 28: ... your local Huawei sales representative or use the Computing Product Compatibility Checker to determine the components to be used 2 4 2 Drive Numbering Figure 2 11 Drive numbering 2 4 3 RAID Controller Card The RAID controller card supports RAID configuration RAID level migration and drive roaming FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 2 Hardware Description Issue 17 2...

Page 29: ...oE device may cause link communication failure or even damage the NIC Contact your local Huawei sales representative or use the Computing Product Compatibility Checker to determine the components to be used 2 5 2 PCIe Slots Figure 2 12 PCIe Slot 2 5 3 PCIe Slot Description NOTE The PCIe slots mapping to a vacant CPU socket are unavailable Table 2 7 PCIe slot description PCIe Slot CPU PCIe Standa r...

Page 30: ... with PCIe x8 PCIe x4 and PCIe x1 cards The PCIe cards are not forward compatible That is the PCIe slot width cannot be smaller than the PCIe card link width 2 6 Boards 2 6 1 Mainboard Figure 2 13 XH321 V5 mainboard 1 PCIe slot 1 2 Southbridge 3 RAID controller card connector 4 Slimline connector 5 CPU 1 6 DIMM050 connector 7 DIMM040 connector 8 DIMM030 connector 9 DIMM031 connector 10 CPU 2 11 DI...

Page 31: ...32 OPA signal connector 33 M 2 connector port 1 34 TPM signal connector 35 PCIe slot 2 36 NC SI connector 37 SoftRAID key connectora 38 Battery 39 iBMC management network port 40 LOM GE port 4 41 LOM GE port 3 42 UID button indicator 43 Power button indicator 44 2 x 10GE optical ports 45 Optical port indicator 46 Universal connector port a The port is reserved FusionServer Pro XH321 V5 Server Node...

Page 32: ...y controller and six memory channels Built in PCIe controller supporting PCIe 3 0 and 48 lanes per processor Two UPI buses between processors providing up to 10 4 GT s transmission per channel Up to 28 cores per processor Max 3 8 GHz Min 1 375 MB L3 cache per core Max 205 W TDP NOTE The preceding information is for reference only Use the Computing Product Compatibility Checker to obtain specific i...

Page 33: ...fferent types RDIMM and LRDIMM and specifications capacity bit width rank and height cannot be used together NOTE The preceding information is for reference only Use the Computing Product Compatibility Checker to obtain specific information FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 3 Product Specifications Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 24 ...

Page 34: ...atch For details see https kb vmware com s article 1033696 Configure syslog For details see https kb vmware com s article 2003322 The M 2 SSD cannot be used to store data due to poor endurance In write intensive applications the M 2 SSD will wear out in a short time Use enterprise level high endurance HE SSDs or HDDs for data storage The M 2 SSD is not recommended for write intensive service softw...

Page 35: ...ietary PCIe SSD cards to bolster I O performance for applications such as searching caching and download services NOTE The preceding information is for reference only Use the Computing Product Compatibility Checker to obtain specific information Ports Supports a variety of ports Ports on the front panel One universal connector port One system management port Two GE electrical ports Two 10GE optica...

Page 36: ...TE If a single fan is faulty The highest operating temperature supported is 30 C 86 F If the power consumption of the processor is 125 W or higher the system performance may be affected or a temperature alarm may be generated When the server is configured with 6248R or 6258R processors the highest operating temperature is 30 C 86 F The highest operating temperature varies depending on the server c...

Page 37: ...ronment meets the requirements of ISO 14664 1 Class 8 There is no explosive conductive magnetic or corrosive dust in the equipment room NOTE It is recommended that the particulate pollution in the equipment room be monitored by a professional agency 3 3 Physical Specifications Table 3 3 Physical specifications Category Description Dimensions H x W x D 40 5 mm x 177 9 mm x 545 5 mm 1 59 in x 7 00 i...

Page 38: ...software and hardware may cause performance inconsistency between the application layer and test software layer If the customer has requirements on the performance of specific application software contact Huawei sales personnel to apply for POC tests in the pre sales phase to determine detailed software and hardware configurations If the customer has requirements on hardware performance consistenc...

Page 39: ...forklift Take protective measures if a Class A product is used in residential areas as it is likely to cause radio interference Personal Safety Only personnel certified or authorized by Huawei are allowed to install the hardware Stop any operation that may cause personal injury or equipment damage report the problem to a project supervisor immediately and take protective measures Do not operate th...

Page 40: ... 5 2 Conductive objects to be removed Figure 5 3 shows how to wear an ESD wrist strap a Secure the ESD wrist strap around your wrist b Fasten the strap buckle and ensure that the ESD wrist strap is in contact with your skin c Insert the ground terminal attached to the ESD wrist strap into the jack on the grounded rack or chassis FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 5...

Page 41: ...andles of the installed modules such as the PSUs fan modules drives and the mainboard Handle the equipment with care Exercise caution when using tools that could cause personal injury If the device is configured with active and standby PSUs connect power cables of active and standby PSUs to different power distribution units PDUs to ensure reliable system operating Properly ground a device before ...

Page 42: ... a number of organizations Table 5 1 Maximum weight one person is permitted to carry Organization Weight kg lb European Committee for Standardization CEN 25 55 13 International Organization for Standardization ISO 25 55 13 National Institute for Occupational Safety and Health NIOSH 23 50 72 Health and Safety Executive HSE 25 55 13 General Administration of Quality Supervision Inspection and Quaran...

Page 43: ...d point of the workbench Monitor the temperature and humidity in the equipment room The heating system may reduce the humidity and increases static electricity indoors Place the product in an ESD bag to avoid direct contact during transportation and storage Before transporting electrostatic sensitive components to a work area that is not affected by static electricity store them in their original ...

Page 44: ...least 1 megohm 10 For grounding purposes wear the wrist strap tightly against your skin Use a heel grounded toe grounded or shoe grounded ESD strap when working in a standing position When standing on a conductive floor or electrostatic dissipative floor mat tie a strap on your feet Use conductive maintenance tools Use a folding tool mat that dissipates static electricity and a portable field serv...

Page 45: ...ng 7 3 Cabling of the SATA Controller Integrated in the Southbridge 7 4 PCIe Card Cabling 7 1 TPM Cabling Figure 7 1 TPM cabling FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 7 Internal Cabling Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 36 ...

Page 46: ...Screw in RAID Controller Card Cabling Figure 7 2 Screw in RAID controller card cabling Table 7 2 Screw in RAID controller card cabling No Huawei BOM Code Description 1 04052070 Signal cable for connecting the screw in RAID controller card Port A to the Slimline connector 2 04052070 Signal cable for connecting the screw in RAID controller card Port B to the Slimline connector FusionServer Pro XH321...

Page 47: ...uthbridge Table 7 3 Cabling of the SATA Controller Integrated in the Southbridge No Huawei BOM Code Description 1 04052068 Signal cable for connecting the southbridge to the Slimline connector FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 7 Internal Cabling Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 38 ...

Page 48: ...n 1 04080606 Signal cable for connecting the PCIe card to the NC SI connector NOTE This cable is required only when the PCIe card is an SP380 NIC BOM code 02312JBD FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 7 Internal Cabling Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 39 ...

Page 49: ...d Stored in the warehouses near sites based on the contract service type Service Level Agreement SLA and service sites Non Regular Spare Part NRSP Mechanical parts accessories and cables Generally safety stock is not kept for NRSPs NRSPs are provided on demand however the lead time is not committed Stored in a country s warehouses based on the contract service type FusionServer Pro XH321 V5 Server...

Page 50: ...try s warehouses based on the contract service type Lower level parts vary with actual demands NSP SUB It is not a spare part but has RSPs or NRSPs at lower levels This part is not stored Lower level parts vary with actual demands 8 2 Obtaining Tools Ensure that the following tools are available ESD wrist strap or ESD gloves Protective gloves ESD bags Box cutter M3 Phillips screwdriver PH1 screwdr...

Page 51: ...powered on but the server node is not installed install the server node The server node will be automatically powered on after it is installed If the chassis is powered on but the server node is not running properly hold the power button the front panel for six seconds to power off the service node forcibly and then press the power button to power it on If the chassis is powered on and the server ...

Page 52: ...ered off forcibly wait for more than 10 seconds for the server node to power off completely Do not power on the server node again before it is completely powered off Forced power off may damage user programs or unsaved data Exercise caution when performing this operation The server node can be powered off in any of the following ways Power off the chassis The server node will be powered off along ...

Page 53: ...esktop of a Server b On the KVM screen click or on the toolbar c Choose Power Off or Forced Power Off A dialog box is displayed d Click Yes 8 3 3 Removing an XH321 V5 Procedure Step 1 Determine the position of the server node in the chassis Figure 8 2 Positions and slot numbers of the XH321 V5 server nodes Step 2 Power off the server node For details see 8 3 2 Power Off Procedure Step 3 Remove the...

Page 54: ... force when using the die casting pliers to prevent damage to the optical or network cables or optical modules 1 Use the die casting pliers to clamp the optical cable connector and remove it from the port Figure 8 4 Removing optical cables using die casting pliers 2 Use the die casting pliers to clamp the optical module unlock the security pin and remove the optical module FusionServer Pro XH321 V...

Page 55: ...the ejector release button on the server node See 1 in Figure 8 6 2 Fully open the ejector lever See 2 in Figure 8 6 3 Holding the ejector lever pull the server node out of the chassis horizontally See 3 in Figure 8 6 4 Close the ejector lever Figure 8 6 Removing the server node Step 6 Place the removed server node in an ESD bag FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8...

Page 56: ...s and slot numbers of the XH321 V5 server nodes NOTE Install the lower server nodes and then the upper server nodes Step 2 Remove filler modules NOTE Perform this operation only when filler modules are installed 1 Press the ejector release button on the filler module See 1 in Figure 8 8 2 Fully open the ejector lever See 2 in Figure 8 8 3 Holding the ejector lever pull the filler module from the c...

Page 57: ...vice damage or personal injury hold the bottom of the device with both hands when installing it 1 Press the ejector release button on the server node See 1 in Figure 8 9 2 Fully open the ejector lever See 2 in Figure 8 9 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 48 ...

Page 58: ...1 in Figure 8 10 4 Close the ejector lever See 2 in Figure 8 10 Figure 8 10 Installing a server node Step 4 Connect the multi port cable as required FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 49 ...

Page 59: ...cal module use die casting pliers to clamp the optical module and insert it into the optical port until you hear a click 2 Use die casting pliers to clamp the optical connector and insert it into the optical port of an optical module until you hear a click Step 6 Power on the server node For details see 8 3 1 Power On Procedure Step 7 Check indicator status For details see 2 1 2 Indicators and But...

Page 60: ...e Air Duct Step 5 Determine the position of the M 2 SSD Figure 8 12 Positions of M 2 SSDs Step 6 Remove the TPM For details see 8 4 15 Removing the TPM Step 7 Remove the M 2 SSD 1 Use an M2 5 Phillips screwdriver to loosen the screw that secures the M 2 SSD See 1 in Figure 8 13 2 Tilt the M 2 SSD 20 to 30 upwards and pull it out See 2 in Figure 8 13 FusionServer Pro XH321 V5 Server Node Maintenanc...

Page 61: ...ver node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Remove the air duct For details see 8 4 23 Removing the Air Duct FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 52 ...

Page 62: ...ts ESD bag Step 8 Install the M 2 SSD 1 Tilt the M 2 SSD 20 to 30 install it in the slot and press it firmly See 1 in Figure 8 15 2 Use an M2 5 Phillips screwdriver to tighten the screw to secure the M 2 SSD See 2 in Figure 8 15 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 53 ...

Page 63: ...n XH321 V5 Step 12 Power on the server node For details see 8 3 1 Power On Procedure End 8 4 3 Removing the Screw in RAID Controller Card Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03...

Page 64: ...4 5 Removing the Supercapacitor Server with a Screw in RAID Controller Card Step 7 Remove the cable from the RAID controller card For details see 7 Internal Cabling Step 8 Remove the RAID controller card 1 Loosen the screws that secure the RAID controller card See 1 in Figure 8 17 2 Remove the RAID controller card from the connector on the mainboard See 2 in Figure 8 17 FusionServer Pro XH321 V5 S...

Page 65: ...ep 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Take the component out of its ESD bag Step 5 Check whether a supercapacitor is configured If yes go to Step 6 If no go to Step 7 Step 6 Install the supercapacitor For details see 8 4 6 Installing the Supercapacitor FusionServer Pro XH321 V5 Server Node Maintenance and S...

Page 66: ...t the RAID controller card vertically into the connector on the mainboard See 1 in Figure 8 19 2 Tighten the screws to secure the RAID controller card See 2 in Figure 8 19 Figure 8 19 Installing a screw in RAID controller card FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 57 ...

Page 67: ...r details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Determine the position of the supercapacitor Step 5 Remove the air duct for the supercapacitor Step 6 Remove the supercapacitor holder 1 Loosen the screws on the server node panel See 1 in Figure 8 20 2 Remove the panel See 2 in...

Page 68: ...oving a supercapacitor Step 9 Place the removed component in an ESD bag NOTICE Place only one component in an ESD bag End 8 4 6 Installing the Supercapacitor Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 59 ...

Page 69: ...nnect the cable between the supercapacitor and the screw in RAID controller card Step 8 Install the supercapacitor and its holder 1 Install the supercapacitor holder into the node panel and tighten the screws to secure the supercapacitor holder See 1 in Figure 8 23 2 Install the node panel See 2 in Figure 8 23 3 Tighten the screws to secure the panel See 3 in Figure 8 23 Figure 8 23 Installing a s...

Page 70: ...ion of the Avago SAS3004iMR RAID controller card Step 5 Remove the M 2 FRU from the Avago SAS3004iMR RAID controller card For details see 8 4 9 Removing an M 2 FRU on the Avago SAS3004iMR PCIe RAID Controller Card Step 6 Remove the Avago SAS3004iMR RAID controller card For details see 8 4 19 Removing a PCIe Card Step 7 Place the removed component in an ESD bag NOTICE Place only one component in an...

Page 71: ...ystem installed you can remove it without powering off the server node If the drive to be removed has the operating system installed and is in a RAID array with redundancy and the removal does not cause a RAID array fault you can remove it without powering off the server node If the drive to be removed has the operating system installed but it is not in a RAID array or it is in a RAID array withou...

Page 72: ...8 25 Figure 8 25 Removing an M 2 FRU Step 3 Place the removed component in an ESD bag NOTICE Place only one component in an ESD bag Step 4 Install an M 2 FRU filler module to shield the EMC NOTE Perform this step only when an M 2 FRU is not installed immediately End FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technolo...

Page 73: ...U filler module NOTE Perform this step only when you configure an M 2 FRU filler module Step 3 Install an M 2 FRU on the Avago SAS3004iMR PCIe RAID controller card 1 Take a spare M 2 FRU out of its ESD bag 2 Open the handle of the M 2 FRU and push the M 2 FRU into the slot of the Avago SAS3004iMR RAID control card See 1 in Figure 8 27 3 Close the handle of the M 2 FRU See 2 in Figure 8 27 FusionSe...

Page 74: ...s being located or RAID is being rebuilt Steady yellow The M 2 FRU is faulty or not detected M 2 FRU activity indicator Off The M 2 FRU is not in position or is faulty Blinking green Data is being read written or synchronized Steady green The M 2 FRU is inactive Step 5 Restore the drive data NOTE Perform this operation only after you replace a drive FusionServer Pro XH321 V5 Server Node Maintenanc...

Page 75: ...on the processor socket and damage it Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Remove the air duct For details see 8 4 23 Removing the Air Duct Step 5 Determine the position of the processor Figure 8 29 Positions of processo...

Page 76: ...apply excessive force 5 Holding the other end of the carrier remove the carrier and processor in the arrow direction See 4 and 5 in Figure 8 30 Figure 8 30 Removing the heat sink and processor carrier Step 7 Remove the processor NOTE When removing a processor hold the processor and its carrier to prevent the processor from falling off To prevent damage to the pins and procesors do not touch the pi...

Page 77: ...m this operation only when a processor is not installed immediately The processor socket cannot be exposed and must be covered by a processor or cover during transportation and storage 1 Aligning the processor socket cover with the guide pin on the processor socket place the cover on the processor socket See 1 in Figure 8 33 2 Press the positions shown by 2 in Figure 8 33 until you hear a click Fu...

Page 78: ...on the processor socket and damage it The same model of processors must be used in a server Before installing a processor clean the surface of the processor the heat sink and the processor carrier If there is residual thermal compound clean it Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 P...

Page 79: ...ure 8 34 Positions of processors Step 7 Install the processor on the carrier 1 Take a processor carrier out of an ESD bag 2 Aligning the notch in one edge of the processor with the protrusion on the processor carrier secure it Ensure that the processor corner with a triangle mark is in the corner of the processor carrier with a notched triangle FusionServer Pro XH321 V5 Server Node Maintenance and...

Page 80: ...ssor is tilted gently press the tilted part to fasten it Step 8 Apply thermal compound 1 Determine the area on the processor that will be in contact with the heat sink and evenly apply 0 4 ml of thermal compound on the area NOTE When applying thermal compound place the processor carrier on the desktop The thermal compound injector has volume marks The two line five dot s shape and X shape patterns...

Page 81: ...is evenly and fully coated Figure 8 38 Smeared thermal compound layer Step 9 Install the processor carrier to the conjoined heat sink 1 Aligning the processor corner with a triangle mark with the corner of the heat sink with a notched triangle buckle the processor carrier on the heat sink FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Co...

Page 82: ... sink 1 Remove the processor socket cover NOTICE When removing the protective cover pull it out vertically to prevent damage to the pins on the processor socket 2 Shine a light at various angles onto the processor socket to check for twisted pins foreign matter and pad damage NOTICE If bent pins or foreign matters are found or the bonding pad is damaged stop the operation and contact technical sup...

Page 83: ...the label on the heat sink See 1 in Figure 8 40 5 Use a Phillips screwdriver to tighten the four screws marked 2 in the label in the middle of the heat sink in a diagonal sequence See 2 in Figure 8 40 Figure 8 40 Installing the heat sink Step 11 Install the air duct For details see 8 4 24 Installing the Air Duct Step 12 Install the server node For details see 8 3 4 Installing an XH321 V5 Step 13 P...

Page 84: ...ove the air duct For details see 8 4 23 Removing the Air Duct Step 5 Determine the position of the memory module Figure 8 41 Memory slots Step 6 Remove the memory module 1 Open the memory module ejectors on the memory slot See 1 in Figure 8 42 2 Remove the memory module from the slot See 2 in Figure 8 42 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue ...

Page 85: ...End 8 4 14 Installing a Memory Module NOTICE If the processor power is 200 W or 205 W a wide conjoined heat sink is required which occupy memory channels DIMM001 and DIMM031 Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co ...

Page 86: ...en the two memory module ejectors on the memory slot Figure 8 44 Opening the memory module ejectors 2 Align the edge connector of the memory module with the alignment key of the memory slot and insert the memory module in the slot The two memory ejectors are closed automatically after the memory module is firmly seated NOTE Do not touch the edge connector on a memory module with bare hands Before ...

Page 87: ...her the new component is normal For details see FusionServer Pro X6000 Server iBMC User Guide NOTE If information about the new memory module is displayed the replacement is successful If information about the original memory module is displayed access the BIOS and check memory information For details see Huawei Server Purley Platform BIOS Parameter Reference End FusionServer Pro XH321 V5 Server N...

Page 88: ...Server Purley Platform BIOS Parameter Reference If a broken or disfigured screw is found on a TPM take appropriate measures to protect data Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Remove the air duct For details see 8 4 23 ...

Page 89: ...bling 2 Release the buckle on the TPM in the arrow direction and push the TPM backwards until it cannot move forward See 1 in Figure 8 47 3 Remove the TPM from the dowel pins on the side wall of the server node and take it out upwards See 2 in Figure 8 47 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd...

Page 90: ... the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Remove the air duct For details see 8 4 23 Removing the Air Duct Step 5 Determine the position of the TPM FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 0...

Page 91: ...Module Step 8 Install M 2 SSDs For details see 8 4 2 Installing an M 2 SSD Step 9 Install the TPM 1 Align the TPM with the dowel pins on the side wall of the server node and insert the TPM in the arrow direction FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 82 ...

Page 92: ...p 13 Power on the server node For details see 8 3 1 Power On Procedure Step 14 Enable TPM 1 Access the BIOS interface For details see Huawei Server Purley Platform BIOS Parameter Reference 2 Select Security 3 Select TPM Operation and press Enter 4 Select Enable 5 Press F10 The system displays Exit Saving changes dialog box is displayed 6 Select Yes The settings are saved 7 Restart the server Fusio...

Page 93: ...iguration file contains ciphertext change the ciphertext to plaintext and delete the comment tags from the line For details about how to import and export configurations see FusionServer Pro X6000 Server iBMC User Guide The BIOS configuration file exported from the iBMC contains the configuration list For details see Configuration File Description in FusionServer Pro X6000 Server iBMC User Guide I...

Page 94: ...rds until the battery ejects and remove the battery vertically See 1 and 2 in Figure 8 51 Figure 8 51 Removing the battery Step 9 Place the removed component in an ESD bag NOTICE Place only one component in an ESD bag End FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 85 ...

Page 95: ...e the position of the battery Figure 8 52 Position of the battery Step 6 Remove the PCIe riser module For details see 8 4 22 Installing a PCIe Riser Module Step 7 Holding the battery with the text side facing upwards insert the left end of the battery into the slot and gently press down the whole battery into the slot Figure 8 53 Installing the battery FusionServer Pro XH321 V5 Server Node Mainten...

Page 96: ...the line For details about how to import and export configurations see FusionServer Pro X6000 Server iBMC User Guide The BIOS configuration file exported from the iBMC contains the configuration list For details see Configuration File Description in FusionServer Pro X6000 Server iBMC User Guide If the iBMC cannot be logged in to or the BIOS configuration file is not exported before the battery is ...

Page 97: ... module and remove the PCIe card in the arrow direction See 5 in Figure 8 55 Figure 8 55 Remove a PCIe card Step 7 Place the removed component in an ESD bag NOTICE Place only one component in an ESD bag End FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 88 ...

Page 98: ... 4 Take the component out of its ESD bag Step 5 Determine the position of the PCIe card Figure 8 56 Position of the PCIe card Step 6 Hold the PCIe riser module and insert the PCIe card module into the PCIe riser module in the arrow direction See 1 in Figure 8 57 Figure 8 57 Installing a PCIe card FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 ...

Page 99: ...wer off the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 Step 3 Place the server node on the ESD workstation Step 4 Determine the position of the PCIe riser module Step 5 Remove the PCIe riser module 1 Use a Phillips screwdriver to loosen the screws on the cover of the server node See 1 in Figure 8 58 2 Remove the co...

Page 100: ...ve the PCIe card For details see 8 4 19 Removing a PCIe Card Step 8 Place the removed PCIe riser module and PCIe card in different ESD bags NOTICE Place only one component in an ESD bag Step 9 Install a filler panel NOTE Perform this operation only when a PCIe riser module is not installed immediately FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 ...

Page 101: ...ling a PCIe Riser Module Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 92 ...

Page 102: ... Determine the position of the PCIe riser module Step 5 Remove the filler panel NOTE Perform this operation only when a filler panel is installed Keep the removed filler panel properly FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 93 ...

Page 103: ...e card For details see 8 4 20 Installing a PCIe Card Step 8 Connect the cable between the PCIe card and the NC SI port For details see 7 4 PCIe Card Cabling FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 94 ...

Page 104: ... arrow direction See 3 in Figure 8 61 4 Use a Phillips screwdriver to tighten the screws on the node panel See 4 in Figure 8 61 Figure 8 61 Installing the PCIe riser module Step 10 Install the server node For details see 8 3 4 Installing an XH321 V5 Step 11 Power on the server node For details see 8 3 1 Power On Procedure End 8 4 23 Removing the Air Duct Procedure Step 1 Power off the server node ...

Page 105: ...ep 6 Place the removed component in an ESD bag NOTICE Place only one component in an ESD bag End 8 4 24 Installing the Air Duct Procedure Step 1 Power off the server node For details see 8 3 2 Power Off Procedure Step 2 Remove the server node For details see 8 3 3 Removing an XH321 V5 FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 8 Parts Replacement Issue 17 2021 03 09 Copyri...

Page 106: ... 8 3 4 Installing an XH321 V5 Step 8 Power on the server node For details see 8 3 1 Power On Procedure End 8 4 25 Removing the Mainboard and Server Node Case If you cannot log in to the iBMC of the mainboard to be replaced you do not need to perform Step 1 to Step 3 Procedure Step 1 Record the firmware iBMC BIOS and CPLD versions of the mainboard to be replaced On the iBMC WebUI choose Information...

Page 107: ...d information You need to manually configure the password for the iBMC user For details about how to import and export the configuration file see FusionServer Pro X6000 Server iBMC User Guide The iBMC and BIOS configuration files exported from the iBMC contain the configuration list For details see Configuration File Description in FusionServer Pro X6000 Server iBMC User Guide If the iBMC and BIOS...

Page 108: ...Step 11 Remove the PCIe riser module For details see 8 4 21 Removing a PCIe Riser Module Step 12 Remove the PCIe card For details see 8 4 19 Removing a PCIe Card Step 13 Remove the TPM For details see 8 4 15 Removing the TPM Step 14 Remove M 2 SSDs For details see 8 4 1 Removing an M 2 SSD Step 15 Remove memory modules For details see 8 4 13 Removing a Memory Module Step 16 Remove processors For d...

Page 109: ...ace only one component in an ESD bag End 8 4 26 Installing the Mainboard and Server Node Case Procedure Step 1 Take the component out of its ESD bag Step 2 Place the server node on the ESD workstation Step 3 Determine the position of the mainboard and the server node case Figure 8 65 Mainboard and server node case Step 4 Connect all the cables to the mainboard For details see 7 Internal Cabling St...

Page 110: ...e For details see 8 3 1 Power On Procedure Step 16 Check the indicator status If the health status indicator is steady green the server node is running properly For details see 2 1 2 Indicators and Buttons Step 17 To burn the product SN in the new mainboard contact technical support NOTICE If the product SN is not burnt in the new mainboard the iBMC and OS cannot obtain the product SN This may aff...

Page 111: ... contain the configuration list For details see Configuration File Description in FusionServer Pro X6000 Server iBMC User Guide The iBMC supports import and export of configuration files from version 2 32 If the iBMC BIOS configuration files are not imported before you replace the mainboard manually configure the iBMC and BIOS data after the replacement Step 20 Upgrade the mainboard firmware iBMC ...

Page 112: ...fault occurs on a server collect logs for fault diagnosis Fault diagnosis Fault diagnosis rules and tools help Huawei technical support engineers and maintenance engineers to analyze and rectify faults according to alarms and hardware fault symptoms Software and firmware upgrade Software and firmware upgrade packages can be downloaded by server model and installed as needed Preventive maintenance ...

Page 113: ...u can use any of the following methods to query the IP address of the iBMC management network port BIOS iBMC WebUI For details see FusionServer Pro X6000 Server iBMC User Guide iBMC CLI Run the ipmcget d ipinfo command For details see FusionServer Pro X6000 Server iBMC User Guide Procedure Step 1 Access the BIOS Step 2 Choose Advanced IPMI iBMC Configuration and press Enter The IPMI iBMC Configura...

Page 114: ...istrator can also unlock a user account using the CLI For security purposes change the initial password upon the first login and periodically change the password If resources fail to be obtained due to unstable network connection the iBMC WebUI may be displayed abnormally If this occurs refresh the browser and log in to the iBMC WebUI again Procedure Step 1 Check that the client for example a loca...

Page 115: ...RE 1 8 U144 Mozilla Firefox 39 0 to 54 0 Windows Server 2008 R2 64 bit Internet Explorer 9 0 to 11 0 JRE 1 7 U45 JRE 1 8 U45 JRE 1 8 U144 Mozilla Firefox 39 0 to 54 0 Google Chrome 21 0 to 44 0 Windows Server 2012 64 bit Internet Explorer 10 0 to 11 0 JRE 1 7 U45 JRE 1 8 U45 JRE 1 8 U144 Mozilla Firefox 39 0 to 54 0 Google Chrome 21 0 to 44 0 Red Hat Enterprise Linux 6 0 64 bit Mozilla Firefox 39 ...

Page 116: ... is a problem with this website s security certificate is displayed click Continue to this website not recommended If a security alert is displayed you can ignore this message or perform any of the following to shield this alert Import a trust certificate and a root certificate to the iBMC For details see Importing the iBMC Trust Certificate and Root Certificate in FusionServer Pro Xxx iBMC User G...

Page 117: ...e and attempt to log in again If the login still fails choose Tools Internet Options Advanced in the menu bar and click Reset to restore default settings of Internet Explorer Then attempt to log in again Log in as an LDAP user NOTICE Before login ensure that the following settings meet the requirements A domain controller exists on the network and a user domain and LDAP users have been created on ...

Page 118: ...BMC WebUI V600 and Later Versions Scenarios Log in to the iBMC WebUI The following uses Internet Explorer 11 0 as an example NOTE A maximum of four users can log in to the iBMC WebUI at the same time By default the system timeout period is 5 minutes If no operation is performed on the WebUI within 5 minutes the user will be automatically logged out of the WebUI The system locks a user account if t...

Page 119: ...xplorer 11 0 Microsoft Edge AdoptOpenJDK 8u222 JRE AdoptOpenJDK 11 0 6 JRE Mozilla Firefox 45 0 to 79 0 Google Chrome 55 0 to 84 0 Windows Server 2008 R2 64 bit Internet Explorer 11 0 AdoptOpenJDK 8u222 JRE AdoptOpenJDK 11 0 6 JRE Mozilla Firefox 45 0 to 79 0 Google Chrome 55 0 to 84 0 Windows Server 2012 64 bit Internet Explorer 11 0 AdoptOpenJDK 8u222 JRE AdoptOpenJDK 11 0 6 JRE Mozilla Firefox ...

Page 120: ... brackets Examples IPv4 address 192 168 100 1 IPv6 address fc00 64 If the message There is a problem with this website s security certificate is displayed click Continue to this website not recommended If a security alert is displayed you can ignore this message or perform any of the following to shield this alert Import a trust certificate and a root certificate to the iBMC For details see Import...

Page 121: ...r group The default user is Administrator and the default password is Admin 9000 When Domain is Local iBMC the maximum length of the user name is 20 characters When Domain is not Local iBMC the maximum length of the user name is 255 characters FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 112 ...

Page 122: ...bout the domain controller The iBMC provides only the access function for LDAP users On the User Security LDAP page of the iBMC WebUI the LDAP function is enabled and the user domain and the LDAP user who belong to the user domain are set a Select a language b Enter the LDAP user name and password NOTE LDAP user name In this case Domain can be Automatic matching or a specified domain LDAP user nam...

Page 123: ... user a Optional On the iBMC login page switch to the target language b Enter the Kerberos user name and password NOTE Kerberos user name In this case Domain can be Automatic matching or a specified domain Kerberos user name Domain name In this case Domain can be Automatic matching or a specified domain and uppercase letters must be used in the domain name When you log in to the iBMC WebUI as a Ke...

Page 124: ... a server using the iBMC Remote Virtual Console Procedure Step 1 Log in to the iBMC WebUI For details see 10 2 Logging In to the iBMC WebUI Step 2 On the menu bar choose Remote Console The Remote Console page is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 115 ...

Page 125: ...tions performed by the other user HTML5 Integrated Remote Console Private allows only one local user or VNC user to access and manage the server at a time HTML5 Integrated Remote Console Shared allows two local users or five VNC users to access and manage the server at a time Each user can view the operations performed by the other user HTML5 supports only Internet Explorer 10 0 and later versions...

Page 126: ...Figure 10 4 Real time operation console Java FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 117 ...

Page 127: ...te Virtual Console Procedure Step 1 Log in to the iBMC WebUI For details see 10 2 Logging In to the iBMC WebUI Step 2 Click Start in the Virtual Console area in the lower right corner of the Home page FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 118 ...

Page 128: ...ons performed by the other user HTML5 Integrated Remote Console Private allows only one local user or VNC user to access and manage the server at a time HTML5 Integrated Remote Console Shared allows two local users or five VNC users to access and manage the server at a time Each user can view the operations performed by the other user HTML5 supports only Internet Explorer 10 0 and later versions F...

Page 129: ...Figure 10 7 Real time operation console Java FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 120 ...

Page 130: ...onsole provides easier operations than the Remote Virtual Console You can download the Independent Remote Console at Independent Remote Console 10 3 2 1 Versions Earlier than V600 10 3 2 1 1 Windows The following Windows OS versions are supported Windows 7 32 bit or 64 bit Windows 8 32 bit or 64 bit Windows 10 32 bit or 64 bit Windows Server 2008 R2 32 bit or 64 bit FusionServer Pro XH321 V5 Serve...

Page 131: ... V228 and earlier versions support only local users iBMC V228 and later versions support local users and LDAP domain users In versions earlier than iBMC V228 the port number is the RMCP service port number In iBMC V228 and later versions the port number is the HTTPS service port number The default port number can be omitted Step 4 Select a login mode Shared Mode allows two users to access and mana...

Page 132: ...ning dialog box will no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed Figure 10 11 Real time desktop End 10 3 2 1 2 Ubuntu The following Ubuntu OS versions are supported FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 1...

Page 133: ...assword NOTE The network address can be in either of the following formats iBMC management network port IPv4 or IPv6 address Port number The IPv6 address must be enclosed in brackets Example fc 64 444 or 192 168 100 1 444 iBMC domain name address Port number iBMC V228 and earlier versions support only local users iBMC V228 and later versions support local users and LDAP domain users In versions ea...

Page 134: ...13 Security warning Step 8 Click Yes NOTE Click No to return to the login page Click Import CA to import a CA certificate cer crt or pem After the CA certificate is imported the security warning dialog box will no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Ser...

Page 135: ... Open the console and set the folder where the Independent Remote Console is stored as the working folder Step 3 Grant the execute permission on the Independent Remote Console chmod x KVM sh Step 4 Open the Independent Remote Console KVM sh The Independent Remote Console login page is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 0...

Page 136: ...n versions earlier than iBMC V228 the port number is the RMCP service port number In iBMC V228 and later versions the port number is the HTTPS service port number The default port number can be omitted Step 6 Select a login mode Shared Mode allows two users to access and manage the server node at the same time Each user can view the operations performed by the other user Private Mode allows only o...

Page 137: ...top End 10 3 2 1 4 Red Hat The following Red Hat OS versions are supported RHEL 6 9 RHEL 7 3 Procedure Step 1 Configure an IP address for the client local PC to enable communication with the iBMC management network port Step 2 Open the console and set the folder where the Independent Remote Console is stored as the working folder Step 3 Grant the execute permission on the Independent Remote Consol...

Page 138: ...iBMC V228 and later versions support local users and LDAP domain users In versions earlier than iBMC V228 the port number is the RMCP service port number In iBMC V228 and later versions the port number is the HTTPS service port number The default port number can be omitted Step 6 Select a login mode Shared Mode allows two users to access and manage the server or node at the same time Each user can...

Page 139: ...ll no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed Figure 10 20 Real time desktop End 10 3 2 2 V600 and Later Versions 10 3 2 2 1 Windows The following Windows OS versions are supported FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Tech...

Page 140: ...port IPv4 or IPv6 address Port number The IPv6 address must be enclosed in brackets Example fc 64 444 or 192 168 100 1 444 iBMC domain name address Port number Local and LDAP domain users are supported The preferred port number is the HTTPS service port number and then the RMCP service port number The default port number can be omitted Step 4 Select a login mode Shared Mode allows two users to acc...

Page 141: ... imported the security warning dialog box will no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed Figure 10 23 Real time desktop End 10 3 2 2 2 Ubuntu The following Ubuntu OS versions are supported FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Hu...

Page 142: ...d password NOTE The network address can be in either of the following formats iBMC management network port IPv4 or IPv6 address Port number The IPv6 address must be enclosed in brackets Example fc 64 444 or 192 168 100 1 444 iBMC domain name address Port number Local and LDAP domain users are supported The preferred port number is the HTTPS service port number and then the RMCP service port number...

Page 143: ...t CA to import a CA certificate cer crt or pem After the CA certificate is imported the security warning dialog box will no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed Figure 10 26 Real time desktop End FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Cop...

Page 144: ...displayed Figure 10 27 Independent Remote Console login page Step 5 Enter the network address user name and password NOTE The network address can be in either of the following formats iBMC management network port IPv4 or IPv6 address Port number The IPv6 address must be enclosed in brackets Example fc 64 444 or 192 168 100 1 444 iBMC domain name address Port number Local and LDAP domain users are ...

Page 145: ...ed Figure 10 28 Security warning Step 8 Click Yes NOTE Click No to return to the login page Click Import CA to import a CA certificate cer crt or pem After the CA certificate is imported the security warning dialog box will no longer be displayed You are advised to periodically update the certificate for security purposes The server desktop is displayed FusionServer Pro XH321 V5 Server Node Mainte...

Page 146: ...tep 2 Open the console and set the folder where the Independent Remote Console is stored as the working folder Step 3 Grant the execute permission on the Independent Remote Console chmod x KVM sh Step 4 Open the Independent Remote Console KVM sh The Independent Remote Console login page is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2...

Page 147: ... port number is the HTTPS service port number and then the RMCP service port number The default port number can be omitted Step 6 Select a login mode Shared Mode allows two users to access and manage the server or node at the same time Each user can view the operations performed by the other user Private Mode allows only one user to access and manage the server or node at a time Step 7 Click Conne...

Page 148: ...played Figure 10 32 Real time desktop End 10 4 Logging In to the CLI 10 4 1 Logging In to the CLI Using PuTTY over a Network Port Scenarios Use PuTTY to access a server over a local area network LAN NOTE You can obtain the PuTTY software from the chiark home page You are advised to use PuTTY of the latest version PuTTY of an earlier version may cause login failures FusionServer Pro XH321 V5 Server...

Page 149: ...llows Host Name or IP address Enter the IP address of the server to be accessed for example 191 100 34 32 Port Retain the default value 22 Connection type Retain the default value SSH Close window on exit Retain the default value Only on clean exit NOTE Configure Host Name and Saved Sessions and click Save You can double click the saved record in Saved Sessions to log in to the server next time Fu...

Page 150: ...m initial configuration of the server The server is inaccessible over a network port NOTE You can obtain the PuTTY software from the chiark home page You are advised to use PuTTY of the latest version PuTTY of an earlier version may cause login failures Procedure Step 1 On the PC double click PuTTY exe The PuTTY Configuration window is displayed Step 2 In the navigation tree choose Connection Seri...

Page 151: ...Open The PuTTY screen is displayed Then the message login as is displayed prompting you to enter a user name NOTE If this is your first login to the server the PuTTY Security Alert dialog box is displayed Click Yes to proceed Step 7 Enter the user name and password If the login is successful the server host name is displayed on the left of the prompt End FusionServer Pro XH321 V5 Server Node Maint...

Page 152: ...ess Enter Step 4 Select IIO Configuration and press Enter Step 5 Select Intel R VMD Technology and press Enter Step 6 Select Intel R VMD Config and press Enter Step 7 Select Auto and press Enter Step 8 Press F10 The Save Changes Exit dialog box is displayed Step 9 Select Yes and press Enter to save the settings The server automatically restarts for the settings to take effect End 10 5 2 Disabling ...

Page 153: ...l Console click on the menu bar Step 3 Choose Forced System Reset or Forced Power Cycle The Select an Option dialog box is displayed NOTICE A forced restart or power cycle may cause data loss or program damage Before performing a forced restart or power cycle ensure that no service interruption risk exists Step 4 Click Yes The server node starts to be forcibly restarted or power cycled Step 5 Duri...

Page 154: ... press F12 Enter the password in the displayed dialog box To access the Smart Provisioning GUI press F6 The screen for entering the BIOS password is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Operations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 145 ...

Page 155: ...rt the server node to unlock it The Main screen of the Setup Utility program is displayed End 10 6 2 Accessing the BIOS V6XX or Later Procedure Step 1 Log in to the desktop of the server node For details see 10 3 Logging In to the Desktop of a Server Step 2 On the Remote Virtual Console click on the menu bar Step 3 Choose Forced System Reset or Forced Power Cycle The Select an Option dialog box is...

Page 156: ... Delete or F4 when the information shown in Figure 10 37 is displayed Figure 10 37 BIOS startup screen NOTE To go to the Boot Manager screen press F11 or F3 To boot from the network press F12 Enter the password in the displayed dialog box To access the Smart Provisioning GUI press F6 The screen for entering the BIOS password is displayed FusionServer Pro XH321 V5 Server Node Maintenance and Servic...

Page 157: ...curity purposes change the administrator password periodically The system will be locked if an incorrect password is entered three consecutive times You can restart the server node to unlock it The Front Page screen is displayed Step 7 Select BIOS Configuration by pressing arrow keys The Main screen is displayed End FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 10 Common Oper...

Page 158: ...t Website Technical documents are available at Huawei enterprise website Huawei Carrier Product and Service website Self Service Platform and Community Learn more about servers and communicate with experts at Computing Product Information Service Platform for specific server product documentation Huawei Enterprise iKnow for Q A about products Huawei Enterprise Support Community Servers for learnin...

Page 159: ...he technical support personnel of your local Huawei office 11 2 Product Information Table 11 1 provides common information about Huawei servers Table 11 1 Product information Item Description How to Obtain Server product documentation Server user guide which provides information about the structure specifications and installation of the server Visit Support Intelligent Servers select a product mod...

Page 160: ...install hardware Intelligent Computing Product Hardware Installation Multimedia Computing Product Memory Configuration Assistant The Computing Product Memory Configuration Assistant is an online application The DIMM installation sequence in a graphical manner will be displayed after the product name CPU quantity and DIMM quantity are specified Computing Product Memory Configuration Assistant 11 4 ...

Page 161: ...rmware Download link Enterprise users Smart Provisioning Carrier users Contact the technical support personnel of your local Huawei office FusionDirector For details see FusionDirector Specifications List FusionDirector is intelligent O M management software for the entire server lifecycle It provides functions such as intelligent deployment management asset management version management fault man...

Page 162: ...n and alarm management The iBMC implements fault detection and alarm management ensuring stable uninterrupted 24 7 system operation Virtual KVM and virtual media The iBMC provides virtual KVM and virtual media facilitating remote maintenance Web based user interface WebUI The iBMC provides a web based UI for setting and querying device information System breakdown screenshots and video playback Th...

Page 163: ...ng system OS and the hardware to interact with the keyboard display and other input output I O devices The BIOS data is stored on the Serial Peripheral Interface SPI flash memory The BIOS performs a power on self test POST initializes CPU and memory checks the I O and boot device and finally boots the OS The BIOS also provides the advanced configuration and power interface ACPI and hot swap The Hu...

Page 164: ...Figure 12 1 BIOS in the system FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide 12 Software and Configuration Utilities Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 155 ...

Page 165: ...Table A 1 SN description Callout No Description 1 SN ID two characters which is 21 2 Material identification code 8 characters that is the processing code 3 Vendor code two characters 10 indicates Huawei and other values indicate outsourcing vendors FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide A Appendix Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 156 ...

Page 166: ...mpliant 7 Internal model that is product name A 2 Operating Temperature Limitations NOTE For configurations not listed in the table the operating temperature range is 5 C 41 F to 30 C 86 F For special configuration and temperature requirements contact Huawei technical support A 2 1 Operating Temperature Limitations Supercapacitor for the RAID Controller Card Configured Table A 2 Operating temperat...

Page 167: ...125 W P 165 W 32 C 89 6 F P 125 W 35 C 95 F 8 Q 16 165 W P 205 W 30 C 86 F 140 W P 165 W 32 C 89 6 F P 140 W 35 C 95 F 0 Q 8 165 W P 205 W 32 C 89 6 F P 165 W 35 C 95 F X6000 V5 C10 12 x 3 5 SAS backplane 8 Q 12 165 W P 205 W N A 125 W P 165 W 32 C 89 6 F P 125 W 35 C 95 F 4 Q 8 165 W P 205 W 30 C 86 F 140 W P 165 W 32 C 89 6 F P 140 W 35 C 95 F 0 Q 4 165 W P 205 W 32 C 89 6 F P 165 W 35 C 95 F Fu...

Page 168: ...165 W 30 C 86 F 30 C 86 F 125 W P 140 W 35 C 95 F 30 C 86 F P 125 W 35 C 95 F 35 C 95 F 0 Q 8 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 35 C 95 F X6000 V5 C00 24 SAS backplane 16 Q 24 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 30 C 86 F 35 C 95 F for 10GE optical modules 8 Q 16 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 35 C 95 F 0 Q 8 P 165 W 35 C 95 F 35 C 95 F X6000 ...

Page 169: ...6 F 30 C 86 F 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 35 C 95 F X6000 V5 C10 24 SAS backplane 16 Q 24 165 W P 205 W N A N A 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 30 C 86 F 35 C 95 F for 10GE optical modules 8 Q 16 165 W P 205 W 30 C 86 F 30 C 86 F 140 W P 165 W 35 C 95 F 30 C 86 F P 140 W 35 C 95 F 35 C 95 F 0 Q 8 165 W P 205 W 35 C 95 F 30 C 86 F P 165 W 35 C 95 F 35 C 9...

Page 170: ...6 F P 165 W 35 C 95 F 35 C 95 F A 2 3 Operating Temperature Limitations Avago SAS3004iMR RAID Controller Card M 2 FRU Configured Table A 4 Operating temperature specifications with Avago SAS3004iMR RAID controller card M 2 FRU configured Drive Backplane Drive Quantity Q CPU Power P Maximum Temperature X6000 V5 C00 24 x 2 5 NVMe backplane 16 Q 24 140 W P 165 W 28 C 82 4 F P 140 W 32 C 89 6 F 8 Q 16...

Page 171: ... W 35 C 95 F 0 Q 8 165 W P 205 W 28 C 82 4 F P 165 W 35 C 95 F X6000 V5 C10 24 x 2 5 SAS backplane 16 Q 24 165 W P 205 W N A 140 W P 165 W 30 C 86 F P 140 W 35 C 95 F 8 Q 16 165 W P 205 W 28 C 82 4 F P 165 W 35 C 95 F 0 Q 8 165 W P 205 W 32 C 89 6 F P 165 W 35 C 95 F X6000 V5 C10 12 x 3 5 SAS backplane 8 Q 12 165 W P 205 W N A 140 W P 165 W 30 C 86 F P 140 W 35 C 95 F 4 Q 8 165 W P 205 W 30 C 86 F...

Page 172: ...onjoined heat sinks Drives 24 DIMMs 16 35 C 95 F X6000 V5 C10 24 x 2 5 NVMe backplane Intel Xeon Scalable Skylake and Cascade Lake processor 165 W P 205 W Wide conjoined heat sinks 0 drives 8 DIMMs 12 30 C 86 F Intel Xeon Scalable Skylake and Cascade Lake processor P 165 W Narrow conjoined heat sinks Drives 24 DIMMs 16 35 C 95 F Wide conjoined heat sinksb 0 drives 8 DIMMs 12 30 C 86 F X6000 V5 C10...

Page 173: ...F 0 drives 4 DIMMs 12 35 C 95 F Intel Xeon Scalable Skylake and Cascade Lake processor P 165 W Narrow conjoined heat sinks Drives 12 DIMMs 16 35 C 95 F Wide conjoined heat sinksb 4 drives 8 DIMMs 12 30 C 86 F 0 drives 4 DIMMs 12 35 C 95 F a The C00 model does not support Intel Xeon Scalable Gold 6144 Gold 6146 Gold 6244 and Gold 6252N processors b When Intel Xeon Scalable Gold 6144 Gold 6146 Gold ...

Page 174: ...Protection Detects memory command and address errors Memory Demand and Patrol Scrubbing Corrects errors upon detection If these errors are not corrected promptly uncorrectable errors may occur Memory Mirroring Improves system reliability Single Device Data Correction Provides a single device multi bit error correction capability to improve memory reliability Device Tagging Degrades and rectifies D...

Page 175: ...the error information to the OS over the APEI of the Advanced Configuration and Power Interface ACPI and locates the error unit improving system availability Error Injection Support Injects errors to verify various RAS features Machine Check Architecture Provides software recovery for uncorrectable errors to improve system availability eMCA Gen2 Improves system availability OOB access to MCA regis...

Page 176: ... The BIOS provides physical unit information for DIMM errors and the OS traces and predicts errors and isolates error memory pages FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide A Appendix Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 167 ...

Page 177: ...he objects E ejector lever A part on the panel of a device used to facilitate installation or removal of the device Ethernet A baseband local area network LAN architecture developed by Xerox Corporation by partnering with Intel and DEC Ethernet uses the Carrier Sense Multiple Access Collision Detection CSMA CD access method and allows data transfer over various cables at 10 Mbit s The Ethernet spe...

Page 178: ...r It seals the front and rear of the chassis to ensure optimal ventilation and electromagnetic compatibility EMC Peripheral Component Interconnect Express PCIe A computer bus PCI which uses the existing PCI programming concepts and communication standards but builds a faster serial communication system Intel is the main sponsor for PCIe PCIe is used only for internal interconnection A PCI system c...

Page 179: ...r A special computer that provides services for clients over a network system event log SEL Event records stored in the system used for subsequent fault diagnosis and system recovery B 5 U Z U U A unit defined in International Electrotechnical Commission IEC 60297 1 to measure the height of a cabinet or chassis 1 U 44 45 mm UltraPath Interconnect UPI A point to point processor interconnect develop...

Page 180: ...tor Extensions B BBU backup battery unit BIOS Basic Input Output System BMC baseboard management controller C CD calendar day CE Conformite Europeenne CIM Common Information Model CLI command line interface FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide C Acronyms and Abbreviations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 171 ...

Page 181: ...d correcting ECMA European Computer Manufacturer Association EDB Execute Disable Bit EN European Efficiency ERP enterprise resource planning ETS European Telecommunication Standards C 2 F J F FB DIMM Fully Buffered DIMM FC Fiber Channel FCC Federal Communications Commission FCoE Fibre Channel over Ethernet FTP File Transfer Protocol FusionServer Pro XH321 V5 Server Node Maintenance and Service Gui...

Page 182: ...rol Message Protocol IDC Internet Data Center IEC International Electrotechnical Commission IEEE Institute of Electrical and Electronics Engineers IGMP Internet Group Message Protocol IOPS input output operations per second IP Internet Protocol IPC intelligent power capability IPMB Intelligent Platform Management Bus IPMI Intelligent Platform Management Interface FusionServer Pro XH321 V5 Server N...

Page 183: ...controller N NBD next business day NC SI Network Controller Sideband Interface C 4 P T P PCIe Peripheral Component Interconnect Express PDU power distribution unit PHY physical layer PMBUS power management bus POK power OK PWM pulse width modulation FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide C Acronyms and Abbreviations Issue 17 2021 03 09 Copyright Huawei Technologies Co ...

Page 184: ...puter System Interface SATA Serial Advanced Technology Attachment SCM supply chain management SDDC single device data correction SERDES serializer deserializer SGMII serial gigabit media independent interface SMI serial management interface SMTP Simple Mail Transfer Protocol SNMP Simple Network Management Protocol SOL serial over LAN SONCAP Standards Organization of Nigeria Conformity Assessment P...

Page 185: ...ted platform module C 5 U Z U UDIMM unbuffered dual in line memory module UEFI Unified Extensible Firmware Interface UID unit identification light UL Underwriter Laboratories Inc USB Universal Serial Bus V VCCI Voluntary Control Council for Interference by Information Technology Equipment VGA Video Graphics Array VLAN virtual local area network VRD voltage regulator down FusionServer Pro XH321 V5 ...

Page 186: ...trical and electronic equipment WSMAN Web Service Management FusionServer Pro XH321 V5 Server Node Maintenance and Service Guide C Acronyms and Abbreviations Issue 17 2021 03 09 Copyright Huawei Technologies Co Ltd 177 ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands