Huawei ES3000 V5 User Manual Download Page 28

Page: 28 / 248

Parameter

Specifications

Humidity

Storage humidity: 5% RH to 95% RH

Operating humidity: 5% RH to 95% RH

Vibration

Operating vibration: 2.17 G

RMS

(5 Hz to 700 Hz)

Non-operating vibration: 3.13 G

RMS

(5 Hz to 800 Hz)

Shock

Operating shock: 1000 G/0.5 ms

Non-operating shock: 1000 G/0.5 ms

2.5 System Requirements

This section describes the ES3000 V5 requirements for server software and hardware.

Hardware Requirements

The minimum requirements for server hardware are as follows:

2 GB memory

To use the ES3000 V5 NVMe PCIe SSD disk, the server must provide at least one 2.5-
inch SSD disk slot that supports the PCIe SFF-8639 connector.

To use the ES3000 V5 NVMe PCIe SSD card, the server must provide at least one
HHHL (167.5 mm x 70 mm x 18.6 mm) PCIe 3.0 x4 card slot.

Software Requirements

Use the

Huawei Server Compatibility Checker

to check the OS types and versions

supported by the ES3000 V5.

OSs listed in this section support common I/O services. To obtain the list of OSs that
support hot swap, see

4.1 OSs Supporting Hot Swap

. To obtain the list of OSs that can

be installed on the ES3000, see

G Installing an OS on the ES3000

ES3000 V5 NVMe PCIe SSD
User Guide

2 Getting to Know the ES3000 V5

Issue 07 (2019-03-19)

Summary of Contents for ES3000 V5

Page 1: ...ES3000 V5 NVMe PCIe SSD User Guide Issue 07 Date 2019 03 19 HUAWEI TECHNOLOGIES CO LTD ...

Page 2: ...s document may not be within the purchase scope or the usage scope Unless otherwise specified in the contract all statements information and recommendations in this document are provided AS IS without warranties guarantees or representations of any kind either express or implied The information in this document is subject to change without notice Every effort has been made in the preparation of th...

Page 3: ...document is intended for l Server installation engineers l Server maintenance engineers Symbol Conventions The symbols that may be found in this document are defined as follows Symbol Description Indicates an imminently hazardous situation which if not avoided will result in death or serious injury Indicates a potentially hazardous situation which if not avoided may result in minor or moderate inj...

Page 4: ...ssue Date Description 07 2019 03 19 Modified notes on the hot swap function 06 2019 03 11 Modified command description 05 2018 12 22 Modified the Information Collection chapter 04 2018 11 13 Added information about SSD cards 03 2018 10 23 Modified descriptions of the command line and updated output of some commands 02 2018 08 06 Modified some commands and added operations for querying formatting p...

Page 5: ... PCIe SSD Disk 21 3 2 2 Installing the ES3000 V5 NVMe PCIe SSD Card on a Riser Card 23 3 2 3 Installing the ES3000 V5 NVMe PCIe SSD Card on the Mainboard 24 3 3 Confirming Driver Information 25 3 4 Verifying the Upgrade Package Integrity 26 3 5 Transferring Files by Using WinSCP 26 3 6 Installing the Driver 28 3 6 1 Installing the ES3000 V5 Driver in Linux 28 3 6 2 Installing the ES3000 V5 Driver ...

Page 6: ... Tool Package1 79 5 4 1 Upgrading the Tool Package in Linux 79 5 4 2 Upgrading the Tool Package in Windows 79 5 4 3 Upgrading the Tool in ESXi 80 5 5 Upgrading the Controller Firmware 80 5 5 1 Upgrade Paths 80 5 5 2 Upgrading the SSD Firmware in Linux PCIe SSD 81 5 5 3 Upgrading the SSD Firmware in Windows PCIe SSD 82 5 5 4 Upgrading the SSD Firmware in VMware PCIe SSD 84 5 6 Querying Health Statu...

Page 7: ...ting the Capacity of an SSD 115 6 4 Formatting 117 6 4 1 Formatting a Logical Drive 117 6 4 2 Querying Formatting Parameter Support Information 118 6 4 3 Formatting an SSD 122 6 5 Safely Erasing User Data of an SSD 124 6 6 Querying and Setting the Power Consumption Level 126 6 6 1 Querying the Power Consumption Level of an SSD 126 6 6 2 Setting the Power Consumption Level of an SSD 126 6 7 Obtaini...

Page 8: ...cified SSD 151 A Supported NVMe Commands 152 A 1 Supported NVMe Management Commands 152 A 2 Supported NVMe I O Commands 153 A 3 Supported Get Log Page Commands 153 A 4 SMART Properties 153 A 5 Supported Set Feature Commands 164 A 6 Supported Vendor Specific Command 166 A 6 1 New Commands 166 A 6 1 1 Query Commands 166 A 6 1 1 1 Obtaining Logs and Statistical Information 167 A 6 1 1 2 Obtaining SSD...

Page 9: ...0 Querying UniqueID Information 196 A 6 2 2 Configuration Commands 197 A 6 2 2 1 Deleting SMART Information 198 A 6 2 2 2 Configuring the SSD Capacity 199 A 6 2 2 3 Recovering Data 199 A 6 2 2 4 Configuring the Number of Namespaces 199 B Out of Band Management 200 B 1 NVMe Basic Management Command 200 B 2 VPD 204 B 3 Temperature Register 210 C SNMP Function 211 C 1 Overview 211 C 2 Configuring the...

Page 10: ...H 4 Obtaining Technical Support 236 I Acronym or Abbreviation 237 ES3000 V5 NVMe PCIe SSD User Guide Contents Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd ix ...

Page 11: ...ot represent all the safety instructions They are only supplements to the safety instructions Local Safety Regulations When operating Huawei equipment you must follow the local laws and regulations The safety instructions in this document are only supplements to the local laws and regulations Basic Requirements l Installation and maintenance personnel must understand basic safety precautions to av...

Page 12: ...stem must be installed in an access controlled location l The device must be fixed securely on the floor or to other immovable objects such as walls and mounting racks before operation l When installing the unit always make the ground connection first and disconnect it at the end l Do not block the ventilation while the device is operating Keep a minimum distance of 5 cm between the device and the...

Page 13: ...a Do not touch the device with your hands When operating the device within this area take electrostatic discharge ESD preventive measures For example wear an ESD preventive wrist strap 1 3 Electrical Safety This section describes safety notice about high voltage thunderstorm high electrical leakage power cable fuse electrostatic discharge High Voltage The high voltage power supply provides power f...

Page 14: ...yes l Before installing or removing the power cable turn off the power switch l Before connecting a power cable check that the label on the power cable is correct Fuse If a fuse is to be replaced the new fuse must be of the same type and specifications Electrostatic Discharge The static electricity generated by human bodies may damage the electrostatic sensitive components on boards for example th...

Page 15: ...o wear an ESD wrist strap Figure 1 1 Wearing an ESD wrist strap 1 4 Certification ES3000 V5 SSD Disk RoHS China RoHS Europe REACH WEEE CE FCC IC NRTL UL VCCI RCM KSA KUCAS SONCAP UNBS ACAP CB KCC EAC PCI SIG and NVMe protocol consistency ES3000 V5 SSD Card RoHS China RoHS Europe REACH WEEE CE FCC IC NRTL UL VCCI RCM KSA KUCAS SONCAP UNBS ACAP CB Warning ES3000 V5 NVMe PCIe SSD User Guide 1 Safety ...

Page 16: ...in a residential environment this product may cause radio interference in which case the user may be required to take adequate measures ES3000 V5 NVMe PCIe SSD User Guide 1 Safety Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 6 ...

Page 17: ...ol logs all the set operations on the ES3000 V5 l Security assurance A complete data clearance tool is provided to destroy data at the NAND flash memory chip level preventing data leakage l Firmware upgrade Firmware upgrade through in band management channels ensures upgrade reliability and network security The upgrade tool verifies the integrity of the ES3000 V5 firmware upgrade file to ensure th...

Page 18: ...nsists of the ES3500P V5 and ES3600P V5 series The ES3000 V5 NVMe PCIe SSD disk is a 2 5 inch PCIe 3 0 x 4 SSD disk that uses the HiSilicon Hi1812E chip and complies with the SFF 8639 standard This SSD disk can be installed in universal servers such as rack servers Figure 2 1 shows the appearance of the ES3000 V5 NVMe PCIe SSD disk Figure 2 1 ES3000 V5 NVMe PCIe SSD disk Label The label on an ES30...

Page 19: ...d half height half length HHHL PCIe 3 0 x 4 card It can be installed on universal servers such as rack servers Figure 2 3 shows the appearance of the ES3000 V5 NVMe PCIe SSD Figure 2 3 ES3000 V5 NVMe PCIe SSD card 2 3 Features The ES3000 V5 NVMe PCIe SSD provides the following NVMe features l Huawei ES3000 V5 SSD supports the NVMe 1 3 specification ES3000 V5 NVMe PCIe SSD User Guide 2 Getting to K...

Page 20: ...rprise hot swap on some V5 servers only ES3600C V5 series support orderly hot swap on some V5 servers l Uses multiple technologies to ensure device reliability Enhanced Low Density Parity Check LDPC algorithm provides higher error correction capability than that required by flash chips and ensures device reliability Embedded RAID like algorithm implements channel based error correcting Data can be...

Page 21: ...V5 and ES3600P V5 series The default power consumption level 0 ensures optimal performance A non default power consumption level is not recommended NOTE Logical Block Addressing LBA is a general mechanism used to describe the block of data on a storage device It is generally used on drives An LBA format indicates an LBA size of the LV as which an NVMe SSD is formatted An LBA block has a variety of...

Page 22: ...e a The maximum power consumption is the test result with the maximum sequential write bandwidth Power consumption specifications in the preceding table are test results based on CentOS 7 0 and the standard storage test tool fio 2 6 with the ES3500P V5 formatted into a 512 byte LBA block device Reliability Specifications Table 2 2 lists the reliability specifications for the ES3500P V5 series Tabl...

Page 23: ...perature of 55 C 131 F 2000 GB l 155 LFM at an ambient temperature of 40 C 104 F l 650 LFM at an ambient temperature of 55 C 131 F 4000 GB l 237 LFM at an ambient temperature of 40 C 104 F l 650 LFM at an ambient temperature of 55 C 131 F 8000 GB l 237 LFM at an ambient temperature of 40 C 104 F l 650 LFM at an ambient temperature of 55 C 131 F Humidity l Storage humidity 5 RH to 95 RH l Operating...

Page 24: ...andby power consumption l 6400 GB 21 W maximum power consumption and 5 7 W standby power consumption Form factor Standard 2 5 inch drive a The maximum power consumption is the test result with the maximum sequential write bandwidth Power consumption specifications in the preceding table are test results based on CentOS 7 0 and the standard storage test tool fio 2 6 with the ES3600P V5 formatted in...

Page 25: ...t The highest operating temperature decreases by 1 C 1 8 F for every increase of 220 m 721 78 ft in altitude Air speed 800 GB l 155 LFM at an ambient temperature of 40 C 104 F l 650 LFM at an ambient temperature of 55 C 131 F 1600 GB l 155 LFM at an ambient temperature of 40 C 104 F l 650 LFM at a ambient temperature of 55 C 131 F 3200 GB l 237 LFM at an ambient temperature of 40 C 104 F l 650 LFM...

Page 26: ... maximum power consumption 13 8 W standby power consumption 5 W l 3200 GB maximum power consumption 18 8 W standby power consumption 5 5 W l 6400 GB maximum power consumption 21 W standby power consumption 5 7 W Form factor Standard HHHL PCIe card Dimensions 167 5 mm 68 9 mm 18 6 mm 6 6 in x 2 7 in x 0 7 in a The maximum power consumption is the test result with the maximum sequential write bandwi...

Page 27: ...to 70 C 32 F to 158 F Altitude l Operating 305 m to 5486 m 1000 64 ft to 17998 47 ft l Non operating 305 m to 12192 m 1000 64 ft to 39999 51 ft The highest operating temperature decreases by 1 C 1 8 F for every increase of 220 m 721 78 ft in altitude Air speed 800 GB l 155 LFM at an ambient temperature of 40 C 104 F l 650 LFM at an ambient temperature of 55 C 131 F 1600 GB l 155 LFM at an ambient ...

Page 28: ...k the server must provide at least one 2 5 inch SSD disk slot that supports the PCIe SFF 8639 connector l To use the ES3000 V5 NVMe PCIe SSD card the server must provide at least one HHHL 167 5 mm x 70 mm x 18 6 mm PCIe 3 0 x4 card slot Software Requirements l Use the Huawei Server Compatibility Checker to check the OS types and versions supported by the ES3000 V5 l OSs listed in this section supp...

Page 29: ...3 2 Installing the ES3000 V5 3 3 Confirming Driver Information 3 4 Verifying the Upgrade Package Integrity 3 5 Transferring Files by Using WinSCP 3 6 Installing the Driver 3 7 Installing the Tool Package 3 1 Installation and Configuration Process The installation and configuration process includes the overall procedures for installing and configuring the ES3000 V5 You can learn about ES3000 V5 ins...

Page 30: ...3 2 2 Installing the ES3000 V5 NVMe PCIe SSD Card on a Riser Card If the PCIe card slots are on the mainboard the ES3000 V5 NVMe PCIe SSD card needs to be installed on the mainboard For details see 3 2 3 Installing the ES3000 V5 NVMe PCIe SSD Card on the Mainboard Preparations l Stop all services on the server l Unpack and inspect the devices Check that the items shipped are complete and intact ba...

Page 31: ... details run the man fstab command to view help information After enabling the automatic mounting of SSD partitions you do not need to run the umount command before installing or uninstalling the driver and inserting or removing the SSD Follow up Procedure After the ES3000 V5 is connected to power the startup process takes about 30 seconds If the activity indicator associated with the ES3000 V5 NV...

Page 32: ...onverter See 1 in Figure 3 3 Step 6 Use two screws to secure the ES3000 V5 to the converter See 2 in Figure 3 3 Figure 3 3 Securing the ES3000 V5 to the converter ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 22 ...

Page 33: ... 9 Close the ejector lever to push the ES3000 V5 into place See 2 in Figure 3 5 Figure 3 5 Installing the ES3000 V5 End 3 2 2 Installing the ES3000 V5 NVMe PCIe SSD Card on a Riser Card Step 1 Wear ESD gloves Step 2 Turn off the power to the server and remove the chassis cover Step 3 Take the new ES3000 V5 out of the ESD bag ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue...

Page 34: ... Step 6 Turn on the power to the server End 3 2 3 Installing the ES3000 V5 NVMe PCIe SSD Card on the Mainboard Step 1 Wear ESD gloves Step 2 Turn off the power to the server and remove the chassis cover Step 3 Hold the upper edge of the ES3000 V5 align its connecting part with the PCIe slot and vertically insert the ES3000 V5 downwards into the PCIe slot slowly See Figure 3 7 ES3000 V5 NVMe PCIe S...

Page 35: ... SSD and SSD provided by other vendors use the NVMe driver integrated in the OS and you do not need to install Huawei s NVMe driver l If Huawei s NVMe driver has been installed in Linux and you need to use SSD provided by other vendors uninstall Huawei s NVMe driver l For Windows Server 2008 R2 or Windows Server 2012 if the KB2990941 patch has been installed the OS is integrated with the NVMe driv...

Page 36: ...espectively For details about the verification tool and method visit the following link Software digital signature Open Verify validation tool 3 5 Transferring Files by Using WinSCP This section describes how to transfer files by using WinSCP You can also transfer files by using other software Scenarios Use WinSCP to transfer files from a PC to a remote host Prerequisites Conditions The File Trans...

Page 37: ...ning Continue connecting and add host key to cache is displayed Click Yes The WinSCP file transfer page is displayed l By default the C Documents and Settings Administrator My Documents directory of the local PC is opened in the left pane and the root directory of the remote host is opened in the right pane Step 4 In the right pane choose a directory of the remote host for storing files for exampl...

Page 38: ..._123 el7 redhat x86_64 rpm or kmod hiodriver 5 0 0 6 3 10 0_123 el7 oracle x86_64 rpm l SLES driver package name hiodriver kmp Kernel feature Driver version Supported kernel version System distributor Hardware platform rpm such as hiodriver kmp default 5 0 0 6 3 12 28_4 suse12 x86_64 rpm l Ubuntu or Debian driver package name hiodriver Driver version Supported kernel version System distributor Har...

Page 39: ... loop Driver package name command to mount the ISO package to a directory for example tmp citrix The following uses Citrix 6 2 as an example mount o loop hiodriver 3 3 9 1 6u2 iso tmp citrix c Access the mount directory where the ISO package is located for example tmp citrix cd tmp citrix d Run install sh in tmp citrix to install the driver install sh e Run the following command to query driver in...

Page 40: ... Step 2 again to install the driver Step 6 Check whether the following message is displayed The message indicates that the original NVMe driver fails to be uninstalled during the installation Warning fail to uninstall the old nvme kernel module Please uninstall the old nvme kernel module manually or reboot the system l If yes go to Step 9 l If no go to Step 7 Step 7 Check whether the following mes...

Page 41: ...n systemctl restart irqbalance service NOTE l For RHEL 6 CentOS 6 and SLES 12 you can add IRQBALANCE_ARGS hintpolicy exact to the end of the etc sysconfig irqbalance file content to better balance interrupts l To configure the irqbalance service in other OSs refer to documents related to the OSs If the OS cannot find the service it indicates that the OS does not use the irqbalance service for bala...

Page 42: ...ou have correctly installed the ES3000 V5 on the server l You have downloaded the driver package and verified its integrity For details see 3 4 Verifying the Upgrade Package Integrity l You have uploaded the driver package to the server by using WinSCP or other software Procedure Select an operation based on the OS type l For Windows Server 2012 R2 go to 1 in Install the driver l For Windows Serve...

Page 43: ... Device Manager is displayed as shown in Figure 3 10 Figure 3 10 Device Manager The following are three possible cases ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 33 ...

Page 44: ...ed updates The Installed Updates window is displayed as shown in Figure 3 11 Figure 3 11 Installed Updates 7 Right click the KB2990941 patch in the Microsoft Windows column and choose Uninstall from the shortcut menu Install the driver 1 Double click the driver installation file hiodriver_V5 3 5 2_2008R2_x84_64 msi NOTE There are four driver installation file types for four Windows versions indica...

Page 45: ...ndow is displayed as shown in Figure 3 13 Figure 3 13 End User License Agreement window 3 Select I accept the terms in the License Agreement and click Next ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 35 ...

Page 46: ...3 14 Choose Setup Type window 4 Click Complete The Ready to install window is displayed as shown in Figure 3 15 Figure 3 15 Ready to install window ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 36 ...

Page 47: ...o 1 in Install the driver If no contact Huawei technical support For details see H Getting Help Figure 3 17 Expanding Storage controllers 8 Expand Disk drives Check whether NVMe HWE32P43032M000N SCSI Disk Device exists under Disk drives See Figure 3 18 If yes the driver has been successfully installed If no contact Huawei technical support for help For details see H Getting Help Figure 3 18 Expand...

Page 48: ...age in a directory The following is an example command esxcli software vib install v tmp hiodriver 5 0 0 6 1OEM 550 0 0 2768847 x86_64 vib NOTE l In the driver file name 5 0 0 6 is the driver version The actual driver version may differ l Enter a complete directory after v l If the Could not find a trusted signer error occurs indicating that the driver version is not certified by ESXi IOVP add no ...

Page 49: ... or Debian the package name is hioadm version number release number hardware architecture deb for example hioadm 5 0 0 6 1 amd64 deb l Citrix driver software package hioadm Version number System version iso for example hioadm citrix 6u2 5 0 0 8 iso Preparations l You have correctly installed the ES3000 V5 on the server l You have downloaded the tool package and verified its integrity l You have us...

Page 50: ... completes f Run xe update apply uuid 320232df 7adb 4cbe a7a3 8515240879e1 to apply the package Step 4 Run the following commands to obtain help information about the tool or query tool information hioadm h or hioadm help rpm qa grep hioadm For Citrix 7 1 run xe update list to query tool information End 3 7 2 Installing the Tool Package Windows Operation Scenario Install the tool package for manag...

Page 51: ...iver version number This procedure uses hioadm_V5 3 5 4_x86_64 msi as an example The installation interface is displayed as shown in Figure 3 19 Figure 3 19 Installation interface Step 3 Click Next The End User License Agreement window is displayed as shown in Figure 3 20 ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 41...

Page 52: ...rms in the License Agreement and click Next The Destination Folder window is displayed as shown in Figure 3 21 Figure 3 21 Destination Folder window ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 42 ...

Page 53: ...re 3 22 Figure 3 22 Ready to install window Step 6 Click Install to start installation When the installation is complete the Completing Setup Wizard window is displayed as shown in Figure 3 23 ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 43 ...

Page 54: ...ool package The following procedure assumes that the tool package is in the tmp directory and the tool package name is hioadm 5 0 1 0 1OEM vib Run the following command to install the ES3000 V5 tool package esxcli software vib install v tool package in a directory The following is an example command esxcli software vib install v tmp hioadm 5 0 1 0 1OEM vib ES3000 V5 NVMe PCIe SSD User Guide 3 Inst...

Page 55: ...r occurs indicating that the tool version is not certified by ESXi IOVP add no sig check at the end of the command to skip signature check Step 3 After the tool is installed run the following command to obtain help information about the tool hioadm h End ES3000 V5 NVMe PCIe SSD User Guide 3 Installation and Configuration Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 45 ...

Page 56: ...conds before swapping or inserting another SSD l SSD hot swap may affect services running on the SSD Before performing a hot swap notify system maintenance personnel to ensure service security l If you insert the ES3000 V5 SSD when it is not installed on a drive tray it may fail to be detected l After the ES3000 V5 SSD is removed wait for at least 3 seconds before inserting it again If you insert ...

Page 57: ...ream Windows OSs Windows Server 2012 R2 Windows Server 2016 l Mainstream Linux OSs RHEL 7 3 and 7 4 4 2 Orderly Hot Swap You can insert the ES3000 V5 SSDs to running servers For Linux and Windows notify the OS before hot removing the ES3000 V5 For ESXi OSs notify the OS before hot swapping the ES3000 V5 4 2 1 Orderly Hot Swap of a Drive in Linux Operation Scenario In Linux perform the operations d...

Page 58: ...el Parameter Configuration on Ubuntu 14 04 15 04 Kernel Parameter Configuration on RHEL 7 3 7 4 The following procedure uses RHEL 7 3 as an example Step 1 Log in to the OS as an administrator Step 2 Run the vim boot grub2 grub cfg command to open the boot configuration file Step 3 Enter i Step 4 In the boot configuration file add pciehp pciehp_force 1 pci pcie_bus_perf to the boldface part shown i...

Page 59: ...44afff501d4492e8c752cca7ff44971 root dev mapper rhel root ro crashkernel auto rd lvm lv rhel root rd lvm lv rhel swap rhgb quiet pciehp pciehp_force 1 pci pcie_bus_perf initrd16 initramfs 0 rescue 144afff501d4492e8c752cca7ff44971 img END etc grub d 10_linux Some code omitted Step 5 Press Esc enter wq and press Enter to save the edits Step 6 Restart the OS for the settings to take effect Step 7 Go ...

Page 60: ...d0 msdos1 if x feature_platform_search_hint xy then search no floppy fs uuid set root hint bios hd0 msdos1 hint efi hd0 msdos1 hint baremetal ahci0 msdos1 hint hd0 msdos1 b64fedb9 b1e9 465d 994f 9cee287c1d57 else search no floppy fs uuid set root b64fedb9 b1e9 465d 994f 9cee287c1d57 fi linux16 vmlinuz 0 rescue 17e12da386e9435aa7f2b6b718079678 root UUID b054ccd0 c3d3 47eb a449 f897f63f1b76 ro rd lv...

Page 61: ...ke effect Step 7 Go to Notifying the OS End Kernel Parameter Configuration on Oracle Enterprise Linux 7 3 7 4 The following procedure uses Oracle 7 3 as an example Step 1 Log in to the OS as an administrator Step 2 Run the vim boot grub2 grub cfg command to open the boot configuration file Step 3 Enter i Step 4 In the boot configuration file add pciehp pciehp_force 1 pci pcie_bus_perf to the boldf...

Page 62: ...gb quiet LANG en_US UTF 8 pciehp pciehp_force 1 pci pcie_bus_perf initrd16 initramfs 3 10 0 514 el7 x86_64 img menuentry Oracle Linux Server 0 rescue ced47f5d0b3a4b6c875666af895ea929 with Linux 7 3 class oracle class gnu linux class gnu class os unrestricted menuentry_id_option gnulinux 0 rescue ced47f5d0b3a4b6c875666af895ea929 advanced 544c61b2 0ee7 4071 b081 084072d6dd85 load_video insmod gzio i...

Page 63: ...s uuid set root hint bios hd0 msdos1 hint efi hd0 msdos1 hint baremetal ahci0 msdos1 hint hd0 msdos1 fcf9f115 8f51 496e a7a8 905f03350b4e else search no floppy fs uuid set root fcf9f115 8f51 496e a7a8 905f03350b4e fi linux16 vmlinuz 3 10 0 123 el7 x86_64 root UUID eba3dd96 8be3 46ce a8d9 21699e3fbedb ro crashkernel auto vconsole font latarcyrheb sun16 rd lvm lv ol swap rd lvm lv ol root vconsole k...

Page 64: ...e3 46ce a8d9 21699e3fbedb ro crashkernel auto vconsole font latarcyrheb sun16 rd lvm lv ol swap rd lvm lv ol root vconsole keymap us rhgb quiet pciehp pciehp_force 1 pci pcie_bus_perf initrd16 initramfs 0 rescue a6e5b98906f643a09908dd24f3b775f7 img if x default Oracle Linux Server with Unbreakable Enterprise Kernel 3 8 13 35 3 1 el7uek x86_64 then default Advanced options for Oracle Linux Server O...

Page 65: ... with Linux 4 4 21 69 default hotkey 2 class sles class gnu linux class gnu class os menuentry_id_option gnulinux 4 4 21 69 default advanced 9e0fffba d347 4e34 9800 59271fd5ce73 load_video set gfxpayload keep insmod gzio insmod part_msdos insmod btrfs set root hd0 msdos2 if x feature_platform_search_hint xy then search no floppy fs uuid set root hint bios hd0 msdos2 hint efi hd0 msdos2 hint bareme...

Page 66: ...d to open the boot configuration file Step 3 Enter i Step 4 In the boot configuration file add pciehp pciehp_force 1 pci pcie_bus_perf to the boldface part shown in the following Between the added content and its preceding content a space is required and no line feed is allowed Some code omitted BEGIN etc grub d 10_linux menuentry SLES12 class sles12 class gnu linux class gnu class os menuentry_id...

Page 67: ...ci pcie_bus_perf echo Loading initial ramdisk initrd boot initrd 3 12 28 4 default menuentry SLES12 with Linux 3 12 28 4 default recovery mode hotkey 3 class sles12 class gnu linux class gnu class os menuentry_id_option gnulinux 3 12 28 4 default recovery dcbfdecb bcd9 4003 ba51 a95c7e31c56e load_video set gfxpayload keep insmod gzio insmod part_msdos insmod btrfs set root hd0 msdos2 if x feature_...

Page 68: ...ss os class xen menuentry_id_option xen gnulinux 3 12 28 4 xen advanced dcbfdecb bcd9 4003 ba51 a95c7e31c56e insmod part_msdos insmod btrfs set root hd0 msdos2 if x feature_platform_search_hint xy then search no floppy fs uuid set root hint bios hd0 msdos2 hint efi hd0 msdos2 hint baremetal ahci0 msdos2 hint hd0 msdos2 dcbfdecb bcd9 4003 ba51 a95c7e31c56e else search no floppy fs uuid set root dcb...

Page 69: ...GIN etc grub d 20_linux_xen depends on whether Xen Hypervisor is selected during OS installation Step 5 Press Esc enter wq and press Enter to save the edits Step 6 Restart the OS for the settings to take effect Step 7 Go to Notifying the OS End Kernel Parameter Configuration on SLES 11 SP4 Step 1 Log in to the operating system OS as an administrator Step 2 Run the vim etc init d boot local command...

Page 70: ...01 63 default root dev sda2 showopts ide nodma apm off noresume edd off powersaved off nohz off highres off processor max_cstate 1 nomodeset x11failsafe vga 0x314 pci pcie_bus_perf initrd boot initrd 3 0 101 63 default Some code omitted NOTE Whether the file content includes title Xen depends on whether Xen Hypervisor is selected during OS installation Step 9 Press Esc enter wq and press Enter to ...

Page 71: ...8 497b ae46 014a53fe3cb5 else search no floppy fs uuid set root e2afe344 3108 497b ae46 014a53fe3cb5 fi linux boot vmlinuz 4 4 0 21 generic root UUID e2afe344 3108 497b ae46 014a53fe3cb5 ro pciehp pciehp_force 1 pci pcie_bus_perf initrd boot initrd img 4 4 0 21 generic submenu Advanced options for Ubuntu menuentry_id_option gnulinux advanced e2afe344 3108 497b ae46 014a53fe3cb5 menuentry Ubuntu wi...

Page 72: ...Linux 4 4 0 21 generic linux boot vmlinuz 4 4 0 21 generic root UUID e2afe344 3108 497b ae46 014a53fe3cb5 ro recovery nomodeset pciehp pciehp_force 1 pci pcie_bus_perf echo Loading initial ramdisk initrd boot initrd img 4 4 0 21 generic END etc grub d 10_linux Some code omitted Step 5 Press Esc enter wq and press Enter to save the edits Step 6 Restart the OS for the settings to take effect Step 7 ...

Page 73: ...ot UUID b7889e8e dbc4 4fe9 bf5c dab9c20a93f8 ro pciehp pciehp_force 1 pci pcie_bus_perf initrd boot initrd img 3 13 0 24 generic submenu Advanced options for Ubuntu menuentry_id_option gnulinux advanced b7889e8e dbc4 4fe9 bf5c dab9c20a93f8 menuentry Ubuntu with Linux 3 13 0 24 generic class ubuntu class gnu linux class gnu class os menuentry_id_option gnulinux 3 13 0 24 generic advanced b7889e8e d...

Page 74: ...ect Step 7 Go to Notifying the OS End Notifying the OS Step 1 Log in to the OS as the root user Step 2 Stop all services accessing the ES3000 V5 SSD Step 3 If you have used the mount command to attach the SSD partitions to certain paths use the umount command to detach the partitions Step 4 Determine the server where the SSD is installed l If the SSD is installed on a V3 server go to Step 5 l If t...

Page 75: ...ws OS before hot removing the ES3000 V5 SSD Preparations l You have correctly installed the ES3000 V5 SSD on the server l You have installed the Windows tool package For details see 3 7 2 Installing the Tool Package Windows Procedure Windows GUI Step 1 Log in to the OS as an administrator or administrator group member Step 2 Stop all services accessing the ES3000 V5 SSD Step 3 Double click the Hio...

Page 76: ... message indicating the OS is accessing the ES3000 V5 SSD stop related services before clicking eject again You can hot remove the ES3000 V5 SSD only when a success message is displayed End Windows CLI Step 1 Log in to the OS as an administrator or administrator group member Step 2 Stop all services accessing the ES3000 V5 SSD and make all logical drives offline ES3000 V5 NVMe PCIe SSD User Guide ...

Page 77: ...open the Windows CLI as shown in Figure 4 3 Figure 4 3 Opening the Windows CLI Step 4 Run the following command to instruct the OS to power off the device hioadm hotplug d devicename r In this command devicename indicates the name of the device to be removed such as nvme0 and nvme1 ES3000 V5 NVMe PCIe SSD User Guide 4 Hot Swap Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 67 ...

Page 78: ...o power off the device hioadm hotplug d devicename r In this command devicename indicates the name of the device to be removed such as nvme0 and nvme1 NOTE After the previous command is executed you can remove the SSD if the green indicator is off and the yellow indicator blinks at the frequency of 0 5 Hz End Hot Inserting an SSD Step 1 Insert the ES3000 V5 into the target slot Step 2 Log in to th...

Page 79: ...000 V5 installed is a V5 server l The VMD function is enabled in the BIOS For details about surprise hot swap operations see the server version requirements and operation guide of each vendor ES3000 V5 NVMe PCIe SSD User Guide 4 Hot Swap Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 69 ...

Page 80: ...nformation Collection 5 8 Securely Erasing SSD Data 5 9 Querying SSD Lifespans 5 10 Common Faults 5 11 Indicator Status Description 5 12 Removing an ES3000 V5 5 1 Uninstalling the Driver This section describes how to uninstall the ES3000 V5 driver on various operating systems OSs 5 1 1 Uninstalling the Driver in Linux Operation Scenario If you want to stop using or replace the ES3000 V5 uninstall ...

Page 81: ...o to Step 5 Step 5 Check whether the following message is displayed The message indicates that the NVMe driver built in the OS fails to be loaded Warning fail to install the in kernel nvme kernel module Please install the in kernel nvme kernel module manually or reboot the system l If yes go to Step 6 l If no go to Step 10 Step 6 Check whether restarting the OS will affect services l If yes go to ...

Page 82: ...y following the procedure described in this section Do not uninstall the ES3000 V5 driver by using Device Manager If you have uninstalled the ES3000 V5 driver by using Device Manager you still need to uninstall the driver by following the procedure described in this section before installing the driver again To install the driver see 3 6 2 Installing the ES3000 V5 Driver in Windows Uninstalling th...

Page 83: ...End 5 1 3 Uninstalling the Driver in ESXi Operation Scenario If you want to stop using or replace the ES3000 V5 uninstall the driver first Procedure Step 1 Log in to the OS as an administrator or a member of the administrator group Step 2 Stop or exit the programs or files that are accessing the ES3000 V5 Step 3 Run the following command to uninstall the ES3000 V5 driver package esxcli software vi...

Page 84: ...atform deb such as hiodriver 5 0 3 3 3 19 0_15_generic ubuntu amd64 deb l UVP driver package name hiodriver kmp xen Driver version Supported kernel version System distributor Hardware platform rpm Preparations Visit http support huawei com enterprise choose Product Support Enterprise Data Center Servers Accelerator Components ES3000 V5 click the Software Download tab and download the software pack...

Page 85: ...g fail to install the new nvme kernel module Please install the new nvme kernel module manually or reboot the system l If yes go to Step 8 l If no go to Step 12 Step 8 Check whether restarting the OS will affect services l If yes go to Step 11 l If no restart the OS and go to Step 12 Step 9 Check whether restarting the OS will affect services l If yes go to Step 10 l If no restart the OS and go to...

Page 86: ... l To configure the irqbalance service in other OSs refer to documents related to the OSs Check whether errors are reported after running the command for restarting the irqbalance service l If yes go to Step 15 l If no no further action is required Step 15 Contact Huawei technical support for help For details see H Getting Help End 5 2 2 Upgrading the ES3000 V5 Driver in Windows Operation Scenario...

Page 87: ...ling the Tool Package 5 3 1 Uninstalling the Tool Package in Linux Operation Scenario Uninstall the tool package to free up space when ES3000 V5 management is not needed Procedure Step 1 Log in to the operating system OS as an administrator Step 2 Run the uninstallation command based on the OS type l For RHEL CentOS OEL or SLES run the rpm e hioadm command l For Ubuntu or Debian run the dpkg r hio...

Page 88: ...ed as shown in Figure 5 3 Figure 5 3 Uninstall or change a program Step 4 Find and right click the tool package and choose Uninstall A confirmation dialog box is displayed as shown in Figure 5 4 Figure 5 4 Confirming the uninstallation Step 5 Click Yes End 5 3 3 Uninstalling the Tool in ESXi The tool is integrated into the driver package Uninstalling the driver package also uninstalls the tool You...

Page 89: ...ure deb for example hioadm 5 0 3 5 1 amd64 deb Preparations l You have correctly installed the ES3000 V5 on the server l You have downloaded the new tool package version and verified its integrity l You have used WinSCP or other software to upload the tool package to the server such as the home directory Procedure Step 1 Log in to the operating system OS as an administrator Step 2 Go to the direct...

Page 90: ...ge the ES3000 V5 when a new tool version is available Procedure Step 1 Run the hioadm v command to check whether the current tool version is the latest Step 2 Install the latest version For details see 3 7 3 Installing the Tool ESXi NOTE The tool upgrade procedure is the same as the tool installation procedure The system will automatically use the new version to replace the earlier version End 5 5...

Page 91: ... file to the server Procedure Step 1 Log in to the OS as an administrator Step 2 Run the following command to view the name of the SSD to be upgraded hioadm info The command output contains information similar to the following NVMe SSD Information Controller Namespace nvme0 0503023HDCN107C80013 nvme0n1 512 0GB nvme1 0503023HDCN107C80014 nvme1n1 512 0GB Step 3 Run the following command to view the ...

Page 92: ...oller firmware is upgraded hioadm updatefw d nvme0 End 5 5 3 Upgrading the SSD Firmware in Windows PCIe SSD Operation Scenario You are advised to upgrade the controller firmware immediately after a new version is released Preparations Before upgrading the firmware check that l You have uploaded the ES3000 V5 software package to a directory on the server for example the D directory l Stop or exit t...

Page 93: ...mn current indicates the firmware version in use whereas next indicates the firmware version that takes effect after next reset Step 6 Move the firmware upgrade file for example ES3000V5_FW_2151 bin to the path where hioadm exe resides Then select an upgrade mode based on the current SSD firmware version For details see 5 5 1 Upgrade Paths Reset activation mode Run the hioadm exe updatefw d nvme0 ...

Page 94: ...nvme1n1 512 0GB Step 3 Run the following command to view the controller firmware version of the SSD nvme0 is used as an example If there are multiple PCIe SSDs perform steps 3 to 8 for each SSD hioadm updatefw d nvme0 The command output contains information similar to the following slot version activation 1 1011 2 1011 current NOTE In the activation column current indicates the firmware version in...

Page 95: ...strator group Step 2 Run the following command to query the health status of the SSD controller such as nvme0 hioadm info d nvme0 device status in the command output indicates the SSD controller health status l healty indicates that the SSD is healthy l warning indicates that the SSD is faulty root localhost tool hioadm info d nvme0 Namespace 1 size 3200 6GB 3200631791616Byte formatted LBA size 51...

Page 96: ...s healthy Step 3 Run the following command to query the SMART information of the SSD controller such as nvme0 hioadm info d nvme0 s The command output is similar to the following root localhost tool hioadm info d nvme0 s critical warning no warning composite temperature 308 degrees Kelvin 35 degrees Celsius available spare 100 available spare threshold 10 percentage used 0 data units read 41908532...

Page 97: ...ave other logs file succeeded opt hio nvme0_other log logs has been saved End 5 7 3 One Click Log Collection Scenario Collect SSD and system operation logs when maintaining SSD devices Prerequisite You have downloaded the ES3000_V5_Tool_x x x x zip package from the Software Download tab at http support huawei com enterprise Product Support Enterprise Data Center Servers Accelerator Components ES30...

Page 98: ...ain all log files from the scratch hio okc directory Step 4 Optional Delete the log files after use to free drive space End 5 8 Securely Erasing SSD Data Operation Scenario Collect SSD logs when maintaining SSD devices Preparations Before performing operations stop all I O services of the SSD and perform the following l For Linux OSs unmount all logical drives l For ESXi OSs delete all partitions ...

Page 99: ...e secure operate succeeded Please remove and then install the NVMe driver or reboot the OS End 5 9 Querying SSD Lifespans Scenario Query the lifespan of an SSD Procedure Step 1 Log in to the OS as an administrator or a member in the administrator group Step 2 Run the following command to query the SMART information of the SSD controller such as nvme0 hioadm info d nvme0 s The command output is sim...

Page 100: ... Typical fault causes are as follows l The drive is not securely installed l The drive is installed in a non NVMe slot l A drive hardware port fault occurs 5 10 2 Drive Identification Faults or Other Faults Drive identification faults indicate scenarios where the host OS disk manager for example fdisk l in Linux and the device manager in Windows cannot properly identify ES3000 SSDs when hardware c...

Page 101: ...mine the SMART status of the drive For details see 6 1 3 Querying Basic Information About a Device6 1 3 Querying Basic Information About a Device Query the SMART information about nvme0 root localhost tool hioadm info d nvme0 s critical warning no warning composite temperature 308 degrees Kelvin 35 degrees Celsius available spare 100 available spare threshold 10 percentage used 0 data units read 4...

Page 102: ...ons l Media placed in read only mode Causes include capacitor failures l Volatile memory backup device failure Causes include capacitor voltage below 28 V or above 35 V End Fault Diagnosis Instance The available space is smaller than the threshold Step 1 Check whether the value of available spare in the SMART information exceeds 10 If the value is greater than or equal to 10 stop using the drive a...

Page 103: ...on and contact Huawei technical support End 5 11 Indicator Status Description 5 11 1 ES3000 SSD Disk Indicator State Descriptions The indicators on the front panel of the drive bay display the working status of the ES3000 V5 as shown in Figure 5 6 The amber blue indicator is the fault indicator and the green indicator is the working status indicator Table 5 2 describes the indicators when the SSD ...

Page 104: ...nt view Table 5 3 ES3000 SSD card indicator state descriptions Activity Indicator Green Fault Indicator Yellow Description Off Off The NVMe SSD card is not powered on On Off The NVMe SSD card is working 2 Hz Off Data access to the NVMe SSD card is in progress Off On Loading the NVMe SSD card firmware fails Off 0 5 Hz The NVMe SSD card firmware is being initialized On Blinking Reserved On On The NV...

Page 105: ...card For details see 5 12 3 Removing an ES3000 V5 NVMe PCIe SSD Card from a Riser Card If PCIe card slots are on the mainboard the SSD card needs to be removed from the mainboard For details see 5 12 4 Removing an ES3000 V5 NVMe PCIe SSD Card from the Mainboard Preparations Before removing an ES3000 V5 NVMe PCIe SSD make the following preparations l Wear ESD gloves Used to prevent ESD damage when ...

Page 106: ...e current environment supports 4 1 OSs Supporting Hot Swap4 1 OSs Supporting Hot Swap Wear ESD gloves Procedure Step 1 Determine the position of the drive in the server and stop the services on the drive to be removed Step 2 Perform operations described in 4 2 Orderly Hot Swap Step 3 Press the ejector release button See 1 in Figure 5 9 ES3000 V5 NVMe PCIe SSD User Guide 5 Maintenance and Upgrade I...

Page 107: ...NVMe PCIe SSD Card from a Riser Card Prerequisites Wear ESD gloves Procedure Step 1 Turn off the power to the server and remove the chassis cover Step 2 Remove all the cables from the riser card Step 3 Hold the upper part of an ES3000 V5 NVMe PCIe SSD card and remove it slowly ES3000 V5 NVMe PCIe SSD User Guide 5 Maintenance and Upgrade Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 97 ...

Page 108: ...Prerequisites Wear ESD gloves Procedure Step 1 Turn off the power to the server and remove the chassis cover Step 2 Open the latch on the PCIe card Step 3 Hold the upper part of an ES3000 V5 NVMe PCIe SSD card and remove it upwards slowly See Figure 5 11 ES3000 V5 NVMe PCIe SSD User Guide 5 Maintenance and Upgrade Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 98 ...

Page 109: ... Removing an ES3000 V5 NVMe PCIe SSD card Step 4 Place the removed drive into an ESD bag End ES3000 V5 NVMe PCIe SSD User Guide 5 Maintenance and Upgrade Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 99 ...

Page 110: ...r cmd minor cmd1 parameter1 minor cmd2 parameter2 major cmd can be any of the following l info information query l format formatting l secure secure erasure of data l updatefw firmware upgrade management l capacity capacity management l log device log management l powerstate power consumption management l temperature temperature management l writemode write mode management deprecated l selftest di...

Page 111: ...d up as an operations log old file overwriting the existing operations log old file The log file content includes the command start time remote client IP address user name command command output time and operation result The operation result is one of the following l SUCCEEDED The command was successfully executed l FAILED The command failed to be executed l CANCELED The command was canceled becau...

Page 112: ...uerying or Setting the Atomic Write Status 6 1 Querying Information 6 1 1 Querying the SSD List Function This command is used to query the SSD list Format hioadm info Parameters None Usage Guidelines None Example Query the list of existing SSDs Linux root localhost tool hioadm info NVMe SSD Information Controller Namespace nvme0 0503023HDCN107C80013 nvme0n1 512 0GB nvme0n2 512 0GB nvme1 0503023HDC...

Page 113: ...m Response The command output in the preceding example is explained as follows l Controller names nvme0 nvme1 and sda l Serial numbers 0503023HDCN107C80013 0503023HDCN107C80014 and HS000000000000253015 l Drive names nvme0n1 nvme0n2 and sda l Capacities 512 0 GB nvme0 nvme0n1 and sda in the command output are values of the parameter after d in the commands 6 1 2 Querying Basic Information About an ...

Page 114: ...10 6 1 3 Querying Basic Information About a Device Function This command is used to query basic information about an SSD SSD controller or logical drive If you query information about an SSD information about all of its logical drives is also displayed Format hioadm info d devicename Parameters Parameter Description Value devicename Name of a device Examples nvme0 and nvme1 Usage Guidelines None E...

Page 115: ...ze Available capacity of the namespace Unit GB In Linux and ESXi 1 GB equals 1000 MB In Windows 1 GB equals 1024 MB Example 3200 6 GB formatted LBA size Formatted logical block address LBA size of the namespace Unit byte Example 512 byte formatted metadata size Formatted metadata size of the namespace Unit byte Example 0 byte 8 byte 64 byte protection type Protection type NOTE This parameter is di...

Page 116: ...odel number Model Example HWE52P431T0L002N firmware version Firmware version Example 1030 NVMe version NVMe version Example 1 3 device status Device health status Values are as follows l healthy l warning l unknown NOTE If an unknown status occurs contact Huawei technical support for help For details see H Getting Help Example healthy 6 1 4 Querying the SMART Information About a Device Function Th...

Page 117: ... 0 number of error information log entries 0 warning composite temperature time 0 min critical composite temperature time 0 min data status OK System Response Parameter Parameter Description Value critical warning Critical warning such as overtemperature and insufficient redundant space Example no warning There are the following types of critical warning l The available space is smaller than the t...

Page 118: ...controller busy time Busy period of the controller Unit minute Example 0 mins power cycles Number of power ons Example 89 times power on hours Power on period Example 1164 h unsafe shutdowns Number of unsafe shutdowns Example 35 times media and data integrity errors Number of data errors Example 0 number of error information log entries Number of error log entries Example 0 warning composite tempe...

Page 119: ...les nvme0 and nvme1 Usage Guidelines None Example Query the advanced information about nvme0 root localhost hioadm info d nvme0 a manufacturer Huawei WWN 50022a1083846072 serial number 025KCA10J7800101 model number HWE56P436T4M002N part number 03032YRE pcb version A controller id 1 PCI slot 0 highest temperature 324 degrees Kelvin 51 degrees Celsius exceed temperature threshold count 0 exceed temp...

Page 120: ...umber Model Example HWE32P430016M00N pcb version PCB version Example A controller id Controller ID Example 1 PCI slot PCI slot number Example 6 NOTE The value may be which indicates that the physical slot in lspci is empty highest temperature Highest temperature recorded Example 329 degree K 56 degree C exceed temperature threshold count Number of times that the temperature threshold is exceeded E...

Page 121: ...blocks count Total number of blocks including bad blocks Example 134144 bad blocks count Total number of bad blocks Example 2383 factory bad blocks count Number of bad blocks before delivery Example 0 flash program failed count Number of flash program failures Example 0 flash erase failed count Number of flash erasure failures Example 0 maximum data transfer size Maximum data transfer size Example...

Page 122: ...not displayed the version does not change after the next reset 6 2 2 Upgrading the Firmware of an SSD Function This command is used to upgrade the firmware of an SSD If you run the following command with activeflag set to 1 that is firmware will be activated immediately services may be stopped Before running this command ensure that stopping services has no adverse impact Format hioadm updatefw d ...

Page 123: ...d nvme0 f ES3000_FW_2 08 bin s 2 a 0 slot version activation 1 1010 next 2 1011 current WARNING You have selected slot 2 to update the firmware image WARNING You have selected the update firmware operation Proceed with the update firmware operation Y N y Please do not remove driver or SSD device Loading OK Downloading and replacing the firmware image at slot 2 succeeded Activating the firmware ima...

Page 124: ...ely might abort IO WARNING You have selected the update firmware operation Proceed with the update firmware Y N y Activating the firmware image at slot 1 succeeded 6 3 Querying and Setting the Capacity The capacity size affects the write performance A small capacity results in a high random write IOPS The ES3000 allows you to set the capacity within a specified range NOTE Setting the capacity may ...

Page 125: ...ce of the SSD Unit GB In Linux and ESXi 1 GB equals 1000 MB In Windows 1 GB equals 1024 MB Example 1600 3 GB 1600321314816 Byte maximum capacity Original available capacity with one decimal place of the SSD Unit GB In Linux and ESXi 1 GB equals 1000 MB In Windows 1 GB equals 1024 MB Example 1600 3 GB 1600321314816 Byte 6 3 2 Setting the Capacity of an SSD Function This command is used to set the c...

Page 126: ...ect the namespace settings automatically restore to the defaults that is only one namespace of the configured size exists and is attached to controller 1 Example Set the current capacity of nvme0 to its original capacity root localhost tool hioadm capacity d nvme0 s 0 current capacity 512 0GB 512000000000Byte maximum capacity 1600 3GB 1600321314816Byte You have selected to restore original capacit...

Page 127: ...he Data Integrity Field DIF function do not format drives into the DIF format Otherwise problems such as operating system resetting will occur 6 4 1 Formatting a Logical Drive Function Perform quick formatting of a logical drive Logical drives support only quick formatting Secure formatting is not allowed Format hioadm format d diskname Parameters Parameter Description Value diskname Name of a log...

Page 128: ...nvme0n1 root localhost tool hioadm format d nvme0n1 WARNING The format operation will clear all data WARNING You have selected the format operation Proceed with the format Y N y Formatting OK The format operate succeeded The current LBA Format type 0 lba 512 metadata 0 6 4 2 Querying Formatting Parameter Support Information Function Query formatting parameter support information including LBA size...

Page 129: ...indicates the metadata size Example 0 bytes l Relative Performance indicates relative performance Example Best l in use indicates the formatting combination currently in use Example LBA Format 0 Data Size 512 bytes Metadata Size 0 bytes Relative Performance Best in use This example indicates that the device supports the combination of 512 B sectors 0 B metadata and best relative performance and th...

Page 130: ...ormation type is type 0 and that the setting is currently in use Protection Information Location l This parameter specifies supported DIF verification information locations l Protection information transferred as the last eight bytes of metadata DIF verification information is transferred as the first eight bytes of metadata Protection information transferred as the first eight bytes of metadata D...

Page 131: ...etadata Size 0 bytes Relative Performance Best in use LBA Format 1 Data Size 512 bytes Metadata Size 8 bytes Relative Performance Better LBA Format 2 Data Size 4096 bytes Metadata Size 64 bytes Relative Performance Better LBA Format 3 Data Size 4096 bytes Metadata Size 0 bytes Relative Performance Best LBA Format 4 Data Size 4096 bytes Metadata Size 8 bytes Relative Performance Better Protection I...

Page 132: ...ze is 0 B l 2 The sector size is 512 B and the metadata size is 8 B l 3 The sector size is 4 KB and the metadata size is 64 B l 4 The sector size is 4 KB and the metadata size is 8 B l 5 The sector size is 520 B and the metadata size is 0 B l 6 The sector size is 4160 B and the metadata size is 0 B If this parameter is not specified the sector size and type are not changed NOTE The Linux operating...

Page 133: ...ransferred as a part of the extended data LBA l 0 Metadata is transferred as a part of separate cache NOTE If Metadata Size is set to 0 this parameter is unavailable Usage Guidelines l Before running this command stop all I O services of the SSD and perform the following For Linux unmount all logical drives Stop all the processes on the logical drives that use the SSD and run the fuser command to ...

Page 134: ...format operation Proceed with the format Y N y Formatting OK The format operate succeeded Please remove and then install nvme driver or reboot the OS The current LBA Format type 0 lba 512 metadata 0 6 5 Safely Erasing User Data of an SSD Function This command is used to safely erase user data of an SSD After safely erasing an SSD the sector type and quantity of logical drives are not changed Exerc...

Page 135: ... The following information in this example the SSD has two logical drives indicates that a process is using an SSD logical drive Stop the process root localhost tool fuser dev nvme0n1 dev nvme0n2 dev nvme0n1 14789 dev nvme0n1 14812 For ESXi delete all partitions of all logical drives For Windows make the logical drives offline as shown in Figure 6 3 Figure 6 3 Making a logical drive offline Exampl...

Page 136: ... SSD Format hioadm powerstate d devicename Parameters Parameter Description Value devicename Name of an SSD Example nvme0 Usage Guidelines None Example Query the power consumption level of the nvme0 SSD root localhost hioadm powerstate d nvme0 Power states 0 25W 1 18W 2 14W 6 6 2 Setting the Power Consumption Level of an SSD Function This command is used to set the power consumption level of an SS...

Page 137: ...fault power consumption level is not recommended Usage Guidelines None Example Set the nvme0 SSD power consumption level to unlimited root localhost tool hioadm powerstate d nvme0 s 0 Set power state succeeded Power states 0 25W 1 18W 2 14W 6 7 Obtaining Logs 6 7 1 Obtaining Logs of a Device Function This command is used to obtain logs of an SSD and generate a log file Format hioadm log d devicena...

Page 138: ...dm exe the key log file name is nvme0_key the full log file name is nvme0_fw the core dump file name is nvme0_coredump Example Obtain the key logs of the nvme0 SSD root localhost tool hioadm log d nvme0 t 0 firmware key log saving OK save firmware key log file succeeded opt hio nvme0_key log logs has been saved 6 7 2 Obtaining All Logs of a Device At a Time Function Obtain and export all logs of a...

Page 139: ...rying and Setting Temperature Thresholds 6 8 1 Querying the Temperature Information About a Device Function This command is used to query the temperature and temperature thresholds of an SSD Format hioadm temperature d devicename i index Parameters Parameter Description Value devicename Name of a device Example nvme0 index Number of a temperature sensor l 0 overall temperature l 1 controller tempe...

Page 140: ... 5 composite overtemperature threshold time Period of SSD overtemperature Unit minute Example 500 min controller highest temperature Highest controller temperature recorded Unit K and C Example 363 degree K 90 degree C controller overtemperature threshold count Number of controller overtemperature occurrences Example 5 controller overtemperature threshold time Period of controller overtemperature ...

Page 141: ... this parameter is not specified this parameter value is not changed Either the overtemperature or undertemperature threshold must be specified NOTE ES3000 SSDs have two alarm severities The second alarm severity restricts drive I O speed If an overtemperature alarm occurs check server heat dissipation and ambient temperature underthreshol d Undertemperature threshold l Unit K l Value range Overal...

Page 142: ...ors temperature information in degree Kelvin degree Celsius current over threshold under threshold composite 303 30 333 60 273 0 controller1 303 30 358 85 273 0 6 9 Querying and Setting the Write Mode 6 9 1 Querying the Write Mode of an SSD Function This command is used to query the write mode of an SSD Format hioadm writemode d devicename Parameters Parameter Description Value devicename Name of ...

Page 143: ...the nvme0 SSD root localhost host hioadm selftest d nvme0 selftest result firmware abnormal passed NAND flash passed NOR flash passed DDR UNC failed ibuf UNC passed capacitor failed blank block passed life passed NOR flash patrol passed firmware UNC passed system area passed bad block passed controller passed powerdown fiq passed format fail passed data check passed ES3000 V5 NVMe PCIe SSD User Gu...

Page 144: ...pacitor test result Values are passed and failed Example failed blank block Blank block test result Values are passed and failed Example passed life Life test result Values are passed and failed Example passed NOR flash patrol NOR patrol test result Values are passed and failed Example passed firmware UNC Firmware UNC test result Values are passed and failed Example passed system area System area ...

Page 145: ...sage Guidelines None Example Query the electronic label information about the nvme1 SSD root localhost hioadm label d nvme1 BoardType HICASPWBD BarCode 032YRH10J9000104 Item 03032YRH Description Finished Board ES3000 V5 HICASPWBD ES3600C 800GB 3 DWPD PCIE 3 0 X4 Vendor ID 19e5 Device ID 3714 1 Model number HWE56P43800M002N HH HL Card NVMe SSD Manufactured 2018 09 30 VendorName Huawei IssueNumber 0...

Page 146: ...y not be able to recover all data l Common recovery will recover original user data as much as possible To ensure data reliability SSD disks or cards will work in read only mode after common recovery Forcible recovery will erase all user data If forcible recovery is successful SSD disks will enter the healthy state Example Recover data for the nvme0 SSD root localhost host hioadm recovery d nvme0 ...

Page 147: ...r a device that supports only one namespace the The number of namespaces supported has been exceeded message is displayed when you create a namespace 6 13 1 Querying the Namespace Distribution of an SSD Function This command is used to query the namespace distribution of an SSD Format hioadm namespace d devicename Parameters Parameter Description Value devicename Name of a device Example nvme0 Usa...

Page 148: ...D Example 1 capacity Capacity one decimal place of a namespace Unit GB Example 200 0 0 GB shared Whether the namespace is shared The value is no no controller id IDs of controllers attached to a namespace Example 1 6 13 2 Creating a Namespace for an SSD Function This command is used to create a namespace for an SSD Format hioadm namespace d devicename t operationtype s size f shareflag Parameters ...

Page 149: ...espace succeeded namespace id 1 6 13 3 Attaching a Namespace to an NVMe Controller Function This command is used to attach a namespace to an NVMe controller Format hioadm namespace d devicename t operationtype i namespaceid c controllerid Parameters Parameter Description Value devicename Name of a device Example nvme0 operationtype Operation Type l 0 Create a namespace l 1 Delete a namespace l 2 A...

Page 150: ...ching the namespace succeeded Please remove and then install the NVMe driver or reboot the OS System Response If attach failed is displayed the operation failed The failure causes are as follows l The namespace is already attached to the NVMe controller l The NVMe controller ID is invalid 6 13 4 Detaching a Namespace from an NVMe Controller Function This command is used to detach a namespace from ...

Page 151: ...ations for the namespace You do not need to stop I O services or perform the following operations for other namespaces l For Linux unmount the logical drive l For ESXi delete all partitions of the logical drive l For Windows make the logical drive offline as shown in Figure 6 4 Figure 6 4 Making a logical drive offline If a Huawei developed driver is installed the setting takes effect immediately ...

Page 152: ...mat hioadm namespace d devicename t operationtype i namespaceid Parameters Parameter Description Value devicename Name of a device Example nvme0 operationtype Operation Type l 0 Create a namespace l 1 Delete a namespace l 2 Attach a namespace l 3 Detach a namespace l 6 Set the number of namespaces Select 1 in this command namespaceid ID of a namespace Usage Guidelines Before running this command s...

Page 153: ... the Number of Namespaces for an SSD Function This command is used to set the number of namespaces that can be created on a specified SSD Format hioadm namespace d devicename t operationtype n num Parameters Parameter Description Value devicename Name of a device Example nvme0 operationtype Operation type l 0 Create a namespace l 1 Delete a namespace l 2 Attach a namespace l 3 Detach a namespace l...

Page 154: ...mat hioadm loadhist d devicename n days Parameters Parameter Description Value devicename Name of a device Example nvme0 days Number of days for query The value ranges between 1 and 360 for example 20 Usage Guidelines None Example Query the amount of written data of nvme0 root localhost tool hioadm loadhist d nvme0 n 20 disk write per day 2 80 System Response Parameter Description Value disk write...

Page 155: ...ample Query the time of a specified SSD nvme0 root localhost tool hioadm synctime d nvme0 Current device time Thu Mar 22 10 46 09 EDT 2018 6 15 2 Synchronizing the Time of a Specified SSD Function This command is used to synchronize system time to a specified SSD Format hioadm synctime d devicename s Parameters Parameter Description Value devicename Indicates the name of an SSD Example nvme0 or sd...

Page 156: ...ormat hioadm restorefeatures d devicename Parameters Parameter Description Value devicename Indicates the name of an SSD Example nvme0 Usage Guidelines None Example Restore the default settings of a specified SSD nvme0 linux mazy hioadm restorefeatures d nvme0 The restorefeatures operation will recover all features Arbitration Power Management Temperature Threshold Error Recovery Interrupt Coalesc...

Page 157: ...e l 0 reserved l 1 exit failure mode l 2 block erase operation l 3 overwrite operation l 4 crypto erase operation NOTE ES3000 V5 whose firmware versions earlier than 2151 supports type 1 and 2 Firmware versions later than 2151 support types 1 2 and 3 For details about how to query the firmware version of an SSD see 6 2 1 Querying the Firmware Version of an SSD release Indicates whether to reassign...

Page 158: ...ration In other cases this parameter is ignored pattern Indicates the overwrite pattern l A 32 bit value is used for overwriting l The default value is 0xFFFFFFFF NOTE This parameter is available only when action is 3 overwrite operation In other cases this parameter is ignored Usage Guidelines None Example Erase a specified SSD nvme0 linux mazy hioadm sanitize d nvme0 m 0 a 2 Set sanitize succeed...

Page 159: ...peration succeeded the NVMe driver has been removed from the OS please remove it manually 6 18 2 Orderly Hot Inserting a Specific SSD Function This command is used to orderly hot insert a specific SSD This command applies only to NVMe SSDs on ESXi Format hioadm hotplug d devicename i Parameters Parameter Description Value devicename Indicates the name of an SSD Example nvme0 Usage Guidelines l Bef...

Page 160: ...6 2 1 Querying the Firmware Version of an SSD 6 19 1 Querying the Atomic Write Status of a Specified SSD Function This command is used to query the atomic write status of a specified SSD Format hioadm atomicwrite d devicename Parameters Parameter Description Value devicename Name of an SSD Example nvme0 Usage Guidelines None Example Query the atomic write status of the nvme0 SSD root localhost too...

Page 161: ...scription Value devicename Name of an SSD Example nvme0 flag Atomic write status l 0 disabled l 1 enabled Usage Guidelines None Example Set the atomic write status of the nvme0 SSD to enabled root localhost tool hioadm atomicwrite d nvme0 f 1 enable atomic write succeeded System Response None ES3000 V5 NVMe PCIe SSD User Guide 6 Command Description Issue 07 2019 03 19 Copyright Huawei Technologies...

Page 162: ...tion Queue l Create I O Completion Queue l Identify l Abort l Set Feature l Get Feature l Asynchronous Event Request The ES3000 V5 also supports the following optional management commands defined by NVMe 1 3 l Namespace Management l Firmware Commit l Firmware Image Download l Device Self test l Namespace Attachment l Directive Send l Directive Receive l Format NVM Huawei drivers and tools must be ...

Page 163: ...0h ES3000 V5 also supports the following custom log command l Vendor Unique SMART Health Information Log Identifier C0h l Vendor Unique Addition SMART Health Information Log Identifier CAh A 4 SMART Properties The ES3000 V5 supports the same SMART properties as the NVMe 1 3 protocol Table1 Table2 and Table3 list the SMART properties Table A 1 SMART properties log identifier 02h Byt es Description ...

Page 164: ...7 3 2 Data Units Read indicates the number of 512 byte sectors read from the controller by the host 1 indicates that the host has read 1000 512 byte sectors which do not include metadata If an LBA is not 512 bytes the controller converts it into 512 byte sectors hexadecimal for calculation 63 4 8 Data Units Written indicates the number of 512 byte sectors read into the controller by the host 1 ind...

Page 165: ...ntire component becomes greater than the ultra high temperature threshold CCTEMP field in controller identify information in minutes 201 200 Temperature Sensor 1 indicates the value detected by temperature sensor 1 Unit Kelvin 203 202 Temperature Sensor 2 indicates the value detected by temperature sensor 2 Unit Kelvin 205 204 Temperature Sensor 3 indicates the value detected by temperature sensor...

Page 166: ...harge cycles of the capacitor 28 Capacitor health status Valu e Description 0h The capacitor is working properly 1h The capacitor has failed FFh 2h Reserved 29 Reserved 33 30 PCIe refclock PLL lock loss count 37 34 CRC error count of PCIe port 0 41 38 Reserved 45 42 E2E validation failure count of the controller 47 46 Number of failed CEs 51 48 Total number of blocks 55 52 Total number of bad bloc...

Page 167: ...7 272 Number of FLASH RR failures 303 288 Number of FLASH RR operations 319 304 Reserved 335 320 Number of UNC errors found during background inspections 351 336 Reserved 367 352 Background inspection data volume unit GB 383 368 Number of UNC errors recorded in the pending list 399 384 Reserved 415 400 Reserved 419 416 Deep rebuild flag 423 420 Reserved 427 424 Before delivery bad block count 1 43...

Page 168: ... count 6 527 520 GC read data volume unit GB 531 528 Forcible data migration count 535 532 Memory error count 539 536 Free Block Num 1 543 540 Free Block Num 2 547 544 Free Block Num 3 551 548 Free Block Num 4 555 552 Free Block Num 5 559 556 Free Block Num 6 563 560 Free Block Num 7 567 564 Free Block Num 8 571 568 Average erase count 575 572 Maximum erase count 583 576 Read correction count 7 58...

Page 169: ... Description 0 Quick formatting failure Secure Erase Settings 0 1 Secure formatting failure Secure Erase Settings 1 2 Rebuild failure 1 3 Rebuild failure 2 4 Rebuild failure 3 5 Rebuild failure 4 6 Rebuild failure 5 7 Power on check failure 1 8 Power on check failure 2 9 Power on check failure 3 10 Power on check failure 4 11 Internal error 1 12 Internal error 2 13 Internal error 3 14 Internal err...

Page 170: ...failure 31 Reserved 32 Reserved 33 Deep rebuild is complete and enter the read only mode 34 Internal error 15 35 Internal error 16 63 3 6 Reserved 659 652 Alarm historical value bit definition same as that of 643 636 667 660 Exception historical value bit definition same as that of 651 644 671 668 Port 0 cold reset count 675 672 Reserved 679 676 Port 0 warm reset count 683 680 Reserved 687 684 Por...

Page 171: ...firmware Bit Description 0 Reserved 1 Firmware health of slot 1 2 Firmware health of slot 2 31 4 3 Reserved 732 Drive status Table A 3 Customized SMART properties Log Identifier CAh Bytes Description 0 Program Fail Count 2 1 Reserved 3 Normalized Value Program Fail 4 Reserved 10 5 Current Raw Value Program Fail 11 Reserved 12 Erase_Fail_Count 14 13 Reserved 15 Normalized Value Erase_Fail_Count ES3...

Page 172: ...e End To End Error Detection 47 Reserved 48 CRC Error Count 50 49 Reserved 51 Normalized Value CRC Error 52 Reserved 58 53 Current Raw Value CRC Error 59 Reserved 60 Timed Workload Media Wear Count 62 61 Reserved 63 Normalized Value Timed Workload Media Wear 64 Reserved 70 65 Current Raw Value Timed Workload Media Wear 71 Reserved 72 Timed Workload Host Reads Count 74 73 Reserved 75 Normalized Val...

Page 173: ...ent Raw Value Thermal Throttle Status 107 Reserved 108 Retry Buffer Overflow Count 110 109 Reserved 111 Normalized Value Retry Buffer Overflow 112 Reserved 118 113 Current Raw Value Retry Buffer Overflow 119 Reserved 120 Pll Lock Loss Count 122 121 Reserved 123 Normalized Value Pll Lock Loss 124 Reserved 130 125 Current Raw Value Pll Lock Loss 131 Reserved 132 Nand Bytes Written Count 134 133 Rese...

Page 174: ... working and health status The ES3000 V5 can also use the following Set Features commands defined by NVMe 1 3 to change the feature parameter settings l Arbitration Feature Identifier 01h l Power Management Feature Identifier 02h l Temperature Threshold Feature Identifier 04h l Error Recovery Feature Identifier 05h l Number of Queues Feature Identifier 07h l Interrupt Coalescing Feature Identifier...

Page 175: ...Temperature sensor 2 not supported 0011b Temperature sensor 3 not supported 0100b Temperature sensor 4 not supported 0101b Temperature sensor 5 not supported 0110b Temperature sensor 6 not supported 0111b Temperature sensor 7 not supported 1000b Temperature sensor 8 not supported 1001b 1110b Reserved 1111b All sensors 15 0 0 Temperature Threshold TMPTH indicates the SSD or controller temperature t...

Page 176: ...dvised to use the newly defined interfaces customized Opcode 0xC2 and Opcode 0xC5 Table A 6 Opcode Opcode 07 Opcode 06 02 Opcode 01 00 Opcode O M Namespa ce Identifier Used Command Generic Command Function Data Transfer 1b 100 00b 00b C0h O NO legacy query command 1b 100 00b 01b C1h O NO legacy configuration command 1b 100 00b 10b C2h O NO new query command 1b 100 00b 01b C5h O NO new configuratio...

Page 177: ...ded the value is 0h Table A 10 Command Dword 12 Bit Description 31 00 Sub_OpCode subcommand number A 6 1 1 1 Obtaining Logs and Statistical Information This command is used to obtain vendor logs from the SSD Each command sent from the host can obtain a maximum of 4KB of logs The host obtains the total log size through the vendor log length query command and then sends commands repeatedly to obtain...

Page 178: ...00 Offset obtained log offset value NOTE Each command sent from the host can obtain a maximum of 4 KB logs If the size exceeds 4 KB the firmware returns the status code NVME_SC_INVALID_FIELD 0x2 Full logs and key logs are composed of a large number of 64 byte log entries whereas CoreDump logs are tiled characters Table A 14 Overall structure of full logs and key logs Byte Description 63 00 Entry 1...

Page 179: ...1 ConnectFlag indicates whether the next log and the unfinished part of the current log can be displayed together 0 indicates yes and 1 indicates no 10 4 Reservd reserved bit 3 0 Level log level Logs are classified into the following levels 0 LOG_LEVEL_DEBUG indicating debugging information 1 LOG_LEVEL_INFO indicating common information which is a normal log 2 LOG_LEVEL_WARNING indicating alarm in...

Page 180: ...e log entries see A 15 whereas CoreDump logs are tiled characters The content of full logs and key logs are parsed by item and written to the corresponding title in the target file The content of CoreDump logs is written to the target file directly A 6 1 1 2 Obtaining SSD ID Parameters This command is used to obtain SSD ID parameters including electronic label information part number PN World Wide...

Page 181: ...during query on network elements The current label version is 3 0 which indicates the number of fields Usually content changes do not cause version changes ArchivesInf oVersion V ersion Number r n 30 6 r n 2 N A Two blank lines are between the information format area and the information description area r n 2 N A Informatio n description area Board Properties r n 20 N A BoardType Model r n 30 18 P...

Page 182: ... n 50 44 Item of a finer granularity This field is reserved for products whose Huawei item codes are insufficient for customers asset management l PN The subcommand is used to set the SSD part number PN The host uses PRP1 and PRP2 to send PN information in the format described in Table4 Table A 20 PN Bytes Description 23 00 SN l WWN The subcommand is used to set the SSD WWN The host uses PRP1 and ...

Page 183: ...0 The data length is specified by Dword10 fixed at 2 Table A 24 FLASH ID BIT Description 63 00 FLASH ID A 6 1 1 3 Obtaining the Enablement Status of Functions This command is used to obtain the enablement status of a customized function through the controller This command uses the PRP1 PRP2 Dword10 and Dword12 fields Enter reserved value 0 for other fields that are not involved Sub_OpCode describe...

Page 184: ...es that the function is disabled 31 01 Reserved A 6 1 1 4 Querying the Maximum Available Capacity of the SSD This command is used to query the maximum available capacity that can be set This command uses the PRP1 PRP2 Dword10 and Dword12 fields Enter reserved value 0 for other fields that are not involved Table A 27 Sub_OpCode Sub_OpCode Description 0x050009h Query the maximum available capacity o...

Page 185: ...n the following table Table A 29 Sub_OpCode Sub_OpCode Description 0x000009h Log parsing package 0x080009h Full logs 0x090009h Key logs 0x120009h CoreDump information 0x420009h Statistical information The obtained valid log length is returned to the host software as Table3 Table A 30 Completion Queue Entry Dword 0 Bit Description 31 0 Valid log length unit byte A 6 1 1 6 Querying the PCB Version T...

Page 186: ... returned to the host through the CQ Dword0 If a bit value is 1 the corresponding item has a fault If a bit value is 0 the corresponding item is passed Table A 34 CQ Dword0 Bit Description 0 Overall test result 1 Firmware error test front end error check whether the drive is in the controller drive mode whether the drive is in read only mode and whether the available space is below the threshold 2...

Page 187: ...Forcibly restore SSDs so that the SSDs can function properly 30 1 Restart or power on and then off SSDs to restore SSDs 31 1 The fault cannot be rectified Contact Huawei support engineers A 6 1 1 8 Obtaining the DIEID of a Chip This command is used to obtain the DIEID which is the unique identifier of a chip and is used to track and manage information of the entire lifecycle including chip manufac...

Page 188: ...days and the result is returned by Dword0 of CQ This command uses the PRP1 PRP2 Dword10 Dword12 and Dword13 fields Enter reserved value 0 for other fields that are not involved Table A 37 Sub_OpCode Sub_OpCode Description 0x0E0000h Query the average DWPD value Table A 38 Command Dword13 Bit Description 31 00 Number of days valid value 1 360 Table A 39 Cq Command Dword0 Bit Description 31 00 DWPD v...

Page 189: ...005h the length is returned by DWORD0 of CQ Table A 41 Cq Command Dword0 Bit Description 31 00 Length The unit is byte When Sub_OpCode delivered to obtain the uniqueID content is 0x10005h the length is returned by Dword10 and must be the same as the length obtained previously The UniqueID content is transmitted by the memory that PRP points to and the content is in the string format Table A 42 Com...

Page 190: ...LBA address and UNC length are returned Each item contains 16 bytes The first 8 bytes indicate the LBA address and the last 8 bytes indicate the UNC length The unit is sector For example the sector size of the 4K data logic is 512 and the corresponding length is 8 A 6 1 1 12 Obtaining the 4 Point Digital Eye Diagram of the PCIe This command is used to obtain the 4 point digital eye diagram of the ...

Page 191: ...1 00 DataRate transmission rate Table A 50 Command Dword 15 Bit Description 31 00 Average Time average count Table A 51 describes the command data format Table A 51 Data format Byte Description 03 00 Upper coordinate 07 04 Lower coordinate 11 08 Left coordinate 15 12 Right coordinate A 6 1 1 13 Obtaining the Full Digital Eye Diagram of the PCIe This command is used to obtain the full digital eye d...

Page 192: ... 13 Bit Description 15 00 Macro 0 corresponds to Macro0 and 1 corresponds to Macro0 31 16 Lane Each macro cell corresponds to two lanes lane 0 and lane 1 Table A 55 Command Dword 14 Bit Description 31 00 DataRate transmission rate Table A 56 Command Dword 15 Bit Description 15 00 IsShowInner 1 to scan the inner eye 2 not to scan the inner eye 31 16 ScanWinInner scanning layers 1 4 Command data for...

Page 193: ...ta structure requires two PRP entries this field represents the second PRP entry address If data of more than two PRP entries is transferred this field points to a PRP list Table A 59 Command Dword 10 Bit Description 31 00 Data_Len indicates the data length in unit of Dword If data transfer is not needed the value is 0h Table A 60 Command Dword 12 Bit Description 31 00 Sub_OpCode subcommand number...

Page 194: ... fields that are not involved If the configured capacity is not aligned by 4 KB the value is rounded up for alignment Table A 63 Dword13 Dword14 Bit Description 63 00 Capacity of the entire SSD unit byte A 6 1 2 3 Recovering Data This command is used to try to recover the SSD from error status or restore data in error status This command uses the Dword12 and Dword13 fields Enter reserved value 0 f...

Page 195: ...cription 0x010000h Configure the NN Value This command involves controller based access and can be run only in PF mode This command uses the Dword13 field to indicate the NN value to be set The NN value ranges from 1 to 64 Enter reserved value 0 for fields that are not involved Table A 67 Command Dword13 Bit Description 31 00 NN Value NN value to be set ranging from 1 to 64 A 6 1 2 5 Setting SSD C...

Page 196: ...n are historical versions and provide the same functions as the New Commands These interfaces are reserved for compatibility with earlier versions For ES3000 V5 and later SSDs the interfaces described in New Commands are recommended A 6 2 1 Query Command This command uses the PRP1 PRP2 and Dword10 to Dword15 fields Enter reserved value 0 for other fields that are not involved Table A 70 PRP1 Bit D...

Page 197: ...ined log length 25h Query the PCB version 29h Obtain the self diagnosis result 2Ah Obtain the chipset DIE ID 57h Obtain the average DWPD value 59h Obtain the unique ID 157h Obtain the pending list A 6 2 1 1 Obtaining Logs and Statistical Information This command is used to obtain vendor logs and statistics information from the SSD Each command sent from the host can obtain a maximum of 4KB of logs...

Page 198: ...0 Data_Len The length of data transmission is in the unit of Dword If the length of the package is not 4 byte aligned then round up the result The actual returned length will be shown in the Dword0 of CQ Table A 77 Command Dword 13 Bit Description 31 00 Offset Log offset 1 indicates 4 KB The valid log length and offset are returned to the host through Dword0 of the CQ Table A 78 Completion Queue E...

Page 199: ... label information The subcommand is used to set the electronic label of the SSD The host uses PRP1 and PRP2 to send electronic label information in the format described in Table 2 Electronic label information In the following table Field indicates the combination of the field name equal sign field value and carriage return For example VendorName Huawei r n is a string where VendorName is the fiel...

Page 200: ...uct model number BarCode S N r n 30 20 SN on the paper label Item BBO M Code r n 30 23 PN on the paper label Description English Description r n 260 246 The line contains a maximum of 260 bytes Manufacture d Manufact uring Date r n 25 10 Format YYYY MM DD VendorName Manufactu rer Name r n 50 37 FRUs made by Huawei or Huawei s outsourced vendors are all branded by Huawei IssueNumber Issue Number r ...

Page 201: ... described in Table3 Table A 81 PN Bytes Description 23 00 SN l WWN The subcommand is used to set the SSD WWN The host uses PRP1 and PRP2 to send WWN information in the format described in Table4 WWN Table A 82 WWN Bytes Description 15 00 WWN l Model The subcommand is used to set the SSD model number The host uses PRP1 and PRP2 to send model information in the format described in Table5 Model Tabl...

Page 202: ...he controller The PF and VF have the permission to run this command based on controller access This command uses the PRP1 PRP2 Dword10 and Dword12 fields Enter reserved value 0 for other fields that are not involved Sub_OpType indicates the function type Table A 86 Sub_OpType Sub_OpType Description 01h Flash inspection enablement 02h Capacitor inspection enablement 03h DDR inspection enablement 04...

Page 203: ...information to the host through PRP1 and PRP2 The format of the returned data is as follows Table A 88 Querying the maximum available capacity of the SS Bytes Description 7 0 Maximum available capacity of the SSD unit byte A 6 2 1 5 Obtaining the Vendor Log Length This command is used to obtain the length of vendor logs including the log parsing package full logs and key logs This command uses the...

Page 204: ...CQ Bit Description 31 0 PCB version A 6 2 1 7 Obtaining Self Diagnosis Results If a bit value is 1 the corresponding item has a fault If a bit value is 0 the corresponding item is OK This command uses the PRP1 PRP2 Dword10 and Dword12 fields Enter reserved value 0 for other fields that are not involved Table A 92 CQ Dword0 Bit Description 0 Overall test result 1 Firmware error test front end error...

Page 205: ...1 Data loss risks exist 27 1 SSD disks can be restored but risks exist 28 1 SSD disks can be restored 29 1 Forcibly restore SSD disks so that the SSD disks can function properly 30 1 Restart or power on and then off SSD disks to restore SSD disks 31 1 The fault cannot be rectified Contact Huawei support engineers A 6 2 1 8 Obtaining the DIEID of a Chip This command is used to obtain the DIEID whic...

Page 206: ...e A 94 Command Dword13 Bit Description 31 00 Number of days valid value 1 360 Table A 95 Cq Command Dword0 Bit Description 31 00 DWPD value 100 indicates 1 If the average daily written data volume is 0 8 TB and the drive user volume is 1 6 TB the DWPD value is 0 5 and the interface result is 50 A 6 2 1 10 Querying UniqueID Information This interface is used to obtain the UniqueID information The h...

Page 207: ... The UniqueID content is transmitted by the memory that PRP points to and the content is in the string format Table A 98 Command Dword10 Bit Description 31 00 Length The unit is Dword A 6 2 2 Configuration Commands This command uses the PRP1 PRP2 and Dword10 to Dword15 fields Enter reserved value 0 for other fields that are not involved Table A 99 PRP1 Bit Description 63 00 PRP Entry 1 PRP1 indica...

Page 208: ...ransfer is not needed the value is 0h Table A 102 Command Dword 12 Bit Description 31 16 Sub_OpType subcommand operation type 15 00 Sub_OpCode subcommand number This command currently supports the following Sub_Opcodes Table A 103 Sub_Opcode Sub_OpCode Description 01h Delete vendor defined logs and information 02h Configure the SSD capacity 1Ah Recover data 1Dh NN value A 6 2 2 1 Deleting SMART In...

Page 209: ...o recover data of the SSD The recovery may take up to 20 minutes Before recovering data run the hioadm info d devicename a command and verify that data status in the command output is need recovery This command uses the Dword12 field Enter reserved value 0 for fields that are not involved A 6 2 2 4 Configuring the Number of Namespaces This command is used to configure the number of namespaces NN f...

Page 210: ...rts out of band management contact the server vendor B 1 NVMe Basic Management Command The I2C address of NVMe out of band management commands is 0x6A excluding the R W bits Table B 1 NVMe basic management data structure Command Code Offset Byte Description 0 00 Length of Status Indicates number of additional bytes to read before encountering PEC This value should always be 6 06h in implementation...

Page 211: ...ration A controller reset is required to resume normal operation Port 0 PCIe Link Active Bit 3 is set to 1 to indicate the first port s PCIe link is up i e the Data Link Control and Management State Machine is in the DL_Active state If cleared to 0 then the PCIe link is down Port 1 PCIe Link Active Bit 2 is set to 1 to indicate the second port s PCIe link is up If cleared to 0 then the second port...

Page 212: ...presented in twos complement 59 C to 1 C 74 2 F to 30 2 F 04 Percentage Drive Life Used PDLU Contains a vendor specific estimate of the percentage of NVM subsystem NVM life used based on the actual usage and the manufacturer s prediction of NVM life A value of 100 indicates that the estimated endurance of the NVM in the NVM subsystem has been consumed but may not indicate an NVM subsystem failure ...

Page 213: ...byte The default value is 0 34 Current power consumption Unit W 35 Reserved byte The default value is 0x80 36 Reserved byte The default value is 0 37 Reserved byte The default value is 0 38 Reserved byte The default value is 0 46 39 Firmware Version 8 byte firmware version number in ASCII code 54 47 Bootloader Version 8 byte bootloader version number in ASCII code 55 PEC An 8 bit CRC calculated ov...

Page 214: ...beginning of each page to prevent overwriting data The ES3000 V5 VPD information meets the requirements of the NVME MI1 0 protocol Table B 3 VPD definition Byte Name 07 00 Common Header 119 08 Product Info Area Vendor Specific 120 MultiRecord Info Area Vendor Specific Internal Use Area not supported Vendor Specific Internal Use Area not supported Vendor Specific Internal Use Area not supported Tab...

Page 215: ...o 256 and then taking the 2 s complement of this sum When the checksum and the sum of the bytes module 256 are added the result should be 0h Table B 5 NVMe MultiRecordArea Byte Factory Default Description 00 0Bh NVMe Record Type ID 01 2h Bit 7 end of list record format version 2 02 28h Record Length RLEN This field indicates the length of the MultiRecord Area in bytes 03 Impl Spec Record Checksum ...

Page 216: ...ndard Height Half Length FHHL 35 CEM add in card Standard Height Full Length FHFL 36 47 Reserved 48 M 2 module unknown 49 M 2 module 2230 50 M 2 module 2242 51 M 2 module 2260 52 M 2 module 2280 53 M 2 module 22110 54 63 Reserved 64 BGA SSD unknown 65 239 Reserved 240 255 Vendor Specific 12 07 00h Reserved 13 00h Initial 1 8V Power Supply Requirements This field specifies the initial 1 8V power su...

Page 217: ...ssage 20 00h Maximum 5V Power Supply Requirements This field specifies the maximum 5V power supply requirements in Watts A value of zero indicates that the power supply voltage is not used 21 19h Initial 12V Power Supply Requirements This field specifies the initial 12V power supply requirements in Watts prior to receiving a Set Slot Power message 22 19h Maximum 12V Power Supply Requirements This ...

Page 218: ...s multirecord This field shall be set to zero in this version of the specification 06 00h PCIe Port Number This field contains the PCIe port number This is the same value as that reported in the Port Number field in the PCIe Link Capabilities Register 07 01h Port Information This field indicates information about the PCIe Ports in the device Bits 7 to 1 are reserved Bit 0 if set to 1 indicates tha...

Page 219: ...Reserved 16 PCIe x16 17 31 Reserved 32 PCIe x32 33 255 Reserved 10 00h MCTP Support This field contains a bit vector that specifies the level of support for the NVMe Management Interface Bits 7 to 1 are reserved Bit 0 if set to 1 indicates that MCTP based management commands are supported on the PCIe port 11 01h Ref Clk Capability This field contains a bit vector that specifies the PCIe clocking m...

Page 220: ...ts 1 to 12 represent temperature values If bit 11 represents 128ºC 262 4 F bit 1 represents 0 125ºC 32 225 F Convert the read values into temperature values by referring to Table B 7 Table B 7 Read data format of the 0x05 register Binary Temperature xxx0 0111 1101 000x 125 C 257 F xxx0 0110 0011 110x 99 75 C 211 55 F xxx0 0101 0101 000x 85 C 185 F xxx0 0010 0111 000x 39 C 102 2 F xxx0 0000 1111 11...

Page 221: ...efore exercise caution when using the utility The ES3000 V5 provides the SNMP extension agent to support the SNMP function The extension defines the device management information base MIB SNMP clients can query device status through the SNMP extension agent C 2 Configuring the SNMP Function in Windows Preparations Before configuring the SNMP function check that l You have correctly installed the E...

Page 222: ... SNMP software version by running the following command root snmpd v l You have determined the directory storing the snmpd conf file as follows root net snmp config snmpconfpath usr local share snmp Installing the Software Step 1 Log in to the OS as the root user Step 2 Access the software package directory such as the root directory Step 3 Install the Linux driver and tool package For details see...

Page 223: ...e the following content assuming that the Net SNMP version is 5 7 dlmod es3000V3 usr lib hioadmsnmp_5 7 so End C 4 Querying SNMP Information This section describes the software for querying ES3000 V5 SNMP information and where to obtain the software Using the Net SNMP Command Set You need to install net snmp utils before using the Net SNMP command set to query ES3000 V5 SNMP information You can us...

Page 224: ...prise 2011 2 319 3 1 1 1 Gauge32 1 enterprise 2011 2 319 3 1 2 1 STRING nvme0 enterprise 2011 2 319 3 1 3 1 INTEGER 1 enterprise 2011 2 319 3 1 4 1 STRING 024YVJ10J4800386 enterprise 2011 2 319 3 1 5 1 STRING HWE52P432T0L002N enterprise 2011 2 319 3 1 6 1 STRING 1530 enterprise 2011 2 319 3 1 7 1 STRING 1 3 enterprise 2011 2 319 3 1 8 1 STRING healthy enterprise 2011 2 319 3 1 9 1 STRING no warnin...

Page 225: ...9 3 1 7 NVMe version number OCTET STRING Read only deviceStatus 1 3 6 1 4 1 2011 2 319 3 1 8 Device health status OCTET STRING Read only criticalWarning 1 3 6 1 4 1 2011 2 319 3 1 9 Alarm information OCTET STRING Read only compositeTempe rature 1 3 6 1 4 1 2011 2 319 3 1 10 Overall temperature OCTET STRING Read only maxCapacity 1 3 6 1 4 1 2011 2 319 3 1 11 Maximum capacity Unsigned32 Read only cu...

Page 226: ...tus OCTET STRING Read only powerState 1 3 6 1 4 1 2011 2 319 3 1 20 Power status OCTET STRING Read only writeMode 1 3 6 1 4 1 2011 2 319 3 1 21 Write mode OCTET STRING Read only NOTE The SNMP system creates a table instance for each SSD The table node OID is 1 iso 3 org 6 dod 1 internet 4 private 1 enterprises 2011 huawei 2 products 319 ES3000V5 3 cardTable ES3000 V5 NVMe PCIe SSD User Guide C SNM...

Page 227: ...erating system hardware and driver must meet the following requirements l Supported OSs Windows Server 7 SP2 Windows Server 10 Windows Server 2012 R2 and Windows Server 2016 RHEL 7 3 or later SLES 12 SP3 or later l Hardware V5 series CPUs are required To use the VROC function purchase the Intel VROC KEY and install it on the server For details about how to purchase and install the VROC KEY see the...

Page 228: ...d download the required software package After decompressing the software package view the corresponding user guide For Linux RSTe_NVMe_for_Linux_SW_User_Guide_xx pdf For Windows RSTe VROC Users Manual_xx pdf ES3000 V5 NVMe PCIe SSD User Guide D Using the VROC Function on the ES3000 V5 Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 218 ...

Page 229: ...ter Description Value device Name of a device Examples nvme0 and nvme1 Usage Guidelines Run this command under the opt hio directory Generated logs will be saved in this directory The structure of the opt hio directory is as follows opt hio 20160615 nvme0_coredump_172050 log nvme0_fw_172050 log nvme0_fw_172050 tmp nvme0_key_172050 log nvme0_key_172050 tmp The tmp files contain original log data Th...

Page 230: ...ading log coredump 39 100 done Parsing log fw 100 done Parsing log key 100 done opt hio 20181224 nvme1_fw_164009 log opt hio 20181224 nvme1_key_164009 log opt hio 20181224 nvme1_coredump_164009 log ES3000 V5 NVMe PCIe SSD User Guide E Obtaining Device Logs by Using an Open Source Tool Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 220 ...

Page 231: ...0 exe file in the tool installation directory l If the NET Framework version is 4 0 or 4 5 double click the HioadmGUI_DotNet_4 0 exe file in the tool installation directory BaseInfo Page This page shows the basic information about controllers and namespaces If the namespace starts with NVMe an NVMe SSD disk is used as shown in Figure F 1 Figure F 1 BaseInfo page of NVMe ES3000 V5 NVMe PCIe SSD Use...

Page 232: ...es a namespace associated with the controller ns indicates a namespace not associated with any controllers See Figure F 2 Figure F 2 BaseInfo page of NS Figure F 3 BaseInfo page of ns ES3000 V5 NVMe PCIe SSD User Guide F Introduction to Windows GUI Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 222 ...

Page 233: ...th no firmware upgrade image selected on the ES3000 Figure F 5 Update page with a firmware upgrade image selected l select image Select an ES3000 or SAS HSSD firmware upgrade image provided by Huawei ES3000 V5 NVMe PCIe SSD User Guide F Introduction to Windows GUI Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 223 ...

Page 234: ...shown in Figure F 6 Figure F 6 NVMeDiskMgnt page Functions on this page are the same as the operations described in 6 4 Formatting 6 3 Querying and Setting the Capacity and 6 5 Safely Erasing User Data of an SSD If setting cap is set to 0 the current capacity is set to the original capacity SelfTest Page This page is used to diagnose the ES3000 as shown in Figure F 7 ES3000 V5 NVMe PCIe SSD User G...

Page 235: ...in 6 10 Querying the Diagnosis Result of a Specified SSD NS Mgnt Page This page is used to manage only NVMe namespaces as shown in Figure F 8 Figure F 8 NS Mgnt page ES3000 V5 NVMe PCIe SSD User Guide F Introduction to Windows GUI Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 225 ...

Page 236: ...city You can also attach the namespace to specified controllers during the creation See Figure F 9 l Delete a namespace l Attach a namespace to controllers l Detach a namespace from controllers Figure F 9 Information about NS1 DevMgnt Page This page is used to manage the ES3000 as shown in Figure F 10 ES3000 V5 NVMe PCIe SSD User Guide F Introduction to Windows GUI Issue 07 2019 03 19 Copyright Hu...

Page 237: ...storical temperature number of overtemperature occurrences and overtemperature duration l threshold setting Set temperature thresholds l controller View controller temperature information including the highest historical temperature number of overtemperature occurrences and overtemperature duration ES3000 V5 NVMe PCIe SSD User Guide F Introduction to Windows GUI Issue 07 2019 03 19 Copyright Huawe...

Page 238: ... the earlier version has a bug Therefore you need to upgrade the kexec tools to a version later than 2 0 0 300 To install Windows on the ES3000 V5 follow the procedure in the section because there are some special requirements If you format the ES3000 V5 into 4 KB sectors by following the instructions in 6 4 2 Formatting an SSD create partitions manually when installing a Windows OS For details ab...

Page 239: ...Bit on the ES3000 V5 Step 1 Download the hiodriver_2012_x86_64 imgdriver image and mount it to the virtual floppy drive of the server Step 2 During the installation select Load driver on the screen shown in Figure G 2 ES3000 V5 NVMe PCIe SSD User Guide G Installing an OS on the ES3000 Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 229 ...

Page 240: ...Selecting Load driver Step 3 Click Next to install the driver as shown in Figure G 3 ES3000 V5 NVMe PCIe SSD User Guide G Installing an OS on the ES3000 Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 230 ...

Page 241: ...4 Figure G 4 Selecting the ES3000 V5 If the ES3000 V5 has previous partition information such as OS data the installation may fail You can perform the following to prevent or solve the installation failure ES3000 V5 NVMe PCIe SSD User Guide G Installing an OS on the ES3000 Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 231 ...

Page 242: ...d press Enter as shown in Figure G 6 Figure G 6 Running the diskpart command Step 7 Enter select disk indicates the number of the ES3000 V5 drive for OS installation and press Enter as shown in Figure G 7 ES3000 V5 NVMe PCIe SSD User Guide G Installing an OS on the ES3000 Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 232 ...

Page 243: ...ng Windows Server 2012 R2 64 Bit on the ES3000 V5 Windows Server 2012 R2 is built with the NVMe driver Therefore the driver image does not need to be mounted for OS installation Note that the ES3000 V5 is identified as 64 drives during the OS installation Select the drive with a total size of greater than 0 for installing the OS ES3000 V5 NVMe PCIe SSD User Guide G Installing an OS on the ES3000 I...

Page 244: ... the installation may fail You can solve the problem in the same way as that for installing Windows Server 2012 If the INACCESSIBLE_BOOT_DEVICE blue screen error is displayed due to other operations perform the following procedure provided by Microsoft Mount the Windows Server 2012 R2 image and hiodriverhiodriver_v _2012R2_x84_64 img file Boot from the CD ROM Select Repair your computer Select Tro...

Page 245: ...port Huawei technical support engineers may assist you in performing some operations to further collect fault information or rectify the fault Before contacting technical support engineers prepare the spare parts for boards and port modules screwdrivers screws serial cables and network cables H 3 Using Product Documentation Huawei provides the documents delivered with the equipment This document p...

Page 246: ...ut products l Huawei Enterprise Support Community Servers for learning and discussion News For notices about product life cycles warnings and updates visit Product Bulletins Cases Learn about server applications at Knowledge Base Huawei Technical Support If a fault persists after taking the above measures obtain technical support in the following ways l Contact Huawei customer service center Enter...

Page 247: ...and correcting H HHHL Half height Half length I IOPS I O operations per second L LBA Logical Block Addressing N NVMe NVM Express M MLC multi level cell MTBF mean time between failures MIB Management Information Base ES3000 V5 NVMe PCIe SSD User Guide I Acronym or Abbreviation Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 237 ...

Page 248: ...y of independent disks S SNMP Simple Network Management Protocol SSD solid state drive V VROC Intel Virtual RAID on CPU ES3000 V5 NVMe PCIe SSD User Guide I Acronym or Abbreviation Issue 07 2019 03 19 Copyright Huawei Technologies Co Ltd 238 ...

Search results

Summary of Contents for ES3000 V5

Reviews:

Related manuals for ES3000 V5

Brands by name

Popular brands

Huawei ES3000 V5, User Manual

Search results

Summary of Contents for ES3000 V5

Reviews:

Related manuals for ES3000 V5

Brands by name

Popular brands