manualshive.com logo in svg

Inventec P47 series, Service Manual

Share
Download
Inventec P47 series Service Manual Download Page 106

 

K800G4 BMC Specification 

6 

 

3.3

 

  Watchdog Event Logging 

Any BMC watchdog timer expired would result in a SEL event being generated to indicate the watchdog timer timeout 
occurred and which timer (BMC hardware, OS or FRB2) caused the timeout action. 
 

3.4

 

  FRB2 Timeout Detection 

The FRB2 timer is implemented with IPMI Watchdog Timer. The BMC supports BMC Watchdog Timer Commands. The 
following is the behavior of FRB2 timer. 
 

3.5

 

  Started From System Reset 

a.

 

BIOS tell BMC to enable Watchdog Timer of FRB2. And BIOS continues the POST process. The Watchdog Timer of 
FRB2 is set to 480 sec (set by BIOS).   

b.

 

During the execution of POST, in some circumstances, BIOS is forced to stop the Watchdog Timer for FRB2 for a while. 
There are 2 cases the FRB2 timer could be paused by BIOS: 

 

Upon enter BIOS Setup menu. 

 

Option ROM initial, etc… (It can be customized to tune the initialed time.) 

c.

 

In order to prevent the Watchdog Timer for FRB2 from expiring, BIOS has to reset the watchdog countdown within 
appropriate time interval. Finally, BIOS executes its last POST task (INT 19h). Then BIOS has to disable the Watchdog 
Timer of FRB2. 

 

3.6

 

  Started From System Warm Reboot 

No matter system cold reset or warm reset, it would be monitored by a BIOS initiated watchdog timer. Their behavior is 
similar to the above descriptions. 
 

3.7

 

  When FRB2 Timeout Happened

 

If the Timer expired from FRB2 over 3 times, BMC will “Power down” the system as the policy to avoid system “Hard Reset” 
again and again. 

Sensor Type 

Event/Reading Type 

Sensor Specific Offset 

Event 

Watchdog2(0x23) 

Sensor-specific(0x6F) 

0x01 

BIOS/FRB2 timeout 

generated hard reset 

Event Data 1 

Event Data 2 

Event Data 3 

 

0xC1 

[3:0]Timer use = 0x01 

BIOS/FRB2 

0xFF 

 

 

3.8

 

  Watchdog Timer 

BIOS code will be executed when CPU ready. So if BIOS wouldn’t tell BMC code is fetched in 20 seconds, it causes the 
watchdog timer expired. After this stage, BIOS needs to set FRB2 timer enable. After BIOS POST complete, BIOS also 
disable FRB2 timer. If user enables “OS load timer”, BIOS needs to enable “OS Load Timer” after FRB2. 

 

 

 

Summary of Contents for P47 series

Page 1: ...P N 2017 MNU 000016 June 2018 Revision A 2U Server System Service Manual P47 ...

Page 2: ... This manual provides the information in relation to the set up and installation of the product herein Nothing herein may be construed as granting any right or license relating to any intellectual property rights of this manual or product Unless otherwise provided in the Purchase and Sale Agreement for this product manufacturer and distributor of this product will not be liable whatsoever relating...

Page 3: ......

Page 4: ...out 1 6 1 6 3 Front View 1 8 1 6 4 Back View 1 9 1 6 5 Buttons and System LED Information 1 9 1 6 6 System Thermal Solution 1 12 2 Hardware Operations 2 1 2 1 Before You Start 2 1 2 1 1 Power Off 2 1 2 2 Rear Chassis Cover 2 2 2 2 1 To remove the rear chassis cover 2 2 2 2 2 To install the rear chassis cover 2 3 2 3 Front Top Cover 2 5 2 3 1 To remove the front top cover 2 5 2 3 2 To install the f...

Page 5: ...2 14 OCP Card Optional 2 28 2 14 1 To remove the OCP card 2 28 2 14 2 To install the OCP card 2 29 2 15 Expansion Cards 2 29 2 15 1 To remove the expansion card 2 30 2 15 2 To install the expansion card 2 31 2 16 Fan Duct 2 32 2 16 1 To remove the fan duct 2 32 2 16 2 To install the fan duct 2 33 3 Connectors 3 1 3 1 Backplane Connectors 3 1 3 2 OCP Card Connectors 3 2 4 Cable Routing 4 1 Appendix...

Page 6: ...About This Manual Conventions Safety Symbols Safety Precautions Regulatory and Integration Information Rack Mount Instructions ...

Page 7: ...the following symbols used throughout this manual Warning Information to prevent injury to yourself when trying to complete a task Caution Information to prevent damage to the components when trying to complete a task Important Information that you must follow to complete a task Note Tips and information to aid in completing a task ...

Page 8: ...ing To reduce the risk of injury from electric shock hazards do not open this enclosure Indicates the presence of a hot surface or hot component Warning To reduce the risk of injury from a hot component allow the surface to cool before touching it Any RJ45 receptacle marked with this symbol indicates a network interface connection Warning To reduce the risk of electric shock fire or damage to the ...

Page 9: ...the server make sure that all cables are correctly connected and power cords are not damaged If any damage is detected contact your dealer as soon as possible To avoid short circuits keep paper clips screws and staples away from connectors slots sockets and circuitry Before opening the chassis panels make sure all power cords are unplugged Avoid dust humidity and extreme temperatures place the ser...

Page 10: ...for those devices are unplugged before the signal cables are connected If possible disconnect all power cords from the existing system before you add a device Use one hand when possible to connect or disconnect signal cables to prevent a possible shock from touching two surfaces with different electrical potentials This product is equipped with a three wire power cable and plug for user safety Use...

Page 11: ...s evaluated as Information Technology Equipment ITE which may be installed in server rooms computer rooms and similar commercial type locations The suitability of this product for other product categories and environments such as medical industrial residential alarm systems and test equipment other than an ITE application may require further evaluation Do not attempt to recharge the battery Do not...

Page 12: ...y and is controlled by the authority responsible for the location Clean dry and free of airborne particles other than normal room dust Well ventilated and away from sources of heat including direct sunlight and radiators Away from sources of vibration or physical shock Isolated from strong electromagnetic fields produced by electrical devices In regions that are susceptible to electrical storms we...

Page 13: ...quipment EN 60950 1 Safety of Information Technology Equipment Including Electrical Business Equipment European Committee for Electrotechnical Standardization CENELEC UL 60950 1 Safety of Information Technology Equipment UL 94 Tests for Flammability of Plastic Materials for Parts in Devices Appliances GB4943 Safety of Information Technology Equipment Product EMC Compliance This product has been te...

Page 14: ...ctions may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at personal expense Declaration of Conformity for Products Marked with the FCC Logo United States Only This device complies with Part 15 of the FCC Rules Operation and is subject t...

Page 15: ...roved for use in that country The power cord must be rated for the product and for the voltage and current marked on the product s electrical ratings label The voltage and current rating of the cord should be greater than the voltage and current rating marked on the product In addition the cross sectional area of the wires must be a minimum of 1 00mm or 18AWG and the length of the cords must be be...

Page 16: ...ma specified by the manufacturer Reduced Air Flow Installation of the equipment in a rack should be such that the amount of air flow required for safe operation of the equipment is not compromised Mechanical Loading Mounting of the equipment in the rack should be such that a hazardous condition is not achieved due to uneven mechanical loading Circuit Overloading Consideration should be given to th...

Page 17: ...Audience Assumptions Manual Organization Packing Checklist Specifications Product Features System Overview Chapter 1 Introduction ...

Page 18: ...rdware Operations The operation of the components on the chassis such as power supply power distribution board system fans backplane and riser card Connectors Information about connectors on the various boards in the system Cable Connections How to connect cables correctly Appendix China RoHS Regulations information 1 3 Packing Checklist Make sure you have all the components shipped with your syst...

Page 19: ... Length 73 66cm Weight Max Weight 24 96KG Temperature Operating System 5 C 35 C Non operating System 40 C 70 C Humidity Operating System 20 80 Non operating System 10 90 Voltage 100 240VAC input 1600Watt 1 power supply 100 240VAC input 3200Watt 2 power supplies Current 10 4A 1 power supply 20 8A 2 power supplies 1 5 Product Features Figure 1 1 Product Introduction with OCP Card ...

Page 20: ...assive Backplanes System Fan Number of fan cage 6 Processor 1x AMD Naples SP3 processor Up to 150W 200W thermal design power TDP Up to 4 Die 8 core per Die Up to 8 16 bit IO links System Memory 16x DDR4 DIMM slots 8 channels per processor Supports ECC registered DIMM RDIMM at 1866 2133 2400 2667MHz BMC Aspeed AST2500 BMC Onboard Storage Ports Up to 8x SATA 6Gb s ports 2x M 2 NVMe Connecors 2x U 2 ...

Page 21: ...out Figure 1 3 System Overview with OCP Card 1 Fan Duct 2 System Fans 3 2 5 HDD Backplane 4 2 5 HDD Bays 5 Front Panel 2 6 Front Panel 1 7 OCP Card Optional 8 Motherboard 9 Expansion Card Assembly of GPU 3 4 10 Expansion Card Assembly of GPU 2 11 Expansion Card Assembly of GPU 1 12 Power Supply 1 13 Power Supply 2 ...

Page 22: ...thout OCP Card 1 Fan Duct 2 System Fans 3 2 5 HDD Backplane 4 2 5 HDD Bays 5 Front Panel 2 6 Front Panel 1 7 Motherboard 8 Expansion Card Assembly of GPU 3 4 9 Expansion Card Assembly of GPU 2 10 Expansion Card Assembly of GPU 1 11 Power Supply 1 12 Power Supply 2 ...

Page 23: ...ayout of the motherboard is shown below Each connector and major components are identified by number Figure 1 5 Connectors and Component Locations of Motherboard 1 Management Port J34 2 VGA Connector J35 3 UID Button SW2 4 Serial Port J67 5 Rear Single USB Port J33 ...

Page 24: ...ctor G2 J19 21 PCI E x8 Slimline Connector G3 J20 22 HDD Backplane Power Connector J53 23 SATA Slimline Connector Port 1 G2 J71 24 AMD HDT connector J47 25 Fan Connector 1 J37 26 Fan Connector 2 J38 27 Fan Connector 3 J39 28 Fan Connector 7 J98 29 DIMM Slots J3 J61 J5 J60 J6 J7 J8 J9 30 Fan Connector 4 J40 31 Front Panel USB Connector J57 32 PCI E x8 Slimline Connector G1 J24 33 Fan Connector 5 J4...

Page 25: ...d 1 Front Panel 1 2 QSFP Port 1 3 QSFP Port 2 4 HDDs 5 Front Panel 2 Figure 1 7 Front View 42 OCP Power Connector J99 43 PCI E x8 Slimline Connector G0 J27 44 PCI E x8 Slimline Connector G0 J32 45 PCI E x8 Slimline Connector G1 J26 46 PCI E x4 Slimline Connector P0 J31 47 System Battery BH1 48 RTC Connector J56 49 CPLD JTAG connector J55 50 M 2 Connector J51 51 M 2 Connector J72 52 PCI E X16 Gen3 ...

Page 26: ...equipped with system LED indicators and buttons located on the front panels The front panel status LEDs allow constant monitoring of basic system functions while the server is operating These LEDs provide visual cues to the status of power and ID of motherboard 1 AC Power Connector 2 2 AC Power Connector 1 3 Rear Single USB Port 4 QSFP Port 1 5 QSFP Port 2 6 Serial Port For BMC debug use only 7 UI...

Page 27: ...th OCP Card Figure 1 10 Front Panel Buttons and LEDs 1 ID Button LED 2 Power Button LED 3 Reset Button 4 USB 3 0 Port 0 5 USB 3 0 Port 1 Figure 1 11 Back View LEDs 1 AC Power LED 2 2 AC Power LED 1 3 UID LED 4 Speed LED of Management Port 5 Link Activity LED of Management Port ...

Page 28: ...h LED Green On System is ready and in normal operation Red On Non recoverable failure e g Non recoverable temperature voltage threshold VRD hot asserted Minimum number of fans does not present or failed It could last even when power off Blinking Critical alarm e g Critical temperature voltage threshold Power fault It could last even when power off Off AC power is disconnected or system power off i...

Page 29: ...t 0 1 Green On Link at first level speed Amber On Link at others speeds Off No link Link Activity LED of Base T port 0 1 Green On Link ON without accessing Blinking Link ON with accessing Off No link 1 6 6 System Thermal Solution This server provides a thermal solution to keep proper cooling The components in the following figure must be installed in place Figure 1 12 Thermal Solution 1 Fan Duct ...

Page 30: ...2 Hardware Operations Before You Start Rear Chassis Cover Front Top Cover Cable Guide Stiffener Motherboard Power Supplies System Fans 2 5 HDDs 8x2 5 HDD Backplane Front Panels OCP Card Expansion Cards Fan Duct ...

Page 31: ...n Press the power button to toggle the server to standby The power LED in green turns off Figure 2 1 Pressing the Power Button The components shown in this chapter are mainly for your reference Please take the actual shipment as standard To reduce the risk of injury from electric shock remove the power cord to completely disconnect power from the system Moving the Power On Off switch to the Off po...

Page 32: ...and disassembly making the replacement of internal components very convenient Reminder Before you remove or install the rear chassis cover please follow the step below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off 2 2 1 To remove the rear chassis cover Release the screw on the chassis cover Push the retaining clip along the di...

Page 33: ...move the cover from the chassis Figure 2 4 Removing the Chassis Cover 2 2 2 To install the rear chassis cover Locate the chassis cover onto the chassis Simultaneously pull up the retaining clip completely with the biggest angle and align the locking tabs on the cover to the corresponding notches on the chassis Figure 2 5 Locating the Cover onto the Chassis ...

Page 34: ...ip and simultaneously the cover automatically slides back into place Figure 2 6 Secure the retaining clip Secure the chassis cover with one screw Figure 2 7 Tightening the Screw This system must be operated with the chassis cover installed to ensure proper cooling ...

Page 35: ...ll the front top cover please follow the step below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover 2 3 1 To remove the front top cover Press down the retaining clip Remove the front top cover along the direction of the arrow Figure ...

Page 36: ... with the locking tabs on the cover snap into the corresponding notches on the chassis Figure 2 9 Installing the Front Top Cover 2 4 Cable Guide The locations of cable guide on the server is shown below Figure 2 10 Cable Guide Location This system must be operated with the chassis cover installed to ensure proper cooling ...

Page 37: ...tep 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the fan duct To remove the fan duct see 2 16 Fan Duct Step 4 Disconnect all necessary cables 2 4 1 To remove the cable guide Remove the two screws that secure the cable guide Lift the cable guide out of the chassis Figure 2 11 Removing the Cable Guide 2 4 2 To install the cable guide Rever...

Page 38: ...eps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the fan duct To remove the fan duct see 2 16 Fan Duct 2 5 1 To remove the stiffener Release the screws Lift the stiffener out of the chassis Figure 2 13 Removing...

Page 39: ...ollow the steps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the fan duct To remove the fan duct see 2 16 Fan Duct Step 4 Remove the stiffener To remove the stiffener see 2 5 Stiffener Step 5 Remove the cable g...

Page 40: ...ift the system fan assembly out of the chassis Figure 2 15 Removing the System Fan Assembly Remove the 8 screws that secure the motherboard Remove the motherboard out of the chassis along the direction of the arrow Figure 2 16 Removing the Motherboard 2 6 2 To install the motherboard Reverse the steps above to install the motherboard ...

Page 41: ...eps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the fan duct To remove the fan duct see 2 16 Fan Duct Step 4 Remove the stiffener To remove the stiffener see 2 5 Stiffener Step 5 Remove the cable guide To remo...

Page 42: ...eat Sink 2 7 2 To install the heat sink Reverse the steps above to install the heat sink Before you put the heat sink on top of the installed processor please do not forget to check if the grease is complete on bottom of the heat sink The heat sink used in the figure below is just for your reference Please choose an appropriate heat sink depending on the whole system requirement ...

Page 43: ...screw marked with 2 around it Remove the screw marked with 1 around it Figure 2 3 Releaing the Force Frame Open the force frame Hold the two clips of the rail frame and lift them up Figure 2 4 Opening the Force Frame To release the force frame you must remove the screws following the sequence of 3 2 and 1 marked around them ...

Page 44: ... the rail frame Figure 2 6 Removing the CPU Package 2 7 4 To install the processor Reverse the steps above to install the processor When installing the CPU package make sure to hold the handle but not the processor To secure the force frame you must install the screws following the sequence of 1 2 and 3 marked around them ...

Page 45: ...ore you remove or install the system memory please follow the steps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the fan duct To remove the fan duct see 2 16 Fan Duct Step 4 Remove the stiffener To remove the s...

Page 46: ...re Operations 2017 MNU 000016 2 16 There are sixteen DIMMs on the motherboard to support the processor The DIMM sequence of the sixteen DIMM sockets is respectively shown below Figure 2 8 DIMM Socket Sequence ...

Page 47: ...the retaining clips outward This action releases the module and partially lifts it out of the socket Lift out the DIMM Figure 2 9 Lifting the DIMM out of the Socket 2 8 2 To install the system memory Unlock a DIMM socket by pressing the retaining clips outward Figure 2 10 Pressing the Retaining Clips Outward ...

Page 48: ...gure 2 11 Inserting the DIMM into the Socket DIMMs fit in only one direction DO NOT force a DIMM into the socket to avoid damaging the DIMM 2 9 Power Supplies This server is designed with single 1600W 2000W power supply or two 1600W 2000W redundant power supplies with system throttling mode The location of power supplies on the server is shown below Figure 2 17 Power Supply Locations ...

Page 49: ...e of the power supply along the direction of the arrow Pull down the power supply handle At the same time pull out the power supply The power supply takes considerable force to remove Figure 2 18 Removing the Power Supply 2 9 2 To install the power supply Insert the replacement power supply firmly into the bay The retaining clip should snap Connect the AC power cord to the replacement power supply...

Page 50: ...emperature for the motherboard backplane and disk drives The location of system fans is shown below Figure 2 20 System Fan Locations Reminder Before you remove or install the system fan please follow the steps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis co...

Page 51: ...tem fans Unlock the system fan by clamping the locking clip inward Take the system fan out of the system fan cage Figure 2 21 Removing the System Fan 2 10 2 To install the system fans Place the system fan into the system fan cage Figure 2 22 Installing the System Fan ...

Page 52: ...or installing a HDD The location of the 2 5 HDDs on the server is shown below Figure 2 23 2 5 HDD Locations 2 11 1 To remove the 2 5 HDD Push the release button Pull the lever open Slide the HDD assembly out of the HDD bay Figure 2 24 Sliding out the 2 5 HDD Assembly Take note of the drive tray orientation before sliding it out The tray will not fit back into the bay if inserted incorrectly ...

Page 53: ...Hardware Operations 2017 MNU 000016 2 23 Lift the HDD out of the tray Figure 2 25 Removing the HDD 2 11 2 To install the 2 5 HDD Place the HDD to the HDD tray Figure 2 26 Placing the HDD ...

Page 54: ...e 8x2 5 HDDs backplanes which can support up to eight 2 5 HDDs with adapter bracket in the system The design incorporates a hot swappable feature to allow easy replacement of HDDs The SATA or SAS connectors on each backplane connect to the motherboard to provide power and indicate HDD access and failure The location of 8x2 5 HDDs backplane is shown below Figure 2 28 8x2 5 HDDs Backplane Location M...

Page 55: ...sis cover see 2 2 Rear Chassis Cover Step 3 Remove the front top cover To remove the front top cover see 2 3 Front Top Cover Step 4 Remove the HDDs To remove a HDD see 2 11 2 5 HDDs Step 5 Disconnect all the necessary cables 2 12 1 To remove the 8x2 5 HDDs backplane Remove the screws that secure the backplane Remove the backplane along the direction of the arrow Figure 2 29 Removing the 8x2 5 HDDs...

Page 56: ...30 Left Front Panel Location The location of right front panel on the server is shown below Figure 2 31 Right Front Panel Location Reminder Before you remove or install the front panel please follow the steps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off ...

Page 57: ...e front panel assembly Remove the front panel assembly from the chassis Figure 2 32 Removing the Front Panel Assembly Unscrew the front panel Remove the front panel Figure 2 33 Removing the Front Panel 2 13 2 To install the front panel Reverse the steps above to install the front panel ...

Page 58: ...he server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the front top cover To remove the front top cover see 2 3 Front Top Cover Step 4 Disconnect all the necessary cables 2 14 1 To remove the OCP card Unscrew the OCP card Remove the OCP card f...

Page 59: ...ies on the server are shown below Figure 2 36 Expansion Card Assembly Locations Reminder Before you remove or install the expansion cards please follow the steps below Step 1 Make sure the server is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover ...

Page 60: ...he expansion card Unscrew the expansion card assembly Lift the expansion card assembly out of the chassis Figure 2 37 Removing the Expansion Card Assembly Push the locking clip along the direction of the arrow to release the clip Figure 2 38 Releasing the Clip ...

Page 61: ...Unscrew the GPU Remove the GPU Install the slot covers Figure 2 39 Removing the GPU Remove the screw that secures the riser card Remove the riser card Figure 2 40 Removing the Riser Card 2 15 2 To install the expansion card Reverse the steps above to install the expansion card ...

Page 62: ...er is not turned on or connected to the AC power To power off the server see 2 1 1 Power Off Step 2 Remove the rear chassis cover To remove the rear chassis cover see 2 2 Rear Chassis Cover Step 3 Remove the stiffener To remove the stiffener see 2 5 Stiffener 2 16 1 To remove the fan duct Lift the fan duct out of the chassis Figure 2 42 Removing the Fan Duct Make sure that the fan duct on the serv...

Page 63: ...Hardware Operations 2017 MNU 000016 2 33 2 16 2 To install the fan duct Reverse the steps above to install the fan duct ...

Page 64: ...Backplane Connectors OCP Card Connectors Chapter 3 Connectors ...

Page 65: ...Connectors 2017 MNU 000016 3 1 3 Connectors 3 1 Backplane Connectors Figure 3 1 8x2 5 HDDs Backplane ...

Page 66: ... Connector 7 Slimline Connector 1 3 Slimline Connector 4 8 Backplane SATA Connectors 2 4 Slimline Connector 3 9 Backplane SATA Connectors 1 5 Backplane Power Connector 3 2 OCP Card Connectors Figure 3 2 OCP Card 1 QSFP Ports 3 Slimline Connector 1 2 Slimline Connector 2 4 NCSI Connector ...

Page 67: ...Chapter 4 Cable Routing ...

Page 68: ...Cable Routing 2017 MNU 000016 4 1 4 Cable Routing ...

Page 69: ...China RoHS Regulations Taiwan BSMI Electromagnetic Emissions Notices Appendix 1 ...

Page 70: ...Appendix 2017 MNU 000010 I Appendix 1 Hazardous Substances Free Regulations and Electromagnetic Emissions Notices China RoHS Regulations Figure I China RoHS Regulations ...

Page 71: ...Appendix 2017 MNU 000010 II Taiwan BSMI Figure II Taiwan BSMI ...

Page 72: ...st harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instructions may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be requir...

Page 73: ...ce Products bearing the CE marking comply with applicable EU Directives Compliance with such directives is assessed using applicable European Harmonized Standards Notices for Japan VCCI Notice Class A EMI Warning Message Power Cord Statement Notices for Korea Class A EMI Warning Message ...

Page 74: ...Appendix 2017 MNU 000010 V Notices for Taiwan BSMI Notices Class A EMI Warning Message Notices for Russia Class A EMI Warning Message ...

Page 75: ...Appendix 2 BIOS SPEC ...

Page 76: ...BIOS Setup User Manual K800QG4 rev 0 01 Mar 2018 ...

Page 77: ...BIOS Setup User Manual Table of Contents Table of Contents 1 Main 1 2 Advanced 2 3 Platform 8 4 Socket 10 5 BMC 19 6 Security 20 7 Boot 21 8 Save Exit 22 ...

Page 78: ...that BIOS image was created Platform Basic info of sytem platform Processor Basic info of system Processor Total Memory Total Memory size System Date Set the Date Use Tab to switch between Date elements Default Ranges Year 1998 9999 Months 1 12 Days dependent on month System Time Set the Time Use Tab to switch between Time elements ...

Page 79: ...hange Attempt Order Change Attempt Order Driver Health Driver Health status Trusted Computing Trusted Computing Settings Note Please remember to install TPM module in advance Security Device Support Enables or Disables BIOS support for security device O S will not show Security Device TCG EFI protocol and INT1A interface will not be available Enable Disable TPM State Enable Disable Security Device...

Page 80: ...ial Port Console Redirection Serial Port Console Redirection configuration Console Redirection Serial Port Console Redirection Enable Disable Note POST will not be full screen until this item is disabled Console Redirection Settings The settings specify how the host computer and the remote computer which the user is using will exchange data Both computers should have the same or compatible setting...

Page 81: ...Odd parity bit is 0 if num of 1 s in the data bits is odd Mark parity bit is always 1 Space Parity bit is always 0 Mark and Space Parity do not allow for error detection They can be used as an additional data bit None Even Odd Mark Space Stop Bits Stop bits indicate the end of a serial data packet A start bit indicates the beginning The standard setting is 1 stop bit Communication with slow device...

Page 82: ...ubsystem Settings PCI Settings Common for all Devices SR IOV Support If system has SR IOV capable PCIe Devices this option Enables or Disables Single Root IO Virtualization Support Enable Disable Network Stack Configuration Network Stack Settings Network Stack Enable Disable UEFI Network Stack Enable Disable Ipv4 PXE Support Enable Ipv4 PXE Boot Support If disabled IPV4 PXE boot option will not be...

Page 83: ...ted Enable Disable PXE boot wait time Wait time to press ESC key to abort the PXE boot 0 5 Media detect count Number of times presence of media will be checked 1 50 CSM Configuration CSM Configuration CSM Support Enable Disable CSM Support Enable Disable Boot option filter This option controls Legacy UEFI ROMs priority UEFI and Legacy Legacy only UEFI only Network Controls the execution of UEFI an...

Page 84: ...execution of UEFI and Legacy Video OpROM UEFI Legacy Note BIOS would no t display info to monitor under legacy OS if you select UEFI mode NVMe Configuration NVMe status and configuration USB Configuration USB devices and configuration USB Mass Storage Driver Support Enable or Disable USB mass storage driver support Enable Disable ...

Page 85: ...er On Last State PCH SATA Configuration PCH SATA Serial Advanced Technology Transport Configuration SATA Controller Enable or Disable SATA Controller Enable Disable Configure SATA as This will configure SATA as RAID or AHCI AHCI RAID PCH sSATA Configuration PCH sSATA Server Serial Advanced Technology Transport Configuration Miscellaneous Configuration Miscellaneous items configuration Active Video...

Page 86: ...ble Enable Server ME Configuration Intel ME Management Engine Configuration General ME Configuration Configure Server ME Technology Parameters Oper Firmware Version Version of operational firmware selected to run Current State ME firmware Current State bits 3 0 in MEFS1 Error Code ME firmware Error Code bits 15 12 in MEFS1 Recovery Cause Server ME firmware recovery cause bits 10 8 in MEFS2 ...

Page 87: ...able Disable Logical Processor threads Disable Enable Enable Intel R TXT Enables Intel R Trusted Execution Technology Disable Enable VMX Enables the Vanderpool Technology takes effect after reboot Disable Enable Enable SMX Enables Safer Mode Extensions Disable Enable Extended APIC Enable disable extended APIC support Disable Enable Force x2APIC IDs Force use of 8 bit APIC IDs Disable Enable AES NI...

Page 88: ...ble UPI Configuration Displays and provides option to change the UPI Settings UPI General Configuration UPI General Configuration UPI Status UPI Status Current UPI Link Frequency Intel Ultra Path Interconnect Intel UPI Link Frequency Memory Configuration Memory Configuration Memory Frequency Maximum Memory Frequency Selections in Mhz Do not select Reserved Auto 1866 2133 2400 2666 Enable ADR Async...

Page 89: ... Ras Settings Static Virtual Lockstep Mode Enable Static Virtual Lockstep mode Disable Enable Mirror mode Mirror Mode will set entire 1LM 2LM memory in system to be mirrored consequently reducing the memory capacity by half Mirror Enable will disable XPT Prefetch Disable Mirror Mode 1LM Mirror Mode 2LM Memory Rank Sparing Some memory ranks in each DDR channel can be set aside as spare memory durin...

Page 90: ... and leaky bucket 0 0x7FFF SDDC Plus One In SDDC Plus One mode BIOS copies the entire rank to the ECC DRAM device once it is in SDDC plus1 mode there is no error correction for the rank it is on error detection mode only Disable Enable SDDC Enable Disable SDDC single device data correction Disable Enable ADDDC Sparing ADDDC Adaptive Double Device Data Correction is an improved implementation of Le...

Page 91: ...rval is programmed using the P Code mail box command MAILBOX_BIOS_CMD_MC_PATROL_SCRUB_INTERVAL passing the scrub interval and mc number Disable Enable Patrol Scrub Interval Selects the number of hours 1 24 required to complete full scrub A value of zero means auto 0 24 NGN Configuration Displays and provides option to change the NGN settings NGNVM DIMM Secure Erase Unit Erases the persistent memor...

Page 92: ...election Auto Gen 1 2 5GT s Gen 2 5 GT s Gen 3 8 GT s PCI E Port Link Status PCIE link current status PCI E Port Link Max PCIE max width PCI E Port Link Speed PCIE link current speed Socket1 Configuration Configuration for 2nd socket IOAT Configuration I O Acceleration Technology I OAT Server consolidation requires large numbers of virtual machines VMs per physical server Intel I OAT helps ensure ...

Page 93: ...DCA DCA Enable Disable for this specific socket Disable Enable DMA Select DMA Direct memory access Enable Disable for each CB device Disable Enable No Snoop No Snoop Enable Disable for each CB device Disable Enable Intel VT for Directed I O VT d Intel VT for Directed I O VT d Configuration Intel VT for Directed I O VT d Enable Disable Intel Virtualization Technology for Directed I O VT d by report...

Page 94: ... in this Stack Disable Enable VMD for Volume Management Device on Socket 1 Intel VMD for Volume Management Device on 2nd socket Advanced Power Management Configuration Displays and provides option to change the Power Management Settings CPU P State Control P State Control Configuration Sub Menu include Turbo XE and etc SpeedStep Pstates Enable Disable EIST P States Disable Enable EIST PSD Function...

Page 95: ...nced Halt State C1E Core C1E auto promotion Control Takes effect after reboot Disable Enable Package C State Control Set the limit on the C State package register Package C State Package C State limit C0 C1 state C2 state C6 non Retention state C6 Retention state No Limit Auto Memory Power Thermal Configuration Displays and provides option to change the Memory Settings ...

Page 96: ... 2 Timer is disabled Do Nothing Reset Power Down Power Cycle View FRU information View FRU information BMC network configuration Configure BMC network parameters Configuration Address source Select to configure LAN channel parameters statically or dynamically by BIOS or BMC Unspecified option will not modify any BMC network parameters during BIOS phase Unspecified Static DynamicBmcDhcp DynamicBMCN...

Page 97: ...BIOS Setup User Manual Security 20 6 Security Admin passwaord user password setting for BIOS boot or enter setup Administrator Password Set Administrator Password User Password Set User Password ...

Page 98: ... Priorities Bootup NumLock State Select the keyboard NumLock state on off Quiet Boot Enables or disables Quiet Boot option Disable Enable Boot Option Priorities Sets the system boot order USB Devices BBS Priorities Set the order of the legacy devices in this group ...

Page 99: ...Reset the system after saving the changes Discard Changes and Reset Reset system setup without saving any changes Save Changes Save changes Discard Changes Discard changes Restore Defaults Restore Load Default values for all the setup options Save as User Defaults Save the changes done so far as User Defaults Restore User Defaults Restore the User Defaults to all the setup options Boot Override Fo...

Page 100: ...Appendix 3 BMC SPEC ...

Page 101: ...K800G4 BMC Specification 1 K800G4 BMC Specification June 22th 2018 ...

Page 102: ...7 6 LAN 8 6 1 The LAN Related Information 8 6 2 Default User Account 8 6 3 Default LAN Channel Configuration 8 7 System Event Log SEL 9 7 1 Time Synchronization 9 7 2 The Behavior when SEL is full 9 8 Serial Interface 10 8 1 Serial Over LAN SOL 10 8 2 SOL Information 10 8 3 SOL Configuration 10 8 4 The Behavior of SOL 10 9 RMCP IPMI v1 5 RMCP IPMI v2 0 and Payload Support 11 9 1 Payload Type List ...

Page 103: ...2 7 Sensor Device Commands 26 12 8 FRU Device Commands 26 12 9 SDR Device Commands 26 12 10 SEL Device Commands 27 12 11 LAN Device Commands 27 12 12 Serial Modem Device Commands 27 12 13 Command Forwarding Commands 27 12 14 Bridge Management Commands ICMB 27 12 15 Discovery Commands ICMB 28 12 16 Bridging Commands ICMB 8 28 12 17 Event Commands ICMB 8 28 12 18 OEM Commands for Bridge NetFn 28 12 ...

Page 104: ...m resides on the system main board It can keep monitoring the system status even at power off status If necessary it is possible for the BMC to reboot the system BMC is connected to the various sensors via IPMB Intelligent Platform Management Bus IPMB is essentially a subset of I2C bus System software could communicate with the BMC via KCS interface Keyboard Controller Style The detailed informati...

Page 105: ... means failed 0b means unknown 7 1b Cannot access SEL device 6 1b Cannot access SDR Repository 5 1b Cannot access BMC FRU device 4 1b IPMB signal lines do not respond 3 1b SDR Repository empty 2 1b Internal Use Area of BMC FRU corrupted 1 1b controller update boot block firmware corrupted 0 1b controller operational firmware corrupted 3 BMC Watchdog A watchdog timer WDT is a device that performs a...

Page 106: ... FRB2 from expiring BIOS has to reset the watchdog countdown within appropriate time interval Finally BIOS executes its last POST task INT 19h Then BIOS has to disable the Watchdog Timer of FRB2 3 6 Started From System Warm Reboot No matter system cold reset or warm reset it would be monitored by a BIOS initiated watchdog timer Their behavior is similar to the above descriptions 3 7 When FRB2 Time...

Page 107: ... far away Because of security consideration remote clients are required to login before sending IPMI commands Users from System Interface and IPMB are regarded as local users No user authentication is required for these two channels The following is a brief table of BMC supported channels For detailed implementation of LAN and Serial Modem channels refer to their respective chapters of IPMI v2 0 S...

Page 108: ...ser Account The BMC have no any constraints on user account such as password retention limits password strength etc follow IPMI spec User ID User Name Password Payload Access Privilege Notes Default state 1 NULL NULL CALLBACK NULL User Name Disabled 2 admin admin SOL ADMINISTRATOR Default administrator account Enabled 6 3 Default LAN Channel Configuration LAN parameter Value Notes IP address 0 0 0...

Page 109: ... to generate SMI to BIOS then BIOS would send the Set SEL Time command with the current system time to sync RTC timer c The BMC would automatically sync its RTC timer every 24 hours to maintain the tolerance within 3sec via issue a SMI to BIOS Those events generated before the timestamp be set would show Pre Init Time Stamp as its timestamp BMC will log SEL no matter the timestamp be set or not 7 ...

Page 110: ...cription Channel 0x01 Baud Rate 9600 19200 38400 57600 115200 default BPS Flow control Hardware flow control Payload port 623 8 3 SOL Configuration LAN parameter Value Notes SOL Enable Disabled Disable SOL payload type SOL Authentications Administrator Administrator access over SOL Non Volatile Bit Rate 115200 115200 baud rate Volatile Bit Rate 115200 115200 baud rate Gratuitous ARP Interval 4 sec...

Page 111: ...us OEM Disable Straight Password Enable MD5 Enable MD2 Enable None Authentication Enable 9 3 RMCP Payloads Algorithm Admin Operator User Callback OEM Authentication None O X O O X HMAC SHA1 O O O O X HMAC MD5 X X X X X HMAC SHA2 X X X X X Integrity None O O O O X HMAC SHA1 96 O O O O X HMAC MD5 128 O O X X X MD5 128 X X X X X HMAC SHA2 128 X X X X X Confidentiality None O O O O X AES CBC 128 O O O...

Page 112: ...ution of the server Following table shows the different combinations Resolution Frequency 16 bit color 32 bit color 640x480 60 Yes Yes 800x600 60 Yes Yes 1024x768 60 Yes Yes 1280x800 60 Yes Yes 1280x1024 60 Yes Yes 1440x900 60 Yes Yes 1600x1200 60 Yes Yes 1680x1050 60 Yes Yes 1920x1080 60 Yes Yes 1920x1200 60 Yes Yes The color depth can be chosen at any time while the console redirection is active...

Page 113: ...800G4 BMC Specification 13 Step 4 Select Remote Control from menu bar Browser pops new window for KVM Step 5 Click button Start KVM If you want to terminate the KVM service pleaser click Stop KVM button ...

Page 114: ...or Record 8 9 10 11 12 13 14 Sensor Number 0x56 0x57 0x58 0x59 0x44 0x45 0x46 Sensor Name CPU_ABS_TEMP_01 CPU_ABS_TEMP_02 CPU_ABS_TEMP_03 CPU_ABS_TEMP_04 CPU_TEMP_01 CPU_TEMP_02 CPU_TEMP_03 Sensor Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x03 0x03 0x03 0x03 0x03 0x03 0x03 Instance 0x01 0x02 0x03 0x04 0x00 0x00 0x00 Initialization 0x7F 0x7F 0x7...

Page 115: ...or Record 22 23 24 25 26 27 28 Sensor Number 0x36 0x37 0x38 0x39 0x64 0x65 0x6C Sensor Name FAN_SPEED_07 FAN_SPEED_08 FAN_SPEED_09 FAN_SPEED_10 PSU_TEMP_01 PSU_TEMP_02 POWER_WATTS Sensor Type 0x04 0x04 0x04 0x04 0x01 0x01 0x03 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x1D 0x1D 0x1D 0x1D 0x0A 0x0A 0x0A Instance 0x00 0x00 0x00 0x00 0x04 0x05 0x00 Initialization 0x7F 0x7F 0x7F 0x7F 0x7...

Page 116: ...CPU_VOLTS_02 CPU_VOLTS_03 CPU_VOLTS_04 SYS_VOLTS_01 SYS_VOLTS_02 Sensor Type 0x02 0x02 0x02 0x02 0x02 0x02 0x02 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x07 0x07 0x07 0x07 0x07 0x07 0x07 Instance 0x00 0x00 0x00 0x00 0x00 0x00 0x00 Initialization 0x7F 0x7F 0x7F 0x77 0x7F 0x7F 0x7F Capabilities 0x68 0x68 0x68 0x68 0x68 0x68 0x68 AE LTR Mask 0x2204 0x0325 0x0325 0x0325 0x0325 0x2204 0...

Page 117: ...PU_VR_Temp_03 CPU_VR_Temp_04 MEM_TEMP_01_A0 Sensor Type 0x07 0x07 0x01 0x01 0x01 0x01 0x01 Event Type 0x6F 0x6F 0x01 0x01 0x01 0x01 0x01 Entity ID 0x03 0x03 0x07 0x07 0x07 0x07 0x08 Instance 0x00 0x00 0x03 0x04 0x05 0x06 0x00 Initialization 0x63 0x63 0x77 0x77 0x77 0x77 0x77 Capabilities 0x40 0x40 0x68 0x68 0x68 0x68 0x48 AE LTR Mask 0x0002 0x0002 0x0280 0x0280 0x0280 0x0280 0x6A14 AE LTR BIT 1 1 ...

Page 118: ...Sensor Name MEM_TEMP_01_E0 MEM_TEMP_01_E1 MEM_TEMP_01_F0 MEM_TEMP_01_F1 MEM_TEMP_02_A0 MEM_TEMP_02_A1 MEM_TEMP_02_B0 Sensor Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x08 0x08 0x08 0x08 0x08 0x08 0x08 Instance 0x00 0x00 0x00 0x00 0x00 0x00 0x00 Initialization 0x77 0x77 0x77 0x77 0x77 0x77 0x77 Capabilities 0x48 0x48 0x48 0x48 0x48 0x48 0x48 AE ...

Page 119: ...e MEM_TEMP_02_F0 MEM_TEMP_02_F1 MEM_TEMP_03_A0 MEM_TEMP_03_A1 MEM_TEMP_03_B0 MEM_TEMP_03_B1 MEM_TEMP_03_C0 Sensor Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x08 0x08 0x08 0x08 0x08 0x08 0x08 Instance 0x00 0x00 0x00 0x00 0x00 0x00 0x00 Initialization 0x77 0x77 0x77 0x77 0x77 0x77 0x77 Capabilities 0x48 0x48 0x48 0x48 0x48 0x48 0x48 AE LTR Mask 0...

Page 120: ...me MEM_TEMP_04_A0 MEM_TEMP_04_A1 MEM_TEMP_04_B1 MEM_TEMP_04_B0 MEM_TEMP_04_C0 MEM_TEMP_04_C1 MEM_TEMP_04_D0 Sensor Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Event Type 0x01 0x01 0x01 0x01 0x01 0x01 0x01 Entity ID 0x08 0x08 0x08 0x08 0x08 0x08 0x08 Instance 0x00 0x00 0x00 0x00 0x00 0x00 0x00 Initialization 0x77 0x77 0x77 0x77 0x77 0x77 0x77 Capabilities 0x48 0x48 0x48 0x48 0x48 0x48 0x48 AE LTR Mask ...

Page 121: ... 0x00 UNC 0x00 0x00 0x00 0x00 0x00 0x00 0x00 UC 0x52 0x52 0x52 0x52 0x52 0x00 0x00 UNR 0x55 0x55 0x55 0x55 0x55 0x00 0x00 Resolution Table A16 Full Sensor Record 106 107 108 109 110 111 112 Sensor Number 0x12 0x13 0x14 0x15 Sensor Name HDD_STATUS_03 HDD_STATUS_04 HDD_STATUS_05 HDD_STATUS_06 Sensor Type 0x0D 0x0D 0x0D 0x0D Event Type 0x6F 0x6F 0x6F 0x6F Entity ID 0x04 0x04 0x04 0x04 Instance 0x00 0...

Page 122: ...5 0 8 15 Table B2 Compact Sensor Record 8 9 10 11 12 13 14 Sensor Number 0xB5 0xB6 0xB7 0xFE 0xFA 0xFD 0xF2 Sensor Name MEM_STATUS_06 MEM_STATUS_07 MEM_STATUS_08 SEL WATCHDOG2 PWR_UNIT_STATUS PSU_REDUNANCY Sensor Type 0x0C 0x0C 0x0C 0x10 0x23 0x09 0x08 Event Type 0x6F 0x6F 0x6F 0x6F 0x6F 0x6F 0x0B Entity ID 0x20 0x20 0x20 0x06 0x00 0x0A 0x0A Instance 0x00 0x00 0x00 0x02 0x09 0x00 0x09 Initializati...

Page 123: ... Event Type 0x6F 0x6F 0x6F 0x6F Entity ID 0x22 0x2E 0x0C 0x2E Instance 0x00 0x00 0x00 0x00 SRS SD 0x0000 0x8101 0x0000 0x0000 SRS SD BIT 0 8 15 AM LTR Assertion Event Mask Lower Threshold Reading Mask DEM UTR De assertion Event Mask Upper Threshold Reading Mask DE UTR Discrete Reading Mask Settable Threshold Mask Readable Threshold Mask LNR Lower non recoverable Threshold LC Lower critical Thresho...

Page 124: ...t On 20 5 App 06h 05h X Yes Set ACPI Power State 20 6 App 06h 06h X Yes Get ACPI Power State 20 7 App 06h 07h X Yes Get Device GUID 20 8 App 06h 08h X X Yes Get NetFn Support 21 2 App 06h 09h X Yes Get Command Support 21 3 App 06h 0Ah X Yes Get Command Sub function Support 21 4 App 06h 0Bh X Yes Get Configurable Commands 21 5 App 06h 0Ch X Yes Get Configurable Command Sub functions 21 6 App 06h 0D...

Page 125: ...s Get Channel Payload Version 24 8 App 06h 4Eh X Yes Get Channel OEM Payload Info 24 9 App 06h 4Fh X Yes Master Write Read 22 11 App 06h 52h X Yes Get Channel Cipher Suites 22 15 App 06h 53h P P P P Yes Suspend Resume Payload Encryption 24 3 App 06h 54h X9 Yes Set Channel Security Keys 22 25 App 06h 55h X Yes Get System Interface Capabilities 22 9 App 06h 56h X 12 4 Chassis Device Commands Support...

Page 126: ...le 35 11 S E 04h 29h X Yes Re arm Sensor Events 35 12 S E 04h 2Ah X Yes Get Sensor Event Status 35 13 S E 04h 2Bh X Yes Get Sensor Reading 35 14 S E 04h 2Dh X Yes Set Sensor Type 35 15 S E 04h 2Eh X Yes Get Sensor Type 35 16 S E 04h 2Fh X Yes Set Sensor Reading And Event Status 35 17 S E 04h 30h X 12 8 FRU Device Commands Support Command IPMI Spec Ref NetFn CMD C U O A Yes Get FRU Inventory Area I...

Page 127: ...ransport 0Ch 13h X No Set PPP UDP Proxy Transmit Data 25 5 Transport 0Ch 14h S S S S No Get PPP UDP Proxy Transmit Data 25 6 Transport 0Ch 15h S S S S No Send PPP UDP Proxy Packet 25 7 Transport 0Ch 16h S S S S No Get PPP UDP Proxy Receive Data 25 8 Transport 0Ch 17h S S S S No Serial Modem Connection Active 25 9 Transport 0Ch 18h b b b b No Callback 25 10 Transport 0Ch 19h 7 X7 No Set User Callba...

Page 128: ...nds ICMB Bridge 02h C0h FEh 12 19 Other Bridge Commands Support Command IPMI Spec Ref NetFn CMD C U O A No Error Report optional ICMB Bridge 02h FFh X 1 This command is sent using the Broadcast format on IPMB See command description for details 2 A User can use a Send Message command to deliver a message to system software but Operator privilege is required to use it to access other channels 3 Com...

Page 129: ...alid it will not be executed BMC will select another SD storage for boot If Both of ROMs part are fail BMC SOC will reboot by WDT again 13 3 Boot Time From the point of reset the BMC boot to fully operational state and be responsive to the host on KCS channel within 30 seconds And LAN channel interface will be enable after 78 seconds 13 4 Boot Process a The SoC is going to fetch UBOOT code after A...

Reviews:

No comments

Brands by name

Popular brands

Load more brands