manualshive.com logo in svg

Cisco APIC L3, Installation And Service Manual

Share
Download
Cisco APIC L3 Installation And Service Manual Download Page 123

A P P E N D I X

C

GPU Card Installation

This appendix contains configuration rules for the supported GPU cards.

•

Server Firmware Requirements, on page 115

•

GPU Card Configuration Rules, on page 115

•

Requirement For All GPUs: Memory-Mapped I/O Greater Than 4 GB, on page 115

•

Replacing a Single-Wide GPU Card, on page 116

•

Installing Drivers to Support the GPU Cards, on page 118

Server Firmware Requirements

The following table lists the minimum server firmware versions for the supported GPU cards.

Cisco IMC/BIOS Minimum Version Required

GPU Card

3.1(3)

NVIDIA Tesla
P4

GPU Card Configuration Rules

Note the following rules when populating a server with GPU cards.

• You can install up to two NVIDIA Tesla P4 GPU cards in the server, in PCIe slots 1 and 2.

• Use the UCS power calculator at the following link to determine the power needed based on your server

configuration:

http://ucspowercalc.cisco.com

• Do not mix different brands or models of GPU cards in the server.

Requirement For All GPUs: Memory-Mapped I/O Greater Than
4 GB

All supported GPU cards require enablement of the BIOS setting that allows greater than 4 GB of
memory-mapped I/O (MMIO).

Cisco APIC M3/L3 Server Installation and Service Guide

115

Summary of Contents for APIC L3

Page 1: ...ver Installation and Service Guide First Published 2018 10 25 Americas Headquarters Cisco Systems Inc 170 West Tasman Drive San Jose CA 95134 1706 USA http www cisco com Tel 408 526 4000 800 553 NETS 6387 Fax 408 527 0883 ...

Page 2: ... are encouraged to try to correct the interference by using one or more of the following measures Reorient or relocate the receiving antenna Increase the separation between the equipment and receiver Connect the equipment into an outlet on a circuit different from that to which the receiver is connected Consult the dealer or an experienced radio TV technician for help Modifications to this product...

Page 3: ...ble Management Arm Optional 12 Reversing the Cable Management Arm Optional 13 Initial Server Setup 14 Connecting to the Server Locally For Setup 15 Connecting to the Server Remotely For Setup 16 Setting Up the System With the Cisco IMC Configuration Utility 17 NIC Mode and NIC Redundancy Settings 19 Updating the BIOS and Cisco IMC Firmware 20 Accessing the System BIOS 21 Smart Access Serial 21 Sma...

Page 4: ...at SAS SATA Drives Considerations 34 Replacing a SAS SATA Drive 35 Replacing a Front Loading NVMe SSD 36 Front Loading NVMe SSD Population Guidelines 36 Front Loading NVME SSD Requirements and Restrictions 37 Enabling Hot Plug Support in the System BIOS 37 Replacing a Front Loading NVMe SSD 38 Installing a PCIe Cable For Front Loading NVMe SSDs 39 Replacing HHHL Form Factor NVMe Solid State Drives...

Page 5: ...acing Power Supplies 66 Replacing AC Power Supplies 66 Replacing a PCIe Card 67 PCIe Slot Specifications 67 Replacing a PCIe Card 68 Cisco Virtual Interface Card VIC Considerations 70 Replacing an mLOM Card 71 Replacing an mRAID Riser Riser 3 72 Replacing a SAS Storage Controller Card RAID or HBA 74 Storage Controller Card Firmware Compatibility 74 Replacing a SAS Storage Controller Card RAID or H...

Page 6: ...ns 99 A P P E N D I X B Supported Storage Controllers and Cables 99 Storage Controller Card Firmware Compatibility 100 Write Cache Policy for Cisco 12G SAS Modular RAID Controller 101 Mixing Drive Types in RAID Groups 101 RAID Controller Migration 102 Storage Controller and Backplane Connectors 102 Embedded SATA RAID Controller 104 Embbeded SATA RAID Requirements 104 Embedded SATA RAID Controller ...

Page 7: ...ater Than 4 GB 115 Replacing a Single Wide GPU Card 116 Installing Drivers to Support the GPU Cards 118 1 Updating the Server BIOS 118 2 Updating the GPU Card Drivers 119 Installation For Cisco UCS Manager Integration 121 A P P E N D I X D Installation For Cisco UCS Manager Integration 121 Cisco APIC M3 L3 Server Installation and Service Guide vii Contents ...

Page 8: ...Cisco APIC M3 L3 Server Installation and Service Guide viii Contents ...

Page 9: ...form factor SFF drives with 10 drive backplane Supports up to 10 2 5 inch SAS SATA drives Drive bays 1 and 2 support NVMe SSDs External Features This topic shows the external features of the server versions Cisco APIC M3 and L3 Server SFF Drives Front Panel Features The following figure shows the front panel features of the small form factor drive versions of the server Cisco APIC M3 L3 Server Ins...

Page 10: ...mperature status LED 9 Power button power status LED 3 Pull out asset tag 10 Unit identification button LED 4 KVM connector used with KVM cable that provides one DB 15 VGA one DB 9 serial and two USB connectors 11 System status LED 5 Power supply status LED 6 Cisco APIC M3 and L3 Server Rear Panel Features The rear panel features are the same for all versions of the server Cisco APIC M3 L3 Server ...

Page 11: ...ort 1 port 3 and port 4 from left to right Note All ports must have the same speed either 10 Gigabit or 25 Gigabit Port 0 and port 1 is one pair corresponding to eth2 1 on APIC and port 2 and port 3 is another pair corresponding to eth2 2 on APIC Only one connection is allowed for each pair For example you can connect one cable to either port 0 or port 1 and connect another cable to either port 2 ...

Page 12: ...ble connectors for front loading NVMe SSDs x8 lane 12 DIMM sockets on motherboard 12 per CPU 4 VIC 1455 with external 10 25 Gigabit Ethernet ports 4 13 CPUs and heatsinks up to two 5 Available empty PCIe slot 14 Mini storage module socket Supports either an SD card module with two SD card slots or an M 2 module with two NVMe or SATA M 2 SSD slots 6 PCIe cable connectors for front loading NVMe SSDs...

Page 13: ...o IMC firmware Depending on your Cisco IMC settings Cisco IMC can be accessed through the 1 Gb dedicated management port the 1 Gb 10 Gb Ethernet LAN ports or a Cisco virtual interface card Baseboard management Rear panel One 1 Gb Ethernet dedicated management port RJ 45 connector Two 1 Gb 10 Gb BASE T Ethernet LAN ports RJ 45 connectors The dual LAN ports can support 1 Gbps and 10 Gbps depending o...

Page 14: ...o 10 2 5 inch SAS SATA drives Drive bays 1 and 2 support NVMe SSDs Storage front panel The server has these internal storage options One USB port on the motherboard Mini storage module socket optionally with either SD card module Supports up to two SD cards M 2 SSD module Supports either two SATA M 2 SSDs or two NVMe M 2 SSDs One micro SD card socket on PCIe riser 1 Storage internal The server has...

Page 15: ...ines Before you install operate or service a server review the Regulatory Compliance and Safety Information for important safety information Note IMPORTANT SAFETY INSTRUCTIONS This warning symbol means danger You are in a situation that could cause bodily injury Before you work on any equipment be aware of the hazards involved with electrical circuitry and be familiar with standard practices for p...

Page 16: ... rack the servers using rail kits Physically placing the units on top of one another or stacking without the use of the rail kits blocks the air vents on top of the servers which could result in overheating higher fan speeds and higher power consumption We recommend that you mount your servers on rail kits when you are installing them into the rack because these rails provide the minimal spacing r...

Page 17: ...erly install the chassis in a grounded rack the switch is grounded because it has a metal to metal connection to the rack Alternatively you can ground the chassis by using a customer supplied grounding cable that meets your local and national installation requirements we recommend 6 AWG wire for U S installations connected to the chassis with a grounding lug provided in the switch accessory kit an...

Page 18: ...e bottom of the rack If the rack is provided with stabilizing devices install the stabilizers before mounting or servicing the unit in the rack Statement 1006 Warning Step 1 Attach the inner rails to the sides of the server a Align an inner rail with one side of the server so that the three keyed slots in the rail align with the three pegs on the side of the server b Set the keyed slots over the p...

Page 19: ...rear mounting pegs enter the rear rack post holes from the inside of the rack post e Attach the second slide rail assembly to the opposite side of the rack Ensure that the two slide rail assemblies are at the same height and are level front to back f Pull the inner slide rails on each assembly out toward the rack front until they hit the internal stops and lock in place Step 4 Insert the server in...

Page 20: ...he screw threads into the static part of the rail on the rack post and prevents the server from being pulled out Repeat for the opposite slam latch Installing the Cable Management Arm Optional The cable management arm CMA is reversible left to right To reverse the CMA see Reversing the Cable Management Arm Optional on page 13 before installation Note Step 1 With the server pushed fully into the ra...

Page 21: ...he end of the width adjustment slider onto the end of the stationary slide rail that is attached to the rack post Slide the tab over the end of the rail until it clicks and locks Step 5 Open the hinged flap at the top of each plastic cable guide and route your cables through the cable guides as desired Reversing the Cable Management Arm Optional Step 1 Rotate the entire CMA assembly 180 degrees le...

Page 22: ...ables the 1 Gb 10 Gb Ethernet ports and the ports on any installed Cisco virtual interface card VIC to access the Cisco Integrated Management Interface Cisco IMC If you want to use the 10 100 1000 dedicated management ports to access Cisco IMC you can connect to the server and change the NIC mode as described in Setting Up the System With the Cisco IMC Configuration Utility on page 17 The NIC redu...

Page 23: ...cable Step 1 Attach a power cord to each power supply in your server and then attach each power cord to a grounded power outlet Wait for approximately two minutes to let the server boot to standby power during the first bootup You can verify system power status by looking at the system Power Status LED on the front panel The system is in standby power mode when the LED is amber Step 2 Connect a US...

Page 24: ...Step 1 Attach a power cord to each power supply in your server and then attach each power cord to a grounded power outlet Wait for approximately two minutes to let the server boot to standby power during the first bootup You can verify system power status by looking at the system Power Status LED on the front panel The system is in standby power mode when the LED is amber Step 2 Plug your manageme...

Page 25: ...terfaces are both enabled You must select the default Active Active NIC redundancy setting in the following step In this NIC mode DHCP replies are returned to both the shared LOM ports and the Cisco card ports If the system determines that the Cisco card connection is not getting its IP address from a Cisco UCS Manager system because the server is in standalone mode further DHCP requests from the ...

Page 26: ...nning of the range of six contiguous MAC addresses Note The static IPv4 and IPv6 settings include the following The Cisco IMC IP address For IPv6 valid values are 1 127 The gateway For IPv6 if you do not know the gateway you can set it as none by entering two colons The preferred DNS server address For IPv6 you can set this as none by entering two colons Step 4 Optional Make VLAN settings Step 5 P...

Page 27: ...tings Table 1 Valid NIC Redundancy Settings For Each NIC Mode Valid NIC Redundancy Settings NIC Mode Active active Shared LOM EXT None Dedicated Active active Active standby Shared LOM Active active Active standby Cisco Card This server has the following NIC mode settings that you can choose from Shared LOM EXT default This is the shared LOM extended mode the factory default setting With this mode...

Page 28: ...tive default All Ethernet ports are utilized simultaneously The Shared LOM EXT mode must use only this NIC redundancy setting Shared LOM and Cisco Card modes can each use either Active standby or Active active settings Updating the BIOS and Cisco IMC Firmware When you upgrade the BIOS firmware you must also upgrade the Cisco IMC firmware to the same version or the server does not boot Do not power...

Page 29: ...e This feature allows you to switch between host serial and Cisco IMC CLI This feature has the following requirements A serial cable connection which can use either the RJ 45 serial connector on the server rear panel or a DB 9 connection when using the KVM cable Cisco PID N20 BKVM on the front panel KVM console connector Console redirection must be enabled in the server BIOS Terminal type must be ...

Page 30: ...nected when you enable Smart Access USB Note You can use USB 3 0 based devices but they will operate at USB 2 0 speed We recommend that the USB device have only one partition The file system formats supported are FAT16 FAT32 MSDOS EXT2 EXT3 and EXT4 NTFS is not supported The front panel KVM connector has been designed to switch the USB port between Host OS and BMC Smart Access USB can be enabled o...

Page 31: ...lacing Components on page 32 Service Headers and Jumpers on page 85 Status LEDs and Buttons This section contains information for interpreting front rear and internal LED states Front Panel LEDs Figure 9 Front Panel LEDs Table 2 Front Panel LEDs Definition of States States LED Name Cisco APIC M3 L3 Server Installation and Service Guide 23 ...

Page 32: ...lowing insertion or the driver is unloading following an eject command Amber The drive has failed Amber blinking A drive Locate command has been issued in the software NVMe SSD drive fault NVMe solid state drive SSD drive tray LEDs have different behavior than SAS SATA drive trays Note 1 NVMe Off No drive activity Green blinking There is drive activity NVMe SSD activity 2 NVMe Off There is no AC p...

Page 33: ...more power supplies are in a degraded operational state Amber blinking One or more power supplies are in a critical fault state Power supply status 6 Green All fan modules are operating properly Amber blinking One or more fan modules breached the non recoverable threshold Fan status 7 Off The Ethernet LOM port link is idle Green One or more Ethernet LOM ports are link active but there is no activi...

Page 34: ...ernet link status on both LAN1 and LAN2 2 Off Link speed is 10 Mbps Amber Link speed is 100 Mbps Green Link speed is 1 Gbps 1 Gb Ethernet dedicated management link speed 3 Off No link is present Green Link is active Green blinking Traffic is present on the active link 1 Gb Ethernet dedicated management link status 4 Off The unit identification function is not in use Blue blinking The unit identifi...

Page 35: ... DC input 12 V main power off 12 V standby power off Green blinking 12 V main power off 12 V standby power on Green solid 12 V main power on 12 V standby power on Amber blinking Warning threshold detected but 12 V main power on Amber solid Critical error detected 12 V main power off for example over current over voltage or over temperature failure Power supply status one LED each power supply unit...

Page 36: ...d Amber Fan has a fault or is not fully seated Green Fan is OK 1 CPU fault LEDs one behind each CPU socket on the motherboard These LEDs operate only when the server is in standby power mode Amber CPU has a fault Off CPU is OK 2 Preparing For Component Installation This section includes information and tasks that help prepare the server for component installation Cisco APIC M3 L3 Server Installati...

Page 37: ...ll power cords from all power supplies in the server Caution You can shut down the server by using the front panel power button or the software management interfaces Shutting Down Using the Power Button Step 1 Check the color of the Power button LED Amber The server is already in standby mode and you can safely remove power Green The server is in main power mode and must be shut down before you ca...

Page 38: ...em performs a graceful shutdown and the server goes to standby mode which is indicated by an amber Power button LED Step 3 If a service procedure instructs you to completely remove power from the server disconnect all power cords from the power supplies in the server Removing the Server Top Cover Step 1 Remove the top cover a If the cover latch is locked use a screwdriver to turn the lock 90 degre...

Page 39: ...ver latch 2 Top cover 1 Serial number label location 3 Serial Number Location The serial number for the server is printed on a label on the top of the server near the front Hot Swap vs Hot Plug Some components can be removed and replaced without shutting down and removing power from the server This type of replacement has two varieties hot swap and hot plug Cisco APIC M3 L3 Server Installation and...

Page 40: ...r covers are in place Statement 1029 Warning When handling server components handle them only by carrier edges and use an electrostatic discharge ESD wrist strap or other grounding device to avoid damage Caution You can press the unit identification button on the front panel or rear panel to turn on a flashing blue unit identification LED on both the front and rear panels of the server This button...

Page 41: ...nectors for front loading NVMe SSDs x8 lane 12 DIMM sockets on motherboard 12 per CPU 4 VIC 1455 with external 10 25 Gigabit Ethernet ports 4 13 CPUs and heatsinks up to two 5 Available empty PCIe slot 14 Mini storage module socket Supports either an SD card module with two SD card slots or an M 2 module with two NVMe or SATA M 2 SSD slots 6 PCIe cable connectors for front loading NVMe SSDs on PCI...

Page 42: ...2 For example in a SFF 10 drive server drive bays 5 and 10 cannot be controlled by embedded SATA RAID Note Keep an empty drive blanking tray in any unused bays to ensure proper airflow You can mix SAS SATA hard drives and SAS SATA SSDs in the same server However you cannot configure a logical volume virtual drive that contains a mix of hard drives and SSDs That is when you create a logical volume ...

Page 43: ...nfigure BIOS Parameters dialog select the Advanced tab Step 5 Go to the LOM and PCIe Slot Configuration section Step 6 Set the PCIe Slot HBA Option ROM to UEFI Only Step 7 Click Save Changes The dialog closes Step 8 Under BIOS Properties set Configured Boot Order to UEFI Step 9 Under Actions click Configure Boot Order Step 10 In the Configure Boot Order dialog click Add Local HDD Step 11 In the Ad...

Page 44: ...ve removed from drive tray 4 Release button 2 Replacing a Front Loading NVMe SSD This section is for replacing 2 5 inch or 3 5 inch form factor NVMe solid state drives SSDs in front panel drive bays To replace HHHL form factor NVMe SSDs in the PCIe slots see Replacing HHHL Form Factor NVMe Solid State Drives Front Loading NVMe SSD Population Guidelines The front drive bay supports 2 5 inch NVMe SS...

Page 45: ...Ds with a SAS RAID controller because NVMe SSDs interface with the server via the PCIe bus UEFI boot is supported in all supported operating systems Hot insertion and hot removal are supported in all supported operating systems except VMWare ESXi Enabling Hot Plug Support in the System BIOS Hot plug OS informed hot insertion and hot removal is disabled in the system BIOS by default If the system w...

Page 46: ...the face of the drive tray c Grasp and open the ejector lever and then pull the drive tray out of the slot d Remove the four drive tray screws that secure the SSD to the tray and then lift the SSD out of the tray If this is the first time that front loading NVMe SSDs are being installed in the server you must install PCIe cable CBL NVME C220FF before installing the drive Note Step 2 Install a new ...

Page 47: ...nects the front panel drive backplane to the PCIe riser 2 board on the PCIe riser assembly If the server was ordered with 2 5 inch form factor NVMe SSDs this cable was preinstalled at the factory No action is required If you are adding 2 5 inch form factor NVMe SSDs for the first time you must order and install the cable as described in the following procedure Cisco APIC M3 L3 Server Installation ...

Page 48: ...n the front panel drive bays see Replacing a Front Loading NVMe SSD on page 36 HHHL SSD Population Guidelines Observe the following population guidelines when installing HHHL form factor NVMe SSDs Two CPU systems You can populate up to 2 HHHL form factor SSDs using PCIe slots 1 2 One CPU systems In a single CPU system PCIe riser 2 slot 2 is not available Therefore the maximum number of HHHL form f...

Page 49: ... as described in Removing the Server Top Cover on page 30 d Use two hands to grasp the external riser handle and the blue area at the front of the riser e Open the hinged plastic card retainer that secures the rear panel tab of the card f Pull evenly on both ends of the HHHL form factor NVME SSD to remove it from the socket on the PCIe riser If the riser has no SSD remove the blanking panel from t...

Page 50: ...ules The seven fan modules in the server are numbered as shown in Serviceable Component Locations on page 3 Each fan module has a fault LED next to the fan connector on the motherboard This LED lights green when the fan is correctly seated and is operating OK The LED lights amber when the fan has a fault or is not correctly seated Tip Cisco APIC M3 L3 Server Installation and Service Guide 42 Maint...

Page 51: ...d point toward the rear of the server b Press down gently on the fan module to fully engage it with the connector on the motherboard c Replace the top cover to the server d Replace the server in the rack replace cables and then fully power on the server by pressing the Power button Replacing Memory DIMMs DIMMs and their sockets are fragile and must be handled with care to avoid damage during insta...

Page 52: ...of CPUs and the number of DIMMs per CPU If your server has two CPUs balance DIMMs evenly across the two CPUs as shown in the table The table below lists recommended configurations Using 5 7 9 10 or 11 DIMMs per CPU is not recommended Note Table 4 DIMM Population Order Populate CPU2 Slots Populate CPU 1 Slot Number of DIMMs per CPU Recommended Configurations Black 2 Slots Blue 1 Slots Black 2 Slots...

Page 53: ...B 128GB You cannot mix DIMM speeds in a bank for example A1 B1 Pairs of DIMMs must be identical same PID and revision You can mix speeds but DIMMs will run at the speed of the slowest DIMMs CPUs installed in the channel DIMM speed For example 2666 GHz You cannot mix DIMM types in a bank You cannot mix DIMM types in a channel DIMM type RDIMMs or LRDIMMs Memory Mirroring The CPUs in the server suppo...

Page 54: ... by pressing the Power button Replacing CPUs and Heatsinks This section contains CPU configuration rules and the procedure for replacing CPUs and heatsinks CPU Configuration Rules This server has two CPU sockets on the motherboard Each CPU supports six DIM channels 12 DIMM slots The server can operate with one CPU or two identical CPUs installed The minimum configuration is that the server must ha...

Page 55: ...sting CPU heatsink assembly from the server a Shut down and remove power from the server as described in Shutting Down and Removing Power From the Server on page 29 b Slide the server out the front of the rack far enough so that you can remove the top cover You might have to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server ...

Page 56: ...e heatsink from the CPU assembly the CPU assembly includes the CPU and the CPU carrier a Place the heatsink with CPU assembly so that it is oriented upside down as shown below Note the thermal interface material TIM breaker location TIM BREAKER is stamped on the CPU carrier next to a small slot Cisco APIC M3 L3 Server Installation and Service Guide 48 Maintaining the Server Replacing a CPU and Hea...

Page 57: ... the TIM breaker slot Use caution to avoid damaging the heatsink surface Caution d Gently rotate the screwdriver to lift up on the CPU until the TIM on the heatsink separates from the CPU Do not allow the screwdriver tip to touch or damage the green CPU substrate Note e Pinch the CPU carrier inner latch at the corner opposite the TIM breaker and push up to disengage the clip from its slot in the h...

Page 58: ...with extreme care to avoid damaging pins Caution Figure 22 CPU Assembly Tool CPU Assembly and Heatsink Alignment Features Angled corner on heatsink pin 1 alignment feature 4 CPU assembly tool 1 Triangle cut into carrier pin 1 alignment feature 5 CPU assembly CPU in plastic carrier 2 Angled corner on CPU assembly tool pin 1 alignment feature 6 Heatsink 3 Step 4 Apply new TIM to the heatsink The hea...

Page 59: ...t heatsink for your CPUs to ensure proper cooling Use UCSC HS C220M5 for standard performance CPUs 150 W and less Note the wattage described on the heatsink label Caution Step 5 With the CPU assembly on the CPU assembly tool set the heatsink onto the CPU assembly Note the pin 1 alignment features for correct orientation Push down gently until you hear the corner clips of the CPU carrier click onto...

Page 60: ... captive nuts that secure the heatsink to the motherboard standoffs Alternate tightening the heatsink nuts evenly so that the heatsink remains level while it is lowered Tighten the heatsink nuts in the order shown on the heatsink label 1 2 3 4 The captive nuts must be fully tightened so that the leaf springs on the CPU socket lie flat Caution e Replace the top cover to the server f Replace the ser...

Page 61: ...5 for CPUs 150 W and less Heatsink cleaning kit UCSX HSCK Scenario 3 You have a damaged CPU carrier the plastic frame around the CPU CPU Carrier UCS M5 CPU CAR 1 flat head screwdriver for separating the CPU from the heatsink Heatsink cleaning kit UCSX HSCK Thermal interface material TIM kit for M5 servers UCS CPU TIM A CPU heatsink cleaning kit is good for up to four CPU and heatsink cleanings The...

Page 62: ...g You must transfer these covers to the system that you are returning as described in this procedure Caution Step 1 When moving an M3 or L3 CPU to a new server you do not have to separate the heatsink from the CPU Perform the following steps a Use a T 30 Torx driver to loosen the four captive nuts that secure the assembly to the board standoffs Alternate loosening the heatsink nuts evenly so that ...

Page 63: ...vers from the new system to the system that you are returning a Remove the socket covers from the replacement system Grasp the two recessed finger grip areas marked REMOVE and lift straight up Keep a firm grasp on the finger grip areas at both ends of the cover Do not make contact with the CPU socket pins Note Cisco APIC M3 L3 Server Installation and Service Guide 55 Maintaining the Server Moving ...

Page 64: ...n the cover except the two points described Pressing elsewhere might damage the socket pins Caution c Press down on the two circular markings next to the word INSTALL that are closest to the two threaded posts see the following figure Press until you feel and hear a click You must press until you feel and hear a click to ensure that the dust covers do not come loose during shipping Note Cisco APIC...

Page 65: ...board align the assembly over the CPU socket as shown below Note the alignment features The pin 1 angled corner on the heatsink must align with the pin 1 angled corner on the CPU socket The CPU socket posts must align with the guide holes in the assembly Cisco APIC M3 L3 Server Installation and Service Guide 57 Maintaining the Server Moving an M5 Generation CPU ...

Page 66: ...e the heatsink to the board standoffs Alternate tightening the heatsink nuts evenly so that the heatsink remains level while it is lowered Tighten the heatsink nuts in the order shown on the heatsink label 1 2 3 4 The captive nuts must be fully tightened so that the leaf springs on the CPU socket lie flat Note Replacing a Mini Storage Module The mini storage module plugs into a motherboard socket ...

Page 67: ...ver as described in Removing the Server Top Cover on page 30 Step 4 Remove a carrier from its socket a Locate the mini storage module carrier in its socket just in front of power supply 1 b At each end of the carrier push outward on the clip that secures the carrier c Lift both ends of the carrier to disengage it from the socket on the motherboard d Set the carrier on an anti static surface Step 5...

Page 68: ...figured in a RAID 1 array through the Cisco IMC interface SD slot 1 is on the top side of the carrier SD slot 2 is on the underside of the carrier the same side as the carrier s motherboard connector Step 1 Power off the server and then remove the mini storage module carrier from the server as described in Replacing a Mini Storage Module Carrier on page 59 Step 2 Remove an SD card a Push on the to...

Page 69: ...bedded SATA RAID utility If M 2 NVMe SSDs are installed in the M 2 module the embedded SATA controller is automatically disabled You cannot control M 2 SATA SSDs in the server with a HW RAID controller Note Step 1 Power off the server and then remove the mini storage module carrier from the server as described in Replacing a Mini Storage Module Carrier on page 59 Step 2 Remove an M 2 SSD a Use a 1...

Page 70: ...ro SD card The socket is on the top of PCIe riser 1 under a flexible plastic cover e Use your fingertip to push open the retainer on the socket cover far enough to provide access to the Micro SD card then push down and release the Micro SD card to make it spring up f Grasp the Micro SD card and lift it from the socket Step 2 Install a new Micro SD card a While holding the retainer on the plastic c...

Page 71: ...p cover You might have to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server from the rack Caution c Remove the top cover from the server as described in Removing the Server Top Cover on page 30 d Locate the USB socket on the motherboard directly in front of PCIe riser 2 e Grasp the USB drive and pull it horizontally to free ...

Page 72: ... Setup Utility by pressing the F2 key when prompted during bootup Step 2 Navigate to the Advanced tab Step 3 On the Advanced tab select USB Configuration Step 4 On the USB Configuration page select USB Ports Configuration Step 5 Scroll to USB Port Internal press Enter and then choose either Enabled or Disabled from the dialog box Step 6 Press F10 to save and exit the utility Cisco APIC M3 L3 Serve...

Page 73: ...over You might have to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server from the rack Caution c Remove the top cover from the server as described in Removing the Server Top Cover on page 30 d Locate the RTC battery The vertical socket is directly in front of PCIe riser 2 e Remove the battery from the socket on the motherboa...

Page 74: ... you have ordered a server with power supply redundancy two power supplies you do not have to power off the server to replace a power supply because they are redundant as 1 1 Note Do not mix power supply types or wattages in the server Both power supplies must be identical Note Step 1 Remove the power supply that you are replacing or a blank panel from an empty bay a Perform one of the following a...

Page 75: ...erver press the Power button to boot the server to main power mode Figure 33 Replacing AC Power Supplies Power supply handle 2 Power supply release lever 1 Replacing a PCIe Card Cisco supports all PCIe cards qualified and sold by Cisco PCIe cards not qualified or sold by Cisco are the responsibility of the customer Although Cisco will always stand behind and support the C Series rack mount servers...

Page 76: ...ard there are prerequisite considerations See Cisco Virtual Interface Card VIC Considerations on page 70 Note RAID controller cards install into a separate mRAID riser See Replacing a SAS Storage Controller Card RAID or HBA on page 74 Note Step 1 Remove an existing PCIe card or a blank filler panel from the PCIe riser a Shut down and remove power from the server as described in Shutting Down and R...

Page 77: ... retainer open align the new PCIe card with the empty socket on the PCIe riser PCIe riser 1 slot 1 has a long card guide at the front end of the riser Use the slot in the long card guide to help support a full length card b Push down evenly on both ends of the card until it is fully seated in the socket c Ensure that the card s rear panel tab sits flat against the riser rear panel opening and then...

Page 78: ...Virtual Interface Card VIC Considerations This section describes VIC card support and special considerations for this server If you use the Cisco Card NIC mode you must also make a VIC Slot setting that matches where your VIC is installed The options are Riser1 Riser2 and Flex LOM See NIC Mode and NIC Redundancy Settings on page 19 for more information about NIC modes Note Table 8 VIC Support and ...

Page 79: ...t you can remove the top cover You might have to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server from the rack Caution c Remove the top cover from the server as described in Removing the Server Top Cover on page 30 d Remove the mRAID riser to provide access to the mLOM socket below the riser To remove the mRAID riser use b...

Page 80: ...l riser that is used for either a Cisco modular storage controller card RAID or HBA or the SATA interposer card for embedded software RAID This riser plugs into a dedicated motherboard socket and provides a horizontal socket for the installed card This riser can be ordered as the following options UCSC XRAIDR 220M5 Replacement unit for this mRAID riser UCSC MRAID1GB KIT Kit for first time addition...

Page 81: ...urface d Remove any card from the riser Open the blue card ejector lever that is on the edge of the card and then pull the card straight out from its socket on the riser Step 3 Install a new mRAID riser a Install your card into the new riser Close the card ejector lever on the card to lock it into the riser b Connect cables to the installed card See Storage Controller and Backplane Connectors on p...

Page 82: ...the same time Note Storage Controller Card Firmware Compatibility Firmware on the storage controller RAID or HBA must be verified for compatibility with the current Cisco IMC and BIOS versions that are installed on the server If not compatible upgrade or downgrade the storage controller firmware using the Host Upgrade Utility HUU for your firmware release to bring it to a compatible level See the ...

Page 83: ...Remove any existing card from the riser a Disconnect cables from the existing card b Open the blue card ejector lever on the back side of the card to eject it from the socket on the riser c Pull the card from the riser and set it aside Step 4 Install a new storage controller card to the riser a With the riser upside down set the card on the riser b Push on both corners of the card to seat its conn...

Page 84: ...co IMC release for instructions on downloading and using the utility to bring server components to compatible levels HUU Guides Note Figure 39 mRAID Riser Internal Riser 3 Location Card ejector lever 3 External blue handle 1 Two pegs on inner chassis wall 2 Replacing the Supercap RAID Backup This server supports installation of one supercap unit The unit mounts to a bracket that is in the middle o...

Page 85: ... cover from the server as described in Removing the Server Top Cover on page 30 Step 2 Remove an existing supercap a Disconnect the supercap cable from the existing supercap b Push aside the securing tab to open the hinged latch that secures the supercap to its bracket on the removable air baffle c Lift the supercap free of the bracket and set it aside Step 3 Install a new supercap a Set the new s...

Page 86: ...equires a SATA interposer card that plugs into a horizontal socket on a dedicated mRAID riser internal riser 3 You cannot use a hardware RAID controller card and the embedded software RAID controller at the same time Note Step 1 Prepare the server for component installation Cisco APIC M3 L3 Server Installation and Service Guide 78 Maintaining the Server Replacing a SATA Interposer Card ...

Page 87: ...ejector lever on the back side of the card to eject it from the socket on the riser c Pull the card from the riser and set it aside Step 4 Install a new card to the riser a With the riser upside down set the card on the riser b Push on both corners of the card to seat its connector in the riser socket c Close the card ejector lever on the card to lock it into the riser Step 5 Return the riser to t...

Page 88: ...h so that you can remove the top cover You might have to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server from the rack Caution c Remove the top cover from the server as described in Removing the Server Top Cover on page 30 Step 2 Remove an existing intrusion switch a Disconnect the intrusion switch cable from the socket on...

Page 89: ...n 1 Installing a Trusted Platform Module TPM The trusted platform module TPM is a small circuit board that plugs into a motherboard socket and is then permanently secured with a one way screw The socket location is on the motherboard below PCIe riser 2 TPM Considerations This server supports either TPM version 1 2 or TPM version 2 0 Field replacement of a TPM is not supported you can install a TPM...

Page 90: ...component remove the server from the rack Caution c Remove the top cover from the server as described in Removing the Server Top Cover on page 30 Step 2 Check if there is a card installed in PCIe riser 2 If no card is installed in PCIe riser 2 you can access the TPM socket Go to the next step If a card is installed in PCIe riser 2 remove the PCIe riser assembly from the chassis to provide clearanc...

Page 91: ...p utility Then navigate to Security Set Administrator Password and enter the new password twice as prompted Note Step 1 Enable TPM Support a Watch during bootup for the F2 prompt and then press F2 to enter BIOS setup b Log in to the BIOS Setup Utility with your BIOS Administrator password c On the BIOS Setup Utility window choose the Advanced tab d Choose Trusted Computing to open the TPM Security...

Page 92: ...Step 2 When prompted press F2 to enter the BIOS Setup utility Step 3 Verify that the prerequisite BIOS values are enabled a Choose the Advanced tab b Choose Intel TXT LT SX Configuration to open the Intel TXT LT SX Hardware Support window c Verify that the following items are listed as Enabled VT d Support default is Enabled VT Support default is Enabled TPM Support TPM State d Do one of the follo...

Page 93: ...ader block J38 1 J39 pin 1 arrow printed on motherboard 7 J38 pin 1 arrow printed on motherboard 2 Boot Cisco IMC from alternate image J39 pins 1 2 8 Clear CMOS J38 pins 9 10 3 Reset Cisco IMC password to default J39 pins 3 4 9 Recover BIOS J38 pins 11 12 4 Reset Cisco IMC to defaults J39 pins 5 6 10 Clear password J38 pins 13 14 5 Cisco APIC M3 L3 Server Installation and Service Guide 85 Maintain...

Page 94: ...o the server The server powers up to standby power mode indicated when the Power LED on the front panel is amber Step 7 Return the server to main power mode by pressing the Power button on the front panel The server is in main power mode when the Power LED is green You must allow the entire server to reboot to main power mode to complete the reset The state of the jumper cannot be determined witho...

Page 95: ... USB drive into a USB port on the server Step 4 Reboot the server Step 5 Return the server to main power mode by pressing the Power button on the front panel The server boots with the updated BIOS boot block When the BIOS detects a valid bios cap file on the USB drive it displays this message Found a valid recovery file Transferring to Cisco IMC System would flash the BIOS image now System would r...

Page 96: ...the server and the screen goes blank for about 10 minutes Do not unplug the power cords during this update Cisco IMC powers on the server after the update is complete Note Step 12 After the server has fully booted power off the server again and disconnect all power cords Step 13 Remove the jumper that you installed If you do not remove the jumper after recovery completion you see the prompt Please...

Page 97: ...e to detach cables from the rear panel to provide clearance If you cannot safely view and access the component remove the server from the rack Caution Step 3 Remove the top cover from the server as described in Removing the Server Top Cover Step 4 Locate header block J39 pins 1 2 as shown in Service Headers and Jumpers Step 5 Install a two pin jumper across J39 pins 1 and 2 Step 6 Reinstall the to...

Page 98: ... is in main power mode when the Power LED is green When you next log in to Cisco IMC you see a message similar to the following Reset to default CIMC password debug functionality is enabled On input power cycle CIMC password will be reset to defaults Note Step 8 Press the Power button to shut down the server to standby power mode and then remove AC power cords from the server to remove all power S...

Page 99: ...ee a message similar to the following CIMC reset to factory defaults debug functionality is enabled On input power cycle CIMC will be reset to factory defaults Note Step 8 To remove the jumper press the Power button to shut down the server to standby power mode and then remove AC power cords from the server to remove all power Step 9 Remove the top cover from the server Step 10 Remove the jumper t...

Page 100: ...Cisco APIC M3 L3 Server Installation and Service Guide 92 Maintaining the Server Using the Reset Cisco IMC to Defaults Header J39 Pins 5 6 ...

Page 101: ...fications The following table lists the physical specifications for the server versions Table 9 Physical Specifications Specification Description 1 7 in 43 2 mm Height 16 9 in 429 0 mm Width Server only 29 5 in 740 3 mm Server with slide rail 31 0 in 787 4 mm Depth length Maximum 37 5 lb 17 0 Kg Minimum 29 0 lb 13 2 Kg Weight Environmental Specifications The following table lists the environmental...

Page 102: ... weighted per ISO7779 LwAd Bels Operation at 73 F 23 C 40 Sound pressure level Measure A weighted per ISO7779 LpAm dBA Operation at 73 F 23 C Power Specifications Do not mix power supply types or wattages in the server Both power supplies must be identical Note You can get more specific power information for your exact server configuration by using the Cisco UCS Power Calculator http ucspowercalc ...

Page 103: ...y standby voltage Climate Savers Platinum Efficiency 80Plus Platinum certified Efficiency rating RSP2 Form factor IEC320 C14 Input connector 1050 W DC Power Supply This section lists the specifications for each 1050 W DC power supply Cisco part number UCSC PSUV2 1050DC Table 12 1050 W DC Specifications Specification Description Nominal range 48 to 60 VDC Range 40 to 72 VDC DC Input Voltage 32 A at...

Page 104: ...tional alternative to the standard power cords Only the approved power cords or jumper power cords listed below are supported Note Table 13 Supported Power Cords Length Meters Length Feet Description 3 5 11 7 CAB 48DC 40A 8AWG DC power cord 48 VDC 40 A 8 AWG Three socket Mini Fit connector to three wire 3 0 9 8 CAB C13 C14 AC AC power cord 10 A C13 to C14 recessed receptacle 2 5 8 2 CAB 250V 10A A...

Page 105: ... cabinet jumper power cord 250 V 10 A C13 to C14 2 5 8 2 CAB 9K10A AU AC power cord 250 V 10 A 3112 plug Australia 2 5 8 2 CAB N5K6A NA AC power cord 200 240 V 6 A North America 2 5 8 2 CAB 250V 10A ID AC power Cord 250 V 10 A India 2 5 8 2 CAB 9K10A SW AC power cord 250 V 10 A MP232 plug Switzerland 2 5 8 2 CAB 250V 10A BR AC power Cord 250 V 10 A Brazil Cisco APIC M3 L3 Server Installation and S...

Page 106: ...ug North America 2 0 6 6 CAB AC L620 C13 AC power cord NEMA L6 20 to C13 connectors 2 5 8 2 CAB 9K10A IT AC power cord 250 V 10 A CEI 23 16 VII plug Italy NA NA R2XX DMYMPWRCORD No power cord PID option for ordering server with no power cord Cisco APIC M3 L3 Server Installation and Service Guide 98 Server Specifications Server Specifications ...

Page 107: ...nd Cables This server supports a single PCIe style SAS RAID or HBA controller that plugs into a dedicated internal riser Alternatively the server has a software based SATA RAID controller embedded in the system Do not mix controller types in the server Do not use the embedded SATA controller and a hardware based RAID controller card at the same time This combination is not supported and could resu...

Page 108: ...ble included with chassis to connect controller to drive backplane No Non RAID This controller is supported only in these server versions SFF 10 drives C220M5SX 10 front loading SAS SATA drives Cisco 12G Modular SAS HBA UCSC SAS M5 External drive cables not sold by Cisco NOTE This HBA does not support optical cables for connection to external storage copper only No Non RAID Supported in all server...

Page 109: ...ervers use the Cisco IMC interface to set Virtual Drive Properties Write Policy See the Managing Storage Adapters section in your Cisco IMC Configuration Guide Cisco IMC GUI and CLI Configuration Guides For Cisco UCS integrated servers use the Cisco UCS Manager interface to set the write cache policy as part of virtual drive configuration in your storage profile Cisco UCS Manager Configuration Gui...

Page 110: ...AID to hardware RAID is not supported and could result in data loss Caution Allowed only before there is data on the drives data migration is not supported 1 Disable embedded RAID in the BIOS 2 Install RAID card 3 Install SAS cables Embedded software RAID Embedded RAID is enabled in the BIOS Not allowed Hardware RAID Embedded RAID is disabled in the BIOS Storage Controller and Backplane Connectors...

Page 111: ...ing figures that illustrate cable connections and which drives are controlled by each cable In the SFF 10 drive version drives 5 and 10 cannot be controlled by the embedded SATA RAID controller Note Figure 45 Embedded RAID Interposer Cable to Drive Backplane Mapping SFF 10 Drive Version Cisco 12G Modular SAS RAID Controller or HBA This HW RAID option can control up to 10 SAS SATA drives in the SFF...

Page 112: ... You can use VMWare in AHCI mode Note The Microsoft Windows Server 2016 Hyper V hypervisor is supported for use with the embedded MegaRAID controller in SW RAID mode but all other hyperviors are not supported All Hypervisors are supported in AHCI mode Note You cannot control the M 2 SATA SSDs in the server with a HW RAID controller Note Embbeded SATA RAID Requirements The embedded SATA RAID contro...

Page 113: ...ware RAID embedded RAID to hardware RAID a controller card is not supported and could result in data loss Migrations from software RAID to hardware RAID are supported only before there is data on the drives or when there are no drives in the server Caution The required drivers for this controller are already installed and ready to use with the LSI SWRAID Configuration Utility However if you will u...

Page 114: ...se one of the options from the dialog LSI SW RAID Enable the embedded pSATA RAID controller This menu option does not appear when the server is set to boot in Legacy mode UEFI mode is required To change the boot mode use the BIOS setting for Boot Options Boot Mode Note Disabled Disable the embedded pSATA RAID controller c For the secondary sSATA controller select M 2 and then choose one of the opt...

Page 115: ...onal drivers for those operating systems Note This section explains how to install the LSI MegaSR drivers for the following supported operating systems Microsoft Windows Server Red Hat Enterprise Linux RHEL SUSE Linux Enterprise Server SLES For the specific supported OS versions see the Hardware and Software Compatibility Matrix for your server release Downloading the MegaSR Drivers The MegaSR dri...

Page 116: ...the ISO image to a disk b Browse the contents of the drivers folders to the location of the embedded MegaRAID drivers OS Storage Intel C600 M5 c Expand the Zip file which contains the folder with the MegaSR driver files d Copy the expanded folder to a USB thumb drive Step 3 Start the Windows driver installation using one of the following methods To install from local media connect an external USB ...

Page 117: ...he device for which you are installing the driver and then click Properties Step 4 On the Driver tab click Update Driver to open the Update Device Driver wizard and then follow the wizard instructions to update the driver Linux Drivers Dowloading the Driver IMG or ISO File See Downloading the MegaSR Drivers on page 107 for instructions on downloading the drivers The Linux driver is included in the...

Page 118: ...x for your server release This topic describes the fresh installation of the RHEL device driver on systems that have the embedded MegaRAID stack If you use an embedded RAID controller with Linux both the pSATA and the sSATA controller must be set to LSI SW RAID mode Note Before you begin Before you install this driver on an embedded controller you must configure a RAID drive group on the embedded ...

Page 119: ...Step 5 Power cycle the target server Step 6 Press F6 when you see the F6 prompt during bootup The Boot Menu window opens Step 7 On the Boot Manager window select the physical disk or virtual disk and press Enter The RHEL installation begins when the image is booted Step 8 Enter one of the following blacklist commands at the boot prompt For RHEL 6 x 32 and 64 bit enter linux dd blacklist isci black...

Page 120: ...are RAID Configuration Utility SATA For sSATA select LSI Software RAID Configuration Utility sSATA Step 1 Prepare the dud img file using one of the following methods To install from physical disk use the procedure in Preparing Physical Thumb Drive for Linux on page 109 then continue with step 4 To install from virtual disk download the Cisco UCS C Series drivers ISO as described in Downloading the...

Page 121: ...r and then press Enter If you mapped the dud img file as a virtual disk choose the location of the virtual disk Yes appears under the F6 Driver heading Step 12 Press Enter to choose Installation Step 13 Press OK The following message is displayed LSI Soft RAID Driver Updates added Step 14 At the menu choose the driver update medium and press the Back button Step 15 Follow the SLES installation wiz...

Page 122: ...Cisco APIC M3 L3 Server Installation and Service Guide 114 Storage Controller Considerations For More RAID Utility Information ...

Page 123: ...IMC BIOS Minimum Version Required GPU Card 3 1 3 NVIDIA Tesla P4 GPU Card Configuration Rules Note the following rules when populating a server with GPU cards You can install up to two NVIDIA Tesla P4 GPU cards in the server in PCIe slots 1 and 2 Use the UCS power calculator at the following link to determine the power needed based on your server configuration http ucspowercalc cisco com Do not mi...

Page 124: ... the BIOS defaults for the server Use this only if you know that the server BIOS is set to use the default enabled setting for this item Step 4 Reboot the server Cisco UCS Manager pushes BIOS configuration changes through a BIOS policy or default BIOS settings to the Cisco Integrated Management Controller CIMC buffer These changes remain in the buffer and do not take effect until the server is reb...

Page 125: ...ith the hinged tab retainer open align the new GPU card with the empty socket on the PCIe riser b Push down evenly on both ends of the card until it is fully seated in the socket c Ensure that the card s rear panel tab sits flat against the riser rear panel opening and then close the hinged tab retainer over the card s rear panel tab Figure 47 PCIe Riser Assembly Hinged card retainer one each slot...

Page 126: ... fully power on the server by pressing the Power button Installing Drivers to Support the GPU Cards After you install the hardware you must update to the correct level of server BIOS and then install GPU drivers and other software in this order 1 Update the server BIOS 2 Update the GPU drivers 1 Updating the Server BIOS Install the latest Cisco UCS C240 M4 server BIOS by using the Host Upgrade Uti...

Page 127: ... the Host Upgrade Utility are at Utility User Guides 2 Updating the GPU Card Drivers After you update the server BIOS you can install GPU drivers to your hypervisor virtual machine Step 1 Install your hypervisor software on a computer Refer to your hypervisor documentation for the installation instructions Step 2 Create a virtual machine in your hypervisor Refer to your hypervisor documentation fo...

Page 128: ...Cisco APIC M3 L3 Server Installation and Service Guide 120 GPU Card Installation 2 Updating the GPU Card Drivers ...

Page 129: ...here Cisco UCS C Series Server Integration with UCS Manager Configuration Guides Refer to the guide that is for the version of Cisco UCS Manager that you are using Also refer to the release notes for Cisco UCS Manager software and C Series Cisco IMC software for any special considerations regarding integration in your release Cisco UCS Manager Release Notes Cisco C Series Software Release Notes Ci...

Page 130: ...Cisco APIC M3 L3 Server Installation and Service Guide 122 Installation For Cisco UCS Manager Integration Installation For Cisco UCS Manager Integration ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands