Oracle Exalogic Elastic Cloud Owner'S Manual Download Page 141

Page: 141 / 342

Configure Ethernet Over InfiniBand

This chapter describes how to set up Ethernet over InfiniBand (EoIB) network

configuration for your Exalogic machine.
For more information about cabling a gateway switch to an external data center 10 Gb

Ethernet switch, see

Connectivity Between Exalogic Machine and External LAN

Through Sun Network QDR InfiniBand Gateway Switch

. For information about cable

and transceiver requirements, see

Transceiver and Cable Requirements

This chapter contains the following topics:

•

Introduction to Virtual NICs (VNICs)

•

Set Up Ethernet Over InfiniBand (EoIB) on Oracle Linux

•

Set Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris

10.1 Introduction to Virtual NICs (VNICs)

A virtual NIC or VNIC maps an Ethernet connector on the Sun Network QDR

InfiniBand Gateway Switch to a network interface within the compute node. A

connector (

0A-ETH-1

0A-ETH-4

, and

1A-ETH-1

1A-ETH-4

) hosts a 10 GbE

port. In the Exalogic machine, on each Sun Network QDR InfiniBand gateway switch,

you can have a maximum of eight 10 GbE uplinks. In most scenarios, the number of 10

GbE uplinks is less than 8.
One or more VNICs can be assigned to a compute node. If more than one VNIC is

assigned, they can be different connectors on the same Sun Network QDR InfiniBand

Gateway Switch or on different Sun Network QDR InfiniBand Gateway Switches. The

default configuration is to map one connector on each of the two switches and to bond

or balance the pair for increased availability.

Note:

In Exalogic, you create VNICs in Sun Network QDR InfiniBand Gateway

Switch manual mode.

10.1.1 VNIC Resource Limit

On each Sun Network QDR InfiniBand Gateway Switch there can be a maximum of

1K VNICs per logical gateway. Logical gateway denotes either a single external

Ethernet port or a defined group of external Ethernet ports. The 1K limit is shared by

all VLANs on the logical gateway and each defined VLAN ID also consumes one of

the 1K VNICs. For more information on VLANs see

Set Up Virtual LANs

To find out how many VNICs are currently defined, use the

showvnics

command

and for VLANs, use

showvlan

Configure Ethernet Over InfiniBand 10-1

Summary of Contents for Exalogic Elastic Cloud

Page 1: ... Cloud Machine Owner s Guide Release 12 2 1 0 0 E18478 21 April 2017 This document describes how to manage and configure an Exalogic machine Release 12 2 1 0 0 is for Oracle Exalogic Elastic Cloud X6 2 X5 2 X4 2 X3 2 and X2 2 ...

Page 2: ...erms and license restrictions applicable to the programs No other rights are granted to the U S Government This software or hardware is developed for general use in a variety of information management applications It is not developed or intended for use in any inherently dangerous applications including applications that may create a risk of personal injury If you use this software or hardware in ...

Page 3: ...1 3 1 4 Operational Procedures for Exalogic Machines 1 4 1 4 1 Non emergency Power Procedure 1 5 1 4 2 Emergency Power off Considerations 1 6 1 4 3 Cautions and Warnings 1 6 2 Site Requirements 2 1 Environmental Requirements 2 1 2 2 Space Requirements 2 1 2 2 1 Receiving and Unpacking Requirements 2 2 2 2 2 Maintenance Access Requirements 2 2 2 3 Flooring Requirements 2 3 2 4 Electrical Power Requ...

Page 4: ...er Rack 4 6 4 4 Exalogic Machine Eighth Rack 4 9 4 5 Default Port Assignments 4 12 5 Add the Exalogic Machine to Your Network 5 1 Default State of the Exalogic Machine Network Configuration 5 1 5 2 Verify the Factory Configuration 5 1 5 3 Prerequisites for Connecting Exalogic Machine to External Ethernet Network 5 5 5 4 Initial Network Configuration of Exalogic Machine 5 5 5 4 1 Configuring the Ci...

Page 5: ...ement Network Diagram for Exalogic Machine 7 2 7 4 ILOM IP Addresses for Exalogic Machine Components 7 3 7 5 Connect to ILOM via the Network 7 3 7 5 1 Connect to the CLI 7 4 7 5 2 Connect to the Web GUI 7 4 7 5 3 Launch a Remote KVM Session 7 4 7 6 Connect to ILOM via a Serial Connection 7 5 7 6 1 Connect to the ILOM of a Compute Node 7 5 7 6 2 Connect to the ILOM of a Sun Network QDR InfiniBand G...

Page 6: ...ntroduction to Virtual NICs VNICs 10 1 10 1 1 VNIC Resource Limit 10 1 10 2 Set Up Ethernet Over InfiniBand EoIB on Oracle Linux 10 2 10 3 Set Up Ethernet Over InfiniBand EoIB on Oracle Solaris 10 7 10 3 1 Set Up Ethernet Over InfiniBand on Oracle Solaris 11 1 10 7 10 3 2 Set Up Ethernet Over InfiniBand on Oracle Solaris 11 2 10 12 11 Set Up Virtual LANs 11 1 Introduction to VLAN 11 1 11 2 Example...

Page 7: ...Subnet Manager in Multirack Configuration Scenarios 14 4 14 5 Check Link Status 14 5 14 6 Verify the InfiniBand Fabric in a Multirack Configuration 14 5 14 6 1 Discover the InfiniBand Network Topology in a Multirack Configuration 14 6 14 6 2 Perform Diagnostics on the InfiniBand Fabric in a Multirack Configuration 14 6 14 6 3 Check for Errors in the InfiniBand Fabric in a Multirack Configuration 1...

Page 8: ...tches in the Fabric 16 5 16 3 4 Determine the SM Priority on an InfiniBand Switch 16 5 16 3 5 Log In to the InfiniBand Switch That Runs Master SM 16 5 16 4 Move from a Default Partition to a Custom Partition 16 5 16 5 Create an IPoIB Partition and Adding Ports 16 6 16 6 Delete a Partition 16 9 16 7 Create a Partition for EoIB and Associating the pkey with a VNIC and VLAN 16 9 16 8 Perform the Post...

Page 9: ...se Manager Ops Center 17 32 17 17 2 Viewing Service Requests 17 33 17 17 3 Filing a Service Request 17 35 18 Install the ASR Software 18 1 About Oracle Auto Service Request ASR 18 1 18 2 Recommended Configuration 18 2 18 3 Before You Begin 18 2 18 4 Prerequisites for Installing ASR Manager 18 2 18 5 Install ASR Manager on a Standalone System 18 2 18 5 1 Install Service Tags for Oracle Linux 18 2 1...

Page 10: ... 2 Compute Nodes D 8 D 9 Parts for Sun Server X3 2 Compute Nodes D 9 D 10 Parts for X4170 M2 Compute Nodes D 10 D 11 Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X5 2 D 11 D 12 Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X4 2 D 12 D 13 Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X2 2 and X3 2 D 12 D 14 Parts for Sun Datacenter InfiniBan...

Page 11: ...Cabling Full Rack E 27 E 3 6 Three Phase PDU Power Cabling Half Rack E 28 E 3 7 Three Phase PDU Power Cabling Quarter Rack E 29 E 3 8 Three Phase PDU Power Cabling Eighth Rack E 30 F Manage Solaris Zones on Exalogic F 1 Requirements F 1 F 2 Terminology F 1 F 3 Create a Solaris Zone F 2 F 3 1 Prerequisites F 2 F 3 2 Set Up a Solaris Zone F 8 F 4 Migrate a Zone to a New Host F 12 G Customize Linux o...

Page 12: ...xii ...

Page 13: ...l Port Connection 7 8 8 1 Network Ports on the Storage Appliance 8 4 8 2 Network Configuration Screen 8 5 8 3 Network Interface Settings 8 6 8 4 igb2 and igb3 in an IPMP Group 8 9 8 5 Creating a New IPMP Group Interface 8 10 8 6 IPMP Network Interface Settings 8 12 8 7 Create Filesystem 8 18 9 1 NIS Service 9 2 9 2 NFS Service 9 3 9 3 NIS Service 9 4 17 1 Oracle Enterprise Manager Home Page 17 4 1...

Page 14: ...gs Page 18 6 18 3 ASR Activation Asset Screen in My Oracle Support 18 7 C 1 Exalogic Eighth Rack Network Connectivity C 2 C 2 Exalogic Quarter Rack Network Connectivity C 3 C 3 Exalogic Half Rack Network Connectivity C 3 C 4 Exalogic Full Rack Network Connectivity C 4 xiv ...

Page 15: ...nd Permissions 8 18 8 4 Case Sensitivity Values 8 19 8 5 Normalization Settings 8 20 12 1 NM2 GW Specifications 12 1 13 1 Running the Subnet Manager in Different Rack Configurations 13 2 14 1 Sun Datacenter InfiniBand Switch 36 Specifications 14 1 D 1 Rack Level Replacement Parts for EL X5 2 D 2 D 2 Rack Level Replacement Parts for EL X4 2 D 2 D 3 Rack Level Replacement Parts for EL X2 2 and X3 2 ...

Page 16: ...Machine Half Rack E 17 E 10 InfiniBand Network Cabling for Exalogic Machine Quarter Rack E 18 E 11 InfiniBand Gateway Switch Inter Connections for Exalogic Machine Quarter Rack E 20 E 12 InfiniBand Network Cabling for Exalogic Machine Eighth Rack E 21 E 13 InfiniBand Gateway Switch Inter Connections for Exalogic Machine Eighth Rack E 22 E 14 Single Phase PDU Power Cabling for Exalogic Machine Full...

Page 17: ...ems that may result from following the average specifications in this document For detailed information about preparing your site for Oracle Exalogic machine deployment consult your hardware specification This preface contains the following sections Audience Revision History Documentation Accessibility Related Documents Conventions Audience This guide is intended for Oracle Exalogic machine custom...

Page 18: ...key Allocation in a Hybrid Rack E18478 14 August 2013 Default Port Assignments Added default ports assignments of the Exalogic rack Initial Configuration Updated note about the initial configuration of the storage appliance Set Up Ethernet Over InfiniBand on Oracle Solaris 11 1 Added Oracle Solaris 11 1 procedure for setting up Ethernet over InfiniBand Oracle Solaris Creating VNICs and Associating...

Page 19: ...pter 16 Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center Added information up front to clarify the scope of the content in this chapter Replacement Units Added replacement parts list for the new hardware components in X3 2 Rebranded the document cater to both X3 2 and X2 2 E18478 09 September 2012 Configure Ethernet Over InfiniBand Updated the procedure to set up EoIB on ...

Page 20: ...trs if you are hearing impaired Related Documents For more information see the following documents Oracle Exalogic Release Notes Oracle Exalogic Machine Multirack Cabling Guide Conventions The following text conventions are used in this document Convention Meaning boldface Boldface type indicates graphical user interface elements associated with an action or terms defined in text or the glossary i...

Page 21: ...leverage an internal InfiniBand fabric that connects all of the processing storage memory and external network interfaces within an Exalogic machine to form a single large computing device Each Exalogic machine is connected to the customer s data center networks via 10 GbE traffic and GbE management interfaces Customers can integrate Exalogic machines with an Exadata machine or additional Exalogic...

Page 22: ... 2 Sun Server X4 2 and Oracle Server X5 2 30 16 8 4 Storage Appliance X6 2 Oracle ZS5 ES storage appliance X2 2 and X3 2 Dual controller Sun ZFS Storage 7320 appliance 60 TB X4 2 and X5 2 Oracle ZS3 ES storage appliance 1 1 1 1 Sun Network QDR InfiniBand Gateway Switches NM2 GW 4 2 2 2 Sun Datacenter InfiniBand Switch 36 NM2 36P 1 1 0 0 Cisco Ethernet management switch X2 2 48 port Cisco Catalyst ...

Page 23: ... the spares kit available to them Note that any parts used by Oracle Service personnel from the spares kit will be replenished by Oracle 1 2 3 Pre cabling for Exalogic X4 2 and newer Machines The eighth quarter and half rack X4 2 and newer machines are pre cabled with a complete set equivalent in number to a full rack of InfiniBand 92 Ethernet 42 and power cables All the cables are connected to th...

Page 24: ...one unit filler NM2 GW InfiniBand Switch Gateway U25 X X X X Vented one unit filler Cisco Ethernet Management Switch U24 X Vented one unit filler NM2 GW InfiniBand Switch Gateway U23 X X X X Vented one unit filler NM2 GW InfiniBand Switch Gateway U21 U22 X X X X Storage Appliance Server Head Storage Appliance Server Head U17 to U20 X X X X Storage Appliance Disk Shelf Storage Appliance Disk Shelf ...

Page 25: ...data center 1 4 1 1 Power On Sequence The power on sequence is as follows 1 Rack including switches Ensure that the switches have had power applied for a few minutes to complete power on configuration before starting the storage controllers and compute nodes 2 Storage server heads attached to the chassis of the storage device Wait for a few minutes for the storage appliance to boot and start NFS s...

Page 26: ... off Switch Emergency power off EPO switches are required when computer equipment contains batteries capable of supplying more than 750 volt amperes for more than five minutes Systems that have these batteries include internal EPO hardware for connection to a site EPO switch or relay Use of the EPO switch will remove power from the Exalogic machine 1 4 3 Cautions and Warnings The following caution...

Page 27: ...e com technetwork server storage engineered systems exalogic index html 2 2 Space Requirements All Exalogic machines use the same rack and have the same space requirements The space requirements are as follows Height 1998 mm 78 66 inches Width 600 mm with side panels 23 62 inches Depth front door handle to rear door handle 1200 mm 47 24 inches Depth doors removed 1112 mm 43 78 inches The minimum c...

Page 28: ...les before entering the data center The entire access route to the installation site should be free of raised pattern flooring that can cause vibration Allow enough space for unpacking it from its shipping cartons Ensure that there is enough clearance and clear pathways for moving the Exalogic machine from the unpacking location to the installation location Table 2 1 lists the access route require...

Page 29: ...nt including PDUs 1004 71 kg 2215 lbs Note Open tiles are required for electrical access 2 4 Electrical Power Requirements Exalogic Machine can operate effectively over a wide range of voltages and frequencies However it must have a reliable power source Damage may occur if the ranges are exceeded Electrical disturbances such as the following may damage Exalogic Machine Fluctuations caused by brow...

Page 30: ...a Center Receptacle NEMA L6 30R IEC309 3P4W IP67 60A 250V AC 3 ph IEC309 60A 3ph 4 Wire Hubbell HBL460R C9W or equivalent Hubbell CS8265C IEC309 3P4W IP67 60A 250V AC 3 ph IEC309 60A 3ph 4 Wire Hubbell HBL460R C9W or equivalent Number of Outlets Per Rack 6 2 6 4 Table 2 5 lists the PDU high voltage requirements Table 2 5 PDU Requirements for High Voltage Specification 15 kVA 3 ph 22 kVA 1 ph 24 kV...

Page 31: ...uit AC current rating PDU power cords are 4 m 13 12 feet long and 1 to 1 5 m 3 3 to 4 9 feet of the cord will be routed within the rack cabinet The installation site AC power receptacle must be within 2 m 6 6 feet of the rack 2 4 2 Circuit Breaker Requirements To prevent catastrophic failures the design of your power system must ensure that adequate power is provided to all of the compute nodes Us...

Page 32: ...3 percent relative humidity For optimal data center rack cooling 45 to 50 percent non condensing Altitude 3048 meters 10000 feet maximum 12000 meters 40000 feet Ambient temperature is reduced by 1 degree Celsius per 300 m above 900 m altitude above sea level Set conditions to the optimal temperature and humidity ranges to minimize the chance of downtime due to component failure Operating an Exalog...

Page 33: ...ely affect air flow and cooling within the rack Relative humidity is the percentage of the total water vapor that can exist in the air without condensing and is inversely proportional to air temperature Humidity goes down when the temperature rises and goes up when the temperature drops For example air with a relative humidity of 45 percent at a temperature of 24 degrees Celsius 75 degrees Fahrenh...

Page 34: ...gher inlet temperature in the servers due to exhaust air recirculation The following is the recommended number of floor tiles Four floor tiles for Exalogic machine full rack Three floor tiles for Exalogic machine half rack One floor tile for Exalogic machine quarter rack and for Exalogic machine eighth rack Figure 2 1 shows a typical installation of the floor tiles for an Exalogic machine full rac...

Page 35: ...tallation of the Exalogic Machine 3 Unpack the Exalogic machine 4 Place the Exalogic machine in its allocated space 5 Perform preliminary checks before connecting the power cords 6 Perform a visual inspection of the hardware 7 Supply rack power and perform the power on self test device checks 8 Switch on the six PDU circuit breakers located on the rear of the PDU A 9 Wait three to five minutes for...

Page 36: ...Go to http download oracle com docs cd E19657 01 index html to download these guides Exalogic machine cabinets can enclose a variety of rack mountable Sun servers storage products and other third party equipment 3 2 2 Exalogic Installation Safety Guidelines Before the Exalogic machine arrives the following safety precautions should be reviewed to ensure the site is safe as well as ready for delive...

Page 37: ...ditioner welding machine or any other equipment that generates loud electronic noises Avoid static electricity at the installation location Static electricity transferred to the Exalogic machine can cause malfunctions Static electricity is often generated on carpets Confirm that the supply voltage and frequency match the electrical ratings indicated on the Exalogic machine Do not insert anything i...

Page 38: ... used to secure the rack to the shipping pallet You can use these mounting brackets to secure the rack permanently to the installation site floor Do not dispose of these brackets as you will not be able to order replacement brackets Figure 3 1 Unpacking Exalogic Machine Rack 3 2 4 Tools for Installation The following tools are required for installation Screwdriver handle magnetic T 30 Torx wrench ...

Page 39: ...her installed equipment Note For more information see Environmental Requirements 4 Install all necessary electrical equipment and ensure that sufficient power is provided See Also The Sun Rack II Power Distribution Units User s Guide for the Sun Rack II Power Distribution Unit PDU power requirements 5 Ensure that the installation site provides adequate air conditioning 6 Operate the air conditioni...

Page 40: ... by pushing on the side panels Pushing on the side panels can tip the rack over This action can cause serious personal injury or death as well as damage to the equipment Never tip or rock the Exalogic machine because the rack can fall over 3 2 6 2 Stabilize the Exalogic Machine After moving the Exalogic machine to the installation site stabilize the rack to ensure that it does not move or tip over...

Page 41: ... feet to the rear rack surface is 32 5 mm 1 28 inches 7 Distance from the center of front casters to the side of the rack is 86 7 mm 3 41 inches 8 Width between the center of the front casters is 426 6 mm 16 80 inches 9 Distance from the center of the rear casters to the rear of the rack is 173 7 mm 6 83 inches 10 Depth between the front and rear casters is 828 6 mm 32 62 inches 11 Distance betwee...

Page 42: ...ite floor using the same mounting brackets that secured the rack to the shipping pallet The rack is secured to the pallet with four mounting brackets Use the front and rear brackets to stabilize the rack to the installation floor Prepare the installation site by drilling four holes into the floor Before permanently stabilizing the Exalogic machine with the mounting brackets pre drill the mounting ...

Page 43: ... different bolt types and strengths 2 Position the Exalogic machine over the pre drilled holes Figure 3 5 shows the bottom view of the Exalogic machine and the location for the mounting hole and floor cut dimensions Figure 3 5 Bottom View of Exalogic Machine Showing Mounting Hole and Floor Cutout Dimensions 1 Distance from mounting bracket to the edge of the rack is 113 mm 4 45 inches Prepare to I...

Page 44: ...es If you plan to route data or PDU power cords down through the bottom of the rack then you need to cut a hole in the installation floor site Cut a rectangular hole below the rear portion of the rack between the two rear casters and behind the rear RETMA Radio Electronics Television Manufacturers Association rails Figure 3 6 shows the base positions of the rack from the bottom Figure 3 6 Base Pos...

Page 45: ...logic machine power distribution units PDUs achieve earth ground through their power cords Final chassis ground is achieved by way of the ground prong when you connect the power cord to a socket For additional grounding attach a chassis earth ground cable to the Exalogic machine The additional ground point enables electrical current leakage to dissipate more efficiently WARNING The PDU power input...

Page 46: ...e 3 8 The attachment point is an adjustable bolt that is inside the rear of the Exalogic machine cabinet on the right side Figure 3 8 Earth Ground Attachment Bolt Location 3 3 Power on the System the First Time Before powering on the system for the first time it is necessary to inspect the machine and connect the power cords This section contains the following topics Inspect the Machine After It I...

Page 47: ...the front of the Exalogic machine See Also Ventilation and Cooling Requirementsfor more information 3 3 2 Connect the Power Cords The following procedure describes how to connect power cords to the Exalogic machine 1 Open the rear cabinet door 2 Ensure that the correct power connectors have been supplied 3 Unfasten the power cord cable ties The ties are for shipping only and are no longer needed 4...

Page 48: ...six power distribution unit PDU circuit breakers located on the rear of the main PDU PDU A inside the Exalogic machine The circuit breakers are on the rear of the Exalogic machine cabinet as shown in Figure 3 11 Press the ON side of the toggle switch Note The 24 kVA PDU has 4 plugs and the 15 kVA PDU has 2 plugs In either case you can use the instructions described in this section to power on your...

Page 49: ...cessor SP powers on and starts up to manage the system The main power is supplied for the remainder of the chassis and fans when the power button on the front of the server chassis is pressed The power button is disabled while the SP is starting The start up state is indicated by a steady one second on one second off blinking pattern of the Power OK LED on the front of the system After the SP has ...

Page 50: ...begins to start the Power OK LED illuminates and remains a steady green 7 Press the soft switches located on the front of the two storage heads in the storage appliance as shown in Figure 3 13 In addition see Figure 3 14 Figure 3 13 Soft Switches on the Server Heads in the Sun ZFS Storage 7320 appliance Figure 3 14 shows the soft switch and LED lights on each of the server heads in the storage app...

Page 51: ...king sure that the storage appliance is up and running press the soft switches located on the front of the compute nodes in your Exalogic machine as shown in Figure 3 13 After power on the LED indicators turn green as shown in Figure 3 14 Note If you are using an Exalogic machine full rack you must switch on thirty compute nodes If you are using an Exalogic machine half rack you must switch on six...

Page 52: ...the Exalogic machine as follows 1 Ensure that you have a laptop with functional USB and network ports 2 Ensure that you have a Category 5E patch cable of maximum length 25 feet and a serial cable of maximum length 15 feet 3 Open the rear cabinet door of the rack 4 Connect the network port of your laptop into an unused input port in the Cisco Ethernet switch This switch is inside a vented filler pa...

Page 53: ...igured using the Oracle Exalogic Configuration Utility For the list of default NET0 IP addresses see Default IP Addresses and Ports If you are using a Linux operating system on your laptop do the following a Log in as a root user b At the command prompt enter the following command to display the network devices such as ETH0 attached to the Exalogic machine ifconfig a The list of network devices or...

Page 54: ...run the Oracle Exalogic Configuration Utility set of tools and scripts you can use the network IP address that you provided as input to the Exalogic Configurator spreadsheet For more information about Oracle Exalogic Configuration Utility see Configure the Exalogic Machine Using ECU 3 5 What Next After powering on the Exalogic machine including compute nodes and storage appliance proceed to config...

Page 55: ... hardware components in an Exalogic machine Full Rack Table 4 1 Default ILOM NET0 and IB Bonded IP for Exalogic Full Rack Unit Component Front View ILOM IP Address NET0 IP Address IB Bonded IP Address 42 Compute Node 192 168 1 132 192 168 1 32 192 168 10 32 41 Compute Node 192 168 1 131 192 168 1 31 192 168 10 31 40 Compute Node 192 168 1 130 192 168 1 30 192 168 10 30 39 Compute Node 192 168 1 12...

Page 56: ...tch 192 168 1 202 Not applicable Not applicable 22 Server head for the storage appliance 192 168 1 116 192 168 1 16 192 168 10 16 21 Server head for the storage appliance 192 168 1 115 192 168 1 15 192 168 10 15 20 Disk storage tray for the storage appliance Not applicable Not applicable Not applicable 19 18 17 16 Sun Network QDR InfiniBand Gateway Switch 192 168 1 201 Not applicable Not applicabl...

Page 57: ... 168 1 205 Not applicable Not applicable PDU A left from rear view 192 168 1 210 Not applicable Not applicable PDU B right from rear view 192 168 1 211 Not applicable Not applicable Note The following table shows the hardware that is included in different Exalogic versions Ex alo gic Ra ck Compute Node s Server Cisco Management Switch Storage Appliance X6 2 Oracle Server X6 2 Cisco 4948E Oracle ZS...

Page 58: ... 4U Solid Filler Not applicable Not applicable Not applicable 37 Not applicable Not applicable Not applicable 36 Not applicable Not applicable Not applicable 35 Not applicable Not applicable Not applicable 34 4U Solid Filler Not applicable Not applicable Not applicable 33 Not applicable Not applicable Not applicable 32 Not applicable Not applicable Not applicable 31 Not applicable Not applicable N...

Page 59: ...applicable 15 Compute Node 192 168 1 114 192 168 1 14 192 168 10 14 14 Compute Node 192 168 1 113 192 168 1 13 192 168 10 13 13 Compute Node 192 168 1 112 192 168 1 12 192 168 10 12 12 Compute Node 192 168 1 111 192 168 1 11 192 168 10 11 11 Compute Node 192 168 1 110 192 168 1 10 192 168 10 10 10 Compute Node 192 168 1 109 192 168 1 9 192 168 10 9 9 Compute Node 192 168 1 108 192 168 1 8 192 168 ...

Page 60: ...ode s Server Cisco Management Switch Storage Appliance X6 2 Oracle Server X6 2 Cisco 4948E Oracle ZS5 ES X5 2 Oracle Server X5 2 Cisco 4948E Oracle ZS3 ES X4 2 Sun Server X4 2 Cisco 4948E F S Oracle ZS3 ES X3 2 Sun Server X3 2 Cisco 4948E F S Sun ZFS 7320 X2 2 Sun Fire X4170 M2 Cisco 4948 Sun ZFS 7320 4 3 Exalogic Machine Quarter Rack Table 4 3 lists the default ILOM NET0 and InfiniBand Bonded IP ...

Page 61: ... applicable Not applicable 35 Not applicable Not applicable Not applicable 34 4U Solid Filler Not applicable Not applicable Not applicable 33 Not applicable Not applicable Not applicable 32 Not applicable Not applicable Not applicable 31 Not applicable Not applicable Not applicable 30 2U Solid Filler Not applicable Not applicable Not applicable 29 Not applicable Not applicable Not applicable 28 2U...

Page 62: ...ch 192 168 1 201 Not applicable Not applicable 15 2U Solid Filler Not applicable Not applicable Not applicable 14 Not applicable Not applicable Not applicable 13 4U Solid Filler Not applicable Not applicable Not applicable 12 Not applicable Not applicable Not applicable 11 Not applicable Not applicable Not applicable 10 Not applicable Not applicable Not applicable 9 Compute Node 192 168 1 108 192 ...

Page 63: ...e that is included in different Exalogic versions Ex alo gic Ra ck Compute Node s Server Cisco Management Switch Storage Appliance X6 2 Oracle Server X6 2 Cisco 4948E Oracle ZS5 ES X5 2 Oracle Server X5 2 Cisco 4948E Oracle ZS3 ES X4 2 Sun Server X4 2 Cisco 4948E F S Oracle ZS3 ES X3 2 Sun Server X3 2 Cisco 4948E F S Sun ZFS 7320 X2 2 Sun Fire X4170 M2 Cisco 4948 Sun ZFS 7320 4 4 Exalogic Machine ...

Page 64: ... Not applicable 35 Not applicable Not applicable Not applicable 34 4U Solid Filler Not applicable Not applicable Not applicable 33 Not applicable Not applicable Not applicable 32 Not applicable Not applicable Not applicable 31 Not applicable Not applicable Not applicable 30 2U Solid Filler Not applicable Not applicable Not applicable 29 Not applicable Not applicable Not applicable 28 2U Solid Fill...

Page 65: ...6 Sun Network QDR InfiniBand Gateway Switch 192 168 1 201 Not applicable Not applicable 15 2U Solid Filler Not applicable Not applicable Not applicable 14 Not applicable Not applicable Not applicable 13 4U Solid Filler Not applicable Not applicable Not applicable 12 Not applicable Not applicable Not applicable 11 Not applicable Not applicable Not applicable 10 Not applicable Not applicable Not app...

Page 66: ...ot applicable Note The following table shows the hardware that is included in different Exalogic versions Ex alo gic Ra ck Compute Node s Server Cisco Management Switch Storage Appliance X6 2 Oracle Server X6 2 Cisco 4948E Oracle ZS5 ES X5 2 Oracle Server X5 2 Cisco 4948E Oracle ZS3 ES X4 2 Sun Server X4 2 Cisco 4948E F S Oracle ZS3 ES X3 2 Sun Server X3 2 Cisco 4948E F S Sun ZFS 7320 X2 2 Sun Fir...

Page 67: ...mpute nodes storage appliance heads and InfiniBand ILOMs HTTPS over TCP 443 Web user configurable Any Compute nodes storage appliance heads and InfiniBand ILOMs IPMI over UDP 623 IPMI Intelligent Platform Management Interface Any Compute nodes and storage appliance heads ILOMs TCP 5120 ILOM remote console CD Any Compute nodes and storage appliance heads ILOMs TCP 5121 ILOM remote console keyboard ...

Page 68: ...vial File Transfer Protocol Compute nodes storage appliance heads and InfiniBand ILOMs Any NTP over UDP 123 Outgoing NTP Compute nodes storage appliance heads and InfiniBand ILOMs Any IPMI over UDP 162 Outgoing IPMI Intelligent Platform Management Interface Platform Event Trap PET Compute nodes storage appliance heads and InfiniBand ILOMs Any Syslog over UDP 514 Outgoing Syslog Compute nodes stora...

Page 69: ...interface Storage management Any NTP over UDP 123 Outgoing NTP Any PDU HTTP over TCP 80 Browser interface Any PDU SNMP over UDP 161 SNMP user configurable Any PDU HTTPS over TCP 443 Browser interface PDU Any SNMP over UDP 162 Outgoing SNMPv2 traps PDU Any Syslog over UDP 514 Outgoing Syslog PDU Any DHCP over UDP 546 DHCP Dynamic Host Configuration Protocol client Default Port Assignments Default I...

Page 70: ...Default Port Assignments 4 16 Machine Owner s Guide ...

Page 71: ...ND0 interfaces Note Exalogic compute nodes are not pre configured with the BOND1 interface for the Ethernet over InfiniBand EoIB connectivity You can configure BOND1 for compute notes during the initial configuration of the Exalogic machine Storage heads configured with NET0 ILOM and BOND0 interfaces Sun Network QDR InfiniBand Gateway Switches referred to as leaf switches configured with the ILOM ...

Page 72: ...1 103 cnode4_host_name el01cn04 cnode4_private_host_name el01cn04 priv cnode4_eth0_IPADDR 192 168 1 4 cnode4_bond0_IPADDR 192 168 10 4 cnode4_SPNET0_IPADDR 192 168 1 104 cnode5_host_name el01cn05 cnode5_private_host_name el01cn05 priv cnode5_eth0_IPADDR 192 168 1 5 cnode5_bond0_IPADDR 192 168 10 5 cnode5_SPNET0_IPADDR 192 168 1 105 cnode6_host_name el01cn06 cnode6_private_host_name el01cn06 priv c...

Page 73: ...168 1 17 cnode15_bond0_IPADDR 192 168 10 17 cnode15_SPNET0_IPADDR 192 168 1 117 cnode16_host_name el01cn16 cnode16_private_host_name el01cn16 priv cnode16_eth0_IPADDR 192 168 1 18 cnode16_bond0_IPADDR 192 168 10 18 cnode16_SPNET0_IPADDR 192 168 1 118 cnode17_host_name el01cn17 cnode17_private_host_name el01cn17 priv cnode17_eth0_IPADDR 192 168 1 19 cnode17_bond0_IPADDR 192 168 10 19 cnode17_SPNET0...

Page 74: ...27 cnode27_private_host_name el01cn27 priv cnode27_eth0_IPADDR 192 168 1 29 cnode27_bond0_IPADDR 192 168 10 29 cnode27_SPNET0_IPADDR 192 168 1 129 cnode28_host_name el01cn28 cnode28_private_host_name el01cn28 priv cnode28_eth0_IPADDR 192 168 1 30 cnode28_bond0_IPADDR 192 168 10 30 cnode28_SPNET0_IPADDR 192 168 1 130 cnode29_host_name el01cn29 cnode29_private_host_name el01cn29 priv cnode29_eth0_IP...

Page 75: ...xadata Database Machine rack Note You can use the Oracle Exalogic Configuration Utility set of tools and scripts to configure the compute nodes storage heads and Sun Network QDR InfiniBand Gateway Switches For more information about Exalogic Configuration Utility see Configure the Exalogic Machine Using ECU You must configure the Cisco Ethernet Switch manually For more information see Configuring ...

Page 76: ...word For the default password contact Oracle Support Switch enable 4 Configure the network for a single VLAN The following is an example of the configuration Switch configure terminal Enter configuration commands one per line End with CNTL Z Switch config interface vlan 1 Switch config if ip address 10 7 7 34 255 255 255 0 Switch config if end Switch Sep 15 14 12 06 309 SYS 5 CONFIG_I Configured f...

Page 77: ...nfig enable password Your_Password Switch config enable secret Your_Password The enable secret you have chosen is thesame as your enable password This is not recommended Re enter the enable secret Switch config end Switch write memory Sep 15 14 25 05 893 SYS 5 CONFIG_I Configured from console by console Building configuration Compressed configuration from 2502 bytes to 1085 bytes OK 8 Set the pass...

Page 78: ...ffset is the hours offset from UTC and minutes offset is the minutes offset from UTC To set summer time hours use the following command clock summer time zone recurring week day month hh mm week day month hh mm offset In the preceding command zone is the time zone to be displayed when summer time is in effect week is the week of the month 1 to 5 or last day is the day of the week month is the mont...

Page 79: ... should show the NTP server synchronized to local time if the Cisco switch is connected to the network and has access to NTP 12 Verify the configuration using the following command Switch show running config The following is an example of the output Building configuration Current configuration 2654 bytes version 12 2 no service pad service timestamps debug datetime msec service timestamps log date...

Page 80: ...rface GigabitEthernet1 46 media type rj45 interface GigabitEthernet1 47 media type rj45 interface GigabitEthernet1 48 media type rj45 interface Vlan1 ip address 10 7 7 34 255 255 255 0 interface Vlan48 no ip address shutdown ip default gateway 10 7 7 1 ip http server control plane line con 0 stopbits 1 line vty 0 4 password Your_Password login line vty 5 15 password Your_Password login ntp server ...

Page 81: ...mory 13 Save the current configuration using the following command Switch copy running config startup config 14 Exit from the session using the following command Switch exit 15 Disconnect the cable from the Cisco console Note To check the configuration attach a laptop computer to port 48 and ping the IP address of the internal management network Initial Network Configuration of Exalogic Machine Ad...

Page 82: ...Initial Network Configuration of Exalogic Machine 5 12 Machine Owner s Guide ...

Page 83: ...e nodes storage appliance and equipment to connect the compute nodes to your network The network connections allow the servers to be administered remotely enable clients to connect to the compute nodes and enable client access to the storage appliance The following table describes the network components and interfaces for each compute node and the storage appliance Table 6 1 Available network comp...

Page 84: ...ts The Cisco Ethernet switch supplied with the Exalogic machine is minimally configured during installation The minimal configuration disables IP routing and sets the following Host name IP address Subnet mask Default gateway Domain name Domain Name Server NTP server Time Time zone Additional configuration such as defining multiple virtual local area networks VLANs or enabling routing may be requi...

Page 85: ...iBand private network This required network connects the compute nodes and the storage appliance through the BOND0 interface to the InfiniBand switches gateways on the Exalogic rack It is the default IP over InfiniBand IPoIB subnet created automatically during the initial configuration of the Exalogic machine Note This network is either based on the default InfiniBand partition or based on a parti...

Page 86: ...6 compute nodes connects 2 Sun Network QDR InfiniBand Gateway Switches in an active passive bond Each compute node is connected to two separate Sun Network QDR InfiniBand Gateway Switches for HA Note that each compute node requires a bond for each external network physical network or VLAN Figure 6 1 shows the network diagram for the Exalogic machine with Oracle Linux operating system Figure 6 1 Ne...

Page 87: ... switches which allow the creation of Exalogic clouds comprising hundreds of racks and tens of thousands of processors 6 3 Important Notes for Oracle Solaris Users If you are using the Oracle Solaris operating system on Exalogic compute nodes keep the following points in mind BOND0 and BOND1 two important terms used in this guide refer to the default interfaces for IP over InfiniBand IPoIB and Eth...

Page 88: ...N VLAN in which the compute node is connected The minimum configuration requires single Bond to single LAN per compute node the number of network connections per Exalogic compute node as outlined in the subsequent table for Full Rack Half Rack and Quarter Rack configurations The following table describes the number of required connections Type of Exalogic Machine Bonded Network Configuration Exalo...

Page 89: ...he network configuration such as host names and IP addresses used during installation is generated from information you supply to your Oracle technical representative in the completed configuration worksheet See Network Configuration Worksheets Configure the new IP addresses in your existing networks only after you have completed the configuration worksheets and received the installation Template ...

Page 90: ...or JMS public and a floating IP address for HTTP private to the Load Balancer In addition T3 typically uses a minimum of two IP addresses One of these IP addresses is private to the Load Balancer and the other floating IP address is exposed to application server clients An application server client initially accesses the application server through a well known virtual IP address such as T3 floatin...

Page 91: ...te node Exalogic machine half rack contains 16 compute nodes One compute node can be a member of only one private network of an application domain quarter rack Minimum of 22 IP addresses At least 2 IP addresses per management network for compute nodes 1 IP address for ILOM 1 IP address for ETH0 2 IP addresses for controllers server heads in the storage appliance 1 IP address per Sun Network QDR In...

Page 92: ...dresses are assigned to Exalogic compute nodes at the time of manufacturing However you can reconfigure the IP addresses by using the Oracle Exalogic Configuration Utility The Oracle Exalogic Configuration Utility and its associated scripts reconfigure InfiniBand Bonded IP addresses based on the input you provide to the Exalogic Configurator spreadsheet For a list of default InfiniBand Bonded IP a...

Page 93: ...en Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch Sun Datacenter InfiniBand Switch 36 part number NM2 36P is used in multirack configuration scenarios an Exalogic machine to another Exalogic machine and an Exalogic machine to an Oracle Exadata Database Machine only This switch is not connected or used in an Exalogic machine single rack Note In the Exalogic mach...

Page 94: ... for IPMP configuration Note For information about the in mpathd daemon and the mpathd configuration file see the in mpathd 1M man page on the Oracle Solaris operating system installed on Exalogic compute nodes For information about ifconfig see the ifconfig 1M man page 6 6 4 2 IPMP Groups An IP multipathing group or IPMP group consists of one or more physical interfaces on the same system that ar...

Page 95: ...hey support eight 10 GB Ethernet ports These ports can be accessed by Exalogic compute nodes through the InfiniBand network through EoIB You can create multiple VLANs per each of these Ethernet ports Each Exalogic compute node can access one or more Ethernet ports on two Sun Network QDR InfiniBand Gateway Switches NM2 GW for HA purposes An Exalogic machine full rack includes 4 gateway switches The...

Page 96: ...y default each Exalogic compute node is configured with one bonded EoIB interface ethX for one external LAN It is BOND1 vnic0 vnic1 which connects to one external LAN such as LAN1 If a vNIC is created at one of the Sun Network QDR InfiniBand Gateway Switches the ethX interface is associated with the vNIC automatically Note You can configure additional EoIB network interfaces for connecting to addi...

Page 97: ...ed by the switch vendor Note Exalogic ships with QSFP transceivers by default Customers may use them on the data center switch side if they use a Sun Oracle 10GbE switch such as the Sun Network 10 GbE Switch 72p 6 6 7 Additional InfiniBand Network Requirements and Specifications Table 6 4 lists additional InfiniBand specifications and cable requirements Table 6 4 HCA Port Specifications and Cable ...

Page 98: ...ine location 8 Inform your Oracle representative when you have completed these steps 6 8 Subnet Manager Requirements for Connecting Exalogic to Exadata See Subnet Manager Operation in Different Rack Configurations 6 9 Network Configuration Worksheets The following networks are used with the Exalogic machine Management network Client access network InfiniBand private network IPoIB The InfiniBand pr...

Page 99: ...gic machine resides Time Zone Valid time zone IP address of the Domain Name Server IP address of one or more network name servers up to four servers Example 10 25 45 123 10 25 45 125 NTP Server IP address of one or more Network Time Protocol servers up to four servers Example 10 12 13 14 10 12 13 15 Search Domains A list of search domains for name lookup up to four domains Example example com exam...

Page 100: ...rksheet includes fields in the Exalogic Configurator spreadsheet that typically require data Table 6 6 Management Network Configuration Worksheet Item Entry Description and Example First Management IP address or First ILOM IP address First IP address in the sequential range used for management network interfaces This network must be distinct from all other networks on the Exalogic machine Example ...

Page 101: ...ration worksheet Fill out this worksheet and provide it to the Oracle representative or use as a reference to complete the initial configuration of your Exalogic machine Information entered in this worksheet is used as input to Oracle Exalogic Configuration Utility This worksheet includes fields in the Exalogic Configurator spreadsheet that typically require data Table 6 8 Private InfiniBand Netwo...

Page 102: ...Network Configuration Worksheets 6 20 Machine Owner s Guide ...

Page 103: ... hardware and software is preinstalled on these components ILOM enables you to actively manage and monitor compute nodes in the Exalogic machine independently of the operating system state providing you with a reliable Lights Out Management LOM system With ILOM you can proactively Learn about hardware errors and faults as they occur Remotely control the power state of your compute node View the gr...

Page 104: ...tions The web interface provides an easy to use browser interface that enables you to log in to the SP then to perform system management and monitoring The command line interface enables you to operate ILOM using keyboard commands and adheres to industry standard DMTF style CLI and scripting protocols ILOM supports SSH v2 0 and v3 0 for secure access to the CLI Using the CLI you can reuse existing...

Page 105: ...guration Utility The Oracle Exalogic Configuration Utility and its associated scripts reconfigure ILOM IP addresses based on the input you provide to the Exalogic Configurator spreadsheet If the Exalogic Configuration Utility fails to reconfigure IP addresses or misconfigures IP addresses you can configure IP addresses manually as described in Configure ILOM IP Addresses Manually 7 5 Connect to IL...

Page 106: ...when prompted The default user name is root The CLI command prompt is displayed 7 5 2 Connect to the Web GUI 1 In the address bar of a web browser type the ILOM IP address of an Exalogic compute node or hardware component and press Enter The login screen is displayed 2 Enter the user name and the password The default user name is root 3 Click Log In The web GUI is displayed 7 5 3 Launch a Remote K...

Page 107: ...om a terminal or a PC running terminal emulation software to the SER MGT port of an Exalogic compute node The cable should be of length 15 feet or less 2 Verify that your terminal or laptop is operational 3 Configure the terminal device or the terminal emulation software to use the following settings 8N1 eight data bits no parity one stop bit 9600 baud default but can be set to any standard rate u...

Page 108: ...re nm2name is the host name of the management controller The name might be the word hostname Even if you do not see the text go to Step 5 5 Log in as the root user The prompt is displayed Note If you do not see this output or prompt there is a problem with the serial configuration the USB to Serial connector or the CLI 7 7 Reconfigure the Network Access This step is required only if your network a...

Page 109: ...ng commands at the ILOM command prompt set pendingipdiscovery static set pendingipaddress ip_address set pendingipnetmask ip_netmask set pendingipgateway ip_gateway set pendingmanagementport SYS MB NET0 set commitpending true 7 7 3 Use the Ipmitool Commands when SP Network Information is Lost If the service processor SP network information gets lost you can use an SSH shell to connect to the ETH0 ...

Page 110: ...al port connection as follows a Establish a serial connection to the SP of a compute node by inserting a serial cable into the SER MGT port on the rear of the compute node b In the terminal window configure the following settings 8N1 Eight data bit no parity one stop bit 9600 Baud Disable hardware control Disable software control c At the login prompt log in as the root user d From your present wo...

Page 111: ...ay set pendingmanagementport SYS MB NET0 set commitpending true 7 8 What Next After configuring ILOM for your Exalogic machine verify the various configurations and proceed to complete the initial configuration of the storage appliance which is included in the Exalogic machine What Next Set Up ILOM on the Compute Nodes 7 9 ...

Page 112: ...What Next 7 10 Machine Owner s Guide ...

Page 113: ... Machine Gathering information such as IP address IP net mask Host name Domain Name Server DNS domain name DNS server IP address Default router IP address and Password for configuring an Ethernet interface on the storage controllers Running the Oracle Exalogic Configuration Utility to reconfigure IP addresses and other network parameters for the storage appliance 8 2 Getting Started You can access...

Page 114: ...appear e Apply the values by pressing ESC 1 or the F1 key or by pressing Enter after confirming the password The final shell configuration screen appears confirming that your appliance is ready for further configuration using the browser user interface BUI 4 Configure the remaining system parameters through a browser running on any client on the same network as the initial interface The management...

Page 115: ...ppliance File systems export a file based hierarchy and can be accessed over NFS over IPoIB in the case of Exalogic machines The project share tuple is a unique identifier for a share within a pool Multiple projects can contain shares with the same name but a single project cannot contain shares with the same name A single project can contain both file systems and LUNs and they share the same name...

Page 116: ...T0 igb0 NET1 igb1 and NET2 igb2 ports on the storage heads are connected to the Cisco management switch which is included in the Exalogic machine The igb0 and igb1 interfaces are reserved for administrative access such as access via a web browser or via command line This configuration ensures that the storage heads are always reachable independent of the load on the network data interfaces and ind...

Page 117: ...rking configuration is displayed as shown in Figure 8 2 Note The interface names and IP addresses shown on the screens in this chapter are examples only You must verify the interface names in your environment and use them accordingly Figure 8 2 Network Configuration Screen The Interfaces section shows the configured network interfaces The green icon indicates that an interface is active on the sto...

Page 118: ...work Interface Settings Note The interface names and IP addresses shown on the screens in this chapter are examples only You must verify the interface names in your environment and use them accordingly 8 4 3 Cluster Network Configuration The cluster is set up in an active passive configuration All resources data interface links and storage pool are owned by the active storage head When the active ...

Page 119: ...e see the Network topic in the Oracle ZFS Storage Appliance Administration Guide 8 4 4 Network Configuration Options You can choose any of the following network configuration options for the storage appliance based on your specific requirements Configure Option 1 ASR Support and Separate Paths for Management and Disaster Recovery Configure Option 2 ASR Support and Shared Path for Management and Di...

Page 120: ...d igb1 Separates the disaster recovery path from the management path Note Ensure that the free hanging cable from the igb3 port is connected to your data center network switch Typically for high availability purposes this cable is connected to a data center switch other than the one that Exalogic s Cisco Management Switch is connected to The bonded interface is a new interface such as dr repl inte...

Page 121: ...ment and use them accordingly In the Properties section if you select the Allow Administration option management is enabled on the interface To create an IPMP Group with two interfaces such as igb2 and igb3 you must click the icon next to Interfaces on the Figure 8 2 The Network Interface screen is displayed as shown in Figure 8 5 Configuration Overview Configure the Storage Appliance 8 9 ...

Page 122: ...on your stand by storage head head 2 is used and the management option is disabled The igb2 and igb3 ports are bonded with IP Multipathing IPMP and the management option is enabled on both igb2 and igb3 This configuration option offers the following benefits Supports Automated Service Request ASR for the storage appliance included in the Exalogic machine using ports igb0 and igb1 Supports disaster...

Page 123: ...paddress 215 or https hostname 215 The login screen appears 3 Type root into the Username field and the administrative password that you entered into the appliance shell kit interface and press the Enter key The Welcome screen is displayed 4 Click the Configuration tab and click NETWORK The default networking configuration is displayed 5 On the network configuration screen Figure 8 2 click the pen...

Page 124: ...low Administration option to enable management traffic on both igb2 and igb3 interfaces 7 Click APPLY 8 4 4 3 Configure Option 3 ASR Support and No Disaster Recovery But with Single Management URL In this custom configuration the igb0 port on your active storage head head 1 is used and the management option is enabled The igb0 port on your stand by storage Configuration Overview 8 12 Oracle Exalog...

Page 125: ... IP address or host name you assigned to the NET0 port of either storage head as follows https ipaddress 215 or https hostname 215 The login screen appears 3 Type root into the Username field and the administrative password that you entered into the appliance shell kit interface and press the Enter key The Welcome screen is displayed 4 Click the Configuration tab and click NETWORK The default netw...

Page 126: ... configuration Common project common Shares NODE_SHARES dumps and general These shares are at the compute node level COMMON_SHARES common patches common general and common images Note This table represents the default configuration of the storage appliance before the Exalogic machine rack configuration is modified at the customer s site Oracle Exalogic Configuration Utility does not alter this con...

Page 127: ...erall settings for the project that are independent of any particular protocol and are not related to access control or snapshots While the CLI groups all properties in a single list this section describes the behavior of the properties in both contexts The project settings page contains three sections Space Usage Users and Groups Inherited Properties and Default Settings File systems and LUNs Tab...

Page 128: ...hecksum Controls the checksum used for data blocks Cache device usage Controls whether cache devices are used for the share Synchronous write bias Controls the behavior when servicing synchronous writes By default the system optimizes synchronous writes for latency which leverages the log devices to provide fast response times Database record size Controls the block size used by the file system Th...

Page 129: ...rty allows the volume size to exceed the amount of available space When set the LUN will consume only the space that has been written to the LUN While this allows for thin provisioning of LUNs most file systems do not expect to get out of space from underlying devices and if the share runs out of space it may cause instability or a corruption on clients or both Volume block size The native block s...

Page 130: ... following permissions can be granted R Read Permission to list the contents of the directory W Write Permission to create files in the directory X Execute Permission to look up entries in the directory If users have execute permissions but not read permissions they can access files explicitly by name but not list the contents of the directory Group Group that is the current group of the directory...

Page 131: ...col being used For NFS FTP and HTTP lookups are case sensitive This is default and prioritizes conformance of the various protocols over cross protocol consistency Insensitive All lookups are case insensitive even over protocols such as NFS that are traditionally case sensitive This setting should only be used where CIFS is the primary protocol and alternative protocols are considered second class...

Page 132: ...Oracle through which you can manage your Sun gear Registration is also a prerequisite for using the PhoneHome service The PhoneHome service communicates with Oracle support to provide Fault reporting the system reports active problems to Oracle for automated service response Depending on the nature of the fault a support case may be opened Details of these events can be viewed in Problems Heartbea...

Page 133: ... appliance will validate the credentials and allow you to choose which of your inventory teams to register with The default team for each account is the same as the account user name prefixed with a 3 Commit your changes Note You can see a log of PhoneHome events in Maintenance Logs PhoneHome If the phone home service is enabled before a valid Oracle Single Sign On account has been entered it will...

Page 134: ...Use the Phone Home Service to Manage the Storage Appliance 8 22 Machine Owner s Guide ...

Page 135: ...port file locking There are known limitations on the NLM based locking and NFSv3 it can cause lock ownership issues under certain multihomed system configurations where the source IP addresses are not deterministic between lock and unlock requests 9 1 2 NFSv4 Unlike NFSv3 NFSv4 is a stateful protocol An NFS client obtains a client ID assigned during the initial negotiation phase and uses it for al...

Page 136: ...isplayed as shown in Figure 9 1 3 Note down the domain name and the server IP address listed on the page You require this information when setting up NIS clients on Exalogic compute nodes If you do not see configuration information on this page it means that you should install an NIS Server Figure 9 1 uses an example NIS domain nisdomain example com Figure 9 1 NIS Service 9 3 Configure the Storage...

Page 137: ...uration on the home page 2 Expand Services on the left navigation menu 3 Click NFS The NFS service configuration page is displayed as shown in Figure 9 2 Figure 9 2 NFS Service Enter information in the fields as follows Minimum supported version NFSv3 Maximum supported version NFSv4 Configure the Storage Appliance Configure NFS Version 4 on Exalogic 9 3 ...

Page 138: ...sdomain example com is entered Figure 9 3 NIS Service The NIS service must be up and running For Server select the Use listed servers option and enter the host name or IP address of the NIS server 9 4 Configure an Exalogic Linux Compute Node to Use NFSv4 Configure an Exalogic Linux compute node as follows 1 Log in to the compute node as root 2 Edit the etc idmapd conf configuration file vi etc idm...

Page 139: ...example domainname nisdomain example com 6 Edit the etc nsswitch conf configuration file vi etc nsswitch conf Change the following entries passwd files passwd files nis shadow files shadow files nis group files group files nis automount files nis nisplus aliases files nis nisplus 7 Restart the rpcidmapd service service rpcidmapd restart 8 Restart the ypbind service by running the following command...

Page 140: ...ommon patches 2 On the compute node create a new directory as follows mkdir p u01 common patches 3 Edit the etc fstab configuration file vi etc fstab Add the following line and enter the correct values for nfs4 and proto tcp el01sn01 export common patches u01 common patches nfs4 rw bg hard nointr rsize 131072 wsize 131072 proto tcp 4 Mount all shared volumes by running the following command mount ...

Page 141: ...DR InfiniBand gateway switch you can have a maximum of eight 10 GbE uplinks In most scenarios the number of 10 GbE uplinks is less than 8 One or more VNICs can be assigned to a compute node If more than one VNIC is assigned they can be different connectors on the same Sun Network QDR InfiniBand Gateway Switch or on different Sun Network QDR InfiniBand Gateway Switches The default configuration is ...

Page 142: ...H 4 Bridge 0 1 up Enabled Bridge 0 Port 1A ETH 1 Bridge 1 2 down Enabled Bridge 0 Port 1A ETH 2 Bridge 1 2 down Enabled Bridge 0 Port 1A ETH 3 Bridge 1 1 up Enabled Bridge 0 Port 1A ETH 4 Bridge 1 1 up Enabled From this output identify the uplinks You can determine that you can use any of the following Ethernet connectors for creating a VNIC 0A ETH 1 0A ETH 2 0A ETH 3 0A ETH 4 1A ETH 3 1A ETH 4 No...

Page 143: ...node You can also specify the bonded IPoIB address of the compute node Example el01cn01 iblinkinfo pl R grep el01cn01 65 15 4X 10 0 Gbps Active LinkUp 121 2 el01cn01 EL C 192 168 10 29 HCA 1 Could be 5 0 Gbps 64 15 4X 10 0 Gbps Active LinkUp 120 1 el01cn01 EL C 192 168 10 29 HCA 1 Could be 5 0 Gbps From the output of the iblinkinfo command note the switch lid value 65 in first column associated wi...

Page 144: ...ress unicast The above address is an example only 6 As ilom admin log in to the gateway switch el01gw04 that you identified in Step 4 7 Run the following command to create a VLAN hostname createvlan connector vlan vlan_ID pkey default Example e101gw04 createvlan 1A ETH 3 vlan 0 pkey default 8 Run the following command to create a VNIC hostname createvnic connector guid compute_node_port_GUID mac u...

Page 145: ...e in the interface file must not be based on the kernel assigned ethX interface name eth4 eth5 and so on Instead Oracle recommends that the interface file name and value of the DEVICE directive in the interface file be derived from the EPORT_ID and IOA_PORT values as follows Note Any other unique naming scheme is also acceptable a Run the following command to find the EPORT_ID mlx4_vnic_info i eth...

Page 146: ...ctive specify the dummy MAC address created in step 5 14 Create an interface file for the second VNIC say eth5 Be sure to name the interface file and specify the DEVICE directive by using a derived interface name and not the kernel assigned name as described earlier In addition be sure to specify the relevant dummy MAC address for the HWADDR directive 15 After creating the interface files create t...

Page 147: ...ort 0A ETH 1 Bridge 0 2 up Enabled Bridge 0 Port 0A ETH 2 Bridge 0 2 up Enabled Bridge 0 Port 0A ETH 3 Bridge 0 1 up Enabled Bridge 0 Port 0A ETH 4 Bridge 0 1 up Enabled Bridge 0 Port 1A ETH 1 Bridge 1 2 down Enabled Bridge 0 Port 1A ETH 2 Bridge 1 2 down Enabled Bridge 0 Port 1A ETH 3 Bridge 1 1 up Enabled Bridge 0 Port 1A ETH 4 Bridge 1 1 up Enabled From this example identify the uplinks In this...

Page 148: ...he switch lids by comparing the first column of the iblinkinfo output to the lid value of the ibswitches command as follows a On the compute node run the ibswitches command on the command line The example output of this command is as follows el01cn02 ibswitches Switch 0x002128548042c0a0 ports 36 SUN IB QDR GW switch el01gw03 enhanced port 0 lid 63 lmc 0 Switch 0x002128547f22c0a0 ports 36 SUN IB QD...

Page 149: ...stconfig 8 To create a VLAN run the following command e101gw05 createvlan 1A ETH 3 vlan 1706 pkey default 9 Note the ID of the VLAN you created by running the showvlan command as follows showvlan Connector LAG VLN PKEY 1A ETH 3 0 ffff 1A ETH 3 1706 ffff In this example the VLAN ID is 1706 10 Run the following command to create a VNIC on the switch el01gw05 createvnic 1A ETH 3 guid 00 21 28 00 01 A...

Page 150: ...teps described in Oracle Solaris Creating VNICs and Associating Them with VLANs 13 Create another VNIC for the same compute node using a connector on a different gateway switch by following steps 1 to 12 Note the name of this interface and its corresponding link For example eoib1 interface with the link net8 14 Delete the following files etc hostname bond1 etc hostname eoib0 etc hostname eoib0 15 ...

Page 151: ...aces for the VNICs you created in step 16 by running the following commands hostname ipadm create ip interface_name Example el01cn02 ipadm create ip eoib0 el01cn02 ipadm create ip eoib1 e Set one of the interfaces as a standby for the bonded interface by running the following command hostname ipadm set ifprop p standby on m ip interface_name Example e101cn02 ipadm set ifprop p standby on m ip eoib...

Page 152: ... Up Ethernet Over InfiniBand on Oracle Solaris 11 2 To set up Ethernet over InfiniBand connectivity for Exalogic compute nodes running Oracle Solaris 11 2 Base Image of EECS 2 0 6 2 0 perform the following procedure 1 Use an SSH client such as PuTTY to log in to a compute node as root For this example log in to el01cn16 as root 2 Run the following command to verify that the image version is EECS 2...

Page 153: ...AGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS net4 2128000 21280001EFF369 1 up el01sw ib02 0a eth 1 7FFF 8206 FFFF el01sw ib02 0a eth 2 el01sw ib02 0a eth 3 root el01cn16 dladm show ib net5 LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS net5 2128000 21280001EFF36A 2 up el01sw ib02 0a eth 1 7FFF 8206 FFFF el01sw ib02 0a eth 2 el01sw ib02 0a eth 3 Make a note of the data that the command displ...

Page 154: ... ib03 port2 net5 el01sw ib02 8 Run the following commands to create the EoIB datalink over net4 and display the results of the procedure root el01cn16 dladm create eoib l net4 g el01sw ib03 c 0A ETH 1 eoib0 root el01cn16 dladm show eoib LINK GWNAME GWPORT GWID FLAGS SPEED MACADDRESS OVER eoib0 el01sw ib03 0a eth 1 506 aH 10000 0 0 0 0 0 0 net4 9 Open another terminal and log in as root to the swit...

Page 155: ...80001eff36a mac 00 14 4F 09 16 02 vNIC created Run the following command to display the result of the creation of the VNIC root el01sw ib02 showvnics grep cn16 108 WAIT IOA N 0021280001EFF36A el01cn16 EL C 192 168 10 16 0000 00 14 4F 09 16 02 NO 0xffff 0A ETH 1 13 From the compute node session run the following command to create a VNIC with a VLAN tag root el01cn16 dladm create vnic l eoib1 v 3066...

Page 156: ...UP N 0021280001EFF369 el01cn16 EL C 192 168 10 16 31744 80 C0 A0 09 16 01 NO 0xffff 0A ETH 1 106 UP H 0021280001EFF369 el01cn16 EL C 192 168 10 16 64513 02 08 20 42 A1 F1 3066 0x8206 0A ETH 1 16 On the switch el01sw ib02 the passive VNIC is not expected to appear until the IPMP group failover process runs From the session to the switch el01sw ib02 run the following command to verify the state of t...

Page 157: ...net link must be configured to support the defined VLANs The benefits include a logical division of workload enforcing security isolation and splitting traffic across several manageable broadcast domains VLANs allow traffic separation from the 10 GbE switch to compute nodes By design Ethernet traffic on one VLAN cannot be seen by any host on a different VLAN To enable communication between two VLA...

Page 158: ...h an external router if they are required to be seen by each other Note On a single Ethernet connection using the connector on the gateway switch you can create up to 4094 VLANs 11 3 Tag the Ethernet Connectors With a VLAN Identifier To tag an Ethernet connector on the gateway switch with a VLAN identifier you must run the createvlan command on the gateway switch that the VLAN will be associated w...

Page 159: ... steps 1 through 6 as described in Set Up Ethernet Over InfiniBand EoIB on Oracle Linux 2 Run the following command to create a VNIC and associate it with a VLAN for example with VLAN 10 as shown in Tag the Ethernet Connectors With a VLAN Identifier createvnic 1A ETH 3 GUID 00 21 28 56 d0 a2 c0 a0 mac a2 c0 a0 a8 1 1 vlan 10 pkey default Where 1A ETH 3 is the Ethernet connector 00 21 28 56 d0 a2 c...

Page 160: ...ation Be sure to create a bonded interface comprising two VNICs on Oracle Linux for high availability purposes 11 5 Oracle Solaris Creating VNICs and Associating Them with VLANs If you wish to associate a VNIC with a VLAN you should provide a VLAN identifier when mapping the MAC address To create a VNIC and associate with a VLAN use the following example procedure 1 On the gateway switch CLI as ro...

Page 161: ... in Set Up Ethernet Over InfiniBand EoIB on Oracle Solaris The two new host administered interfaces are the ones with IDs 1 and 2 with VLANs 10 and 11 respectively Tip After creating the interfaces you can run the dladm command to verify the MAC address on the compute node For example to verify the new interface and its MAC address run one of the following commands on the Oracle Solaris compute no...

Page 162: ...Oracle Solaris Creating VNICs and Associating Them with VLANs 11 6 Machine Owner s Guide ...

Page 163: ...ollowing topics Physical Specifications Access the Command Line Interface CLI of a Gateway Switch Verify the Status of a Gateway Switch Start the Subnet Manager Manually Check Link Status Verify the InfiniBand Fabric Monitor a Gateway Switch Using Web Interface 12 1 1 Physical Specifications This section introduces Sun Network QDR InfiniBand Gateway Switches which are also referred to as leaf swit...

Page 164: ...ollows a Connect a serial terminal terminal server or workstation with a TIP connection to the USB to serial adapter Configure the terminal or terminal emulator with these settings 115200 baud 8 bits No parity 1 Stop bit and No handshaking b Press the Return or Enter key on the serial device several times to synchronize the connection You might see text similar to the following CentOS release 5 2 ...

Page 165: ...asured V1P2 AND 1 16 V Measured 1 2V BridgeX 1 21 V Measured 1 8V 1 80 V Measured 1 2V Standby 1 20 V Voltage test returned OK Starting PSU test PSU 0 present PSU 1 present PSU test returned OK Starting Temperature test Back temperature 23 00 Front temperature 32 62 SP temperature 26 12 Switch temperature 45 maxtemperature 45 Bridge 0 temperature 41 maxtemperature 42 Bridge 1 temperature 43 maxtem...

Page 166: ...Different Rack Configurations For example to set the SM on a gateway switch to priority 5 run the following command setsmpriority 5 The following output is displayed OpenSM 3 2 6_20090717 Reading Cached Option File etc opensm opensm conf Loading Cached Option routing_engine ftree Loading Cached Option sminfo_polling_timeout 1000 Loading Cached Option polling_retry_number 3 Command Line Arguments P...

Page 167: ...p Enabled Bridge 0 0 Port 0A ETH 3 up Enabled Bridge 0 0 Port 0A ETH 4 up Enabled Connector 1A ETH Present Bridge 1 1 Port 1A ETH 1 up Enabled Bridge 1 1 Port 1A ETH 2 up Enabled Bridge 1 0 Port 1A ETH 3 up Enabled Bridge 1 0 Port 1A ETH 4 up Enabled Connector 0B Present Switch Port 19 up Enabled Connector 1B Present Switch Port 21 up Enabled Connector 15B Not present 12 1 6 Verify the InfiniBand ...

Page 168: ...id 14 4xQDR vendid 0x2c9 devid 0x673c sysimgguid 0x3ba000100e38b caguid 0x3ba000100e388 Ca 2 H 0003ba000100e388 nsn33 43 HCA 1 2 3ba000100e38a S 0021283a8389a0a0 23 lid 14 lmc 0 Sun DCS 36 QDR switch localhost lid 15 4xQDR Note The actual output for your InfiniBand fabric will differ from that in the example 12 1 6 2 Perform Diagnostics on the InfiniBand Fabric To perform a collection of tests on ...

Page 169: ...InfiniBand Fabric Use the ibcheckerrors command that uses the topology file to scan the InfiniBand fabric and validate the connectivity as described in the topology file and to report errors as indicated by the port counters On the command line interface CLI enter the following command ibcheckerrors Summary 4 nodes checked 0 bad nodes found 34 ports checked 0 ports have errors beyond threshold Not...

Page 170: ...itch is installed with the default vNICs vnic0 and vnic1 configured on separate Sun Network QDR InfiniBand Gateway Switches for the Ethernet over InfiniBand EoIB BOND1 interface Optionally you can create VLANs and vNICs using the InfiniBand gateway switches What Next 12 8 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 171: ...g system and file system From the prompt on the CLI you can type hardware commands to perform some administrative and management tasks Hardware commands are user friendly and can perform some testing upon the switch chip enabling greater control of a gateway switch and its operation After you log in to the root account the shell prompt appears and you can enter shell commands Enter the hardware co...

Page 172: ...ut only one SM is active at a time The active SM is Master SM others are Standby SMs If the master SM shuts down or fails a standby SM will automatically become the master SM Note In the Exalogic machine the InfiniBand switches both leaf and spine are automatically configured to separate the IP over InfiniBand IPoIB traffic and the Ethernet over InfiniBand EoIB traffic 13 2 2 Subnet Manager Operat...

Page 173: ...sary to benefit from the features of the latest firmware Note that the SM should run on at least two switches in the fabric Consider a configuration that consists of three or more spine switches for example two Exalogic machines connected to two Exadata machines but with varying firmware versions If two or more of the available spine switches across the configuration have the highest firmware vers...

Page 174: ...n the command line interface CLI run the following command getmaster l The output is displayed as in the following example getmaster l Last ring buffer history listed whereismaster daemon is running 20091204 15 00 53 whereismaster started 20091204 15 00 55 No OpenSM Master seen in the system 20091204 15 06 19 OpenSM Master on Switch 0x0002c9000100d050 ports 36 Sun DCS 36 QDR switch o4nm2 36p 2 nor...

Page 175: ... If the master SM is not running on a spine switch log in as a root user to the leaf switch where the master SM is located 3 Disable SM on the switch as described in Disable Subnet Manager on a Switch This step relocates the master SM to another switch in the network 4 Perform the above steps until the master SM relocates to the spine switch Sun Datacenter InfiniBand Switch 36 5 Enable SM on the l...

Page 176: ...icast domain When you create IP subnets ensure that each of the interfaces per Exalogic compute node for these additional IP subnets above the default IPoIB subnet is bonded for high availability HA purposes For more information see the Application Isolation by Subnetting over IPoIB topic in the Oracle Exalogic Enterprise Deployment Guide 13 3 2 Verify the Default Partition You can verify the defa...

Page 177: ...hines together on the same InfiniBand fabric This chapter contains the following topics Physical Specifications Access the CLI of a Sun Datacenter InfiniBand Switch 36 Verify the Switch Status Start the Subnet Manager in Multirack Configuration Scenarios Check Link Status Verify the InfiniBand Fabric in a Multirack Configuration Monitor the Spine Switch Using Web Interface What Next 14 1 Physical ...

Page 178: ...ial communication with the command line interface CLI as follows a Connect a serial terminal terminal server or workstation with a TIP connection to the USB to serial adapter Configure the terminal or terminal emulator with these settings 115200 baud 8 bits No parity 1 Stop bit and No handshaking b Press the Return or Enter key on the serial device several times to synchronize the connection You m...

Page 179: ... BridgeX 1 21 V Measured 1 8V 1 80 V Measured 1 2V Standby 1 20 V Voltage test returned OK Starting PSU test PSU 0 present PSU 1 present PSU test returned OK Starting Temperature test Back temperature 23 00 Front temperature 32 62 SP temperature 26 12 Switch temperature 45 maxtemperature 45 Bridge 0 temperature 41 maxtemperature 42 Bridge 1 temperature 43 maxtemperature 44 Temperature test returne...

Page 180: ...Manager on the spine switch to priority 8 run the following command on the CLI of the spine switch setsmpriority 8 The following output is displayed OpenSM 3 2 6_20090717 Reading Cached Option File etc opensm opensm conf Loading Cached Option routing_engine ftree Loading Cached Option sminfo_polling_timeout 1000 Loading Cached Option polling_retry_number 3 Command Line Arguments Priority 8 Creatin...

Page 181: ... 22 up Enabled Connector 2A Present Switch Port 24 up Enabled Connector 15A Not present Connector 0A ETH Present Bridge 0 1 Port 0A ETH 1 up Enabled Bridge 0 1 Port 0A ETH 2 up Enabled Bridge 0 0 Port 0A ETH 3 up Enabled Bridge 0 0 Port 0A ETH 4 up Enabled Connector 1A ETH Present Bridge 1 1 Port 1A ETH 1 up Enabled Bridge 1 1 Port 1A ETH 2 up Enabled Bridge 1 0 Port 1A ETH 3 up Enabled Bridge 1 0...

Page 182: ...id 0x2c9 devid 0xbd36 sysimgguid 0x21283a8389a0a3 switchguid 0x21283a8389a0a0 21283a8389a0a0 Switch 36 S 0021283a8389a0a0 Sun DCS 36 QDR switch localhost enhanced port 0 lid 15 lmc 0 23 H 0003ba000100e388 2 3ba000100e38a nsn33 43 HCA 1 lid 14 4xQDR vendid 0x2c9 devid 0x673c sysimgguid 0x3ba000100e38b caguid 0x3ba000100e388 Ca 2 H 0003ba000100e388 nsn33 43 HCA 1 2 3ba000100e38a S 0021283a8389a0a0 2...

Page 183: ...mp ibdiagnet log for complete log I Done Run time was 1 seconds Note The actual output for your InfiniBand fabric will differ from that in the example 14 6 3 Check for Errors in the InfiniBand Fabric in a Multirack Configuration Use the ibcheckerrors command that uses the topology file to scan the InfiniBand fabric and validate the connectivity as described in the topology file and to report error...

Page 184: ...the root user 3 Click the Switch Fabric Monitoring Tools tab 4 Click Launch Sun DCS GW Monitor The Fabric Monitor is displayed 14 8 What Next After setting up the Sun Datacenter InfiniBand Switch 36 in a multirack configuration scenario you can proceed to monitor and control the InfiniBand fabric Monitor the Spine Switch Using Web Interface 14 8 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 185: ...r a Node Display Data Counters for a Node Display Low Level Detailed Information for a Node Display Low Level Detailed Information for a Port Map LIDs to GUIDs Perform Comprehensive Diagnostics for the Entire Fabric Perform Comprehensive Diagnostics for a Route Determine Changes to the InfiniBand Topology Determine Which Links Are Experiencing Significant Errors Check All Ports 15 1 1 Identify All...

Page 186: ...in the following example Ca 0x0003ba000100e388 ports 2 nsn33 43 HCA 1 Ca 0x5080020000911310 ports 1 nsn32 20 HCA 1 Ca 0x50800200008e532c ports 1 ib 71 HCA 1 Ca 0x50800200008e5328 ports 1 ib 70 HCA 1 Ca 0x50800200008296a4 ports 2 ib 90 HCA 1 Note The output in the example is just a portion of the full output and varies for each InfiniBand topology 15 1 3 Display the InfiniBand Fabric Topology To un...

Page 187: ...nformation by displaying the GUIDs ports and LIDs of the nodes On the command line interface CLI run the following command ibtracert slid dlid where slid is the LID of the source node and dlid is the LID of the destination node in the fabric The output is displayed as in the following example ibtracert 15 14 From switch 0x0021283a8389a0a0 portnum 0 lid 15 15 Sun DCS 36 QDR switch localhost 23 ca p...

Page 188: ...ort info Lid 15 DR path slid 15 dlid 65535 0 23 LinkState Active PhysLinkState LinkUp LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 4X LinkSpeedSupported 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 10 0 Gbps Note The actual output for your InfiniBand fabric will differ from that in the example 15 1 6 Display Counters ...

Page 189: ...de in the fabric and port is the port of the node For example ibdatacounts 15 23 XmtData 6048 RcvData 6048 XmtPkts 84 RcvPkts 84 Note The actual output for your InfiniBand fabric will differ from that in the example 15 1 8 Display Low Level Detailed Information for a Node If intensive troubleshooting is necessary to resolve a problem you can use the smpquery command to display very detailed inform...

Page 190: ...e a problem you can use the smpquery command to display very detailed information about a port On the command line interface CLI enter the following command smpquery portinfo lid port where lid is the LID of the node in the fabric For example smpquery portinfo 15 23 Mkey 0x0000000000000000 GidPrefix 0x0000000000000000 Lid 0x0000 SMLid 0x0000 CapMask 0x0 DiagCode 0x0000 MkeyLeasePeriod 0 LocalPort ...

Page 191: ...Band fabric Note Creation of the mapping file is not a requirement for InfiniBand administration The following procedure creates a file that lists the LID in hexadecimal the GUID in hexadecimal and the node description 1 Create an inventory file osmtest f c i inventory txt The inventory txt file can be used for other purposes too besides this procedure 2 Create a mapping file cat inventory txt gre...

Page 192: ... in the fabric and dlid is the LID of the destination node The ibdiagpath log file contains the log of the testing 15 1 13 Determine Changes to the InfiniBand Topology If your fabric has a number of nodes that are suspect the osmtest command enables you to take a snapshot inventory file of your fabric and at a later time compare that file to the present conditions Note Although this procedure is m...

Page 193: ... to port0x21283a8620b0f0 Aug 13 19 45 02 375997 B7EF96C0 0x02 osmtest_validate_sa_class_port_info SA Class Port Info base_ver 1 class_ver 2 cap_mask 0x2602 cap_mask2 0x0 resp_time_val 0x10 Aug 13 19 45 02 378991 B7EF96C0 0x01 osmtest_validate_node_data Checkingnode 0x0021283a8620b0a0 LID 0x14 Aug 13 19 45 02 379172 B7EF96C0 0x01 osmtest_validate_node_data Checkingnode 0x0021283a8620b0b0 LID 0x15 A...

Page 194: ...In addition in the output of the ibdiagnet command search for the link_error_recovery_counter string That line contains the recovery error count in hexadecimal The preceding lines identify the node and port with the errors Recovery errors are major errors and the respective links must be investigated for the cause of the rapid symbol error propagation Additionally the ibdiagnet log file contains t...

Page 195: ... the command line interface CLI run the following command ibclearerrors The output is displayed as in the following example Summary 5 nodes cleared 0 errors 15 2 2 Clear Data Counters When you are optimizing the InfiniBand fabric for performance you might want to know how the throughput increases or decreases according to changes you are making to the fabric and Subnet Manager The ibclearcounters ...

Page 196: ...ymbol error generation The ibportstate command can set the speed to 2 5 5 0 or 10 0 GB sec On the command line interface CLI run the following command ibportstate lid port speed value where lid is the LID of the node in the fabric port is the port of the node and value is the speed you want to set Note Adding speed values enables either speed For example speed 7 is 2 5 5 0 and 10 0 GB sec For exam...

Page 197: ... of unreliable operation Partition A connector and port pair are identified as being isolated from the InfiniBand fabric Both the Blacklist and Partition passphrases survive reboot You unlock these passphrases using the enableswitchport command with the reason option Note State changes made with the ibportstate command are not recognized by the disableswitchport enableswitchport or listlinkup comm...

Page 198: ... the port 1 36 For example enableswitchport reason Blacklist 14A Enable Switch port 7 Initial PortInfo Port info DR path slid 65535 dlid 65535 0 port 7 LinkState Down PhysLinkState Disabled LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 4X LinkSpeedSupported 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 2 5 Gbps After Po...

Page 199: ...ions Overview of Partitioning Understand Partition Keys Before You Begin Move from a Default Partition to a Custom Partition Create an IPoIB Partition and Adding Ports Delete a Partition Create a Partition for EoIB and Associating the pkey with a VNIC and VLAN Perform the Post Configuration Steps Important Notes for Combined Exalogic Exadata Fabric Users Partitioning Limitations 16 1 Overview of P...

Page 200: ...idelines for Managing pkey Allocation in a Hybrid Rack The term hybrid rack denotes an Exalogic machine on which half the compute nodes are in a physical configuration and the other half constitutes a virtualized data center For more information about hybrid racks see the Exalogic Elastic Cloud Release Notes On a hybrid Exalogic rack Exalogic Control ensures that a unique pkey is assigned to each ...

Page 201: ...ardless of whether it is on the physical or virtual part will have a unique pkey 16 3 Before You Begin Before you can start creating unique InfiniBand partitions you must complete the following tasks 1 Verify the switch firmware version 2 Gather the port GUIDs of compute nodes and BridgeX ports of gateway switches 3 Identify the InfiniBand switches in your Exalogic machine s InfiniBand fabric and ...

Page 202: ...mpute node Alternatively you can run the following command to display only GUIDs ibstat grep Port GUID 16 3 2 1 Identify BridgeX Ports on Gateway Switches To identify the BridgeX ports on the gateway switches that are connected to your compute nodes run the following command at the command prompt on each gateway switch that your compute node is connected to showgwports This command displays the Br...

Page 203: ...t Manager SM is running getmaster This command displays output as shown in the following example Local SM enabled and running 20111122 08 45 02 Master SubnetManager on sm lid 11 sm guid 0x21283bad45c0a0 SUN IB QDR GW switch el01gw04 10 10 10 10 16 3 5 Log In to the InfiniBand Switch That Runs Master SM After identifying the InfiniBand switch where master SM is running log in to the ILOM shell for ...

Page 204: ...mmand smpartition start 3 Create the myIPoIB partition with the pkey 0x005 with full membership by running the following command smpartition create n myIPoIB pkey 0x8005 m full flag ipoib 4 Run the following command to add the compute node port GUIDs which you noted down in Gather Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches to the myIPoIB partition smpartition add n myIPoIB p...

Page 205: ...he following command to view the changed partition configuration smpartiiton list modified This command displays the new partition with its pkey ports added to the partition and membership type 8 Run the following command to confirm the partition configuration smpartition commit 9 Create interfaces for the ibp0 and the ibp1 network devices and bond them by running these steps a Log in to the Brows...

Page 206: ...hare mounted by following these steps a SSH to a compute node that is a member of the partition as the root user b Determine the active infiniband device by running the following commands on the compute node ifconfig ib0 ifconfig ib1 The active device will have non zero Rx Tx bytes c Configure the network for the active device by running the following commands echo 0x8005 sys class net ib0 create_...

Page 207: ...t on one of the gateway switches run the following command el01gw04 listlinkup grep Bridge The following is an example of the output of the lislinkup command Connector 0A ETH Present Bridge 0 Port 0A ETH 1 Bridge 0 2 up Enabled Bridge 0 Port 0A ETH 2 Bridge 0 2 up Enabled Bridge 0 Port 0A ETH 3 Bridge 0 1 up Enabled Bridge 0 Port 0A ETH 4 Bridge 0 1 up Enabled Bridge 0 Port 1A ETH 1 Bridge 1 2 dow...

Page 208: ...SM lid 6 Capability mask 0x02510868 Port GUID 0x0021280001a0a366 Link layer IB In the output information about two ports is displayed Identify the GUID and Base lid of the port that you want to use for creating the VNIC For the example illustrated in this procedure we will use the port with GUID 0x0021280001a0a366 and Base lid 121 b On the same compute node run the following command to view inform...

Page 209: ...fine a dummy MAC address in the following format last3_octets_of_switchGUID last3_octets_of_computenode_adminIP_in_hex_format Example GUID of switch 00 21 28 56 d0 a2 c0 a0 Last three octets a2 c0 a0 Administrative IP of the compute node that requires the VNIC 192 168 1 1 Last three octets 168 1 1 in hexadecimal notation a8 01 01 MAC address a2 c0 a0 a8 01 01 Note The dummy MAC address should be u...

Page 210: ...vlan 1A ETH 3 vlan 10 pkey 0x005 Where 1A ETH 3 is the Ethernet connector on the gateway switch 10 is the VLAN identifier and 0x005 is the partition key that you created earlier 13 To verify run the following command showvlan The following information is displayed Connector LAG VLN PKEY 1A ETH 3 10 0x005 0A ETH 1 11 ffff 14 As root log in to el01gw04 that you identified in Step 4 Use its IP addres...

Page 211: ...r on a different gateway switch Note the ethX ID of this VNIC too It is recommended that you configure the two EoIB interfaces as a bonded interface such as bond1 19 Create interface files for the VNICs on the compute node To ensure correct failover behavior the name of the VNIC interface file and the value of the DEVICE directive in the interface file must not be based on the kernel assigned ethX...

Page 212: ... HWADDR a2 c0 a0 a8 01 01 MASTER bond1 SLAVE yes Make sure that the name of the interface file ifcfg eth331_1 in the example is the name derived in step 19 For the DEVICE directive specify the device name eth331_1 in the example derived in step 19 For the HWADDR directive specify the dummy MAC address created in step 4 21 Create an interface file for the second VNIC say eth5 Be sure to name the in...

Page 213: ...er on the partitioned network 1 Log in as a root user 2 Run the following commands on the command line cd sys class net ib0 echo 0x8033 create_child 3 Run the following command to verify that the child interface was created ifconfig ib0 8033 4 Specify your setup for the child interface in an ibcfg ib0 8033 file in the etc sysconfig networks scripts directory Note that it is 8033 even if it might b...

Page 214: ...itations when creating non default partitions Once a new partition configuration is successfully committed using the smpartition command on the current master SM the configuration is kept highly available among the defined set of SM instances However all Sun Network QDR InfiniBand Gateway Switches defined to have SM enabled that is defined by the smnodes command on each gateway switch must be oper...

Page 215: ...ise Manager Ops Center 12c documentation at http docs oracle com cd E27363_01 index htm For more information about Enterprise Manager Cloud Control see the Enterprise Manager Cloud Control 12c documentation at http docs oracle com cd E24628_01 index htm Note Exalogic virtual configurations include Exalogic Control which provides the management and monitoring interface for Exalogic The information ...

Page 216: ...ware components of the Exalogic machine 17 2 Key Features Oracle Enterprise Manager 11g Ops Center supports the following key features in the Exalogic machine environment Hardware lifecycle management InfiniBand fabric views and Ethernet network view Console access to launch the browser user interface for managing the storage appliance and the InfiniBand switches Serial console access to the servi...

Page 217: ...11857_01 nav management htm Note After installing Oracle Enterprise Manager Ops Center you can deploy the enterprise controller and proxy controllers on the same machine For more information about site preparation and installation see the Site Preparation and Installation sections in the Oracle Enterprise Manager Ops Center documentation Verifying the IP addresses assigned to each of the managemen...

Page 218: ... 6 Understanding the Workflow Figure 17 2 illustrates the workflow for the management of Exalogic machine hardware components using Oracle Enterprise Manager Ops Center Understanding the Workflow 17 4 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 219: ...Figure 17 2 Management of Exalogic Machine Hardware Using Oracle Enterprise Manager Ops Center Understanding the Workflow Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17 5 ...

Page 220: ...Management topic in the Administration section in the Oracle Enterprise Manager Ops Center 11g documentation This topic describes the following tasks Adding a user Deleting a user Roles and authorizations Assigning a role Configuring a notification profile Deleting a notification profile 17 8 Discovering and Managing Exalogic Machine Hardware This section describes how to discover and manage the h...

Page 221: ...riteria for the following assets in your Exalogic machine individually Compute Nodes Storage appliance Sun Network QDR InfiniBand Gateway Switches Sun Datacenter InfiniBand Switch 36 Note This switch is used in multirack configuration scenarios only It is not connected in a Exalogic machine single rack configuration Discovering and Managing Exalogic Machine Hardware Monitoring the Exalogic Machine...

Page 222: ... the Exalogic machine asset categories name and save the criteria You can check Save Criteria for Future Use to save the criteria for future use Note You can launch the discoveries all at once by multiple selection of the saved criteria 5 Click Discover Assets When the assets are discovered they are listed under the Available to be Managed Assets tab on the home page 6 To add assets to the Assets ...

Page 223: ...extend or modify the groups at any time For more information see Discovering and Managing Exalogic Machine Hardware In addition verify that all Exalogic specific assets are listed under Managed Assets as shown in the example Figure 17 9 17 9 2 Creating the Exalogic Top Level Group To create a top level group for Exalogic machine assets complete the following steps 1 On the right navigation pane na...

Page 224: ...f an Exalogic rack For example you can enter Exalogic el01 as a group name 3 After entering the group information and selecting the location click Next The Summary screen is displayed as in Figure 17 5 Grouping Exalogic Machine Hardware Assets 17 10 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 225: ...ne named Actions click Create Group under Organize The Configure Modify Group screen is displayed 2 In the Configure Group screen in the Group Name field enter a name for the new Exalogic group For example enter ComputeNodes Enter a description in the Description field Select the Inside a user defined Group Subgroup option as Location In the Group Selection Panel select the Exalogic top level grou...

Page 226: ...layed 4 On the Summary screen click Finish The ComputeNodes sub group is created under the Exalogic group 17 9 4 Creating a Sub Group for the Storage Appliance After creating a top level group for the Exalogic machine you can create a sub group for the storage appliance which is included in the Exalogic machine You can create a sub group for this storage appliance as follows Grouping Exalogic Mach...

Page 227: ...defined Group Subgroup option as Location In the Group Selection Panel select the Exalogic top level group that you created in Creating the Exalogic Top Level Group The Configure Group screen should be similar to Figure 17 7 Figure 17 7 Configure Sub Group for Storage Appliance 3 After entering the sub group information and selecting the group location click Next The Summary screen is displayed Gr...

Page 228: ... switches as follows 1 On the right navigation pane named Actions click Create Group under Organize The Configure Modify Group screen is displayed 2 In the Configure Group screen in the Group Name field enter a name for the new Exalogic group For example enter IBswitches Enter a description in the Description field Select the Inside a user defined Group Subgroup option as Location In the Group Sel...

Page 229: ...oup 17 9 6 Adding Assets to a Group After creating the top level Exalogic group and its sub groups you must add Exalogic specific managed assets to a group To add an Exalogic specific managed asset to a group do the following 1 On the All Assets page click the Managed Assets tab Exalogic specific managed assets are listed under All Managed Assets Grouping Exalogic Machine Hardware Assets Monitorin...

Page 230: ...ets to Group button The selected asset is added to the selected group sub group 4 Repeat these steps for each of the Exalogic compute nodes the storage appliance and the InfiniBand switches listed under Managed Assets 17 10 Viewing Exalogic Compute Nodes To view Exalogic compute nodes do the following 1 On the left navigation pane under Assets from the drop down list select the All User Defined Gr...

Page 231: ...formation about that compute node 4 Click the Hardware tab to view detailed information about that compute node This information includes CPU memory network adapter disk power supply and fan tray fan For example see Figure 17 10 Viewing Exalogic Compute Nodes Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17 17 ...

Page 232: ...unch the Service Processor SP console from within this screen click the Console tab If your console connection is not enabled enable it by clicking the Enable the Console Connection icon 7 Click the Connect to the Console icon The SP console for the selected compute node is launched as shown in Figure 17 11 Viewing Exalogic Compute Nodes 17 18 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 233: ...the following 1 On the left navigation pane under Assets from the drop down list select the All User Defined Groups option 2 Select the Exalogic top level group The Exalogic group page is displayed as in Figure 17 9 3 On the left navigation pane select a particular switch under the IBswitches sub group to view information about that switch You should see a page similar to the compute node page sho...

Page 234: ...formation about that switch This information includes ports network adapters connected to the switch and so on The hardware information about the switch is displayed as in Figure 17 13 Viewing InfiniBand Switches 17 20 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 235: ...ck the Connectivity tab 7 If you wish to view information about switch ports click the Port Monitoring tab 8 If you wish to launch the Service Processor SP console from within this interface click the Console tab If your console connection is not enabled enable it by clicking the Enable the Console Connection icon 9 Click the Connect to the Console icon The SP console for the selected switch is la...

Page 236: ...group page is displayed as in Figure 17 9 3 On the left navigation pane select the storage appliance under the Storage sub group to view information about the storage appliance as shown in Figure 17 14 Figure 17 14 Storage Appliance View 4 Click the Hardware tab to view detailed information about the storage appliance This information includes CPU memory network adapters disks and so on Viewing th...

Page 237: ... the corresponding storage appliance specifications in your Exalogic machine 6 Click the Storage Shares tab to view the shares exported file systems configured on the storage appliance as shown in Figure 17 16 Viewing the Storage Appliance Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17 23 ...

Page 238: ...cting an appropriate action from the Operate actions pane as shown in Figure 17 15 17 13 Viewing the InfiniBand Fabric and Its Nodes You can view the InfiniBand fabric and its nodes in Enterprise Manager Ops Center To view the InfiniBand fabric do the following 1 On the left navigation pane under Managed Assets from the drop down list select the option that starts with the Fabric string This strin...

Page 239: ...ticular fabric node 3 From this view you can view detailed information about the fabric node such as Summary Hardware Capabilities Connectivity Problems Monitoring Port Monitoring Console Jobs Configuration Charts and Service Requests 4 Similarly on the left navigation pane under the fabric drop down list click a gateway switch Information about the switch as a fabric node is shown as shown in Fig...

Page 240: ...c 5 To monitor switch ports click the Port Monitoring tab Monitoring information about the switch ports is displayed as shown in Figure 17 19 Viewing the InfiniBand Fabric and Its Nodes 17 26 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 241: ...rt Monitoring Console Jobs Configuration Charts and Service Requests 7 Similarly on the left navigation pane under the fabric drop down list click the storage appliance Information about the storage appliance as a fabric node is shown as shown in Figure 17 20 Viewing the InfiniBand Fabric and Its Nodes Monitoring the Exalogic Machine Using Oracle Enterprise Manager Ops Center 17 27 ...

Page 242: ...automate problem identification and resolution When monitoring is enabled it is connected with a problem management and notification system Monitoring includes a standard set of monitoring rules and attributes many of which are editable In addition you can add custom monitoring attributes and alert conditions A new alert is generated every time an attribute does not meet a monitoring rule When an ...

Page 243: ...er administrator has permissions to edit and add monitoring rules and profiles The following are the main components of a complete monitoring configuration Monitoring Rules Express alerting conditions You can apply one or more rules to an asset in order to monitor the asset and raise an alert when the monitoring rule condition is met Monitoring Profiles A set of monitoring rules targeted to a spec...

Page 244: ...ne The Add Alert Monitoring Rule Parameters window is displayed 4 Select a Rule Type from the drop down menu Threshold Boolean Control Enumerated Control or Expression 5 Select an Asset Type from the drop down menu 6 Complete the Monitored Attribute If you selected Expression the Monitored Attribute option is not available 7 Provide a name and description for the rule that will appear in the Profi...

Page 245: ...he Firmware Compliance Report to determine if the firmware on the asset complies with your firmware profile s specifications If assets do not contain the firmware version identified in the profile you can update the firmware from the report Server provisioning reports Obtain a report of historical server provisioning actions Run this simple ad hoc report to obtain details about the Deployment Plan...

Page 246: ... for Using Oracle Services in Oracle Enterprise Manager Ops Center Viewing Service Requests Filing a Service Request 17 17 1 Prerequisites for Using Oracle Services in Oracle Enterprise Manager Ops Center The following are the prerequisites for using Oracle Services in Oracle Enterprise Manager Ops Center Register your assets with My Oracle Support Register your user as a My Oracle Support user in...

Page 247: ...ane click Message Center 2 Click Open Service Requests My Service Requests or Service Requests Opened by Others to display a list of requests 3 To view details of a particular service request highlight a row then click the View Service Request icon The following graphic shows an example service request Using Oracle Services in Oracle Enterprise Manager Ops Center Monitoring the Exalogic Machine Us...

Page 248: ...Using Oracle Services in Oracle Enterprise Manager Ops Center 17 34 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 249: ...ccessfully filing a service request ticket If the asset is not registered in My Oracle Support the service request job will fail To file a service request from a problem do the following 1 Click Message Center in the Navigation pane 2 Click My Problems or Unassigned Problems 3 Highlight the problem then click the phone icon in the center pane To file a service request from an asset do the followin...

Page 250: ...Using Oracle Services in Oracle Enterprise Manager Ops Center 17 36 Machine Owner s Guide ...

Page 251: ...nable faster problem resolution by eliminating the need to initiate contact with Oracle Support Services for common hardware component failures reducing both the number of phone calls needed and overall phone time required ASR also simplifies support operations by using electronic diagnostic data Easily installed and deployed ASR is completely controlled by you the customer to ensure security ASR ...

Page 252: ...of Exalogic to serve as ASR Manager Identify and verify ASR assets Ensure connectivity to the Internet using HTTPS Note For Exalogic compute nodes ASR only covers what the ILOM monitors 18 4 Prerequisites for Installing ASR Manager The following are the prerequisites for installing ASR Manager on a system outside of Exalogic Root access to the system on which ASR Manager will be installed Operatin...

Page 253: ...n article cmd show type NOT doctype SYSTEMDOC id 1185493 1 2 As root run the following command to install the SASM package On Oracle Linux rpm i SUNWsasm version_num rpm On Oracle Solaris pkgadd d SUNWsasm version_num pkg 18 5 3 Install ASR Package To install the ASR package complete the following steps 1 Download the ASR package from the following URL and unzip https support oracle com CSP main a...

Page 254: ...e Appliance To activate the storage appliance included in your Exalogic machine complete the following steps 1 Ensure that you have a valid My Oracle Support Account 2 Ensure that you can connect to the Internet from your storage appliance using either a direct connection or a web proxy 3 In a web browser enter the IP address or host name you assigned to the NET0 port of either storage head as fol...

Page 255: ...Figure 18 1 Phone Home Tab on the List of Services When you click Phone Home the Phone Home page is displayed as shown in Figure 18 2 Activate the Storage Appliance Install the ASR Software 18 5 ...

Page 256: ...ame for the accessing the web proxy server In the Password field enter your password 10 Click the Apply button 11 When the Service Enable Disable popup is presented select the Enable option Note For information about network configuration for the storage appliance see Network Configuration Options ASR activation for the storage appliance covers ILOM on the storage server heads 18 9 Approve and Ver...

Page 257: ...18 3 You may need to update information about your Exalogic machine as required Figure 18 3 ASR Activation Asset Screen in My Oracle Support 5 Click Approve to complete activation Note An Exalogic machine asset must be in an active ASR state in My Oracle Support in order for Service Request autocreate to work 18 9 2 View and Verify ASR Assets To view and verify ASR assets Exalogic machine componen...

Page 258: ... including compute nodes and storage server heads 2 Log in to My Oracle Support You must see the same set of ASR assets with their correct serial numbers Approve and Verify ASR Activation for Exalogic Machine Assets 18 8 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 259: ...ault network topology Compute nodes and storage heads with NET0 ILOM and BOND0 IPoIB interfaces Sun Network QDR InfiniBand Gateway Switches with the ILOM interface Note Sun Datacenter InfiniBand Switch 36 which is included in Exalogic full and half rack machines is not connected or configured by default This switch is used in multirack configuration scenarios only For more information see Use the ...

Page 260: ...them in the configurator spreadsheet which is used as input by Exalogic Configuration Utility Ensure that all the required cabling is completed You must have connected your Exalogic machine to the data center s Ethernet device such as a switch For more information see Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch Set up VNICs as necessary ...

Page 261: ...ist Reracking Checklists B 1 System Components Checklist Complete the following checklist to ensure that the systems component considerations have been addressed System Components Checklist Items Yes No N A Comment Is it an Exalogic machine full rack Exalogic machine half rack Exalogic machine quarter rack or Exalogic machine eighth rack How many racks will be installed If more than one rack will ...

Page 262: ... the data center room requirements are met For information about the data center requirements see Flooring Requirements Data Center Room Checklist Items Yes No N A Comment Has the Exalogic machine location been allocated Is there a vacant location for the new equipment Does the floor layout meet the equipment maintenance access requirements Is there adequate space available for maintenance Will th...

Page 263: ...ments Data Center Environment Considerations Yes No N A comment Does the computer room air handling meet temperature and humidity requirements Does the installation floor layout satisfy the ventilation requirements Will the equipment be positioned so the exhaust air of one rack does not enter the air intake of another rack Are the perforated floor tiles each rated at 400 CFM or greater Do the data...

Page 264: ...yes then provide details Are there any stairs or ramps in the moving patch for the new hardware Have you confirmed that all route incline angles are within the permitted range Have you confirmed that the access route is free of any obstacles that would expose the device to shock Are all the surfaces acceptable for rolling the new unpacked and packed equipment If a pallet jack is to be use then hav...

Page 265: ...r to Flooring Requirements for requirements B 5 Facility Power Checklist Complete the following checklist to ensure that the facility power requirements are met For information about power requirements see Electrical Power Requirements Facility Power Considerations Yes No N A Comment Do you know the required operating voltage and electric current level of the device and peripherals Will you be usi...

Page 266: ...Power Checklist Considerations Yes No N A Comment Do you have the minimum required power sources Are power outlets available for the new equipment at the designated location Does the power frequency meet the equipment specifications Is there a UPS to power the equipment Is the capacity of the UPS sufficient for the Exalogic machine Does the power source support the power load for the new hardware ...

Page 267: ...e data center personnel Is there security or access control for the data center Are there any security background checks or security clearances required for vendor personnel to access the data center If yes then do you have a recommended agency How many days in advance must background checks be completed Are there any additional security access issues Is computer room access available for installa...

Page 268: ...hen do ground level deliveries require a truck with a side rail lift Is there a time constraint on dock access If yes then provide time constraints Is tail lift required on delivery carrier to unload the equipment at the delivery dock Will any of the following be required to place equipment in computer room Stair walkers Lifters Ramps Steel plates Floor covers Does the delivery carrier require any...

Page 269: ...networking worksheets which is included in Network Configuration Worksheets and provide it to your Oracle technical representative Have you received the site specific installation template from your Oracle technical representative Did you review the installation template and consult with your Oracle technical representative regarding site specific changes if any Did you verify the IP addresses in ...

Page 270: ...racle Installation Service Is there a cart capable of carrying the weight of the servers to move the components and associated cabling from the supplied rack to the customer supplied rack Do the target rack dimensions meet the following requirements Height 42RU shorter racks can be supported with customer provided PDUs Width 600mm Depth 1112mm without front and rear doors Is the target rack at lea...

Page 271: ... of 600 mm is recommended to accommodate the PDU and cable harnesses on the side If the rack is less than 600 mm wide then it must have additional depth to accommodate mounting behind the server CMAs Does the target rack support Oracle cable management arms CMAs Does the target rack support installation of Oracle vented and solid filler panels Can the target rack provide tie downs along the left r...

Page 272: ...r requirements in case one PDU fails Can the customer ensure power loads are evenly distributed across all circuits of a single PDU Can the customer provide appropriate power drops for the PDUs B 10 1 Rack Recycling The following checklist is specific to recycling the rack Recycling Considerations Yes No N A Comment Does the customer want Oracle to take back and recycle the empty rack If yes then ...

Page 273: ...c Eighth Rack Exalogic Quarter Rack Exalogic Half Rack Exalogic Full Rack NOT_SUPPORTED For information about connecting an Exalogic machine with one or more Exalogic or Exadata machines see the Exalogic Elastic Cloud Multirack Cabling Guide C 1 Exalogic Eighth Rack Figure C 1 illustrates the cabling for a single Exalogic eighth rack Cabling Diagrams C 1 ...

Page 274: ...o the second gateway switch The remaining half of the compute nodes are connected with their active ports to the second gateway switch and their passive ports to the first gateway switch This connection is for high availability and load distribution purposes Exalogic machine eighth racks do not contain the Sun Datacenter InfiniBand Switch 36 C 2 Exalogic Quarter Rack Figure C 2 illustrates the cab...

Page 275: ...he second gateway switch The remaining half of the compute nodes are connected with their active ports to the second gateway switch and their passive ports to the first gateway switch This connection is for high availability and load distribution purposes Exalogic machine quarter racks do not contain the Sun Datacenter InfiniBand Switch 36 C 3 Exalogic Half Rack Figure C 3 illustrates the cabling ...

Page 276: ...logic Full Rack Figure C 4 illustrates the cabling for a single Exalogic full rack Figure C 4 Exalogic Full Rack Network Connectivity In Figure C 4 NM2 GW represents a Sun Network QDR InfiniBand Gateway Switch leaf switches Servers refers to compute nodes in the Exalogic machine The Sun Datacenter InfiniBand Switch 36 referred to as spine switch is not connected Exalogic Full Rack C 4 Oracle Exalo...

Page 277: ... from the spares kit will be replenished by Oracle This appendix contains the following topics Rack Level FRUs for Exalogic X5 2 Rack Level FRUs for Exalogic X4 2 Rack Level FRUs for Exalogic X2 2 and X3 2 Parts for ZS3 ES Storage Appliance for Exalogic X5 2 Parts for ZS3 ES Storage Appliance for Exalogic X4 2 Parts for Sun ZFS Storage 7320 Appliance Parts for Oracle Server X5 2 Compute Nodes Part...

Page 278: ...99 Pwrcord Jmpr SR2 1m C14RA 10A C13 FRU 7015400 Pwrcord Jmpr SR2 2m C14RA 10A C13 FRU 530 4436 01 CBL 7 ETHRNT CAT5 CAT5E BLACK FRU 530 4502 01 CBL 10 ETHRNT CAT5 CAT5E BLUE FRU 530 4446 01 5M 10GE QSFP Passive Copp CAB FRU 350 1546 01 ASSY 1U CMA KIT EXADATA X4170 FRU 371 4919 01 ASY SLIDE RAIL LONG 1U 2U STAN FRU 135 1204 01 XCVR 40GBps SR QSFP MTO FRU D 2 Rack Level FRUs for Exalogic X4 2 Tabl...

Page 279: ...1M QSFP Passive Copper CBL FRU D 3 Rack Level FRUs for Exalogic X2 2 and X3 2 Table D 3 lists the rack level replaceable parts for Exalogic X2 2 and X3 2 machines Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 3 Rack Level Replacement Parts for EL X2 2 and X3 2 Part Number Description Coverage under Basic Wa...

Page 280: ...Storage Appliance for Exalogic X5 2 Table D 4 lists the replaceable parts for the ZS3 ES storage appliance which is included in the Exalogic X5 2 machine Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 4 Replacement Parts for Oracle ZS3 ES Appliance Part Number Description Coverage under Basic Warranty 701535...

Page 281: ...e Appliance for Exalogic X4 2 Table D 5 lists the replaceable parts for the ZS3 ES storage appliance which is included in the Exalogic X4 2 machine Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 5 Replacement Parts for Oracle ZS3 ES Appliance Part Number Description Coverage under Basic Warranty 7015351 CPU ...

Page 282: ...e ZS3 ES Storage Appliance Customer Service Manual D 6 Parts for Sun ZFS Storage 7320 Appliance Table D 6 lists the replaceable parts for the Sun ZFS Storage 7320 appliance which is included in the Exalogic machine Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 6 Replacement Parts for Sun ZFS Storage 7320 ap...

Page 283: ... 7 Parts for Oracle Server X5 2 Compute Nodes Table D 7 lists the replaceable parts for the Oracle Server X5 2 compute nodes used in the Exalogic machine Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 7 Replacement Parts for Oracle Server X5 2 Compute Nodes Part Number Description Coverage with Basic Warrant...

Page 284: ...ompute nodes used in the Exalogic machine Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 8 Replacement Parts for Sun Server X4 2 Compute Nodes Part Number Description Coverage with Basic Warranty 7058153 Motherboard Assembly FRU 7069240 FRU CPU Intel E5 2697 12Core 2 7 GHz FRU 7060951 FRU PS AC A256 F 12V 60...

Page 285: ... parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 9 Replacement Parts for Sun Server X3 2 Compute Nodes Part Number Description Coverage with Basic Warranty 7026878 1X 2 9 Ghz Intel 8 Core Xeon E5 2690 135 W FRU 7020774 Heatsink CPU X4170 M3 FRU 7018701 16 GB DIMMs FRU F375 3696 Dual port 4X QDR InfiniBand PCI E HCA CX2 FRU F375 3...

Page 286: ...FF 8087 to SFF 8643 620 mm FRU 7027302 Front Indicator Module Cable FRU 7041996 Disk Data Cable SFF 8087 to SFF 8643 650mm FRU For more information see the Sun Server X3 2 Service Manual D 10 Parts for X4170 M2 Compute Nodes Table D 10 lists the replaceable parts for the X4170 M2 compute nodes used in the Exalogic machine Note All replacement parts shown in the following table are considered FRU w...

Page 287: ...885 PCI E Riser X16 1U FRU F541 4275 Paddle Card CR_FANs FRU F530 4228 CBL MINI SAS 36POS LONG LY FRU F530 4431 CABLE FAN_DATA 1U FRU F530 3927 CBL PDB MB 1U 2U RIBBON FRU F530 4417 CABLE FAN PADDLE BD P FRU F541 4276 FAN MODULE CR 40X56 FRU F541 4274 FAN_DECK 1U CR_FANS FRU F541 2075 BUSS_BAR PWR 1U 2U FRU F150 3993 CR2032 Battery for M B FRU For more information see the Sun Fire X4170 M2 Server ...

Page 288: ...ay Switches used in Exalogic X4 2 machines Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 12 Replacement Parts for Sun Network QDR InfiniBand Gateway Switch for Exalogic X4 2 Machines Part Number Description Coverage with Basic Warranty 7054724 NM2 GW system FRU FRU 300 2299 01 PS AC A247A F 12V 760W FRU 706...

Page 289: ...atacenter InfiniBand Switch 36 used in the Exalogic X5 2 machine full rack and half rack configurations Note All replacement parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 14 Replacement Parts for Sun Datacenter InfiniBand Switch 36 Part Number Description Coverage with Basic Warranty 7052970 Assy System NM2 36P Managed FRU 7065...

Page 290: ... parts shown in the following table are considered FRU when covered under Premier Support of Systems warranty Table D 16 Replacement Parts for Sun Datacenter InfiniBand Switch 36 Part Number Description Coverage with Basic Warranty F541 3495 03 FRU ASSY SYSTEM NM2 36P MANAGE FRU F350 1312 04 FRU FAN MODULE 1RU REVERSE FLO FRU F300 2233 02 FRU PS AC A247 F 12V 760W FRU For more information see the ...

Page 291: ...10 0307 Cooling fan for Cisco Catalyst 4948 switch HS FRU For repair information see the Cisco Catalyst 4948E Switch Installation Guide D 19 Parts for the Gari DE2 24C Disk Enclosure for Exalogic X5 2 Table D 19 lists the replacement parts for the Gari DE2 24C disk enclosure for Exalogic X5 2 Table D 19 Replacement Parts for Gari DE2 24C Disk Enclosure Part Number Description Coverage with Basic W...

Page 292: ...24P C I O Module IOM FRU 7043630 ASSY 2U Chassis with Midplane FRU 530 3883 ASSY 2M 4X MINI SAS CBL SHLD FRU 530 3886 ASSY 0 5M 4X MINI SAS CBL SHLD FRU 7044619 ASSY DE2 24P UNIVERSAL RAIL KIT FRU 7044396 ASSY DE2 24P SAS 73GB SSD LogZ FRU 7066831 Assy DRIVE 4T 3 5 SAS 7200 rpm DE2 24C FRU Parts for the Gari DE2 24C Disk Enclosure D 16 Oracle Exalogic Elastic Cloud Machine Owner s Guide ...

Page 293: ...ls and lacer bars are removed the X4 2 and newer servers are installed with rack rails and cable management arms and the cables power InfiniBand and Ethernet are connected to the newly installed compute nodes The following abbreviations are used in the tables Abbreviation Description Rn Rack n where n is the number of the rack such as R1 Un Unit height in rack where n is the number such as U20 Pn ...

Page 294: ...ute node NET0 11 7 Blue Cat 5 U34 Compute node NET0 12 7 Blue Cat 5 U33 Compute node NET0 13 7 Blue Cat 5 U32 Compute node NET0 14 7 Blue Cat 5 U31 Compute node NET0 15 7 Blue Cat 5 U30 Compute node NET0 16 7 Blue Cat 5 U29 Compute node NET0 17 7 Blue Cat 5 U28 Compute node NET0 18 7 Blue Cat 5 U27 Compute node NET0 19 7 Blue Cat 5 U26 Sun Network QDR InfiniBand Gateway Switch NET0 38 7 Black Cat ...

Page 295: ...Compute node NET0 26 7 Blue Cat 5 U12 Compute node NET0 27 10 Blue Cat 5 U11 Compute node NET0 28 10 Blue Cat 5 U10 Compute node NET0 29 10 Blue Cat 5 U9 Compute node NET0 30 10 Blue Cat 5 U8 Compute node NET0 31 10 Blue Cat 5 U7 Compute node NET0 32 10 Blue Cat 5 U6 Compute node NET0 33 10 Blue Cat 5 U5 Compute node NET0 34 10 Blue Cat 5 U4 Compute node NET0 35 10 Blue Cat 5 U3 Compute node NET0 ...

Page 296: ... 7 Blue Cat 5 U21 Storage appliance server head NET0 22 7 Blue Cat 5 U21 Storage appliance server head NET1 23 7 Blue Cat 5 U21 Storage appliance server head NET2 44 7 Blue Cat 5 U16 Sun Network QDR InfiniBand Gateway Switch NET0 41 7 Black Cat 5 U15 Compute node NET0 24 7 Blue Cat 5 U14 Compute node NET0 25 7 Blue Cat 5 U13 Compute node NET0 26 7 Blue Cat 5 U12 Compute node NET0 27 10 Blue Cat 5 ...

Page 297: ...ion Not applicable External LAN NET 1 10 Blue Cat 5 Not applicable PDU A NET 2 1M Grey Cat 5 Not applicable PDU B NET 3 1M Grey Cat 5 U23 Sun Network QDR InfiniBand Gateway Switch NET0 40 7 Black Cat 5 U22 Storage appliance server head NET0 20 7 Blue Cat 5 U22 Storage appliance server head NET1 21 7 Blue Cat 5 U22 Storage appliance server head NET2 43 7 Blue Cat 5 U21 Storage appliance server head...

Page 298: ...AN NET 1 10 Blue Cat 5 Not applicable PDU A NET 2 1M Grey Cat 5 Not applicable PDU B NET 3 1M Grey Cat 5 U23 Sun Network QDR InfiniBand Gateway Switch NET0 40 7 Black Cat 5 U22 Storage appliance server head NET0 20 7 Blue Cat 5 U22 Storage appliance server head NET1 21 7 Blue Cat 5 U22 Storage appliance server head NET2 43 7 Blue Cat 5 U21 Storage appliance server head NET0 22 7 Blue Cat 5 U21 Sto...

Page 299: ...lists the location ports and cables for the InfiniBand connections for an Exalogic machine full rack Table E 5 InfiniBand Network Cabling for Exalogic Machine Full Rack From Rack Unit Sun Network QDR InfiniBand Gateway Switch Port To Rack Unit Type of Equipment Port Description U26 9A U42 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U26 9B U41 Compute node PCIe 2 P1 3 meter QDR InfiniBand c...

Page 300: ...QDR InfiniBand cable U26 15A U30 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U26 15B U29 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U24 9A U42 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U24 9B U41 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U24 10A U40 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U24 10B U39 Compute node PCIe 2 P2 3 meter QDR InfiniBand cab...

Page 301: ...U29 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U23 7A U28 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U23 7B U27 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U23 8A U22 Storage appliance server head PCIe 0 P2 2 meter QDR InfiniBand cable U23 8B U21 Storage appliance server head PCIe 0 P2 2 meter QDR InfiniBand cable U23 9A U15 Compute node PCIe 2 P1 2 meter QDR InfiniBand c...

Page 302: ...e 2 P2 3 meter QDR InfiniBand cable U23 14B U4 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 15A U3 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 15B U2 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U16 7A U28 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U16 7B U27 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U16 8A U22 Storage appliance server head PCIe 0 P...

Page 303: ... U9 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U16 12B U8 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U16 13A U7 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 13B U6 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 14A U5 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 14B U4 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 15A U3 Compute node PCIe ...

Page 304: ...twork QDR InfiniBand Gateway Switch 6A 2 meter QDR InfiniBand cable U16 6B U23 Sun Network QDR InfiniBand Gateway Switch 6B 2 meter QDR InfiniBand cable U24 4A U26 Sun Network QDR InfiniBand Gateway Switch 4A 1 meter QDR InfiniBand cable U24 4B U26 Sun Network QDR InfiniBand Gateway Switch 4B 1 meter QDR InfiniBand cable U24 5A U26 Sun Network QDR InfiniBand Gateway Switch 5A 1 meter QDR InfiniBan...

Page 305: ...d cable U16 1B U26 Sun Network QDR InfiniBand Gateway Switch 1B 2 meter QDR InfiniBand cable U16 2A U24 Sun Network QDR InfiniBand Gateway Switch 2A 2 meter QDR InfiniBand cable U16 2B U24 Sun Network QDR InfiniBand Gateway Switch 2B 2 meter QDR InfiniBand cable U16 3A U24 Sun Network QDR InfiniBand Gateway Switch 3A 2 meter QDR InfiniBand cable U16 3B U24 Sun Network QDR InfiniBand Gateway Switch...

Page 306: ... InfiniBand Gateway Switch 3A 1 meter QDR InfiniBand cable U23 3B U26 Sun Network QDR InfiniBand Gateway Switch 3B 1 meter QDR InfiniBand cable E 2 2 Exalogic Machine Half Rack This section contains the following tables Table E 8 Table E 9 Table E 8 lists the location ports and cables for the InfiniBand connections for an Exalogic machine half rack Table E 8 InfiniBand Network Cabling for Exalogic...

Page 307: ...U13 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U23 10B U12 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U23 11A U11 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U23 11B U10 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U23 12A U9 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 12B U8 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 13A U7 Compute node PC...

Page 308: ...able U16 9A U15 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 9B U14 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 10A U13 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 10B U12 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 11A U11 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 11B U10 Compute node PCIe 2 P2 2 meter QDR InfiniBand cable U16 12A U9 Co...

Page 309: ... an Exalogic machine half rack Table E 9 InfiniBand Gateway Switch Inter Connections for Exalogic Machine Half Rack From Rack Unit Sun Network QDR InfiniBand Gateway Switch Port To Rack Unit Type of Equipment Port Description U16 4A U23 Sun Network QDR InfiniBand Gateway Switch 4A 2 meter QDR InfiniBand cable U16 4B U23 Sun Network QDR InfiniBand Gateway Switch 4B 2 meter QDR InfiniBand cable U16 ...

Page 310: ...Network Cabling for Exalogic Machine Quarter Rack From Rack Unit Sun Network QDR InfiniBand Gateway Switch Port To Rack Unit Type of Equipment Port Description U23 8A U22 Storage appliance server head PCIe 0 P2 2 meter QDR InfiniBand cable U23 8B U21 Storage appliance server head PCIe 0 P2 2 meter QDR InfiniBand cable U23 12A U9 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 12B U8 Comput...

Page 311: ...PCIe 0 P1 3 meter QDR InfiniBand cable U16 12A U9 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U16 12B U8 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U16 13A U7 Compute node PCIe 2 P1 2 meter QDR InfiniBand cable U16 13B U6 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 14A U5 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 14B U4 Compute node PCIe 2 P1 3 meter QDR ...

Page 312: ... Gateway Switch 5A 2 meter QDR InfiniBand cable U16 5B U23 Sun Network QDR InfiniBand Gateway Switch 5B 2 meter QDR InfiniBand cable U16 6A U23 Sun Network QDR InfiniBand Gateway Switch 6A 2 meter QDR InfiniBand cable U16 6B U23 Sun Network QDR InfiniBand Gateway Switch 6B 2 meter QDR InfiniBand cable E 2 4 Exalogic Machine Eighth Rack This section contains the following tables Table E 12 Table E ...

Page 313: ...ompute node PCIe 2 P2 3 meter QDR InfiniBand cable U23 15B U2 Compute node PCIe 2 P2 3 meter QDR InfiniBand cable U16 8A U22 Storage appliance server head PCIe 0 P1 3 meter QDR InfiniBand cable U16 8B U21 Storage appliance server head PCIe 0 P1 3 meter QDR InfiniBand cable U16 14A U5 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable U16 14B U4 Compute node PCIe 2 P1 3 meter QDR InfiniBand cable ...

Page 314: ...QDR InfiniBand Gateway Switch 6A 2 meter QDR InfiniBand cable U16 6B U23 Sun Network QDR InfiniBand Gateway Switch 6B 2 meter QDR InfiniBand cable U16 USB Serial Not applicab le Open ended Not applicable USB Adapter U16 USB Serial Not applicab le Open ended Not applicable USB Adapter E 3 Power Distribution Unit Cabling Tables This section contains the tables for the Power Distribution Units PDU ca...

Page 315: ...ic Machine Full Rack From Rack Unit Type of Equipment PDU A PDU B Jumper Length U42 Compute node G5 5 G0 1 2M U41 Compute node G5 4 G0 2 2M U40 Compute node G5 3 G0 3 2M U39 Compute node G5 2 G0 4 2M U38 Compute node G5 1 G0 5 2M U37 Compute node G5 0 G0 6 2M U36 Compute node G4 6 G1 0 2M U35 Compute node G4 5 G1 1 2M U34 Compute node G4 4 G1 2 2M U33 Compute node G4 3 G1 3 2M U32 Compute node G4 ...

Page 316: ... 3 2M U10 Compute node G1 2 G4 4 2M U09 Compute node G1 1 G4 5 2M U08 Compute node G1 0 G4 6 2M U07 Compute node G0 6 G5 0 2M U06 Compute node G0 5 G5 1 2M U05 Compute node G0 4 G5 2 2M U04 Compute node G0 3 G5 3 2M U03 Compute node G0 2 G5 4 2M U02 Compute node G0 1 G5 5 2M U01 NM2 36P SW G0 0 G5 6 2M E 3 2 Single Phase PDU Power Cabling Half Rack This section contains the following tables Table ...

Page 317: ... 2M U12 Compute node G1 4 G4 2 2M U11 Compute node G1 3 G4 3 2M U10 Compute node G1 2 G4 4 2M U09 Compute node G1 1 G4 5 2M U08 Compute node G1 0 G4 6 2M U07 Compute node G0 6 G5 0 2M U06 Compute node G0 5 G5 1 2M U05 Compute node G0 4 G5 2 2M U04 Compute node G0 3 G5 3 2M U03 Compute node G0 2 G5 4 2M U02 Compute node G0 1 G5 5 2M U01 NM2 36P SW G0 0 G5 6 2M E 3 3 Single Phase PDU Power Cabling Q...

Page 318: ...M U03 Compute node G0 2 G5 4 2M U02 Compute node G0 1 G5 5 2M E 3 4 Single Phase PDU Power Cabling Eighth Rack This section contains the following tables Table E 17 Table E 17 lists the location ports and cables for the Single Phase PDU connections for an Exalogic machine eighth rack Table E 17 Single Phase PDU Power Cabling for Exalogic Machine Eighth Rack From Rack Unit Type of Equipment PDU A P...

Page 319: ...ack Table E 18 Three Phase PDU Power Cabling for Exalogic Machine Full Rack From Rack Unit Type of Equipment PDU A PDU B Jumper Length U42 Compute node G5 5 G2 1 2M U41 Compute node G5 4 G2 2 2M U40 Compute node G5 3 G2 3 2M U39 Compute node G5 2 G2 4 2M U38 Compute node G5 1 G2 5 2M U37 Compute node G5 0 G2 6 2M U36 Compute node G4 6 G1 0 2M U35 Compute node G4 5 G1 1 2M U34 Compute node G4 4 G1 ...

Page 320: ... node G1 4 G4 2 2M U11 Compute node G1 3 G4 3 2M U10 Compute node G1 2 G4 4 2M U09 Compute node G1 1 G4 5 2M U08 Compute node G1 0 G4 6 2M U07 Compute node G0 6 G3 0 2M U06 Compute node G0 5 G3 1 2M U05 Compute node G0 4 G3 2 2M U04 Compute node G0 3 G3 3 2M U03 Compute node G0 2 G3 4 2M U02 Compute node G0 1 G3 5 2M U01 NM2 36P SW G0 0 G3 6 2M E 3 6 Three Phase PDU Power Cabling Half Rack This se...

Page 321: ...6 2M U12 Compute node G1 4 G4 2 2M U11 Compute node G1 3 G4 3 2M U10 Compute node G1 2 G4 4 2M U09 Compute node G1 1 G4 5 2M U08 Compute node G1 0 G4 6 2M U07 Compute node G0 6 G3 0 2M U06 Compute node G0 5 G3 1 2M U05 Compute node G0 4 G3 2 2M U04 Compute node G0 3 G3 3 2M U03 Compute node G0 2 G3 4 2M U02 Compute node G0 1 G3 5 2M U01 NM2 36P SW G0 0 G3 6 2M E 3 7 Three Phase PDU Power Cabling Q...

Page 322: ...2M U03 Compute node G0 2 G3 4 2M U02 Compute node G0 1 G3 5 2M E 3 8 Three Phase PDU Power Cabling Eighth Rack This section contains the following tables Table E 21 Table E 21 lists the location ports and cables for the Three Phase PDU connections for an Exalogic machine eighth rack Table E 21 Three Phase PDU Power Cabling for Exalogic Machine Eighth Rack From Rack Unit Type of Equipment PDU A PDU...

Page 323: ...c Machine Eighth Rack From Rack Unit Type of Equipment PDU A PDU B Jumper Length U05 Compute node G0 4 G3 2 2M U04 Compute node G0 3 G3 3 2M U03 Compute node G0 2 G3 4 2M U02 Compute node G0 1 G3 5 2M Power Distribution Unit Cabling Tables Cabling Tables E 31 ...

Page 324: ...Power Distribution Unit Cabling Tables E 32 Machine Owner s Guide ...

Page 325: ...tch for Zones on Shared Storage ZOSS over iSCSI available in the My Oracle Support document ID 16514816 F 2 Terminology Table F 1 describes the terms used in this appendix Table F 1 Terminology Term Description Logical Unit A logical unit is a component of a storage system A logical unit is uniquely numbered creating a Logical Unit Number LUN The storage appliance can contain many LUNs LUNs when a...

Page 326: ...nitiator Create the Project and LUN Disable the Write Cache Format the LUN Set Up the Exclusive 10 GbE Network for the Zone F 3 1 1 Create an iSCSI Target You can create an iSCSI target by doing the following 1 Log in to the storage appliance BUI as the root user 2 Click the Configuration tab 3 Click SAN 4 Click iSCSI Targets 5 To create a new iSCSI target click the plus button next to iSCSI Targe...

Page 327: ... F 3 1 2 Create an iSCSI Initiator You can create an iSCSI initiator by doing the following 1 Before you can create an iSCSI initiator you must identify an initiator IQN The initiator IQN is a unique reference number associated with a specific compute node To find the initiator IQN for a compute node do the following a Log in to an Exalogic compute node b Run the iscsiadm list initiator node comma...

Page 328: ...11 Click OK 12 Add the iSCSI initiator to an iSCSI initiator group by dragging and dropping the initiator If required you can create a new iSCSI initiator group F 3 1 3 Create the Project and LUN You can create the project and LUN by doing the following 1 Create a project as described in Create Custom Projects 2 You can create the LUN by doing the following a Next to the Project click Shares b Cli...

Page 329: ... 0 0 0 0 0 0 0 0 0 0 0 4 Restart the compute node by running the reboot command F 3 1 5 Format the LUN Before the compute node can use the LUN you must format the LUN You can format the LUN by doing the following 1 Log in to a compute node as the root user 2 Run the iscsiadm commands to discover the iSCSI targets from the compute node iscsiadm add discovery address IPoIB_address_of_the_storage_app...

Page 330: ...un the format command to start formatting the disk as follows format e c0t600144F09C96CCA900005190BFC4000Ad0 selecting c0t600144F09C96CCA900005190BFC4000Ad0 disk formatted The format prompt appears b Enter fdisk to manipulate the partition tables as follows format fdisk No fdisk table exists The default partition for the disk is a 100 SOLARIS System partition c When prompted enter n to edit the pa...

Page 331: ...c7t0d0 LSI MR9261 8i 2 12 28 87GB pci 0 0 pci8086 340a 3 pci1000 9263 0 sd 0 0 Specify disk enter its number F 3 1 6 Set Up the Exclusive 10 GbE Network for the Zone The zone you want to create should be given access to an exclusive network You should create the necessary VNICs for the zone by doing the following 1 Create a VLAN and VNIC Follow the procedure in Oracle Solaris Creating VNICs and As...

Page 332: ...l and Boot Up the Zone F 3 2 1 Create a Zone You can create a zone by doing the following 1 Log in to the compute node as the root user 2 Run the zonecfg command to configure the zone as follows zonecfg z zone_name Example zonecfg z zone04 Use create to begin configuring a new zone In this example the name of the zone you are creating is zone04 3 Enter create to begin configuring the zone zonecfg ...

Page 333: ...zonecfg zone04 remove anet zonecfg zone04 add net zonecfg zone04 net set physical vnic2_1706 zonecfg zone04 net end zonecfg zone04 add net zonecfg zone04 net set physical vnic3_1706 zonecfg zone04 net end zonecfg zone04 verify zonecfg zone04 commit 5 You can verify the details of the zone by running the info command as follows zonecfg zone04 info zonename zone04 zonepath zones zone04 brand solaris...

Page 334: ...eating IPS image Startup linked 1 1 done Installing packages from exa family origin http localhost 1008 exa family acbd22da328c302a86fb9f23d43f5d10f13cf5a6 solaris origin http install1 release solaris DOWNLOAD PKGS FILES XFER MB SPEED Completed 185 185 34345 34345 229 7 229 7 10 6M s PHASE ITEMS Installing new actions 48269 48269 Updating package state database Done Updating image state Done Creat...

Page 335: ...0 lo0 138 3 48 0 138 3 51 2 U 2 0 bond1 Routing Table IPv6 Destination Mask Gateway Flags Ref Use If 1 1 UH 2 0 lo0 7 Add the IP address of the bond you noted in step 5 by running the following command root zone04 route p add default IP_address_of_bond Example root zone04 route p add default 138 3 48 1 add net default gateway 138 3 48 1 add persistent net default gateway 138 3 48 1 8 Display the r...

Page 336: ...ove one or more iSCSI discovery addresses because logical unit is in use Exported zone zpool zone04_rpool Unconfigured zone storage resource s from iscsi 192 168 14 133 luname naa 600144f09c96cca900005190bfc4000a 4 Create a directory on the storage appliance to which you can export the configuration of the zone mkdir p directory Example mkdir p u01 common general zone04 5 Export the configuration ...

Page 337: ...o image Processing linked 1 1 done Updating non global zone Auditing packages No updates necessary for this image Updating non global zone Zone updated Result Attach Succeeded Log saved in non global zone as zones zone04 root var log zones zoneadm 20130513T135704Z zone04 attach 9 Boot up the zone by running the following command zoneadm z name_of_zone boot Example zoneadm z zone04 boot Note In som...

Page 338: ...Migrate a Zone to a New Host F 14 Machine Owner s Guide ...

Page 339: ...endix contains the following sections RPMs That Must Not Be Modified or Removed Prepare the Compute Nodes for Yum Updates Install Update and Remove RPMs Using Yum G 1 RPMs That Must Not Be Modified or Removed Do not modify or delete the following RPMs outside of an Exalogic upgrade patch set update PSU or patch kernel compat dapl dapl ib bonding ibacm ibutils ibsim infiniband diags libibcm libibma...

Page 340: ...entire directive must be on one line exclude kernel compat dapl dapl ib bonding ibacm ibutils ibsim infiniband diags libibcm libibmad libibumad libibverbs libmlx4 librdmacm libsdp mpi selector mpitests_openmpi_gcc mstflint mvapich ofa ofed openmpi_gcc opensm perftest qperf rds tools sdpnetstat srptools exalogic infinibus This exclusion list includes all of the RPMs listed in RPMs That Must Not Be ...

Page 341: ...name For the yum update or yum remove command if you specify an RPM that is in the exclusion list defined earlier the command will fail Install Update and Remove RPMs Using Yum Customize Linux on the Compute Nodes G 3 ...

Page 342: ...Install Update and Remove RPMs Using Yum G 4 Machine Owner s Guide ...

Search results

Summary of Contents for Exalogic Elastic Cloud

Reviews:

Related manuals for Exalogic Elastic Cloud

Brands by name

Popular brands

Oracle Exalogic Elastic Cloud, Owner'S Manual

Search results

Summary of Contents for Exalogic Elastic Cloud

Reviews:

Related manuals for Exalogic Elastic Cloud

Brands by name

Popular brands