manualshive.com logo in svg

IBM N Series, Hardware Manual

The Nokia N Series is a cutting-edge mobile phone series that encompasses advanced features and functionalities. To ensure seamless usage, a comprehensive User Manual is provided with each device, offering detailed instructions and insights. Download your free User Manual for Nokia N Series from manualshive.com and make the most of your device.

Share
Download
background image

Appendix A. Getting started 

319

2-character country code (Required) []: us
Postal code where business resides []: 

The root volume currently contains 2 disks;  you may add more
disks to it later using the "vol add" or "aggr add" commands.
Now apply the appropriate licenses to the system and install
the system files (supplied on the Data ONTAP CD-ROM or downloaded
from the NOW site) from a UNIX or Windows host.  When you are
finished, type "download" to install the boot image and
"reboot" to start using the system.

Thu May  3 05:33:10 GMT [n3300a: init_java:warning]: Java disabled:  Missing 
/etc/java/rt131.jar.
Thu May  3 05:33:10 GMT [dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk 
drives
Thu May  3 05:33:13 GMT [n3300a: 10/100/1000/e0a:info]: Ethernet e0a: Link up
add net default: gateway 9.11.218.1
Thu May  3 05:33:15 GMT [n3300a: httpd_servlet:warning]: Java Virtual Machine not 
accessible
There are 4 spare disks; you may want to use the vol or aggr command
to create new volumes or aggregates or add disks to the existing volume.
Thu May  3 05:33:15 GMT [mgr.boot.disk_done:info]: Data ONTAP Release 7.2.5.1 boot 
complete. Last disk update written at Thu Jan  1 00:00:00 GMT 1970
Clustered failover is not licensed.
Thu May  3 05:33:15 GMT [cf.fm.unexpectedAdapter:warning]: Warning: clustering is not 
licensed yet an interconnect adapter was found. NVRAM will be divided into two parts 
until adapter is removed
Thu May  3 05:33:15 GMT [cf.fm.unexpectedPartner:warning]: Warning: clustering is not 
licensed yet the node once had a cluster partner
Thu May  3 05:33:16 GMT [mgr.boot.reason_ok:notice]: System rebooted.
Thu May  3 05:33:16 GMT [asup.config.minimal.unavailable:warning]: Minimal Autosupports 
unavailable. Could not read /etc/asup_content.conf
n3300a> Thu May  3 05:33:18 GMT [n3300a: console_login_mgr:info]: root logged in from 
console

4. Add software licenses by running the following command (see Example A-7): 

license add <license> 

Example A-7   Example NFS license

n3300a> license add XXXXXXX
n3300a> Wed May 3 23:19:30 GMT [rc:notice]: nfs licensed

5. Always consider updating firmware and Data ONTAP to the preferred version. For more 

information, see “Updating Data ONTAP” on page 319.

6. Repeat these steps on the second filer for N series with model A20 or A21.

Updating Data ONTAP

To identify the compatible IBM System Storage N series hardware for the supported releases 
of Data ONTAP, see the IBM System Storage N series Data ONTAP Matrix that is available at 
this website:

http://www.ibm.com/support/docview.wss?uid=ssg1S7001786

Update the installed N series storage system to the latest Data ONTAP release. Metrics 
demonstrate reliability over many customer installations and completion of compatibility 
testing with other products.

Summary of Contents for N Series

Page 1: ...ge N series Hardware Guide Roland Tretau Jeff Lin Dirk Peitzmann Steven Pemberton Tom Provost Marco Schwarz Select the right N series hardware for your environment Understand N series unified storage solutions Takestorageefficiencytothe next level ...

Page 2: ......

Page 3: ...International Technical Support Organization IBM System Storage N series Hardware Guide May 2014 SG24 7840 03 ...

Page 4: ...Restricted Rights Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp Fourth Edition May 2014 This edition applies to the IBM System Storage N series portfolio as of October 2013 Note Before using this information and the product it supports read the information in Notices on page xi ...

Page 5: ... 1 4 Data ONTAP 8 supported systems 11 Chapter 2 Entry level systems 13 2 1 Overview 14 2 2 N32x0 common features 15 2 3 N3150 model details 16 2 3 1 N3150 model 2857 A15 16 2 3 2 N3150 model 2857 A25 16 2 3 3 N3150 hardware 16 2 4 N3220 model details 18 2 4 1 N3220 model 2857 A12 18 2 4 2 N3220 model 2857 A22 18 2 4 3 N3220 hardware 18 2 5 N3240 model details 19 2 5 1 N3240 model 2857 A14 19 2 5 ...

Page 6: ...8 5 2 3 Environmental and technical specifications 48 5 3 Expansion unit EXN3200 48 5 3 1 Overview 49 5 3 2 Supported EXN3000 drives 50 5 3 3 Environmental and technical specifications 50 5 4 Expansion unit EXN3500 51 5 4 1 Overview 52 5 4 2 Intermix support 53 5 4 3 Supported EXN3500 drives 53 5 4 4 Environmental and technical specification 54 5 5 Self Encrypting Drive 54 5 5 1 SED at a glance 54...

Page 7: ...ns 88 7 4 Managing an HA pair configuration 89 7 4 1 Managing an HA pair configuration 89 7 4 2 Halting a node without takeover 90 7 4 3 Basic HA pair configuration management 91 7 4 4 HA pair configuration failover basic operations 100 7 4 5 Connectivity during failover 100 Chapter 8 MetroCluster 103 8 1 Overview of MetroCluster 104 8 2 Business continuity solutions 107 8 3 Stretch MetroCluster 1...

Page 8: ...ouble parity stripes 154 10 4 4 RAID DP reconstruction 155 10 4 5 Protection levels with RAID DP 159 10 5 Hot spare disks 163 Chapter 11 Core technologies 165 11 1 Write Anywhere File Layout 166 11 2 Disk structure 167 11 3 NVRAM and system memory 168 11 4 Intelligent caching of write requests 169 11 4 1 Journaling write requests 169 11 4 2 NVRAM operation 170 11 5 N series read caching techniques...

Page 9: ...213 16 1 Administration methods 214 16 1 1 FilerView interface 214 16 1 2 Command line interface 214 16 1 3 N series System Manager 216 16 1 4 OnCommand 216 16 2 Starting stopping and rebooting the storage system 216 16 2 1 Starting the IBM System Storage N series storage system 217 16 2 2 Stopping the IBM System Storage N series storage system 217 16 2 3 Rebooting the system 222 Part 3 Client har...

Page 10: ...2 3 Asymmetric Logical Unit Access 276 19 2 4 Why ALUA 276 Part 4 Performing upgrades 279 Chapter 20 Designing for nondisruptive upgrades 281 20 1 System NDU 282 20 1 1 Types of system NDU 282 20 1 2 Supported Data ONTAP upgrades 283 20 1 3 System NDU hardware requirements 284 20 1 4 System NDU software requirements 284 20 1 5 Prerequisites for a system NDU 286 20 1 6 Steps for major version upgra...

Page 11: ...319 Obtaining the Data ONTAP software from the IBM NAS website 320 Installing Data ONTAP system files 321 Downloading Data ONTAP to the storage system 326 Setting up the network using console 328 Changing the IP address 329 Setting up the DNS 330 Appendix B Operating environment 333 N3000 entry level systems 334 N3400 334 N3220 334 N3240 335 N6000 mid range systems 336 N6210 336 N6240 337 N6270 33...

Page 12: ...x IBM System Storage N series Hardware Guide ...

Page 13: ...d do not in any manner serve as an endorsement of those websites The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Any performance data contained herein was determined in a controlled en...

Page 14: ...s of the International Business Machines Corporation in the United States other countries or both AIX DB2 DS4000 DS6000 DS8000 Enterprise Storage Server IBM Redbooks Redpapers Redbooks logo System Storage System x Tivoli XIV z OS The following terms are trademarks of other companies Intel Intel Xeon Intel logo Intel Inside logo and Intel Centrino logo are trademarks or registered trademarks of Int...

Page 15: ... make installation administration and troubleshooting straightforward In addition this book addresses high availability solutions including clustering and MetroCluster that support highest business continuity requirements MetroCluster is a unique solution that combines array based clustering with synchronous mirroring to deliver continuous availability This Redbooks publication is a companion book...

Page 16: ...m x IBM System Storage storage software and virtualization Tom also is the co author of several other Redbooks and IBM Redpapers He joined IBM in 2010 Marco Schwarz is an IT specialist and team leader for Techline as part of the Techline Global Center of Excellence who lives in Germany He has many years of experience in designing IBM System Storage solutions His expertise spans all recent technolo...

Page 17: ... Corporation International Technical Support Organization Dept HYTD Mail Station P099 2455 South Road Poughkeepsie NY 12601 5400 Stay connected to IBM Redbooks Find us on Facebook http www facebook com IBMRedbooks Follow us on Twitter http twitter com ibmredbooks Look for us on LinkedIn http www linkedin com groups home gid 2130806 Explore new Redbooks publications residencies and workshops with t...

Page 18: ...xvi IBM System Storage N series Hardware Guide ...

Page 19: ...014 Fourth Edition New information The following new information is included The N series hardware portfolio was updated to reflect the October 2013 status quo Information and changed in Data ONTAP 8 1 x have been included High availability and MetroCluster information was updated to include SAS shelf technology Changed information The following changed information is included Hardware information...

Page 20: ...xviii IBM System Storage N series Hardware Guide ...

Page 21: ...t includes the following chapters Chapter 1 Introduction to IBM System Storage N series on page 3 Chapter 2 Entry level systems on page 13 Chapter 3 Mid range systems on page 23 Chapter 4 High end systems on page 33 Chapter 5 Expansion units on page 45 Chapter 6 Cabling expansions on page 59 Chapter 7 Highly Available controller pairs on page 71 Chapter 8 MetroCluster on page 103 Chapter 9 MetroCl...

Page 22: ...2 IBM System Storage N series Hardware Guide ...

Page 23: ...stem Storage N series is designed to deliver high end value with midrange affordability Built in enterprise serviceability and manageability features support customer efforts to increase reliability simplify and unify storage infrastructure and maintenance and deliver exceptional economy This chapter includes the following sections Overview IBM System Storage N series hardware Software licensing s...

Page 24: ...also have simple management interfaces that make installation administration and troubleshooting straightforward The N series unified storage solution supports file and block protocols as shown in Figure 1 1 Converged networking also is supported for all protocols Figure 1 1 Unified storage This type of flexible storage solution offers the following benefits Heterogeneous unified storage solution ...

Page 25: ...pport for online backup and recovery Include added value features such as deduplication to optimize space management All N series storage systems use a single operating system Data ONTAP across the entire platform They offer advanced function software features that provide one of the industry s most flexible storage platforms This functionality includes comprehensive system management storage mana...

Page 26: ...che Creates a flexible caching layer within your storage infrastructure that automatically adapts to changing usage patterns to eliminate bottlenecks Improves application response times for large compute farms speeds data access for remote users or creates a tiered storage infrastructure that circumvents tedious data management tasks FlexClone Provides near instant creation of LUN and volume clone...

Page 27: ...ed management of multiple N series systems Flash Cache Performance Acceleration Module Improves throughput and reduces latency for file services and other random read intensive workloads Offers power savings by using less power than adding more disk drives to optimize performance RAID DP Offers double parity bit RAID protection N series RAID 6 implementation Protects against data loss because of d...

Page 28: ... incremental data in place point in time copies of a LUN or volume with minimal performance effect Enables frequent nondisruptive space efficient and quickly restorable backups SnapVault Exports Snapshot copies to another N series system which provides an incremental block level backup solution Enables cost effective long term retention of rapidly restorable disk based backups Storage Encryption P...

Page 29: ...ibre Channel iSCSI SnapRestore SnapMirror SnapVault FlexClone SnapLock SnapManager Suite Storage Efficiency features Snapshot Copies Point in time copies that write only changed blocks No performance penalty Virtual Copies FlexClone Near zero space instant virtual copies Only subsequent changes in cloned dataset get stored Thin Provisioning FlexVol Create flexible volumes that appear to be a certa...

Page 30: ...nd Cluster Failover Remote License Keys are required for DOT 8 1 the DSM MPIO License key must be installed on Server Protocols Sold Separately iSCSI FCP CIFS NFS License Key Details Each Protocol License Key must be installed separately SnapRestore Includes SnapRestore License Key Details SnapRestore License Key must be installed separately SnapMirror Includes SnapMirror License Key Details SnapM...

Page 31: ...me older N series systems that are suitable to run Data ONTAP 8 Figure 1 5 Supported Data ONTAP 8 x systems Models Supported by Data ONTAP Versions 8 0 and Higher IBM 8 0 8 0 1 8 0 2 8 0 3 8 1 N3220 x N3240 x N3400 x x x x x N5300 x x x x x N5600 x x x x x N6040 x x x x x N6060 x x x x x N6070 x x x x x N6210 x x x x N6240 x x x x N6270 x x x x N7600 x x x x x N7700 x x x x x N7800 x x x x x N7900...

Page 32: ...12 IBM System Storage N series Hardware Guide ...

Page 33: ...tems This chapter describes the IBM System Storage N series 3000 systems which address the entry level segment This chapter includes the following sections Overview N32x0 common features N3150 model details N3220 model details N3240 model details N3000 technical specifications 2 ...

Page 34: ...System Storage N3220 Model A12 Single node Model A22 Dual node Active Active HA Pair The IBM System Storage N3240 Model A14 Single node Model A24 Dual node Active Active HA Pair Table 2 1 provides a comparison of the N3000 series Table 2 1 N3000 series comparison N3000 featuresa a All specifications are for dual controller active active configurations N3150 FAS2220 N3220 FAS2240 2 N3240 FAS2240 4 ...

Page 35: ... A15 a25 2U chassis 12 SAS 3 5 2 One or two controllers each with no mezzanine card N3220 A12 A22 2U chassis 24 SFF SAS 2 5 2 One or two controllers each with Dual FC mezzanine card or Dual 10 GE mezzanine card N3240 A14 A24 4U chassis 24 SATA 3 5 4 Feature code Configuration 2030 Controller with dual port FC Mezzanine Card include SFP 2031 Controller with dual port 10 GbE Mezzanine Card no SFP Ex...

Page 36: ...ver and failback function which helps improve overall availability Model A25 is a 2U rack mountable storage controller 2 3 3 N3150 hardware The N3150 hardware has the following characteristics Specifications single node 2x for dual node 2U standard 19 inch rack mount enclosure single or dual node One 1 73 GHz Intel dual core processor 6 GB random access ECC memory NVRAM 768 MB Four integrated Giga...

Page 37: ...nt view Figure 2 3 shows the N3150 Single Controller in chassis Model A15 Figure 2 3 N3150 Single Controller in chassis Figure 2 4 shows the N3150 Dual Controller in chassis Model A25 Figure 2 4 N3150 Dual Controller in chassis Note The N3150 supports IP protocols only because it lacks any FC ports ...

Page 38: ...at are designed to provide failover and failback function which helps improve overall availability Model A22 is a 2U rack mountable storage controller 2 4 3 N3220 hardware The N3220 hardware has the following characteristics Based on the EXN3500 expansion shelf 24 2 5 SFF SAS disk drives minimum initial order of 12 disk drives Specifications single node 2x for dual node 2U standard 19 inch rack mo...

Page 39: ...I NFS CIFS and FCP support through optional features The N3240 Model A14 is a 4U storage controller that must be mounted in a standard 19 inch rack Model A14 can be upgraded to a Model A24 However this is a disruptive upgrade 2 5 2 N3240 model 2857 A24 N3240 Model A24 is designed to provide identical functions as the single node Model A14 However it includes a second Processor Control Module and C...

Page 40: ...om access ECC memory NVRAM 768 MB Four integrated Gigabit Ethernet RJ45 ports Two SAS ports One serial console port and one integrated RLM port One optional expansion I O adapter slot on mezzanine card 8 Gb FC card provides two FC ports 10 GbE card provides two 10 GbE ports Redundant hot swappable auto ranging power supplies and cooling fans Figure 2 8 shows the front view of the N3240 Figure 2 8 ...

Page 41: ... 2 10 N3240 Dual Controller in chassis Figure 2 11 shows the controller with the 8 Gb FC Mezzanine card option Figure 2 11 Controller with 8 Gb FC Mezzanine card option Figure 2 12 shows the controller with the 10 GbE Mezzanine card option Figure 2 12 Controller with 10 GbE Mezzanine card option ...

Page 42: ...8 MB 1 5 GB 768 MB 1 5 GB 768 MB 1 5 GB Onboard I O ports FC ports Speed N A N A 0 0 0 0 Ethernet ports 4 1 Gb 8 1 Gb 4 1 Gb 8 1 Gb 4 1 Gb 8 1 Gb SAS ports 2 6 Gb 4 6 Gb 2 6 Gb 4 6 Gb 2 6 Gb 4 6 Gb Storage scalability Max expansion shelves 2 5 5 Max disk drives 60 12 internal 48 external 144 24 internal 120 external 144 24 internal 120 external Max raw capacity 240 TB 501 TB 576 TB Max volumes 500...

Page 43: ...rved 23 Chapter 3 Mid range systems This chapter describes the IBM System Storage N series 6000 systems which address the mid range segment This chapter includes the following sections Overview N62x0 model details N62x0 technical specifications 3 ...

Page 44: ...ems help you meet your network attached storage NAS needs They provide high levels of application availability for everything from critical business operations to technical applications You can also address NAS and storage area network SAN as primary and auxiliary storage requirements In addition you get outstanding value These flexible systems offer excellent performance and impressive expandabil...

Page 45: ...more PCIe slots The I O expansion is not available on Cxx models 3 1 3 Functions and features common to all models This section describes the functions and features that are common to all eight models Fibre Channel SAS and SATA attachment All models include Fibre Channel SAS and SATA attachment options for disk expansion units These options are designed to allow deployment in multiple environments...

Page 46: ...ntroller It is similar to the serial console RLM and standard network interfaces Use the e0M interface for network based storage controller administration monitoring activities and ASUP reporting The RLM is used when you require its higher level of support features Host side application data should connect to the appliance on a separate subnet from the management interfaces RLM assisted cluster fa...

Page 47: ... include the IOXM which provides more PCI slots Figure 3 3 shows the IBM N62x0 Controller I O module Figure 3 3 IBM N62x0 Controller I O The different N62x0 models also support different chassis configurations For example a single chassis N6220 might contain a single node C15 model dual nodes C25 or a single node plus IOXM E15 A second chassis is required for the dual node with IOXM models E25 and...

Page 48: ...ibility IBM N62x0 I O Expansion Module IOXM is shown in Figure 3 5 and features the following characteristics Components are not hot swappable Controller panics if it is removed If inserted into running IBM N62x0 IOXM is not recognized until the controller is rebooted 4 full length PCIe v1 0 Gen 1 x8 slots Figure 3 5 IBM N62x0 I O Expansion Module IOXM ...

Page 49: ...rd layout Figure 3 7 shows the IBM N62x0 USB Flash Module which has the following features It is the boot device for Data ONTAP and the environment variables It replaces CompactFlash It has the same resiliency levels as CompactFlash 2 GB density is used It is a replaceable FRU Figure 3 7 IBM N62x0 USB Flash Module ...

Page 50: ...de the path for cluster interconnect traffic The N60xx and N62xx series offer a new architecture that incorporates a dual controller design with the cluster interconnect on the backplane The N62x0 ports c0a and c0b are the ports that you must connect to establish controller communication Use these ports to enable NVRAM mirroring after you set up a dual chassis HA configuration that is N62x0 with I...

Page 51: ...r type 2 3 GHz Intel Quad Core Number of Processors 1 2 1 2 2 4 Memory 12 24 12 24 20 40 NV RAM 1 6 GB 3 2 GB 1 6 GB 3 2 GB 2 GB 4 GB Onboard I O ports FC ports Speed 2 4 Gb 4 4 Gb 2 4 Gb 4 4 Gb 2 4 Gb 4 4 Gb Ethernet ports 2 1 Gb 4 1 Gb 2 1 Gb 4 1 Gb 2 1 Gb 4 1 Gb SAS ports 2 6 Gb 4 6 Gb 2 6 Gb 4 6 Gb 2 6 Gb 4 6 Gb Storage scalability Max FC loops 5 13 13 Max disk drives 480 480 720 Max raw capac...

Page 52: ...32 IBM System Storage N series Hardware Guide ...

Page 53: ...er 4 High end systems This chapter describes the IBM System Storage N series 7000 system which addresses the high end segment This chapter includes the following sections Overview N7x50T hardware IBM N7x50T configuration rules N7000T technical specifications 4 ...

Page 54: ... access simultaneous multiprotocol support expandability upgradability and low maintenance requirements 4 1 1 Common features The N7x50T modular disk storage systems includes the following common features High data availability and system level redundancy that is designed to address the needs of business critical and mission critical applications Single integrated architecture that is designed to ...

Page 55: ...is controller uses a dual node active active configuration which is composed of two controller units in either one or two chassis as required for Metrocluster configuration N7950T The IBM System Storage N6250 includes the Model E25 storage controller This controller uses a dual node active active configuration which is composed of two controller units each with an IOXM in two chassis 4 2 N7x50T ha...

Page 56: ...nfiguration Figure 4 4 IBM N series N7950T configuration 4 2 2 Controller module components Although they differ in processor count and memory configuration the processor modules for the N7550T and N7950T provide the same onboard I O connections The N7950T also includes an I O expansion module IOXM to provide more I O capacity Figure 4 5 on page 37 shows the IBM N series N7x50T controller I O ...

Page 57: ...ystems 37 Figure 4 5 N7x50 controller Figure 4 6 shows an internal view of the IBM N series N7x50T Controller module The N7550T and N7950T differ in number of processors and installed memory Figure 4 6 N7x50 internal view ...

Page 58: ...ot supported on the N7550T model Figure 4 7 shows the IBM N series N7950T I O Expansion Module IOXM Figure 4 7 IBM N series N7950T I O Expansion Module IOXM The N7950T IOXM features the following characteristics All PCIe v2 0 Gen 2 slots Vertical slots have different form factor Not hot swappable Controller panics if removed Hot pluggable but not recognized until reboot Figure 4 8 shows the IBM N ...

Page 59: ... SAS I O card Feature Code 1079 FC and Feature Code 1080 SAS Can mix FC and SAS system boards in slots 11 and 12 FC card ports can be set to target or initiator Horizontal PCIe slots The horizontal PCIe slots include the following characteristics Support standard PCIe adapters and cards 10 GbE NIC new quad port 1 GbE PCIe adapter for N7x50T FC1028 10 GbE unified target adapter 8 Gb Fibre Channel F...

Page 60: ...N series platforms going forward 4 3 5 MetroCluster Gateway and FlexCache MetroCluster and Gateway configurations include the following characteristics Supported MetroCluster two chassis configuration Single enclosure stand alone chassis IBM N series N7950T E22 controller with IOXM Fabric MetroCluster requires EXN4000 shelves The N7x50T series can also function as a Gateway FlexCache uses N7x50T c...

Page 61: ...M8 and SAS I O system boards use the QSFP connector Mixing the cables does not cause physical damage but the cables do not work Label your HA and SAS cables when you remove them 4 3 7 N7x50T SFP modules This section provides detailed information about SFP modules Figure 4 10 shows the 8 Gb SFP modules Figure 4 10 8 Gb SFP modules ...

Page 62: ...42 IBM System Storage N series Hardware Guide Figure 4 11 shows the 10 GbE SFP modules Figure 4 11 10 GbE SFP modules ...

Page 63: ...hine type 2867 C20 2867 E22 Gateway feature FC 9551 Processor type 2 26 GHz Nehalem quad core 2 93 GHz Intel 6 core Number of processors 4 16 cores 4 24 cores Memory 96 GB 192 GB NV RAM 4 GB 8 GB Onboard I O ports FC ports Speed 8 8 Gb 8 8 Gb Ethernet ports 8 10 Gbps 4 1 Gbps 8 10 Gbps 4 1 Gbps SAS ports 0 8 6 Gbps 0 24 6 Gbps Storage scalability Max FC loops 10 14 Max disk drives 1200 1440 Max ra...

Page 64: ...44 IBM System Storage N series Hardware Guide ...

Page 65: ...chapter describes the IBM N series expansion units which also called disk shelves This chapter includes the following sections Shelf technology overview Expansion unit EXN3000 Expansion unit EXN3200 Expansion unit EXN3500 Self Encrypting Drive Expansion unit technical specifications 5 ...

Page 66: ...ability for the IBM System Storage N series systems The EXN3000 is a 4U disk storage expansion unit It can be mounted in any industry standard 19 inch rack The EXN3000 includes the following features Dual redundant hot pluggable integrated power supples and cooling fans Dual redundant disk expansion unit switched controllers Diagnostic and status LEDs 5 2 1 Overview The IBM System Storage EXN3000 ...

Page 67: ...e 5 2 EXN3000 front view The EXN3000 SAS SATA expansion unit is shipped with no disk drives unless disk drives are included in the order In that case the disk drives are installed in the plant The EXN3000 SAS SATA expansion unit can be shipped with no disk drives installed Disk drives that are ordered with the EXN3000 are installed by IBM in the plant before shipping Figure 5 3 shows the rear view...

Page 68: ...200 ships with 48 disk drives per unit The EXN3200 is a disk storage expansion unit for mounting in any industry standard 19 inch rack The EXN3200 provides low cost high capacity SAS disk storage for the IBM N series system storage family The EXN3200 must be ordered with a full complement of 48 disks EXN3000 RPM Capacity SAS 15 K 600 GB 600 GB encrypted SATA 7 2 K 1 TB 2 TB 3 TB 3 TB encrypted 4 T...

Page 69: ...plement of disks Disk drives that are ordered with the EXN3200 are shipped separately from the EXN3200 shelf and must be installed at the customer s location Disk drive bays are numbered horizontally starting from 0 at the upper left position to 23 at the lower right position The EXN3200 SAS SATA expansion unit is shown in Figure 5 4 Figure 5 4 EXN3200 front view Each of the 24 disk bays contains ...

Page 70: ...chnical specifications Table 5 4 shows the environmental and technical specifications Table 5 4 EXN3000 environmental and technical specifications EXN3000 RPM Capacity SATA 7 2 K 3 TB 7 2 K 4 TB Input voltage 100 to 240 V 100 V actual 200 to 240 V 200 V actual Size Worst case 2 PSUa Typical Worst case 2 PSU Typical Per PSU pairb System four PSUc Per PSU pair System four PSU Total input current mea...

Page 71: ...bandwidth per wide port Improved power consumption Power consumption per GB reduced by approximately 30 50 Only SAS drives are supported in the EXN3500 SATA is not supported The following features were not changed Same underlying architecture and FW base as EXN3000 All existing EXN3000 features and functionality Still uses the 3 Gb PCIe Quad Port SAS HBA already 6 Gb capable or onboard SAS ports T...

Page 72: ...nt view The EXN3500 SAS expansion unit can be shipped with no disk drives installed Disk drives ordered with the EXN3500 are installed by IBM in the plant before shipping Disk drives can be of 450 GB and 600 GB physical capacity and must be ordered as features of the EXN3500 Figure 5 8 shows the rear view of the EXN3500 which highlights the connectivity and resiliency Figure 5 8 EXN3500 rear view ...

Page 73: ...ther platforms mixing EXN3500 and EXN3000 w IOM3 or IOM6 is supported Applies only to N3150 and N32x0 not other platforms EXN3000 supports IOM3 and IOM6 modules EXN3500 supports only IOM6 modules the use of IOM3 modules in an EXN3500 is not supported 5 4 3 Supported EXN3500 drives Table 5 5 on page 54 lists the drives that are supported by EXN3500 at the time of this writing Attention Even though ...

Page 74: ...rive SED 600 GB capacity 2 5 inch form factor 10 K RPM 6 GB SAS Encryption that is enabled through disk drive firmware same drive as what is shipping with different firmware Available in EXN3500 and EXN3000 expansion shelf and N3220 internal drives controller Only fully populated 24 drives and N3220 controller EXN3500 RPM Capacity SAS 10 K 450 GB 600 GB 600 GB encrypted 900 GB 900 GB encrypted 1 2...

Page 75: ...raphic shredding of data for non returnable disk NRD disk repurposing scenarios and simplified disposal of the drive through disk destroy commands These processes render a disk unusable This greatly simplifies the disposal of drives and eliminates the need for costly time consuming physical drive shredding All data on the drives is automatically encrypted If you do not want to track where the most...

Page 76: ...he use of many different encryption key management tools for many different business purposes and IT assets Communication with the KMIP server Self encryption uses Secure Sockets Layer SSL certificates to establish secure communications with the KMIP server These certificates must be in Base64 encoded X 509 PEM format and can be self signed or signed by a certificate authority CA Supported key man...

Page 77: ...try in developing and promoting an exciting new security standard Key Management Interoperability Protocol KMIP KMIP is an open standard that is designed to support the full lifecycle of key management tasks from key creation to key retirement IBM Security Key Lifecycle Manager Version 1 0 supports the following operating systems AIX V5 3 64 bit Technology Level 5300 04 and Service Pack 5300 04 02...

Page 78: ... model DS4243 DS4246 DS4486 DS2246 Connectivity SAS SAS SAS Optical SAS support Yes Yes Yes I O Modules IOM3 or IOM6 IOM6 IOM6 MetroCluster support Yes No Yes Form factor Rack units 4 RU 4 RU 2 RU Drives per shelf 24 48 24 Drive form factor 3 5 inch 3 5 inch 2 5 inch Drive carrier Single drive Dual drive Single drive Storage tiers supported Ultra Perf SSD Yes No Yes High Perf HDD Yes No Yes High C...

Page 79: ...rights reserved 59 Chapter 6 Cabling expansions This chapter describes the multipath cabling of expansions and includes the following sections EXN3000 and EXN3500 disk shelves cabling EXN4000 disk shelves cabling Multipath HA cabling 6 ...

Page 80: ...in each N series controller The configuration includes two SAS stacks each of which has three SAS shelves 6 1 1 Controller to shelf connection rules Each controller connects to each stack of disk shelves in the system through the controller SAS ports These ports can be A B C and D and can be on a SAS HBA in a physical PCI slot slot 1 N or on the base controller For quad port SAS HBAs the controlle...

Page 81: ...storage expansion units Figure 6 2 Top and bottom cabling for quad port SAS HBAs SAS cabling is based on the following rules that each controller is connected to the top storage expansion unit and the bottom storage expansion unit in a stack Controller 1 always connects to the top storage expansion unit IOM A and the bottom storage expansion unit IOM B in a stack of storage expansion units Control...

Page 82: ...62 IBM System Storage N series Hardware Guide Figure 6 3 shows how the SAS shelves are interconnected for two stacks with three shelves each Figure 6 3 SAS shelf interconnect ...

Page 83: ...Chapter 6 Cabling expansions 63 6 1 3 Top connections The top ports of the SAS shelves are connected to the HA pair controllers as shown in Figure 6 4 Figure 6 4 SAS shelf cable top connections ...

Page 84: ... failure or shelf controller causes any interruption of service 6 1 5 Verifying SAS connections After you complete the SAS connections in your storage system by using the applicable cabling procedure verify the SAS connections Complete the following steps to verify that the storage expansion unit IOMs have connectivity to the controllers 1 Enter the following command at the system console sasadmin...

Page 85: ...section provides information about cabling the disk shelf ACP connections for a new storage system installation This section also provides information about cabling the optional disk shelf ACP connections for a new storage system installation as shown in Figure 6 6 Figure 6 6 SAS shelf cable ACP connections The following ACP cabling rules apply to all supported storage systems that use SAS storage...

Page 86: ...e IBM System Storage EXN3000 Storage Expansion Unit Hardware and Service Guide which is available at this website http www ibm com storage support nas 6 2 EXN4000 disk shelves cabling This section describes the requirements for connecting an expansion unit to N series storage systems and other expansion units For more information about installing and connecting expansion units in a rack or connect...

Page 87: ...XN4000 disk shelves that are connected to a HA pair with non multipath cabling A single Fibre Channel cable or shelf controller failure might cause a takeover situation Figure 6 7 EXN4000 dual controller non multipath Attention Do not mix Fibre Channel and SATA expansion units in the same loop ...

Page 88: ...abling No single Fibre Channel cable or shelf controller failure causes a takeover situation Figure 6 8 EXN4000 dual controller with multipath Tip For N series controllers to communicate with an EXN4000 disk shelf the Fibre Channel ports on the controller or gateway must be set for initiator Changing the behavior of the Fibre Channel ports on the N series system can be performed by using the fcadm...

Page 89: ...er a failover as shown in Example 6 2 Example 6 2 Clustered system with MPHA connections to disks N6270A storage show disk p PRIMARY PORT SECONDARY PORT SHELF BAY 0a 16 A 0c 16 B 1 0 0c 17 B 0a 17 A 1 1 0c 18 B 0a 18 A 1 2 0a 19 A 0c 19 B 1 3 With only a single connection to the A channel a disk loop is technically a daisy chain When any component fiber cable shelf cable or shelf controller in the...

Page 90: ...70 IBM System Storage N series Hardware Guide ...

Page 91: ...ode This chapter describes aspects of determining HA pair status and HA pair management In Data ONTAP 8 x the recovery capability that is provided by a pair of nodes storage systems is called an HA pair This pair is configured to serve data for each other if one of the two nodes stops functioning Previously with Data ONTAP 7G this function was called an Active Active configuration This chapter inc...

Page 92: ...e 7 1 Standard HA pair configuration In a standard HA pair Data ONTAP functions so that each node monitors the functioning of its partner through a heartbeat signal that is sent between the nodes Data from the NVRAM of one node is mirrored to its partner Each node can take over the partner s disks or array LUNs if the partner fails The nodes also synchronize time 7 1 1 Benefits of HA pairs Configu...

Page 93: ...erconnect that consists of adapters and cable or in systems with two controllers in the same chassis through an internal interconnect The nodes use the interconnect to perform the following tasks Continually check whether the other node is functioning Mirror log data for each other s NVRAM Synchronize each other s time They use two or more disk shelf loops or third party storage in which the follo...

Page 94: ... that each node has sufficient resources to adequately support the workload of both nodes during takeover mode Use the HA Configuration Checker to help ensure that failovers are successful If your system supports remote management by using a Remote LAN Management RLM or Service Processor ensure that you configure it properly Higher numbers of traditional volumes and FlexVols on your system can aff...

Page 95: ...is zoned into two logically independent fabrics is not a dual fabric connection FC single fabric topologies are topologies in which the hosts are attached to the storage systems through a single Fibre Channel fabric The fabric can consist of multiple Fibre Channel switches iSCSI direct attached topologies are topologies in which the hosts are directly attached to the storage controller Direct atta...

Page 96: ...lable each having distinct advantages and requirements Standard HA pairs Mirrored HA pairs Stretch MetroClusters Fabric attached MetroClusters Each of these HA pair types is described in the following sections 7 2 1 Standard HA pairs In a standard HA pair Data ONTAP functions so that each node monitors the functioning of its partner through a heartbeat signal that is sent between the nodes Data fr...

Page 97: ...se Notes for the list of supported systems which is available at this website http www ibm com storage support nas For systems with two controller modules in a single chassis both nodes of the HA pair configuration are in the same chassis and have internal cluster interconnect Storage capacity The number of disks must not exceed the maximum configuration capacity The total storage that is attached...

Page 98: ...d HA pairs maintain two complete copies of all mirrored data These copies are called plexes and are continually and synchronously updated when Data ONTAP writes to a mirrored aggregate The plexes can be physically separated to protect against the loss of one set of disks or array LUNs Mirrored HA pairs use SyncMirror Setup requirements and restrictions for mirrored HA pairs The restrictions and re...

Page 99: ...s of the specified data volumes or file systems that you indicated as being mirrored volumes or file systems in an HA pair Data volume copies are called plexes and are continually and synchronously updated every time Data ONTAP writes data to the disks Plexes are physically separated from each other across separate groupings of disks The Stretch MetroCluster nodes can be physically distant from ea...

Page 100: ...gate must use the same type of storage For example you cannot mirror a Fibre Channel aggregate with SATA storage MetroCluster is not supported on the N3300 N3400 and N3600 platforms The following distance limitations dictate the default speed that you can set If the distance between the nodes is less than 150 meters and you have an 8 Gb FC VI adapter set the default speed to 8 Gb If you want to in...

Page 101: ...abric attached MetroCluster nodes is up to 100 km depending on the switch configuration A fabric attached MetroCluster connects the two controllers nodes and the disk shelves through four SAN switches that are called the Back end Switches The Back end Switches are IBM Brocade Fibre Channel switches in a dual fabric configuration for redundancy Figure 7 5 shows a simplified Fabric attached MetroClu...

Page 102: ...modes of HA pair configurations The first time that you start the HA pair ensure that the nodes are correctly connected and powered up Then use the setup program to configure the systems When the setup program runs on a storage system in an HA pair it prompts you to answer questions specific for HA pairs Consider the following questions about your installation before you proceed through the setup ...

Page 103: ...tems and shelves are on separate power supplies or grids This configuration prevents a single power outage from affecting both controller units and shelves Use VIFs to provide redundancy and improve the availability of network communication The virtual interfaces are set up during initial installation or the subsequent initiation of setup Maintain a consistent configuration between HA pair nodes s...

Page 104: ...omplete the following steps 1 For each required license enter the license and code on both node consoles as shown in the Example 7 2 Example 7 2 Enabling license license add xxxxx where xxxxxis the license code you received for the feature 2 Reboot both nodes by using the reboot command 3 Enable HA pair capability on each node by entering the cf enable command on the local node console 4 Verify th...

Page 105: ...at case the IP addresses from the controller that is taken over can be accessed even if it is down for maintenance LAN interfaces can be configured in the following ways Shared interfaces Dedicated interfaces Standby interfaces Configuring shared interfaces with setup A shared network interface for the local controller and the partner If the partner fails the network interface assumes the identity...

Page 106: ...ce for the partner You do not configure any IP addresses for the e0b interface Example 7 7 Configuring standby network interface Please enter the IP address for Network Interface e0b Should interface e0b take over a partner IP address during failover n y Please enter the IPv4 address or interface name to be taken over by e0b e0b 7 3 6 Setting options and parameters Some options must be the same on...

Page 107: ...st be the same on each node The parameters that are listed in Table 7 2 must be the same so that takeover is smooth and data is transferred between the nodes correctly Table 7 2 Parameters that must be the same in both nodes 7 3 7 Testing takeover and giveback After you configure all aspects of your HA pair configuration complete the following steps to verify that it operates as expected 1 Check t...

Page 108: ...system NVRAM Yes No If an NVRAM adapter fails cluster failover automatically fails over to its partner storage system and serves data from the takeover storage system Processor fan Yes No If the processor fan fails the node gracefully shuts down Cluster failover automatically fails over to its partner storage system and serves data from the takeover storage system Multiple NICs with VIFS virtual i...

Page 109: ...elf is a passive backplane with dual power supplies dual fans dual ESH2s and dual FC AL loops It is the most reliable component in a storage system Power supply No No The storage system and the disk shelf feature dual power supplies If one power supply fails the second power supply automatically takes effect No failover is needed in this situation Fan storage system or disk shelf No No The storage...

Page 110: ...how an HA pair node is halted by using the halt f command You can monitor the entire shutdown process to the LOADER prompt by logging on through the RLM module Doing so gives you console access even during reboot Example 7 8 Halting by using the halt f command itsonas2 cf status Cluster enabled itsonas1 is up itsonas2 cf monitor current time 09Apr2011 01 49 12 UP 8 23 34 29 partner itsonas1 cluste...

Page 111: ...onas2 cf status Cluster enabled itsonas1 is up itsonas2 2 Run the cf takeover command Example 7 10 shows the console output during takeover Example 7 10 cf takeover command itsonas2 cf takeover cf takeover initiated by operator itsonas2 Sat Apr 9 02 00 22 GMT 7 itsonas2 cf misc operatorTakeover warning Cluster monitor takeover initiated by operator Sat Apr 9 02 00 22 GMT 7 itsonas2 cf fsm nfo acce...

Page 112: ...ern syslogd restarted info syslogd Restarted Sat Apr 9 02 00 52 GMT 7 itsonas1 itsonas2 asup smtp host info Autosupport cannot connect to host mailhost Unknown mhost for message SYSTEM CONFIGURATION WARNING Sat Apr 9 02 00 52 GMT 7 itsonas1 itsonas2 asup smtp unreach error Autosupport mail was not sent because the system cannot reach any of the mail hosts from the autosupport mailhost option SYSTE...

Page 113: ...shell itsonas2 itsonas2 itsonas1 vol status Volume State Status Options vol0 online raid_dp flex root Flexvolume_copy online raid_dp flex create_ucode on sis convert_ucode on dedupe online raid_dp flex create_ucode on sis convert_ucode on testdata online raid_dp flex create_ucode on convert_ucode on itsonas2 itsonas1 aggr status Aggr State Status Options aggr0 online raid_dp aggr root itsonas2 its...

Page 114: ...s1 cf fsm stateTransit warning Cluster monitor TAKEOVER UP Tue Apr 12 03 17 16 GMT 7 itsonas1 cf fsm takeoverByPartnerDisabled notice Cluster monitor takeover of itsonas1 by itsonas2 disabled unsynchronized log Tue Apr 12 03 17 16 GMT 7 itsonas1 cf fm timeMasterStatus info Acting as cluster time slave Tue Apr 12 03 17 17 GMT 7 itsonas1 cf fsm takeoverOfPartnerDisabled notice Cluster monitor takeov...

Page 115: ...e takeover by using System Manager and clicking Active Active Configuration Takeover Figure 7 6 System Manager initiating takeover Tip Under normal conditions you do not need to perform takeover or giveback on an IBM N series system Usually you must use it only if a controller must be halted or rebooted for maintenance ...

Page 116: ... shows the Active Active takeover wizard step 1 Click Next to continue Figure 7 7 System Manager initiating takeover Step 1 3 Figure 7 8 shows the Active Active takeover wizard step 2 Click Next to continue Figure 7 8 System Manager initiating takeover Step 2 ...

Page 117: ...Active takeover wizard step 3 Click Finish to continue Figure 7 9 System Manager initiating takeover Step 3 5 Figure 7 10 shows the Active Active takeover wizard final step where takeover was run successfully Click Close to continue Figure 7 10 System Manager takeover successful ...

Page 118: ...splays the status of the takeover The only option at this stage to perform giveback Figure 7 11 System Manager itsonas2 taken over by itsonas1 Starting giveback by using System Manager Figure 7 12 shows how to perform the giveback by using System Manager Figure 7 12 FilerView Start giveback ...

Page 119: ...e 7 13 shows a successfully completed giveback Figure 7 13 System Manager giveback successful Figure 7 14 shows that System Manager now reports the systems back to normal after a successful giveback Figure 7 14 System Manager with systems back to normal ...

Page 120: ...tarted when one of the following conditions occurs An HA pair node that is configured for immediate takeover on panic undergoes a software or system failure that leads to a panic A node that is in an HA pair configuration undergoes a system failure for example NVRAM failure and cannot reboot There is a mismatch between the disks that one node can see and the disks that the other node can see One o...

Page 121: ...ugh the HA pair configuration interconnect adapters At this time the nodes exchange a list of disk shelves that are visible on the A loop and B loop of each node If the B loop shelf count on its partner is greater than its local A loop shelf count the system concludes that it is impaired It then prompts that node s partner to start a takeover ...

Page 122: ...102 IBM System Storage N series Hardware Guide ...

Page 123: ...roCluster is to provide mission critical applications with redundant storage services in case of site specific disasters By synchronously mirroring data between two sites it tolerates site specific disasters with minimal interruption to applications and no data loss This chapter includes the following sections Overview of MetroCluster Business continuity solutions Stretch MetroCluster Fabric Attac...

Page 124: ...troCluster is a fully integrated solution that is easy to administer and is built on proven technology It provides automatic failover to remote data center to achieve the following goals Helps protect business continuity if the primary data center fails Helps reduce dependency on IT staff for manual actions Provides synchronous mirroring up to 100 km Its data replication capabilities are designed ...

Page 125: ...e Provides a mechanism to fail over automatically or administrator driven FC switches Provide storage system connectivity between sites and locations These switches are used for fabric MetroClusters only FibreBridges if you use EXN3000 or EXXN3500 SAS Shelves Cables Multimode fiber optic cables single mode cables are not supported MetroCluster allows the Active Active configuration to be spread ac...

Page 126: ...are a site disaster and start a site failover by using a single command The cf forcetakeover d command starts a takeover of the local partner even in the absence of a quorum of partner mailbox disks This command gives the administrator the ability to declare a site specific disaster and have one node take over its partner s identity without a quorum of disks The following requirements must be in p...

Page 127: ...lity and data mirroring You can place these two systems in separate locations When the distance between the two systems is less than 500 m you can implement Stretch MetroCluster The cabling is direct connected between nodes and shelves FibreBridges are required when SAS Shelves EXN3000 and EXN3500 are used Feature Synchronous SnapMirror MetroCluster SyncMirror Network for Replication Fibre Channel...

Page 128: ...ck per site Because of the SyncMirror feature and the mirrored plexes pool0 and pool1 disk failures have no operational impact All disks including spare disks must be manually assigned to each pool MetroCluster design features the following important considerations Stretch MetroCluster has no imposed spindle limits only the platform limit Take care in planning N6210 MetroCluster configurations bec...

Page 129: ...ified distances are up to 500 m If you have distances greater than 500 m choose Fabric MetroCluster Table 8 2 lists theoretical Stretch MetroCluster distances Table 8 2 Theoretical MetroCluster distances in meters 8 3 2 Cabling Stretch MetroClusters Figure 8 5 shows a Stretch MetroCluster with two EXN4000 FC shelves on each site Figure 8 5 Stretch MetroCluster cabling with EXN4000 Data Rate in Gbp...

Page 130: ...MetroCluster with FibreBridges and SAS Shelves 8 4 Fabric Attached MetroCluster Fabric Attached MetroCluster which is sometimes called Fabric MetroCluster is based on the same concept as Stretch MetroCluster However it provides greater distances up to 100 km by using SAN Fabrics Both nodes in a Fabric MetroCluster are connected through four Fibre Channel switches two fabrics for high availability ...

Page 131: ...ilable for these SFPs for large distances greater than 30 km Leased metro wide transport services from a service provider Typically provisioned by dense wavelength division multiplexer time division multiplexer optical add drop multiplexer DWDM TDM OADM devices Make sure that the device is supported by fabric switch vendor IBM Brocade Dedicated bandwidth between sites mandatory One inter switch li...

Page 132: ... applies N62x0 and N7950T as shown in Table 8 3 Table 8 3 Maximum number of spindles with DOT 8 x and Fabric MetroCluster Platform Number of spindles SAS SATA requires FibreBridges Maximum number of FC disks N6210 480 480 N6240 600 600 N6270 960 840 672 with DOT7 3 2 or 7 3 4 N7950T 1176 840 672 with DOT7 3 2 or 7 3 4 Requirement Fabric MetroClusters need four dedicated FC switches in two fabrics ...

Page 133: ...es that the two fabrics be dedicated to Fabric MetroCluster Extensive testing was done to ensure adequate performance with switches that are included in a Fabric MetroCluster configuration For this reason Fabric MetroCluster requirements prohibit the use of any other model or vendor of Fibre Channel switch than the Brocade included with the Fabric MetroCluster If you use SAS Shelves EXN3000 and EX...

Page 134: ...ting data to the following plexes The local plex on the local shelf that is actively serving data The remote plex on the remote shelf that is normally not serving data On local shelf failure the remote shelf seamlessly takes over data serving operations Both copies or plexes are updated synchronously on writes which ensures consistency 8 5 1 SyncMirror overview The design of IBM System Storage N s...

Page 135: ...d Any of the commands that are shown in Example 8 1 can be used Example 8 1 Verification of SyncMirror itsosj_n1 sysconfig r itsosj_n1 aggr status r itsosj_n1 vol status r To see the volume plex raidgroup relationship use the sysconfig r command as shown in Example 8 2 Use the aggr mirror command to start mirroring the plexes Example 8 2 Viewing the aggregate status n5500 ctr tic 1 sysconfig r Agg...

Page 136: ... 1 FCAL 15000 136000 278528000 280104 573653840 Pool1 spare disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used MB blks Phys MB blks Spare disks for block or zoned checksum traditional volumes or aggregates spare 0c 28 0c 1 12 FC B 1 FCAL 15000 272000 557056000 280104 573653840 Pool0 spare disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used MB blks Phys MB blks Spare disks for bloc...

Page 137: ...RAID set plex Therefore without MetroCluster all of the following rules apply as for a normal cluster If controller A fails partner B takes over If loop A Plex0 on controller A fails controller A continues operation by running through loop B Plex1 If controller A fails and either loop A or loop B Plex0 Plex1 fails you cannot continue MetroCluster protects against the following scenario If controll...

Page 138: ...ate an ISL to high priority cluster interconnect traffic FCVI exchanges must be isolated because they are high priority traffic that should not be subject to any interruption or congestion caused by storage traffic Fabric OS v6 0 0b introduces Traffic Isolation Zones which have the following features Can create a dedicated route Does not modify the routing table Is implemented across the entire da...

Page 139: ...c that enters from a member of the zone The traffic is isolated to only those ISLs that were included in the zone The fabric also attempts to exclude traffic that is not in the TI zone from using ISLs within that TI zone TI Zones are a new feature of Fabric OS v6 0 0b that have the following restrictions TI Zones exist only in the Defined Zoning Configuration TI Zones must be created with Domain I...

Page 140: ...abric MetroCluster environment VI traffic orange is separated from data backend traffic black by TI zones Figure 8 13 TI Zones in MetroCluster environment 8 7 Failure scenarios This section describes some possible failure scenarios and the resulting configurations when MetroCluster is used ...

Page 141: ... 8 14 IBM System Storage N series failure 8 7 2 N series and expansion unit failure Figure 8 15 shows the loss of one site which results from failure of the controller and shelves at the same time Figure 8 15 Controller and expansion unit failure To continue access a failover must be performed by the administrator by running the cfo d command Data access is restored because DC1 mirror was in sync ...

Page 142: ...lure it must be resolved promptly before a more critical failure occurs Figure 8 16 Interconnect failure During this period data access is uninterrupted to all hosts No automated controller takeover occurs Both controller heads continue to serve its LUNs and volumes However mirroring and failover are disabled which reduces data protection When the interconnect failure is resolved mirrors are resyn...

Page 143: ...t be started by using the cfo d command Both primaries now exist at data center 2 and hosting of Host1 is done at data center 2 Figure 8 17 Site failure Attention If the site failure is staggered in nature and the interconnect fails before the rest of the site data loss might occur Data loss occurs because processing continues after the interconnect fails However site failures often occur pervasiv...

Page 144: ...s switches and storage systems are recovered at data center 1 a recovery can be performed A cf giveback command is run to resume normal operations as shown in Figure 8 18 Mirrors are resynchronized and primaries and mirrors are reversed to their previous status Figure 8 18 MetroCluster recovery ...

Page 145: ... Chapter 9 MetroCluster expansion cabling This chapter describes two options for using MetroCluster with SAS connected expansion shelves This chapter includes the following sections FibreBridge 6500N Stretch MetroCluster with SAS shelves and SAS cables 9 ...

Page 146: ...in stretch MetroCluster configurations However both plexes of the same aggregate must use the same type of storage In a fabric MetroCluster configuration only Fibre Channel drive shelves were supported Starting with Data ONTAP 8 1 EXN3000 SAS or SATA and EXN3500 are supported on Fabric MetroCluster and on Stretch MetroCluster through SAS Fibre Channel bridge FibreBridge The FibreBridge as shown in...

Page 147: ...herefore two FibreBridges one for redundancy are required per stack of SAS shelves Current maximum is 10 SAS shelves per stack of SAS or SATA disks Attention At the time of this writing Data ONTAP 8 1 has the following limitations The FibreBridge does not support mixing EXN3000 and EXN3500 in same stack FibreBridge configurations do not support SSD drives The FibreBridge does not support SNMP Shel...

Page 148: ...rmal platform spindle limits apply to the entire MetroCluster configuration However because each controller sees all storage the platform spindle limit applies to the entire configuration For example if the spindle limit for N series N62x0 is n the spindle limit for a N62x0 fabric MetroCluster configuration remains n despite the two controllers ...

Page 149: ...el ports of the N series nodes are connected to the Fibre Channel ports on the FibreBridge FC1 and FC2 SAS ports of the first and last shelf in a stack are connected to the SAS ports SAS port A on the FibreBridge Each stack has two bridges MetroCluster uses at least four FibreBridges Figure 9 4 Stretch MetroCluster with FibreBridges ...

Page 150: ...9 1 3 Administration and management The FibreBridge comes with an easy to use web based ExpressNAV System Manager which provides capabilities for remote configuration management of the bridge diagnostic testing and updating the bridge firmware The ATTO QuickNAV utility can be used to configure the bridge Ethernet management 1 port Use the ATTO ExpressNAV System Manager which requires you to connec...

Page 151: ...500N bridges for which you are replacing SAS cables or hot adding a SAS disk shelf or an existing system for which you are replacing SAS copper cables and FibreBridge 6500N bridges 9 2 1 Before you begin The following overall requirements must be met before any procedures in this document are complete Your system platform disk shelves and version of Data ONTAP that your system is running must supp...

Page 152: ...onnect all eight four pairs of the SC LC or MTRJ breakout connectors to the patch panel If you are using SAS optical singlemode breakout cables the following parameters apply You can use these cables for controller to shelf connections Shelf to shelf connections use multimode QSFP to QSFP cables or multimode MPO cables with MPO QSFP modules The point to point QSFP to QSFP path of a single singlemo...

Page 153: ... controller to patch panel and patch panel to shelf connections Before you begin The following prerequisites must be met before you begin You ordered and received the appropriate type number and length of SAS optical cables that are required for your configuration Your system must have the appropriate number of available SAS ports on each controller If you are using SAS HBAs your system must have ...

Page 154: ...e SAS connections by entering the following command at the Maintenance mode prompt of either controller sasadmin expander_map 10 The next step depends on the output If the output lists all of the IOMs the IOMs all have connectivity Go to step 11 If any IOMs are not shown the output does not show an IOM because it is cabled incorrectly or the output does not show all the IOMs downstream from the in...

Page 155: ...AS cables in a multipath HA configuration The SAS cables that are described in this section include SAS copper and SAS optical cables When you replace SAS cables consider the following important points Replacing a SAS cable means that you are replacing a cable by using the same ports for a controller to shelf or shelf to shelf connection Situations where you might want to replace a SAS cable can i...

Page 156: ... or later than the most current version on the N series Support Site no disk shelf firmware update is needed If the firmware version in the command output is an earlier version than the most current version on the N series Support Site you must update the disk shelf firmware You can run the commands from either controller 4 Replace SAS cables by completing the following steps You can ignore cablin...

Page 157: ...eet so that you know how to cable your disk shelf or stack of shelves to the controller If you are hot adding a disk shelf with SAS optical cables to a stack of disk shelves that is connected with SAS copper cables you can temporarily have both cable types present in the stack After the disk shelf is hot added you must replace the SAS copper cables for the rest of the shelf to shelf connections in...

Page 158: ...e power supplies for each disk shelf and wait for the disk drives to spin up When the disk shelf has the maximum number of supported power supplies all disk drives or carriers spin up at the same time However if one or two power supplies faulted in a disk shelf with four power supplies or if one power supply faulted in a disk shelf with two power supplies disk drives spin up in sets of six at 12 s...

Page 159: ...M A circle port on the last shelf in the stack You can leave the other end of the cable connected to the controller to minimize confusion or replace the cable with a longer cable if needed b Connect daisy chain the IOM A circle port of the last disk shelf in the stack to the IOM A square port of the new disk shelf by using the SAS cables that were included with the new disk shelf c Connect the cab...

Page 160: ...ou hot added a disk shelf with SAS optical cables to a stack of disk shelves that are connected with SAS copper cables replace the SAS copper cables for the rest of the shelf to shelf connections and the controller to shelf connections so that the stack meets the cabling rules 6 Go to the next section Verifying the disk drive and disk shelf firmware versions on page 140 Verifying the disk drive an...

Page 161: ...asadmin expander_map b Locate the disk shelf firmware information for the hot added disk shelf in the output Example 0151 is the disk shelf firmware version for shelf number one Slot A IOM A in the storage system as shown in the following example Expanders on channel 4a Level 3 WWN 500a0980000840ff ID 1 Serial Number SHU0954292G114C Product DS424IOM6 Rev 0151 Slot A c Compare the firmware informat...

Page 162: ...and ACP Cabling Guide from the N series Support Site Disk shelves that are connected with SAS optical cables require a version of disk shelf firmware that supports SAS optical cables Best practice is to update all disk shelves in the storage system with the latest version of disk shelf firmware Commands are entered at the console of either controller unless otherwise noted This procedure assumes t...

Page 163: ...gure 9 6 Stretch MetroCluster using FibreBridge and SAS copper cables Figure 9 7 on page 144 is an example of how a 62xx looks after the system is cabled with SAS optical cables which were replaced the FibreBridge 6500N bridges and SAS copper cables ...

Page 164: ...ould be the fourth line of output 3 Verify that the disk shelves in the storage system have the latest version of disk shelf firmware by completing the following steps a Enter the following command at the storage system console sysconfig v b Locate the disk shelf firmware information for the disk shelves in the output Example 0151 is the disk shelf firmware version for shelf number one for each IO...

Page 165: ...lowing command from either node cf disable 8 If you ordered more HBAs install them now otherwise go to the next step 9 Enter the following command from the system console to perform a clean shutdown halt 10 Select one of the disk shelf stacks and remove the cabling by completing the following steps you can begin on any stack a Remove the controller to bridge FC cables You are removing a total of f...

Page 166: ...onfig The output should be the same as Step 2 the system should be Multi Path HA However the SAS ports and attached disk shelf information changed because disk shelves were moved from FC ports connected through the bridges to SAS ports on the controllers 16 Enable controller failover by entering the following command on either node cf enable 17 Verify that controller failover is enabled by enterin...

Page 167: ...os Other key areas that are covered include cost information special hardware requirements creating RAID groups and converting from RAID 4 to RAID DP This chapter includes a double disk failure recovery scenario This scenario shows how RAID DP allows the RAID group to continue serving data and re create the data on the two failed disks This chapter includes the following sections Background Why us...

Page 168: ...ed seek time performances it did not reduce the amount of contrast between decreased reliability It addition the technology increased bit error rates The result is an increase of potential uncorrectable bit errors and reduced reliability of traditional single parity RAID adequately protecting data Today traditional RAID is stretching past its limitations By increasing the data fault tolerance of v...

Page 169: ...s Disk drives are orders of magnitude larger than they were when RAID was first introduced As disk drives become larger their reliability did not improve and the bit error likelihood per drive increased proportionally with larger media These three factors larger disks unimproved reliability and increased bit errors with larger media have serious consequences for the ability of single parity RAID t...

Page 170: ... disks are lost to parity which reduces usable capacity and increases the total cost of ownership TCO Performance is slower with smaller arrays aggregates and volumes which affects businesses and users The most reliable protection that is offered by single parity RAID is RAID 1 or mirroring In RAID 1 the mirroring process replicates an exact copy of all data on an array aggregate or volume to a se...

Page 171: ...P At the lowest level RAID DP offers protection against two failed disks within the same RAID group It also offers protection from a single disk failure followed by a bad block or bit error before reconstruction completes A higher level of protection is available by using RAID DP with SyncMirror In this configuration the protection level is up to five concurrent disk failures That is you are prote...

Page 172: ... single parity RAID improves data protection Because traditional single parity RAID has a good track record to date the concept of double parity RAID sounds like a better protection scheme This is borne out in the earlier example that used the MTTDL formula But what exactly is RAID DP At the most basic layer RAID DP adds a second parity disk to each RAID group in a volume A RAID group is an underl...

Page 173: ...ty Figure 10 7 represents a traditional RAID 4 group that uses row parity It consists of four data disks the first four columns labeled D and the single row parity disk the last column labeled P The rows represent the standard 4 KB blocks that are used by the traditional RAID 4 implementation The second row is populated with sample data in each 4 KB block Parity that is calculated for data in the ...

Page 174: ...nal parity stripe includes an element from row parity as part of its diagonal parity sum RAID DP treats all disks in the original RAID 4 construct including both data and row parity disks as the same In Figure 10 9 the rest of the data is added for each block and creates corresponding row and diagonal parity stripes Figure 10 9 Block representation of RAID DP corresponding with row and diagonal pa...

Page 175: ...ical theorems and proofs see Double Disk Failure Correction which is available at the following USENIX Organization website http www usenix org Review the double disk failure and subsequent recovery process that is described in 10 4 4 RAID DP reconstruction on page 155 10 4 4 RAID DP reconstruction By using Figure 10 9 as the starting point assume that the RAID group is functioning normally when a...

Page 176: ...construction simulation diagonal blue block The data was re created from the missing diagonal blue block by using the same arithmetic that was previously described 12 7 2 2 1 Now that the missing blue series diagonal information is re created the recovery process switches from using diagonal parity to using horizontal row parity Specifically the top row after the blue block re creates the missing ...

Page 177: ...gray diagonal block in column two row two as shown in Figure 10 13 Figure 10 13 RAID DP reconstruction simulation of gray block column two RAID DP recovers the gray diagonal block in column two row two Adequate information is now available for row parity to re creating the missing horizontal white block one in the first column row two as shown in Figure 10 14 Figure 10 14 RAID DP reconstruction si...

Page 178: ... missing data in the gold stripe as shown in Figure 10 15 Figure 10 15 RAID DP reconstruction simulation of second horizontal block After RAID DP re creates a missing diagonal block the process again switches to re creating a missing horizontal block from row parity When the missing diagonal block in the gold stripe is re created enough information is available to re create the missing horizontal ...

Page 179: ...shows recovery in operation However there are other areas of RAID DP that require further description For example if a double disk failure occurs RAID DP automatically raises the priority of the reconstruction process so that the recovery completes faster The time in reconstruction of data blocks that are generated from two failed disk drives is slightly less than the time to reconstruct data from...

Page 180: ...the default RAID group type The default RAID group type that is used either RAID DP or RAID4 depends on the platform and disk that are used The output that is shown in Figure 10 19 from the vol status command shows a four disk RAID DP RAID group for a traditional volume named test The second parity disk for diagonal parity is denoted as dparity Figure 10 19 Volume status command output of aggregat...

Page 181: ...ifies that diagonal parity creation in progress Figure 10 22 The itso volume in reconstruct status during conversion of diagonal parity RAID DP Calculating the diagonals as part of a conversion to RAID DP takes time and affects performance slightly on the storage controller The amount of time and performance effect for conversions to RAID DP depends on the storage controller and how busy the stora...

Page 182: ...ws itso as RAID DP parity Figure 10 24 Aggregate status of itso as RAID DP parity Figure 10 25 shows the conversion of itso back to RAID4 In this case the conversion is instantaneous because the old RAID4 row parity construct is still in place as a subsystem in RAID DP Figure 10 25 The itso volume conversion from traditional RAID DP to RAID4 Figure 10 26 shows the completed process If a RAID DP gr...

Page 183: ... disk is a storage system disk that is not assigned to a RAID group It does not yet hold data but is ready for use In a disk failure within a RAID group Data ONTAP automatically assigns hot spare disks to RAID groups to replace the failed disks Hot spare disks do not have to be in the same disk shelf as other disks of a RAID group to be available to a RAID group as shown in Figure 10 27 Figure 10 ...

Page 184: ...of the same size as the failed disk If no disk of the same size is available Data ONTAP replaces the failed disk with a spare disk of the next available size up 2 The storage system reconstructs in the background the missing data onto the hot spare disks 3 The storage system logs the activity in the etc messages file on the root volume With RAID DP these processes can be carried out even if two di...

Page 185: ...ibes N series core technologies such as the WAFL file system disk structures and non volatile RAM NVRAM access methods This chapter includes the following sections Write Anywhere File Layout Disk structure NVRAM and system memory Intelligent caching of write requests N series read caching techniques 11 ...

Page 186: ...based storage and the read and write mechanisms to support block storage or LUNs As shown in Figure 11 1 the protocol access layer is above the data placement layer of WAFL This layer allows all of the data to be effectively managed on disk independently of how it is accessed by the host This level of storage virtualization offers significant advantages over other architectures that have tight ass...

Page 187: ...re Closely integrated with N series RAID is the aggregate which forms a storage pool by concatenating RAID groups The aggregate controls data placement and space management activities The FlexVol volume is logically assigned to an aggregate but is not statically mapped to it This dynamic mapping relationship between the aggregate layer and the FlexVol layer is integral to the innovative storage fe...

Page 188: ...tack allows the N series to optimize writes to disk even when writing to double parity RAID Most other storage vendors cache writes at the device driver level The N series uses a multilevel approach to read caching The first level read cache is provided by the system buffer cache Special algorithms decide which data to retain in memory and which data to prefetch to optimize this function The N ser...

Page 189: ...ause Data ONTAP does not overwrite existing disk blocks The NVRAM is used as a journal of the write requests that Data ONTAP received since the last consistency point was created With this approach Data ONTAP reverts to the latest consistency point if a failure occurs It then replays the journal of write requests from NVRAM to bring the system up to date and ensure the data and metadata on disk ar...

Page 190: ...M into two separate buffers When one buffer is full that triggers disk write activity to flush all the cached writes to disk and create a consistency point Meanwhile the second buffer continues to collect incoming writes until it is full and then the process reverts to the first buffer This approach to caching writes in combination with WAFL is closely integrated with N series RAID 4 and RAID DP I...

Page 191: ...illiseconds and it takes a few milliseconds for a disk to respond to an I O Therefore disks are always the performance bottleneck of any storage system because disks are radically slower than any other component on the system When a system starts committing back to back CPs the disks are taking writes as fast as they can That is a platform limit for that system To improve performance when the plat...

Page 192: ... most data intensive environments The system buffer cache and Flash Cache increase read performance within a storage system FlexCache scales read performance beyond the boundaries of any single system s performance capabilities N series deduplication and other storage efficiency technologies eliminate duplicate blocks from disk storage These functions ensure that valuable cache space is not wasted...

Page 193: ...reads Small random reads are the most expensive disk operation because they require a higher number of head seeks per kilobyte than sequential reads Head seeks are a major source of the read latency that is associated with reading from disk Therefore data that is randomly read is a high priority for caching in system memory The default caching behavior for the Data ONTAP buffer cache is to priorit...

Page 194: ...174 IBM System Storage N series Hardware Guide ...

Page 195: ...4 All rights reserved 175 Chapter 12 Flash Cache This chapter provides an overview of Flash Cache and all of its components This chapter includes the following sections About Flash Cache Flash Cache module How Flash Cache works 12 ...

Page 196: ...uired and the power cooling and rack space requirements A Flash Cache module provides another 256 GB 512 GB or 1 TB PAM II of extended cache for your IBM System Storage N series storage subsystem The amount depends on the model Up to eight modules can be installed Each module must be installed on a PCI express slot and uses only another 18 W of power per module Extra rack space and ventilation are...

Page 197: ...t to the disks to retrieve it This process is shown in Figure 12 2 Figure 12 2 Read request without Flash Cache module installed 12 3 2 Data ONTAP clearing space in the system memory for more data When more space was needed in memory Data ONTAP analyzes what it holds and looks for the lowest priority data to clear out to make more space Depending on the workload this data might be in system memory...

Page 198: ...ving useful data in Flash Cache With the addition of Flash Cache modules the data that was cleared previously is now placed in the module Data is always read from disk into memory and then stored in the module when it must be cleared from system memory as shown in Figure 12 4 Figure 12 4 Data is stored in Flash Cache ...

Page 199: ...s needed as shown in Figure 12 5 Figure 12 5 Read request with Flash Cache module installed When the data is there access to it is far faster than having to go to disk This process is how a workload is accelerated as shown in Figure 12 6 Figure 12 6 More storage for WAFL extended cache Need Space in System Memory Read from Disk Data in Module Read from Module Not in Module ...

Page 200: ...180 IBM System Storage N series Hardware Guide ...

Page 201: ...the disk in a manner that precludes the recovery of that data by any known recovery methods It also describes the Data ONTAP disk sanitization feature and briefly addresses data confidentiality technology drivers costs and risks and the sanitizing operation This chapter includes the following sections Data ONTAP disk sanitization Data confidentiality Data ONTAP sanitization operation Disk Sanitiza...

Page 202: ...entiality In every industry IT managers face increasing pressure to ensure the confidentiality of corporate client and patient data In addition companies and managers in certain industries must comply with laws that specify strict standards for handling distributing and the use of confidential client corporate and patient information There are methods and products to aid in data storage and transm...

Page 203: ...o help ensure that data on those disks is physically unrecoverable You might have disks that you intend to remove from one storage system and want to reuse those disks in another appliance 13 2 4 Costs and risks All enterprises must consider the following critical factors when they are deciding on the cost and risk of a hard disk sanitization practices The cost of running sanitization programs on ...

Page 204: ...the first pass 0xaa on the second pass and 0x3c on the third pass The c option specifies the number of cycles of pattern writes The default is one cycle All sanitization process information is written to the log file at etc sanitization log The serial numbers of all sanitized disks are written to etc sanitized_disks Disk sanitization is not supported on solid state drives SSDs It does not work on ...

Page 205: ... 3FP0RJMR0000221844GP Tue Jun 24 02 53 55 Disk 8a 44 S N 3FP0RFAZ00002218446B format completed in 00 13 45 Tue Jun 24 02 53 59 Disk 8a 43 S N 3FP20XX400007313LSA8 format completed in 00 13 49 Tue Jun 24 02 54 04 Disk 8a 45 S N 3FP0RJMR0000221844GP format completed in 00 13 54 Tue Jun 24 02 54 11 Disk 8a 44 S N 3FP0RFAZ00002218446B cycle 1 pattern write of 0x47 completed in 00 00 16 Tue Jun 24 02 5...

Page 206: ...plete 13 4 Disk Sanitization with encrypted disks You can destroy data that is stored on disks by using the disk encrypt sanitize command If you want to return a disk to a vendor but do not want anyone to access sensitive data on it use the disk encrypt sanitize command This process renders the data on the disk inaccessible but the disk can be reused This command works only on spare disks and was ...

Page 207: ... you are sizing an IBM System Storage N series storage system to your environment The following topics are addressed A complete explanation is beyond the scope of this book so only high level planning considerations are presented This chapter includes the following sections Primary issues that affect planning Performance and throughput Summary 14 ...

Page 208: ...und tasks that are running on the storage system for example SnapMirror 14 2 1 Capacity requirements A key measurement of a storage system is the amount of storage that it provides Vendors and installers of storage systems often deal with raw storage capacities However users are often concerned with available capacity only Ensuring that the gap is bridged between raw capacity and usable capacity m...

Page 209: ...00 GB or 48 TB Usable capacity Usable capacity is determined by factoring out the portion of the raw capacity that goes to support the infrastructure of the storage system This capacity includes space that is used for operating system information disk drive formatting file system formatting RAID protection spare disk allocation mirroring and the Snapshot protection mechanism The following example ...

Page 210: ...storage aggregates that then have volumes also called file systems or LUNs allocated on them Normal practice is to treat the nine remaining disks per controller as data disks which creates a single large aggregate on each controller The following 24 available disks are now allocated Spare disk drive 2 1 per controller RAID parity disks 2 2 per controller Data disks 18 9 per controller About 25 of ...

Page 211: ...ardless of disk type behind an N Series Gateway Zone checksum ZCS Zone checksums were used on older N Series SATA storage They imposed only a small capacity overhead with 63 64 sectors remaining available for user data but they had a negative effect on performance Zone checksums are no longer supported Advanced zone checksum AZCS Data ONTAP 8 1 1 and later releases provide support for a new checks...

Page 212: ...configuration In those cases change it to 5 Disk Type Capacity decimal GB Capacity GB binary GiB Checksum type Right sized capacity binary GiB FC SAS SSD 72 68 BCS 512 520 blocks approximately 1 5 66 144 136 132 300 272 265 600 900 1200 SATA or OEM array LUNs with a Gateway 500 465 BCS 8 9 blocks approximately 11 1 413 750 698 620 1000 931 827 2000 1862 1655 3000 2794 2483 MSATA or OEM array LUNs ...

Page 213: ...ntroller and allocate them to a single large volume the resulting capacity is approximately 13 400 GB binary as shown in Figure 14 1 Figure 14 1 Example of raw decimal to usable binary capacity The example in Figure 14 1 is for a small system The ratio of usable to raw capacity varies depending on several factors such as RAID group size disk type and space efficiency features that can be applied l...

Page 214: ...napshot technology optimizes storage because only changed blocks are retained For file access the change rate is typically in the 1 5 range For database applications it might be similar However in some cases it might be as high as 100 Server SAN attached Minor effect on the performance of the server when the Snapshot is created to ensure file system and LUN consistency Negligible ongoing effect on...

Page 215: ...or controller hardware The optimal initial plan is for 50 average usage with peak periods of 70 processor usage In a two node storage cluster this configuration allows the cluster to failover to a single node with no performance degradation If the processors are regularly running at a much higher usage for example 90 performance might still be acceptable However expect some performance degradation...

Page 216: ...itional application is used for network attached storage solutions Because many clients are attached to one or more servers there is little possibility to effectively plan and model in advance of actual deployment However the following common sense considerations can help This environment is characterized by the use of Network File System NFS or Common Internet File System CIFS protocols It is acc...

Page 217: ...erstanding of the type of different workloads so that beneficial balancing can be accomplished Clustered environments always have multiple I O paths available so it is important to balance the workload across these I O paths and across server heads For mission critical environments it is important to plan for the worst case scenario That is running the enterprise when one of the storage systems fa...

Page 218: ...ees have daytime operations Number of storage groups Because a storage group cannot span N series storage systems the number of storage groups affects sizing There is no recommendation on number of storage groups per IBM System Storage N series storage system However the number and type of users per storage group helps determine the number of storage groups per storage system Volume type Are FlexV...

Page 219: ...p and recovery In addition to backup servers all storage systems must be backed up Generally the goal is to have the backup process occur at a time and in a way that minimizes the effect on overall production Therefore many backup processes are scheduled to run during off hours However all of these backups run more or less at the same time Therefore the greatest I O load that is put on the storage...

Page 220: ...ihood of error while providing a mechanism to get the server back into service as quickly as possible That is accept the fact that failures occur but have a plan ready and practiced to recover when they do Spare servers Some enterprises keep spare equipment around in case of failure Generally this is the most expensive solution and is only practical for the largest enterprises An often overlooked ...

Page 221: ... patches together and might introduce new features Major upgrades generally introduce significant new features To remain informed of new software releases subscribe to the relevant sections at the following IBM automatic support notification website https www ibm com support mynotifications Upgrades for Data ONTAP and mechanisms for implementing the upgrade are available at this website http www i...

Page 222: ...eatly challenging and most customers can successfully deploy it themselves by following these guidelines Because of the simplicity that appliances provide if a mistake is made in the initial deployment corrective actions are not difficult or overly disruptive For many years customers iterated their storage system environments into scalable reliable and smooth running configurations Therefore getti...

Page 223: ...allation and software setup To help perform the initial hardware and software setup it also describes the following administrative interfaces Serial console RLM interface SSH connections At a high level the GUI interfaces This part includes the following chapters Chapter 15 Preparation and installation on page 205 Chapter 16 Basic N series administration on page 213 Part 2 ...

Page 224: ...204 IBM System Storage N series Hardware Guide ...

Page 225: ...tem even with limited experience and knowledge of the N series hardware and software features System Manager helps with basic setup and administration tasks and can help you manage multiple IBM N series storage systems from a single application This chapter includes the following sections Installation prerequisites Configuration worksheet Initial hardware setup Troubleshooting if the system does n...

Page 226: ...nment such as host name IP DNS AD and Network Work through this checklist with the customer and inform them about the rack and floor space requirements This process speeds up the installation time because all information was collected beforehand After this process is complete and equipment is delivered to the customer you can arrange an installation date 15 1 2 Before arriving on site Before you a...

Page 227: ...t Type of information Your values Storage system Host name Password Time zone Storage system location Language that is used for multiprotocol storage systems Administration host Host name IP address Interface groups Name of the interface group such as ig0 Mode type single multi or LACP Load balancing type IP based MAC address based or round robin based Number of links number of physical interfaces...

Page 228: ...ype Are jumbo frames supported MTU size for jumbo frames Flow control e0M interface if available IP address Network mask Partner IP address Flow control Router if used Gateway name IPv4 address IPv6 address HTTP Location of HTTP directory DNS Domain name Server address 1 Server address 2 Server address 3 NIS Domain name Server address 1 Server address 2 Server address 3 Type of information Your va...

Page 229: ...ent from default User authentication style 1 Active Directory domain 2 Windows NT 4 domain 3 Windows Workgroup 4 etc passwd or NIS LDAP Windows Active Directory domain Domain name Time server name IP address Windows user name Windows user password Local administrator name Local administrator password CIFS administrator or group Active Directory container BMC MAC address IP address Network mask sub...

Page 230: ...tests 5 Protocol and software license verification activation 6 Firmware updates Disk if applicable Shelf if applicable System RLM BMC 7 Protocol tests and cluster failover tests RLM MAC address IPv4 Address IPv4 Subnet mask IPv4 Gateway IPv6 Address IPv6 Subnet prefix length IPv6 Gateway AutoSupport mailhost AutoSupport recipients ACP Network interface name Domain subnet for network interface Net...

Page 231: ... to the system and verify any changes that you made by running the disk show v command 8 Turn off your controller and disk shelves then turn on the disk shelves For more information about LED responses check the quick reference card that came with the disk shelf or the hardware guide for your disk shelf 9 Complete the following steps to use the onboard diagnostic tests to check that Fibre Channel ...

Page 232: ...NTAP by entering autoboot at the prompt 11 Try starting your system again Table 15 3 shows the next possible steps that you can take Table 15 3 Starting the system If your system Then Starts successfully Proceed to setting up the software Does not start successfully Call IBM technical support The system might not have the boot image downloaded on the boot device ...

Page 233: ...6 Basic N series administration This chapter describes how to perform basic administration tasks on IBM System Storage N series storage systems This chapter includes the following sections Administration methods Starting stopping and rebooting the storage system 16 ...

Page 234: ...cess a pre 8 1 N series through FilerView open your browser and go to the following URL http filername or ip address na_admin To proceed specify a valid user name and password 16 1 2 Command line interface The CLI can be accessed through Telnet or a Secure Shell SSH interface Use the help command or enter a question mark to obtain an overview of available commands Enter help command for a brief de...

Page 235: ... available options of the specified command as shown in Figure 16 1 Figure 16 1 The help and commands The manual pages can be accessed by entering the man command Figure 16 2 shows a detailed description of a command and lists options man command Figure 16 2 Results of a man command ...

Page 236: ... are the supported platforms System Manager release 1 1 supports Data ONTAP 7 2 3 and later The current release is Data ONTAP 8 1 7 mode For more information about System Manager see the following IBM NAS support website http www ibm com storage support nas 16 1 4 OnCommand OnCommand is an operations manager is an N series solution for managing multiple N series storage systems that provides the f...

Page 237: ...s CTRL C for special boot menu Example 16 2 shows the boot options You often boot in normal boot mode Example 16 2 Boot menu 1 Normal Boot 2 Boot without etc rc 3 Change Password 4 Initialize all disks 4a Same as option 4 but create a flexible root volume 5 Maintenance boot Selection 1 5 16 2 2 Stopping the IBM System Storage N series storage system Stopping and rebooting the IBM System Storage N ...

Page 238: ... sends a message to all open client connections This setting can be changed by running the following command options cifs shutdown_msg_level 0 1 2 The following options are available 0 Never send CIFS shutdown messages 1 Send CIFS messages to clients connected and with open files only 2 Send CIFS messages to all open connections default The cifs terminate command shuts down CIFS ends CIFS service ...

Page 239: ...tsosj n1 When you shut down an N series there is no need to specify the cifs terminate command During shutdown this command is run by the operating system automatically Depending on the CIFS message settings messages such as those that are shown in Figure 16 3 are displayed on the affected workstations Figure 16 3 Shut down messages on CIFS clients To restart CIFS run the cifs restart command as s...

Page 240: ...mmand line interface serial console CFE version 1 2 0 based on Broadcom CFE 1 0 35 Copyright C 2000 2001 2002 2003 Broadcom Corporation Portions Copyright C 2002 2003 Network Appliance Corporation CPU type 0x1040102 650MHz Total memory 0x40000000 bytes 1024MB CFE Booting the N series As described in 16 2 1 Starting the IBM System Storage N series storage system on page 217 the IBM System Storage N...

Page 241: ...ram at 0xffffffff80001000 Press CTRL C for special boot menu Interconnect based upon M VIA ERing Support Copyright c 1998 2001 Berkeley Lab http www nersc gov research FTG via Wed Aug 31 19 00 46 GMT cf nm nicTransitionUp info Interconnect link 0 is UP Wed Aug 31 19 00 46 GMT cf nm nicTransitionDown warning Interconnect link 0 is DOWN Data ONTAP Release 7 1H1 Mon Aug 15 16 02 45 PDT 2005 IBM Copyr...

Page 242: ...down These messages are shown in Figure 16 3 on page 219 16 2 3 Rebooting the system The System Storage N series systems can be rebooted from the command line or from the NSM interface Rebooting from the CLI halts the N series and then restarts it as shown in Example 16 10 Example 16 10 Rebooting from the command line interface root itso3775 node1 reboot Broadcast message from root pts 2 Thu Sep 8...

Page 243: ...utility kit software It also describes how to configure a client system to SAN boot from an N series and provides a high level description of host multipathing on the N series platform This part includes the following chapters Chapter 17 Host Utilities Kits on page 225 Chapter 18 Boot from SAN on page 237 Chapter 19 Host multipathing on page 273 Part 3 ...

Page 244: ...224 IBM System Storage N series Hardware Guide ...

Page 245: ... N series storage systems It describes why HUKs are an important part of any successful N series implementation and the connection protocols that are supported It also provides a detailed example of a Windows HUK installation This chapter includes the following sections Host Utilities Kits Host Utilities Kit components Host Utilities functions Windows installation example Setting up LUNs 17 ...

Page 246: ...re on your host computer as described in the Host Utilities release s installation and setup guide Installation of N series Host Utilities is required for hosts that are attached to N series and other storage array to ensure that IBM configuration requirements are met 17 2 Host Utilities Kit components This section provides a high level description of the Host Utility components 17 2 1 What is inc...

Page 247: ...host based utility that helps you configure IBM N series controllers and LUNs The sanlun tool bridges the namespace between host and storage controller and collects and reports storage controller LUN information It then correlates this information with the host device file name or equivalent entity This process assists with debugging SAN configuration issues The sanlun utility is available in all ...

Page 248: ...Interoperability Matrix Some of the hotfixes require a reboot of your Windows host You can wait to reboot the host until after you install or upgrade the Host Utilities When you run the installer for the Windows Host Utilities it lists any missing hotfixes You must add the required hotfixes before the installer can complete the installation process Use the Interoperability Matrix to determine whic...

Page 249: ...hed connections If you are connecting the host s HBA ports directly to the storage system change the media setting of the target ports to loop This task applies to Data ONTAP operating in 7 Mode To display the current setting of the storage system s target ports enter the following command at a storage system command prompt fcp show adapter v The current media type setting is displayed To change t...

Page 250: ...ools Windows Server 2008 R2 The iSCSI initiator is built into the operating system The iSCSI Initiator Properties information is available from Administrative Tools Windows XP guest systems on Hyper V For guest systems on Hyper V virtual machines that access storage directly not as a virtual hard disk mapped to the parent system download and install the iSCSI software initiator You cannot select t...

Page 251: ... Foundation for Windows Select no MPIO only if you are using a single path to the storage system Installing the Host Utilities interactively To install the Host Utilities software package interactively run the Host Utilities installation program and follow the prompts Complete the following steps 1 Check the Interop matrix http www ibm com support docview wss uid ssg1S7003897 for important alerts ...

Page 252: ... 4 Host configuration settings You must collect some host configuration settings as part of the installation process The Host Utilities installer modifies other host settings that are based on your installation choices Fibre Channel and iSCSI identifiers The storage system identifies hosts that are allowed to access LUNs The hosts are identified based on the Fibre Channel worldwide port names WWPN...

Page 253: ...ndow differs depending on the Windows version The iSCSI node name resembles the following example iqn 1991 05 com microsoft server3 17 4 5 Host Utilities registry and parameters settings The Host Utilities require certain registry and parameter settings to ensure that the Windows host correctly handles the storage system behavior The parameters that are set by Windows Host Utilities affect how the...

Page 254: ... must configure these start services so that they depend on the Microsoft iSCSI Initiator service You can create LUNs manually or by running the SnapDrive or System Manager software You can access the LUN by using the Fibre Channel or the iSCSI protocol The procedure for creating LUNs is the same regardless of which protocol you use You must create an initiator group igroup create the LUN and then...

Page 255: ...7 5 4 Adding iSCSI targets To access LUNs when you are using iSCSI you must add an entry for the storage system by using the Microsoft iSCSI Initiator GUI To add a target complete the following steps 1 Run the Microsoft iSCSI Initiator GUI 2 On the Discovery tab create an entry for the storage system 3 On the Targets tab log on to the storage system 4 If you want the LUNs to be persistent across h...

Page 256: ... the newly created LUNs are visible as disks on the host The LUNs are displayed on the Windows host as basic disks under Veritas Enterprise Administrator Accessing LUNs on hosts that use the native OS stack To access a LUN when you are using the native OS stack you must make the LUN visible to the Windows host Complete the following steps 1 Right click My Computer on your desktop and select Manage...

Page 257: ...er HBA communicates with the system BIOS which enables the host to boot from a LUN on the storage system This chapter describes the process to set up a Fibre Channel Protocol FCP SAN boot for your server This process uses a LUN from an FCP SAN attached N series storage system It explains the concept of SAN boot and general prerequisites for using this technique Implementations on the following ope...

Page 258: ...eric server You can then resume operations with the same boot image from the SAN Only some minor reconfiguration is required on the storage system This quick interchange helps reduce downtime and increases host application availability Provisioning for peak usage Because the boot image is available on the SAN it is easy to deploy more servers to temporarily cope with high workloads Centralized adm...

Page 259: ...reboot the server while the original HBA is used for booting purposes This configuration can also be used if the cable or the Fibre Channel switch fails In this scenario use QLogic Fast UTIL or Emulex HBAnyware to select the active HBA Then enable the BIOS scan the BUS to discover the boot LUN and assign the worldwide port name WWPN and LUN ID to the active HBA However when both HBA connections ar...

Page 260: ... storage usage without sacrificing performance RHEL5 can now detect create and install to dm multipath devices during installation To enable this feature add the parameter mpath to the kernel boot line At the initial Linux installation panel enter Linux mpath and press Enter to start the Red Hat installation Windows operating system pagefile placement For Windows 2003 and 2008 configurations store...

Page 261: ...stem fails and cannot boot until the problem is corrected In addition do not create two pagefiles on devices with different performance profiles such as a local disk and a SAN device Attempting to distribute the pagefile in this manner might result in kernel inpage STOP errors In general if the system is paging heavily performance suffers regardless of whether the pagefile is on a SAN device or lo...

Page 262: ...oot process of the IA32 architecture has not changed significantly since the early days of the personal computer Before the actual loading of the operating system from disk takes place the following pre boot process is completed by the host BIOS routines 1 Power on self test The BIOS starts a diagnostic test of all hardware devices for which a routine exists Devices for which the system BIOS does ...

Page 263: ...om systems support myview supportsite wss selectproduct taski nd 7 brandind 5000029 familyind 5364556 typeind 0 modelind 0 osind 0 psid sr conti nue x 1 For more information about Linux Support for FCP iSCSI Host Utilities see this website http www 304 ibm com systems support myview supportsite wss selectproduct taskin d 7 brandind 5000029 familyind 5364552 typeind 0 modelind 0 osind 0 psid sr con...

Page 264: ...OS Utility BootBIOS displays the configuration information for the HBA including the WWPN as shown in Figure 18 3 Figure 18 3 Adapter 02 panel 4 Record the WWPN for the HBA Obtaining the WWPN by using QLogic Fast UTIL To obtain the WWPN by using QLogic Fast UTIL complete the following steps 1 Reboot the host 2 Press Ctrl Q to access BootBIOS ...

Page 265: ...ailable adapters Select the appropriate HBA and press Enter as shown in Figure 18 4 Figure 18 4 Selecting host adapter 4 The Fast UTIL options are displayed Select Configuration Settings and press Enter as shown in Figure 18 5 Figure 18 5 Fast UTIL Options panel ...

Page 266: ...18 7 Enabling host adapter BIOS in Adapter Settings menu 6 Record the WWPN from the Adapter Port Name field Enabling and configuring BootBIOS on the HBA BootBIOS enables the HBA to access the existing BIOS on Intel 32 bit Intel Xeon 64 bit and AMD Opteron 64 bit systems It also enables you to designate a Fibre Channel drive such as a storage system LUN as the host s boot device BootBIOS firmware i...

Page 267: ...up Otherwise press Alt E In Figure 18 8 the x86 BootBIOS is enabled at startup so we press Alt E to access the BIOS Utility QLogic Fast UTIL Enable BootBIOS for QLogic HBAs by using FastUTIL Enabling and configuring Emulex BootBIOS To enable BootBIOS complete the following steps 1 Power on your server and press Alt L to open the Emulex BIOS Utility 2 Select the appropriate adapter and press Enter ...

Page 268: ...ure Adapter s Parameters menu select 1 to enable the BIOS as shown in Figure 18 10 Figure 18 10 Configure the adapter s parameters panel 5 This panel shows the BIOS disabled Select 1 to enable the BIOS as shown in Figure 18 11 Figure 18 11 Enable disable BIOS panel ...

Page 269: ...9 The BIOS is now enabled as shown in Figure 18 12 Figure 18 12 Enable BIOS success panel 6 Press Esc to return to the configure adapter s parameters menu as shown in Figure 18 13 Figure 18 13 Configure adapter s parameters panel ...

Page 270: ...nel 8 The eight boot entries are zero by default The primary boot device is listed first it is the first bootable device Select a boot entry to configure and select 1 as shown in Figure 18 15 Figure 18 15 Configure boot device panel Tip The Emulex adapter supports FC_AL public and private loop and fabric point to point During initialization the adapter determines the appropriate network topology a...

Page 271: ...18 17 Figure 18 17 Starting LUN number panel Clarification In target device failover if the first boot entry fails because of a hardware error the system can boot from the second bootable entry If the second boot entry fails the system boots from the third bootable entry and so on up to eight distinct entries This process provides failover protection by automatically redirecting the boot device wi...

Page 272: ...18 only one LUN is available because SAN zoning is configured to one path as described in 18 2 1 Configuration limits and preferred configurations on page 239 Select 01 to select the primary boot entry and press Enter Figure 18 18 Bootable devices menu 12 After the LUN is selected another menu prompts you to specify how the boot device is identified Use the WWPN for all boot from SAN configuration...

Page 273: ...onfiguration as shown in Figure 18 20 Your HBA s BootBIOS is now configured to boot from a SAN on the attached storage device Figure 18 20 Exit Emulex Boot Utility and saved boot device panel 14 Press Y to reboot your system as shown in Figure 18 21 Figure 18 21 Reboot system confirmation panel ...

Page 274: ...y as shown in Figure 18 22 Figure 18 22 Pressing Ctrl Q for Fast UTIL panel 3 The QLogic Fast UTIL displays the available adapters which are listed in boot order The primary boot device is the first device that is listed If the primary boot device is unavailable the host boots from the next available device in the list Select the first Fibre Channel adapter port and press Enter as shown in Figure ...

Page 275: ...elect Configuration Settings and press Enter as shown in Figure 18 24 Figure 18 24 Configuration settings for QLE2462 adapter panel 5 Select Adapter Settings and press Enter as shown in Figure 18 25 Figure 18 25 Adapter Settings panel ...

Page 276: ...option is disabled press Enter to enable it If this option is enabled go to the next step Figure 18 26 Enabling host adapter BIOS 7 Press Esc to return to the Configuration Settings panel Scroll to Selectable Boot Settings and press Enter as shown in Figure 18 27 Figure 18 27 Accessing selectable boot settings ...

Page 277: ... Enabling selectable boot in Selectable Boot Settings panel 9 Select the entry in the Primary Boot Port Name LUN field as shown in Figure 18 29 Press Enter Figure 18 29 Selecting the Primary Boot Port Name 10 The available Fibre Channel devices are displayed as shown in Figure 18 30 Select the boot LUN 0 from the list of devices and press Enter Figure 18 30 Select Fibre Channel Device panel ...

Page 278: ... prompted to reboot the system as shown in Figure 18 32 Select Reboot system and press Enter Figure 18 32 Exiting the Fast UTIL Configuring the PC BIOS boot order If your host has an internal disk you must enter BIOS setup to configure the host to boot from the LUN You must ensure that the internal disk is not bootable through the BIOS boot order The BIOS setup program differs depending on the typ...

Page 279: ...age memory space to 128 K maximum This limit becomes a concern if you want more devices that require ROM spaced If you have many HBAs in your server you might receive a PCI error allocation message during the boot process To avoid this error disable the boot options in the HBAs that are not being used for SAN boot installation To configure the IBM BIOS setup program complete the following steps 1 ...

Page 280: ...n Figure 18 34 Figure 18 34 Selecting Start Options in Configuration Setup Utility panel 4 Scroll to the PCI Device Boot Priority option and select the slot in which the HBA is installed as shown in Figure 18 35 Figure 18 35 Selecting PCI Device Boot Priority in Start Options panel ...

Page 281: ...Select Hard Drive 5 Configure the LUN as the first boot device 18 2 5 Windows 2003 Enterprise SP2 installation This section describes installation procedures for Windows 2003 Enterprise SP2 Copying the SAN boot drivers When you boot from a LUN you must ensure that the operating system on the LUN has the required HBA driver for booting from a LUN You must download these drivers from the QLogic or E...

Page 282: ...select the supported HBA that you are using and press Enter The driver for the selected HBA is configured in the Windows operating system 7 Follow the prompts to set up the Windows operating system When prompted set up the Windows operating system in a partition that was formatted with NTFS 8 The host system reboots and then prompts you to complete the server setup process as you normally do The r...

Page 283: ... be successful in enumerating the LUNs correctly 18 2 6 Windows 2008 Enterprise installation The Windows 2008 server can be installed in the following installations Full installation Core installation Full installation supports GUI and no roles such as print file or DHCP are installed by default Core installation does not support any GUI It supports only command line and Windows power shell which ...

Page 284: ...2008 full installation option complete the following steps 1 Insert the appropriate Windows Server 2008 installation media into your DVD drive Reboot the server as shown in Figure 18 38 Figure 18 38 Rebooting the server 2 Select an installation language regional options and keyboard input and click Next as shown in Figure 18 39 on page 265 ...

Page 285: ...oot from SAN 265 Figure 18 39 Selecting the language to install regional options and keyboard input 3 Click Install now to begin the installation process as shown in Figure 18 40 Figure 18 40 Selecting Install now ...

Page 286: ...Hardware Guide 4 Enter the product key and click Next as shown in Figure 18 41 Figure 18 41 Entering the product key 5 Select I accept the license terms and click Next as shown in Figure 18 42 Figure 18 42 Accepting the license terms ...

Page 287: ...e 18 43 Figure 18 43 Selecting the Custom installation option 7 If the window that is shown in Figure 18 44 does not show any hard disk drives or if you prefer to install the HBA device driver now click Load Driver Figure 18 44 Where do you want to install Windows window ...

Page 288: ...ve the Windows creates the partition automatically window or click Drive options advanced to create the partition Then click Next to start the installation process as shown in Figure 18 46 Figure 18 46 Installing Windows window When Windows Server 2008 Setup completes the installation the server automatically restarts 11 After Windows Server 2008 restarts you are prompted to change the administrat...

Page 289: ... power on self test POST to check and initialize the hardware It then locates a valid device to boot the system Boot loader If a boot device is found the BIOS loads the first stage boot loader stored in the master boot record MBR into memory The MBR is the first 512 bytes of the bootable device This first stage boot loader is then run to locate and load into memory the second stage boot loader Boo...

Page 290: ...e any special hardware SCSI card or HBA that requires a specific Linux driver If you have a device driver that you must load during the installation process enter Linux dd at the installation boot prompt before the installation wizard is loaded The installation process is similar to local disk installation To set up a Linux SAN boot complete the following steps 1 Insert the Linux installation CD a...

Page 291: ... a single network device However you can still install a iSCSI HBA that offloads certain operations to its own processor The configuration of these are comparable to the boot from FCP 18 3 2 Boot from FCoE FCoE is a protocol that seamlessly replaces the Fibre Channel physical interface with Ethernet FCoE protocol specification uses the enhancements in Data Center Bridging DCB to support the lossle...

Page 292: ...272 IBM System Storage N series Hardware Guide ...

Page 293: ...athing This chapter introduces the concepts of host multipathing It addresses the installation steps and describes the management interface for the Windows Linux and IBM AIX operating systems This chapter includes the following sections Overview Multipathing software options 19 ...

Page 294: ...with no multipath management software and OS with multipath management software Without multipath management software the OS believes that it is connected to two different physical storage devices With multipath management software the OS correctly interprets that both HBAs are connected to the same storage device Figure 19 1 With and without host multipathing Seamlessly recover from a path failur...

Page 295: ... the operating system for example Windows MSDSM Solaris MPxIO AIX MPIO Linux Device Mapper Multipath HP UX PVLinks VMware ESX Server NMP 19 2 1 Third party multipathing solution Third party multipathing solutions are provided by storage vendors or independent software vendors such as Symantec The advantage of the use of multipathing solutions that are provided by storage vendors is that it provide...

Page 296: ...l unit can be accessed from more than one target port One target port might provide full performance access to a logical unit Another target port particularly on a different physical controller might provide lower performance access or might support a subset of the available SCSI commands to the same logical unit Before inclusion of ALUA in the SCSI standards multipath providers had to use vendor ...

Page 297: ...nistrator of the host computer does not need to manually intervene in path management Instead it is handled automatically Running MPIO on the host is still required but no other host specific plug ins are required This process allows the host to maximize I O by using the optimal path consistently and automatically ALUA has the following limitations Can be enabled only on FCP initiator groups Is un...

Page 298: ...278 IBM System Storage N series Hardware Guide ...

Page 299: ...al considerations for nondisruptive upgrades on the N series platform It also provides some high level example procedures for common hardware and software upgrades This part contains the following chapters Chapter 20 Designing for nondisruptive upgrades on page 281 Chapter 21 Hardware and software upgrades on page 295 Part 4 ...

Page 300: ...280 IBM System Storage N series Hardware Guide ...

Page 301: ...objective is to enable upgrade and maintenance of the storage system without affecting the system s ability to respond to foreground I O requests This does not mean that there is no interruption to client I O Rather the I O interruptions are brief enough so that applications continue to operate without the need for downtime maintenance or user notification This chapter includes the following secti...

Page 302: ...rt docview wss uid ssg1S7003776 20 1 1 Types of system NDU The following types of system NDUs are available Major A major version system NDU is an upgrade from one major release of Data ONTAP to another For example an upgrade from Data ONTAP 7 2 x to Data ONTAP 7 3 x is considered a major system NDU Major version NDU is supported only when going from one release to the next in sequence There are o...

Page 303: ... Fibre Channel or iSCSI environment Table 20 2 NDU support for block storage environments Source Data ONTAP version Minor version NDU supported Major version NDU supported 7 1 7 1 1 Yes No 7 1 2 and later Yes Yes 7 2 7 2 1 7 2 2 Yes No 7 2 3 and later Yes Yes 7 3 and later Yes Yesa a Customers who are upgrading from Data ONTAP 7 3 2 can perform major version NDU to Data ONTAP 8 0 and 8 1 releases ...

Page 304: ...eduplication and compression volume dense volume limits for NDU see 20 1 4 System NDU software requirements on page 284 20 1 3 System NDU hardware requirements System NDU is supported on any IBM N series storage controller or gateway hardware platform that supports the HA pair controller configuration Both storage controllers must be identical platforms Systems must be cabled and configured in an ...

Page 305: ...150 N A N5300 150 150 500 150 500 N6040 150 150 500 150 500 N6060 200 300 500 200 500 N5600 250 300 500 300 500 N6070 250 300 500 300 500 N6210 N A 300 500 300 500 N6240 N A 300 500 300 500 N6270 N A 300 500 300 500 N7600 250 300 500 300 500 N7700 250 300 500 300 500 N7800 250 300 500 300 500 N7900 250 300 500 300 500 N7550 N A N A 500 300 500 N7750 N A N A 500 300 500 N7950 N A N A 500 300 500 Re...

Page 306: ...you are upgrading and all host specific items if your environment uses FCP or iSCSI Validating the storage controller system configurations Confirm that both storage controllers are prepared for the NDU operation Validate each individual controller s configuration to identify any issues before upgrading Create a detailed upgrade test procedure document and a back out plan Identify any inconsistenc...

Page 307: ...d at the console of each storage system controller sysstat c 10 x 3 Avoid having the values in the CPU and Disk Util columns above 50 for all 10 measurements that are reported Make sure that no other load is added to the storage system until the upgrade completes Synchronizing date and time Make sure that the date and time are synchronized between the two controllers Although synchronized time is ...

Page 308: ...d kernel image The cf takeover n command also cannot be used across minor releases The cf takeover n command applies only to major version NDU It fails if attempted during a minor NDU or normal takeover The cf giveback f command can be used during system NDU Running this command might be necessary when long running operations or operations that cannot be restarted are running on behalf of the part...

Page 309: ...o so use the storage download shelf adapter_number adapter_number shelf_number command This command informs the user if the upgrade disrupts client I O and offers an option to cancel the operation Systems that use only LRC ESH ESH2 or ESH4 shelf modules in any combination are not disrupted during the upgrade process They are not disrupted regardless of whether the upgrade is performed manually or ...

Page 310: ... O then occurs on the spare nonproduction storage controller rather than on the production system This approach does not eliminate the risk of latent shelf module failure on the systems in which modules are being swapped in It also has no effect on the risk of running different shelf controller firmware even if only for a short time 20 2 4 Upgrading the AT FCX shelf firmware during system reboot T...

Page 311: ...ves that are members of RAID DP or RAID 4 aggregates are upgraded nondisruptively in the background Still upgrade all disk firmware before a Data ONTAP NDU is done 20 3 2 Upgrading the disk firmware non disruptively Nondisruptive upgrades are performed by downloading the most recent firmware from the IBM Support site to the controller s etc disk_fw directory Updates start automatically for any dis...

Page 312: ...hip the firmware upgrade must be performed separately on each node individually in sequence Therefore you must wait for the first node to complete before starting the second Disk firmware can be downloaded only when the cluster is enabled and both nodes can communicate with each other Do not perform any takeover or giveback actions until the firmware upgrade is complete Firmware download cannot be...

Page 313: ...nt capabilities such as remote platform management remote access monitoring troubleshooting and logging The RLM is operational regardless of the state of the controller and is available if the controller has input power The RLM firmware can be updated by the Data ONTAP command line interface or the RLM command line interface Both procedures are nondisruptive upgrades of the RLM firmware Perform no...

Page 314: ...294 IBM System Storage N series Hardware Guide ...

Page 315: ...ights reserved 295 Chapter 21 Hardware and software upgrades This chapter describes high level procedures for some common hardware and software upgrades This chapter includes the following sections Hardware upgrades Software upgrades 21 ...

Page 316: ... cable from the A Output on the last disk shelf in the existing loop to the A Input on the new disk shelf b Connect cable from the B Output on the last disk shelf in the existing loop to the B Input on the new disk shelf 5 The storage system automatically recognizes the hot added disk shelf To remove a disk shelf from a loop shut down both controllers and disconnect the shelf Removing a disk shelf...

Page 317: ...expansion drawer cables except the B loop connections on each head Also leave the cluster interconnect cables if any disconnected b Boot to maintenance mode c Verify that the new controller head sees only the disks on its A loops d Reassign the old controller s disks to the new controller s system ID e Delete the old local mailbox disk 3 Repeat the following process on the partner system a Shut do...

Page 318: ...ss A manual update is not necessary unless the new firmware is not compatible with the storage system disks Update the Data ONTAP kernel The latest system firmware is included with Data ONTAP update packages for CompactFlash based storage systems New disk firmware is sometimes included with Data ONTAP update packages For more information see the Data ONTAP Upgrade Guide which is available at this ...

Page 319: ... installation Before the storage controller NDU is performed complete the following steps 1 Validate the high availability controller configuration 2 Remove all failed disks to allow giveback operations to succeed 3 Upgrade the disk and shelf firmware 4 Verify that system loads are within the acceptable range The load should be less than 50 on each system Table 21 1 shows supported NDU upgrade pat...

Page 320: ...t delete any Snapshots that are made on Data ONTAP 8 0 You must initialize again all SnapVault relationships after the revert because all snapshots that are associated with Data ONTAP 8 0 are deleted SnapMirror sources must be reverted before SnapMirror destinations are reverted A revert cannot be nondisruptive so plan for system downtime Example 21 1 shows details of the revert_to command Example...

Page 321: ...the storage system This installation boots from a Data ONTAP version that is stored on a remote HTTP or Trivial File Transfer Protocol TFTP server Complete the following steps for a netboot installation 1 Upgrade BIOS if necessary as shown in the following example ifconfig e0c addr 10 10 123 mask 255 255 255 0 gw 10 10 123 1 ping 10 10 123 45 flash tftp 10 10 123 45 folder system_type flash 2 Ente...

Page 322: ... mask 255 255 255 0 gw 10 10 123 1 ping 10 10 123 45 4 Netboot from the loader prompt as shown in the following example netboot http 10 10 123 45 home bootimage kernel 5 Enter the NFS root path as shown in the following example 10 10 123 45 vol home web bootimage rootfs img The NFS root path is the IP address of an NFS server that is followed by the export path 6 Press Ctrl C to display the Boot m...

Page 323: ...AN Module Status Online To verify the existing firmware level use the version b command as shown in Example 21 5 Example 21 5 The version command n5500 ctr tic 1 version b 1 x86_elf kernel primary krn OS 7 3 7 1 backup x86_elf kernel primary krn OS 7 3 6P5 1 x86_elf diag diag krn 5 6 1 1 x86_elf firmware deux firmware img Firmware 3 1 0 1 x86_elf firmware SB_XIV firmware img BIOS NABL Firmware 3 0...

Page 324: ... Run the software get command to obtain the Data ONTAP code from an http server A simple freeware http server is sufficient for smaller environments 2 Run the software list command to verify that the code is downloaded correctly 3 Run the software install command with your selected code level 4 Run the download command 5 Run the reboot command to finalize your upgrade Upgrade the boot loader of th...

Page 325: ... post upgrade N6070A version Data ONTAP Release 8 1 7 Mode Wed Apr 25 23 47 02 PDT 2012 N6070A sysconfig Data ONTAP Release 8 1 7 Mode Wed Apr 25 23 47 02 PDT 2012 System ID 0151696979 N6070A partner ID 0151697146 N6070B System Serial Number 2858133001611 N6070A System Rev A1 System Storage Configuration Multi Path HA System ACP Connectivity NA slot 0 System Board Processors 4 Processor type Opter...

Page 326: ...306 IBM System Storage N series Hardware Guide ...

Page 327: ... Copyright IBM Corp 2012 2014 All rights reserved 307 Part 5 Appendixes Part 5 ...

Page 328: ...308 IBM System Storage N series Hardware Guide ...

Page 329: ...e N series storage system This appendix includes the following sections Preinstallation planning Start with the hardware Power on N series Updating Data ONTAP Obtaining the Data ONTAP software from the IBM NAS website Installing Data ONTAP system files Downloading Data ONTAP to the storage system Setting up the network using console Changing the IP address Setting up the DNS A ...

Page 330: ...equirements Also check the document for rack power supplies power requirements and thermal considerations 3 Use the hardware guide to install the following N series storage system Installation and Setup instructions for N series storage system GC26 7784 Hardware and Service Guide for N series storage system GC26 7785 There are separate cabling instructions for single node and Active Active configu...

Page 331: ...p process is recorded in the SNMP location information Use a description that identifies where to find your storage system for example lab 5 row 7 rack B Language used for multiprotocol storage systems Administration host Aclientcomputer that is allowed to access the storage system through a Telnet client or through the Remote Shell protocol Host name IP address Virtual interfaces The virtual netw...

Page 332: ...fault is set to no MTU size for jumbo frames Flow control none receive send full The default is set to full E0M Ethernet interface if available IP address Subnet mask Partner IP address The default is set to no for most installations Flow control none receive send full The default is set to full Router if used Gateway name IP address Would you like to continue setup through Web interface You do th...

Page 333: ...n a local file the generic user accounts can be stored in the NIS or LDAP databases If generic accounts are stored in the local passwd file mapping of a Windows user to a generic UNIX user and mapping of a generic UNIX user to a Windows user work better than when generic accounts are stored in NIS or LDAP NIS group caching NIS group caching is used when access is requested to data with UNIX securi...

Page 334: ...es from a remote console by using a web browser or command line after initial setup otherwise use a serial port The ASCII terminal console enables you to monitor the boot process configure your N series system after it boots and perform system administration User authentication style Active Directory authentication Active Directory domains only Windows NT 4 authentication Windows NT or Active Dire...

Page 335: ... Sequentially power on the N series systems by completing the following steps a Turn on the power to the expansion units only Ensure that you turn them on within 5 minutes of each other b Turn on the N series storage systems 2 Initialize Data ONTAP This step provides information if you want to format all disks on a filer and reinstall Data ONTAP This step can also be used to troubleshoot when a ne...

Page 336: ...ssage Press Ctrl C for special menu is displayed press Ctrl C to access the special boot menu as shown in Example A 2 Example A 2 Boot menu LOADER boot_ontap Loading 0x200000 33342524 0x21cc43c 31409732 0x3fc0a80 2557763 0x42311c3 5 Entry at 0x00200000 Starting program at 0x00200000 cpuid 0x80000000 0x80000004 0x0 0x0 0x0 Press CTRL C for special boot menu Special boot options menu will be availab...

Page 337: ...00974325E5 disk zeroing complete Wed May 2 03 32 01 GMT raid disk zero done notice Disk 0c 00 8 Shelf Bay NETAPP X286_S15K5146A15 NQ06 S N 3LN1322S0000974208ZC disk zeroing complete Wed May 2 03 32 02 GMT raid disk zero done notice Disk 0c 00 1 Shelf Bay NETAPP X286_S15K5146A15 NQ06 S N 3LN11G4G00009742TXB2 disk zeroing complete Wed May 2 03 32 02 GMT raid disk zero done notice Disk 0c 00 9 Shelf ...

Page 338: ... the web interface n Please enter the name or IP address of the default gateway 9 11 218 1 The administration host is given root access to the filer s etc files for system administration To allow etc root access to all NFS clients enter RETURN below Please enter the name or IP address of the administration host Where is the filer located Tucson Do you want to run DNS resolver n Do you want to run ...

Page 339: ...ing Warning clustering is not licensed yet an interconnect adapter was found NVRAM will be divided into two parts until adapter is removed Thu May 3 05 33 15 GMT cf fm unexpectedPartner warning Warning clustering is not licensed yet the node once had a cluster partner Thu May 3 05 33 16 GMT mgr boot reason_ok notice System rebooted Thu May 3 05 33 16 GMT asup config minimal unavailable warning Min...

Page 340: ...vailable at this website http www ibm com support docview wss uid ssg1S7001558 The following methods are used to upgrade storage systems in an Active Active configuration Nondisruptive The nondisruptive update method is appropriate when you must maintain service availability during system updates When you halt one node and allow takeover the partner node continues to serve data for the halted node...

Page 341: ... the system visible via WINS n A filer can be configured for multiprotocol access or as an NTFS only filer Since NFS DAFS VLD FCP and iSCSI are not licensed on this filer we recommend that you configure this filer as an NTFS only filer 1 NTFS only filer 2 Multiprotocol filer Selection 1 2 1 1 CIFS requires local etc passwd and etc group files and default files will be created The default passwd fi...

Page 342: ...e A 10 Share CIFS access n3300a cifs access C root Full Control 1 share s have been successfully modified n3300a cifs shares Mount Point Description ETC etc Remote Administration priv access only HOME vol vol0 home Default Share everyone Full Control C Remote Administration root Full Control 2 Complete the following steps to map the system storage to a drive You must log in as administrator or log...

Page 343: ...tting started 323 b Enter the network mapping address as shown in Figure A 2 Figure A 2 Mapping address c Enter a user name and password to access the storage system as shown in Figure A 3 Figure A 3 Storage access ...

Page 344: ...le 3 Complete the following steps to run the Data ONTAP installer a Go to the drive to which you previously downloaded the software see Obtaining the Data ONTAP software from the IBM NAS website on page 320 b Double click the files that you downloaded A dialog box opens as shown in Figure A 5 Figure A 5 Winzip self extractor ...

Page 345: ...as shown in Figure A 6 Y ETC Figure A 6 Extract path d Ensure that the following options were selected Overwrite files without prompting When done unzipping open Leave the options as they are e Click Unzip A window displays the confirmation messages as files are extracted as shown in Figure A 7 Figure A 7 Extraction finished f Run the script installer as shown in Figure A 8 Figure A 8 Script insta...

Page 346: ...m complete the following steps 1 Install Data ONTAP Run the download command to copy the kernel and firmware data files to the CompactFlash card The download command provides a status message that is similar to Example A 11 Example A 11 Download process n3300a download download You can cancel this operation by hitting Ctrl C in the next 6 seconds download Depending on system load it may take many ...

Page 347: ...Main Memory Used CMOS RAM Status OK Controller B To display the firmware version on the CompactFlash run the following command as shown in Example A 13 version b Example A 13 version b n3300a version b 1 x86_elf kernel primary krn OS 7 2 5 1 1 backup x86_elf kernel primary krn OS 7 2 4L1 1 x86_elf diag diag krn 5 3 1 x86_elf firmware deux firmware img Firmware 3 1 0 1 x86_elf firmware SB_XIV firmw...

Page 348: ...r firmware by using the update_flash command 6 At the firmware environment boot prompt enter bye to reboot the system The reboot uses the new software and if applicable the new firmware as shown in Example A 15 Example A 15 Rebooting the system LOADER bye AMI BIOS8 Modular BIOS Copyright C 1985 2006 American Megatrends Inc All Rights Reserved Portions Copyright C 2006 Network Appliance Inc All Rig...

Page 349: ...dress netmask mask Example A 17 Changing network IP n3300a ifconfig e0a 9 11 218 147 netmask 255 255 255 0 n3300a netstat in Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Collis Queue e0a 1500 9 11 218 24 9 11 218 147 33k 0 13k 0 0 0 e0b 1500 none none 0 0 0 0 0 0 lo 8160 127 127 0 0 1 52 0 52 0 0 0 3 If you want this IP address to be persistent after the N series is rebooted update the etc hos...

Page 350: ...lv conf file Then add or update these entries to the following add name server as shown in Figure A 12 nameserver ip_address Figure A 12 Name server 2 Update or confirm the DNS domain name with the following commands To display the current DNS domain name options dns domainname To update the DNS domain name as shown in Example A 18 on page 331 run the following command options dns domainname domai...

Page 351: ...Sun May 6 03 50 06 GMT n3300a reg options overrideRc warning Setting option dns enable to on conflicts with etc rc that sets it to off Option dns enable is being set to on but this conflicts with a line in etc rc that sets it to off Options are automatically persistent but the line in etc rc will override this persistence so if you want to make this change persistent you will need to change or rem...

Page 352: ...BM System Storage N series Hardware Guide 4 To make this change persistent after filer reboot update the etc rc file to ensure that the name server exists as shown in Figure A 13 Figure A 13 The etc rc file ...

Page 353: ... appendix provides information about the Physical environment and operational environment specifications of N series controller and disk shelves This appendix includes the following sections N3000 entry level systems N6000 mid range systems N7000 high end systems N series expansion shelves B ...

Page 354: ... 68 77 degrees F Non operating 40 70 degrees C 40 158 degrees F Relative humidity Maximum operating range 20 80 non condensing Recommended operating range 40 55 Non operating range 10 95 non condensing Maximum wet bulb 28 degrees C Maximum altitude 3050 m 10 000 ft Wet bulb caloric value 853 Btu hr Maximum electrical power 100 240 V ac 10 4 A per node 47 63 Hz Nominal electrical power 100 120 V ac...

Page 355: ...electrical power 100 240 V ac 8 3 A per node 50 60 Hz Nominal electrical power 100 120 V ac 16 A 200 240 V ac 6 A 50 60 Hz Noise level 66 dBa 1 m 23 degrees C 7 2 bels 1 m 23 degrees C N3240 The IBM System Storage N3240 Model A14 A24 features the following physical specifications Width 44 9 cm 17 7 in Depth 65 7 cm 25 8 in with cable management arms 65 4 cm 25 7 in without cable management arms He...

Page 356: ...eature the following physical specifications Width 44 7 cm 17 6 in Depth 71 3 cm 28 1 in with cable management arms 65 5 cm 25 8 in without cable management arms Height 13 cm 5 12 in times 2 for E21 The IBM System Storage N6240 Models C10 C20 C21 E11 and E21 feature the following operating environment specifications Temperature Maximum range 10 40 degrees C 50 104 degrees F Recommended 20 25 degre...

Page 357: ...e N6240 Models C10 C20 C21 E11 and E21 feature the following operating environment specifications Temperature Maximum range 10 40 degrees C 50 104 degrees F Recommended 20 25 degrees C 68 77 degrees F Non operating 40 70 degrees C 40 158 degrees F Relative humidity Maximum operating range 20 80 non condensing Recommended operating range 40 55 Non operating range 10 95 non condensing Maximum wet bu...

Page 358: ... 80 non condensing Recommended operating range 40 55 Non operating range 10 95 non condensing Maximum wet bulb 28 degrees C Maximum altitude 3050 m 10 000 ft Wet bulb caloric value 1847 Btu hr Maximum electrical power 100 240 V ac 12 8 A per node 50 60 Hz Nominal electrical power 100 120 V ac 4 7 A 200 240 V ac 2 3 A 50 60 Hz Noise level 55 5 dBa 1 m 23 degrees C 7 5 bels 1 m 23 degrees C N7000 hi...

Page 359: ...2270 Btu hr Maximum electrical power 100 240 V ac 12 7 8 A per node 50 60 Hz Nominal electrical power 100 120 V ac 6 9 A 200 240 V ac 3 5 A 50 60 Hz Noise level 66 dBa 1 m 23 degrees C 8 1 bels 1 m 23 degrees C N series expansion shelves This section lists the N series expansion shelves specifications EXN1000 Because the EXN1000 was withdrawn from the market and is no longer sold it is not covered...

Page 360: ...AC 4 4 A 200 240VAC 2 1 A 50 60 Hz SATA drives Noise level 5 7 bels 1 m 23 degrees C SATA drives idle 6 0 bels 1 m 23 degrees C SAS drives idle 6 7 bels 1 m 23 degrees C SATA drives operating 7 0 bels 1 m 23 degrees C SAS drives operating EXN3500 The EXN3500 SAS expansion unit features the following physical specifications Width 447 2 mm 17 6 in Depth 542 6 mm 21 4 in Height 85 3 mm 3 4 in Weight ...

Page 361: ...specifications Width 447 mm 17 6 in Depth 508 mm 20 0 in Height 133 mm 2 25 in Weight 35 8 kg 78 8 lb The EXN4000 FC expansion unit features the following operating environment specifications Temperature Maximum range 10 40 C 50 104 F Recommended 20 25 C 68 77 F Non operating 40 65 C 40 149 F Relative humidity 10 90 non condensing Wet bulb caloric value 1 215 Btu hr fully loaded shelf Electrical p...

Page 362: ...342 IBM System Storage N series Hardware Guide ...

Page 363: ...ith IBM System Storage N series Operation Manager SG24 7734 Using an IBM System Storage N series with VMware to Facilitate Storage and Server Consolidation REDP 4211 Using the IBM System Storage N series with IBM Tivoli Storage Manager SG24 7243 IBM System Storage N series with VMware vSphere 5 SG24 8110 00 IBM System Storage N series and VMware vSphere Storage Best Practices SG24 7871 IBM System ...

Page 364: ...TM Cross Reference http www 304 ibm com support docview wss uid ssg1S7001844 IBM N Series to NetApp Machine type comparison table http www ibm com support techdocs atsmastr nsf WebIndex TD105042 Interoperability matrix http www 304 ibm com support docview wss uid ssg1S7003897 Online resources The following websites are also relevant as further information sources IBM NAS support http www ibm com s...

Page 365: ...5 spine 0 475 0 873 250 459 pages IBM System Storage N series Hardware Guide IBM System Storage N series Hardware Guide IBM System Storage N series Hardware Guide IBM System Storage N series Hardware Guide ...

Page 366: ...IBM System Storage N series Hardware Guide IBM System Storage N series Hardware Guide ...

Page 367: ......

Page 368: ...to high end enterprise storage and data management capabilities with midrange affordability Built in serviceability and manageability features help support your efforts to increase reliability simplify and unify storage infrastructure and maintenance and deliver exceptional economy The IBM System Storage N series systems provide a range of reliable scalable storage solutions to meet various storag...

Reviews:

No comments

Related manuals for N Series

Eaton SmartWire-DT EU5C-SWD-DP Manual preview
SmartWire-DT EU5C-SWD-DP
Brand: Eaton Pages: 88
B+B KNX-GSM Gateway Documentation preview
KNX-GSM Gateway
Brand: B+B Pages: 19
DAUDIN iO-GridM GFGW-RM02N User Manual preview
iO-GridM GFGW-RM02N
Brand: DAUDIN Pages: 28
The Things Network TTN Gateway v1 - EU Assembly Instructions Manual preview
TTN Gateway v1 - EU
Brand: The Things Network Pages: 20
David Clark U9922-G38 User Manual preview
U9922-G38
Brand: David Clark Pages: 11
OBSIDIAN CONTROL SYSTEMS NETRON EP2 Installation Manual preview
NETRON EP2
Brand: OBSIDIAN CONTROL SYSTEMS Pages: 19
Bacharach BACnet Protocol Installation Manual preview
BACnet Protocol
Brand: Bacharach Pages: 12
VOGO VOKKERO ELITE User Manual preview
VOKKERO ELITE
Brand: VOGO Pages: 24
TENETICS, LLC Ceres Manual preview
Ceres
Brand: TENETICS, LLC Pages: 12
RTA 515ENI-N34 Product User Manual preview
515ENI-N34
Brand: RTA Pages: 48
RTA 460PSSC-NNA4 Product User Manual preview
460PSSC-NNA4
Brand: RTA Pages: 78
RTA 460MS-NNA1 Product User Manual preview
460MS-NNA1
Brand: RTA Pages: 58
RTA 460MCBM-NNA4 Product User Manual preview
460MCBM-NNA4
Brand: RTA Pages: 68
RTA 460ETCMM-NNA4 Product User Manual preview
460ETCMM-NNA4
Brand: RTA Pages: 78
RTA 460ESMC Product User Manual preview
460ESMC
Brand: RTA Pages: 75
RTA 460ESDFM-NNA1 Product User Manual preview
460ESDFM-NNA1
Brand: RTA Pages: 74
RTA 460BSDFM-NNA1 Product User Manual preview
460BSDFM-NNA1
Brand: RTA Pages: 72
RTA 460BMSC Product User Manual preview
460BMSC
Brand: RTA Pages: 66

Brands by name

Popular brands

Load more brands