manualshive.com logo in svg

Cisco 9124 - mds multilayer fabric switch, Troubleshooting Manual

Share
Download
Cisco 9124 - mds multilayer fabric switch Troubleshooting Manual Download Page 1

S e n d   d o c u m e n t a t i o n   c o m m e n t s   t o   m d s f e e d b a c k - d o c @ c i s c o . c o m

Americas Headquarters

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706 
USA
http://www.cisco.com
Tel: 408 

526-4000

800 553-NETS (6387)

Fax: 408 

527-0883

Cisco MDS 9000 Family 

 

Troubleshooting Guide, Release 3.x

Cisco MDS SAN-OS for Release 3.3(1) 

May 2008

Text Part Number: OL-9285-05

Summary of Contents for 9124 - mds multilayer fabric switch

Page 1: ... o m Americas Headquarters Cisco Systems Inc 170 West Tasman Drive San Jose CA 95134 1706 USA http www cisco com Tel 408 526 4000 800 553 NETS 6387 Fax 408 527 0883 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x Cisco MDS SAN OS for Release 3 3 1 May 2008 Text Part Number OL 9285 05 ...

Page 2: ...EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES CCDE CCENT Cisco Eos Cisco HealthPresence the Cisco logo Cisco Lumin Cisco Nexus Cisco StadiumVision Cisco TelePresence Cisco WebEx DCE and Welcome to the Human Network are trademarks Changing the Way We Work Live Play and Learn and Cisco Store are service marks and Access Registrar Aironet AsyncOS Bringing the Mee...

Page 3: ...ference i xxxi Installation and Configuration Note i xxxi Obtaining Documentation Obtaining Support and Security Guidelines i xxxii C H A P T E R 1 Troubleshooting Overview 1 1 Overview of the Troubleshooting Process 1 1 Best Practices 1 2 Troubleshooting Basics 1 2 General Steps 1 2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1 3 Common Fabric Manager Tools 1 3 Common...

Page 4: ...Installations 2 6 Troubleshooting Cisco SAN OS Software Upgrades and Downgrades 2 7 Software Installation Reports an Incompatibility 2 7 Diagnosing Compatibility Issues 2 7 Software Installation Ends with Error 2 9 Installing SAN OS Software Using Fabric Manager 2 10 Installing Cisco SAN OS Software from the CLI 2 11 Troubleshooting Cisco SAN OS Software System Reboots 2 13 Power On or Switch Rebo...

Page 5: ...ls in Fabric Manager 3 3 Common Troubleshooting Commands in the CLI 3 3 SSM Issues 3 3 SSM Fails to Boot 3 4 Upgrading the SSI Image 3 4 Verifying the SSI Boot Image 3 5 Using the install ssi Command 3 6 Recovering a Replacement SSM 3 8 SSM Upgrade Is Disruptive 3 9 Installing EPLD Images on Modules 3 9 C H A P T E R 4 Troubleshooting Hardware 4 1 Overview 4 1 SNMP Traps 4 2 Troubleshooting Startu...

Page 6: ...0 Troubleshooting Switching and Services Modules 4 21 Overview of Module Status 4 21 Module Initialization Overview 4 22 Module Bootup 4 23 Image Download 4 23 Runtime Diagnostics 4 24 Runtime Configuration 4 24 Online and Operational 4 24 Analyzing The Logs 4 25 Troubleshooting Module Issues 4 25 Troubleshooting Powered Down Modules 4 26 Diagnosing a Powered Down Module 4 28 Troubleshooting Reloa...

Page 7: ...bleshooting Licensing 6 1 License Overview 6 1 Chassis Serial Numbers 6 2 Grace Period 6 2 Initial Troubleshooting Checklist 6 3 Displaying License Information Using Fabric Manager 6 3 Displaying License Information Using Device Manager 6 4 Displaying License Information Using Fabric Manager Web Client 6 4 Displaying License Information Using the CLI 6 4 Licensing Installation Issues 6 6 One Click...

Page 8: ...tribution Failure 7 10 Regions for Conditional Service 7 11 Changing Regions 7 11 C H A P T E R 8 Troubleshooting Ports 8 1 Overview 8 1 Initial Troubleshooting Checklist 8 1 Limitations and Restrictions 8 4 Overview of the FC MAC Driver and the Port Manager 8 4 Port Manager Overview 8 5 Troubleshooting Port States with the Device Manager 8 6 Device View 8 6 Device Manager Summary View 8 6 Device ...

Page 9: ...9 Troubleshooting N Port Virtualization 9 1 Overview 9 1 Initial Troubleshooting Checklist 9 2 Limitations and Restrictions 9 2 Common CLI Commands for NPV 9 2 Common Problems with NPV 9 4 Moving the Login of an End Device 9 4 NPIV Is Not Enabled 9 5 VSAN Mismatches 9 5 Core NPV Device Is Not a Switch 9 6 NPV Core Switch Port Is Down 9 6 Server Interface is Down 9 6 Waiting on FLOGI from the Serve...

Page 10: ...Isolated ISL Using Fabric Manager 11 7 Resolving an Isolated ISL Using the CLI 11 7 Resolving Fabric Timer Issues Using Fabric Manager 11 9 Resolving Fabric Timer Issues Using the CLI 11 9 Troubleshooting Interop Mode Issues 11 9 Dynamic Port VSAN Membership Issues 11 9 Troubleshooting DPVM Using Fabric Manager 11 10 Troubleshooting DPVM Using the CLI 11 11 DPVM Configuration Not Available 11 11 D...

Page 11: ...ng Device Manager 11 30 Resolving a Mismatched Retransmit Interval on an ISL Using the CLI 11 30 Resolving a Mismatch in Dead Intervals on an ISL Using Fabric Manager 11 31 Resolving a Mismatch in Dead Intervals on an ISL Using the CLI 11 31 Resolving a Region Mismatch Using Fabric Manager 11 32 Resolving a Region Mismatch Using the CLI 11 32 C H A P T E R 12 Troubleshooting SAN Device Virtualizat...

Page 12: ...nding Action Pending Commits 13 17 Error Fabric Is Changing Please Retry the Request Later 13 17 C H A P T E R 14 Troubleshooting Zones and Zone Sets 14 1 Overview 14 1 Troubleshooting Checklist 14 1 Troubleshooting Zone Configuration Issues with Fabric Manager 14 2 Troubleshooting Zone Configuration Issues with the CLI 14 2 Zone and Zone Set Issues 14 4 Host Cannot Communicate with Storage 14 5 R...

Page 13: ...Lock Issues with Fabric Manager 14 23 Resolving Enhanced Zoning Lock Issues with the CLI 14 23 C H A P T E R 15 Troubleshooting Distributed Device Alias Services 15 1 Overview 15 1 Initial Troubleshooting Checklist 15 1 Merge Failure Messages 15 2 Merge Validation Failure Messages 15 2 Commit Failure Messages 15 3 Verifying Device Alias Database Status Using the CLI 15 3 Limitations and Restrictio...

Page 14: ...2 C H A P T E R 17 Troubleshooting RADIUS and TACACS 17 1 AAA Overview 17 1 Initial Troubleshooting Checklist 17 1 Common Troubleshooting Tools in Fabric Manager 17 2 Common Troubleshooting Commands in the CLI 17 2 AAA Issues 17 2 Switch Does Not Communicate with AAA Server 17 2 Verifying RADIUS Configuration Using Fabric Manager 17 4 Verifying RADIUS Configuration Using the CLI 17 4 Verifying TAC...

Page 15: ...fying User Login with System Messages Using the CLI 18 6 User Cannot Create Roles 18 7 User Cannot Create Other Users With Fabric Manager or Device Manager 18 7 User Cannot Access Certain Features 18 8 Verifying Roles Using Device Manager 18 8 Verifying Roles Using the CLI 18 9 User Has Too Much Access 18 10 User Cannot Configure Some VSANs 18 10 Verifying VSAN Restricted Roles Using Fabric Manage...

Page 16: ...o Fabric 19 12 Disabling Autolearn Using Fabric Manager 19 13 Disabling Autolearn Using the CLI 19 13 Port Security Settings Lost After Reboot 19 13 Merge Fails 19 14 Configuring Port Security with Autolearn Using Fabric Manager 19 14 Configuring Port Security with Autolearn Using the CLI 19 15 Fabric Binding Issues 19 15 Switch Cannot Attach to the Fabric 19 16 Verifying Fabric Binding Violations...

Page 17: ...of the Profiles Using the CLI 20 15 Verifying the Establishment of the FCIP Tunnel Using the CLI 20 15 Verifying the Establishment of Default TCP Connections for Each Configured FCIP Tunnel Using the CLI 20 17 Verifying the Statistics of the ASIC Chip on Each Gigabit Ethernet Port Using the CLI 20 17 Ethereal Screen Captures of the TCP Connection and FCIP Tunnels 20 18 One to Three FCIP Tunnel Cre...

Page 18: ...n 20 41 Performing Basic Dynamic iSCSI Troubleshooting 20 41 Useful Show Commands to Debug Dynamic iSCSI Configuration 20 42 Virtual Target Access Control 20 43 Useful Show Commands to Debug Static iSCSI Configuration 20 43 iSCSI TCP Performance Issues 20 48 CLI Commands Used to Access Performance Data 20 49 Understanding TCP Parameters for iSCSI 20 49 Lab Setup 20 50 Configuring from the Bottom S...

Page 19: ...ort Information 21 2 ICMP Information 21 3 ToS Information 21 3 Initial Troubleshooting Checklist 21 4 Common Troubleshooting Tools in Fabric Manager 21 4 Common Troubleshooting Commands in the CLI 21 4 IP ACL Issues 21 4 All Packets Are Blocked 21 5 Re creating IP ACLs Using Fabric Manager 21 5 Re creating IP ACLs Using the CLI 21 6 No Packets Are Blocked 21 7 PortChannel Not Working with ACL 21 ...

Page 20: ...ss 22 15 C H A P T E R 23 Troubleshooting SANTap 23 1 Overview 23 1 Definitions 23 2 Limitations 23 2 Interface Restrictions 23 3 Initial Troubleshooting Checklist 23 3 Common Troubleshooting Tools in Fabric Manager 23 4 Common Troubleshooting Commands in the CLI 23 4 Messages Logs and Databases 23 6 SANTap Issues 23 6 Host Login Problems 23 7 ITL Problems 23 7 Common Mismatch Problems 23 7 C H A ...

Page 21: ...eature 25 2 Initial Troubleshooting Checklist 25 3 Common Troubleshooting Tools in Fabric Manager 25 3 Common Troubleshooting Tools in Device Manager 25 3 Common Troubleshooting Commands in the CLI 25 3 Call Home Issues 25 4 Not Receiving Call Home Alerts 25 4 Configuring an Alert Group Using Fabric Manager 25 4 Configuring an Alert Group Using the CLI 25 5 Configuring the Message Level for a Dest...

Page 22: ...Interface for Fabric Manager Server 26 5 Specifying an Interface for Fabric Manager Client or Device Manager 26 6 Configuring a Proxy Server 26 6 Clearing Topology Maps 26 6 Using Fabric Manager in a Mixed Software Environment 26 7 Troubleshooting Fabric Manager Web Client 26 7 Cannot Download Fabric Manager Web Client 26 7 Allowing network open Users to Download Fabric Manager Web Client 26 8 Cle...

Page 23: ...ng B 5 Using FC Traceroute B 5 Monitoring Processes and CPUs B 7 Viewing Running Processes on Device Manager B 7 Using the show processes CLI Command B 8 Viewing CPU Time In Device Manager B 9 Using the show processes cpu CLI Command B 9 Using the show system resource CLI Command B 10 Using On Board Failure Logging B 10 Configuring OBFL for the Switch B 11 Configuring OBFL for a Module B 12 Displa...

Page 24: ...SH B 22 Using Fibre Channel SPAN B 23 Using Cisco Network Management Products B 24 Cisco MDS 9000 Family Port Analyzer Adapter B 24 Cisco Fabric Analyzer B 25 IP Network Simulator B 27 Using Other Troubleshooting Products B 27 Fibre Channel Testers B 28 Fibre Channel Protocol Analyzers B 28 Using Host Diagnostic Tools B 28 A P P E N D I X C Configuration Limits for Cisco MDS SAN OS Release 3 x C 1...

Page 25: ...fer to the Cisco MDS 9000 Family Release Notes available at the following Cisco Systems website http www cisco com en US products hw ps4159 ps4358 prod_release_notes_list html Table 1 summarizes the new and changed features for the Cisco MDS 9000 Family Troubleshooting Guide Release 3 x and tells you where they are documented The table includes a brief description of each new feature and the relea...

Page 26: ... 2 Chapter 12 Troubleshooting SAN Device Virtualization SNMP test traps Added SNMP test trap information 3 1 2 Chapter 4 Troubleshooting Hardware Digital Certificates Added troubleshooting options for digital certificates 3 0 1 Chapter 24 Troubleshooting Digital Certificates iSCSI Load Balancing Added troubleshooting options for iSCSI load balancing iSLB 3 0 1 Chapter 20 Troubleshooting IP Storage...

Page 27: ... when installing upgrading or rebooting Cisco MDS 9000 Family hardware Chapter 3 Managing Storage Services Modules Describes how to identify and resolve problems that might occur when installing replacing or upgrading storage services modules SSMs Chapter 4 Troubleshooting Hardware Describes how to identify and resolve problems that might occur when replacing modules fans chassis power supplies or...

Page 28: ... to troubleshoot FICON Chapter 17 Troubleshooting RADIUS and TACACS Describes procedures to troubleshoot RADIUS and TACACS Chapter 18 Troubleshooting Users and Roles Describes procedures to troubleshoot role based access control Chapter 19 Troubleshooting FC SP Port Security and Fabric Binding Describes procedures to troubleshoot FC SP port security and fabric binding Chapter 20 Troubleshooting IP...

Page 29: ...isco MDS SAN OS Release 3 x Lists configuration limits for Cisco MDS SAN OS features Chapter Title Description Convention Description boldface font Commands and keywords are in boldface italic font Arguments for which you supply values are in italics Elements in square brackets are optional x y z Optional alternative keywords are grouped in brackets and separated by vertical bars string A nonquote...

Page 30: ...s Interface Images Cisco MDS 9000 Family Release Notes for Cisco MDS SVC Releases Cisco MDS 9000 Family Release Notes for Cisco MDS 9000 EPLD Images Compatibility Information Cisco MDS 9000 SAN OS Hardware and Software Compatibility Information Cisco MDS 9000 Family Interoperability Support Matrix Cisco MDS 9000 Storage Services Module Interoperability Support Matrix Cisco MDS SAN OS Release Compa...

Page 31: ... MDS 9000 Family CLI Quick Configuration Guide Cisco MDS 9000 Family CLI Configuration Guide Cisco MDS 9000 Family Command Reference Cisco MDS 9000 Family Quick Command Reference Cisco MDS 9020 Fabric Switch Configuration Guide and Command Reference Cisco MDS 9000 Family SAN Volume Controller Configuration Guide Troubleshooting and Reference Cisco MDS 9000 Family Troubleshooting Guide Cisco MDS 90...

Page 32: ...uidelines Obtaining Documentation Obtaining Support and Security Guidelines For information on obtaining documentation obtaining support providing documentation feedback security guidelines and also recommended aliases and general Cisco documents see the monthly What s New in Cisco Product Documentation which also lists all new and revised Cisco technical documentation at http www cisco com en US ...

Page 33: ...h Logs page 1 13 Contacting Customer Support page 1 15 Overview of the Troubleshooting Process To troubleshoot your fabric environment follow these general steps Step 1 Gather information that defines the specific symptoms Step 2 Identify all potential problems that could be causing the symptoms Step 3 Systematically eliminate each potential problem from most likely to least likely until the sympt...

Page 34: ... of the logical unit numbers LUNs on an existing subsystem then fabric specific issues such as FSPF ISLs or FCNS do not need to be investigated The fabric components can therefore be eliminated from possible causes of the problem This section contains the following topics General Steps page 1 2 Gathering Information Using Common Fabric Manager Tools and CLI Commands page 1 3 Verifying Basic Connec...

Page 35: ...ager tools and CLI commands that are commonly used to troubleshoot problems within your fabric These tools and commands are a subset of what you may use to troubleshoot your specific problem Each chapter in this guide may include additional tools and commands specific to the symptoms and possible problems covered in that chapter Common Fabric Manager Tools Use the following navigation paths in Fab...

Page 36: ... errors show zoneset active show accounting log Note Use the show running interface CLI command to view the interface configuration in Cisco SAN OS Release 3 0 1 or later The interface configuration as seen in the show running config CLI command is no longer consolidated Note To issue commands with the internal keyword you must have an account that is a member of the network admin group Verifying ...

Page 37: ...Switches Interfaces FC Physical FLOGI In the CLI use the show flogi commands Are the HBA and storage subsystem on the same VSAN In Fabric Manager choose End Devices and verify the VSAN IDs are identical From the CLI use the show vsan membership command Does any single zone contain both devices In Fabric Manager choose the Zone Edit Full Zone Database and select the active zone set in bold for the ...

Page 38: ...nitial tasks to perform while investigating port connectivity issues include Verify correct media copper or optical single mode SM or multimode MM Is the media broken or damaged Is the LED on the switch green Is the active LED on the HBA for the connected device on Basic port monitoring using Device Manager begins with the visual display in the Device View See Figure 1 1 Port display descriptions ...

Page 39: ... Manager Summary View In Device Manager selecting the Summary View expands the information available for port monitoring See Figure 1 2 The display includes the following VSAN assignment For N ports the port World Wide Name pWWN and Fibre Channel ID FC ID of the connected device For ISLs the IP address of the connected switch Speed Frames transmitted and received Percentage utilization for the CPU...

Page 40: ...e Monitoring from the pop up menu In Summary View choose one or more interfaces and click the Monitor tool The initial display shows traffic statistics for the selected interval including bytes and frames transmitted and received Additional tabs include the following Protocol View protocol related traffic and error statistics including link reset counts offline and non operational sequence errors ...

Page 41: ...s Fibre Channel switching modules the 32 port 2 Gbps Fibre Channel switching module the Cisco MDS 9120 20 port 1 2 Gbps Fibre Channel module and the Cisco MDS 9140 40 port 1 2 Gbps Fibre Channel module Primary Troubleshooting Flowchart The flowchart in Figure 1 3 shows the overall troubleshooting process Begin any troubleshooting investigation by checking one of the following four areas Physical p...

Page 42: ...ckets A decimal number for example is represented as dec PORT 3 IF_UNSUPPORTED_TRANSCEIVER Transceiver for interface chars is not supported Use this string to find the matching system message in the Cisco MDS 9000 Family System Messages Reference Each system message is followed by an explanation and recommended action The action may be as simple as No action required It may involve a fix or a reco...

Page 43: ...ric Manager choose Switches Events Syslog in the Physical pane and then click the Servers tab in the Information pane In Device Manager choose Logs Syslog Setup and click the Servers tab in the Syslog dialog box Step 2 Click the Create Row icon in Fabric Manager or Create in Device Manager to add a new syslog server Step 3 Enter the name or IP address in dotted decimal notation for example 192 168...

Page 44: ...bled 172 22 36 211 server severity notifications server facility local1 Step 2 Configure the syslog server a Modify etc syslog conf to handle local1 messages For Solaris there needs to be at least one tab between the facility severity and the action var adm MDS_logs Below is for the MDS 9000 logging local1 notice var adm MDS_logs b Create the log file touch var adm MDS_logs c Restart syslog etc in...

Page 45: ...d system messages and other logged events In Device Manager click Logs to set up and view logs In Fabric Manager select the Logs tab at the bottom of the fabric pane to view log information Learn to use Threshold Manager to alert you that critical statistics have exceeded a set threshold Viewing Logs with the CLI The following CLI commands are available to access and view logs on a switch Musky 95...

Page 46: ...gging nvram CLI command See Example 1 2 Example 1 2 Show logging nvram switch show logging nvram 2005 Sep 16 13 19 20 172 20 150 82 PLATFORM 2 PS_OK Power supply 2 ok Serial number 2005 Sep 16 13 19 20 172 20 150 82 PLATFORM 2 PS_FANOK Fan in Power supply 2 ok 2005 Sep 16 13 19 20 172 20 150 82 PLATFORM 2 FANMOD_FAN_OK Fan module 1 Front fan ok 2005 Sep 16 13 19 20 172 20 150 82 PLATFORM 2 FANMOD_...

Page 47: ...ormation ready to help your service provider assist you as quickly as possible Date you received the switch Chassis serial number located on a label on the right side of the rear panel of the chassis Type of software and release number Maintenance agreement or warranty information Brief description of the problem Brief explanation of the steps you have already taken to isolate and resolve the prob...

Page 48: ... m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 1 16 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 1 Troubleshooting Overview Contacting Customer Support ...

Page 49: ...hooting Cisco SAN OS Software Upgrades and Downgrades page 2 7 Troubleshooting Cisco SAN OS Software System Reboots page 2 13 Recovering the Administrator Password page 2 32 Miscellaneous Software Image Issues page 2 32 Overview Each Cisco MDS 9000 switch ships with an operating system Cisco SAN OS that consists of two images the kickstart image and the system image There is also a module image if...

Page 50: ...Image Version in the Device Manager to view information on images in the directories of the MDS file system Guidelines for Upgrading Not all images need to be updated during an upgrade Use the following checklist to prepare for an upgrade After you have completed the checklist you are ready to upgrade the switches in your fabric Note It is normal for the active supervisor to become the standby sup...

Page 51: ...ee the Installing Cisco SAN OS Software from the CLI section on page 2 11 Using the install all command offers the following advantages You can upgrade the entire switch using the least disruptive procedure with just one command You can receive descriptive information on the intended changes to your system before you continue with the command You have the option to cancel the command Once the effe...

Page 52: ...ions in the module reach a steady state The IPS modules require a five minute delay before the next IPS module upgrade can guarantee a stable state SSM supports nondisruptive upgrades for the Layer 1 and Layer 2 protocols under the following conditions SSM is running Cisco SAN OS Release 2 1 2 or later and upgrading to a later release The SSM hardware has the ELPD image for Release 2 1 2 installed...

Page 53: ...ul collect the details from the show tech support command output and the console output from the installation if available Troubleshooting Fabric Manager Installations This section describes possible problems and solutions for a Fabric Manager installation failure Fabric Manager requires that the appropriate version of Sun JAVA JRE be installed based on the Fabric Manager release Table 2 1 shows t...

Page 54: ...CESS Verifying image bootflash i 1 3 0 104 SUCCESS Extracting system version from image bootflash i 1 3 0 104 SUCCESS Extracting kickstart version from image bootflash b 1 3 0 104 SUCCESS Extracting loader version from image bootflash b 1 3 0 104 SUCCESS switch show install all status This is the log of last installation log of last install Verifying image bootflash b 1 3 0 104 SUCCESS Fabric Mana...

Page 55: ...patibility Issues To view the results of a dynamic compatibility check use the show incompatibility system bootflash filename CLI command Use the show incompatibility CLI command for diagnosis when the install all CLI command warns of compatibility issues During an attempted upgrade the install all CLI command may return the following warning Warning The startup config contains commands not suppor...

Page 56: ...ility is strict because continuing the upgrade might cause the switch to move into an inconsistent state that is configured features might stop working switch show incompatibility system bootflash new image The following configurations on active are incompatible with the system image 1 Feature Index 67 Capability CAP_FEATURE_SPAN_FC_TUNNEL_CFG Description SPAN Remote SPAN feature using fc tunnels ...

Page 57: ... installation See the Installing SAN OS Software Using Fabric Manager section on page 2 10 or the Installing Cisco SAN OS Software from the CLI section on page 2 11 The fabric or switch was configured while the upgrade was in progress Wait until the upgrade is complete before configuring the switch In Device Manager choose Admin CFS or from the CLI use the show cfs lock command to check that there...

Page 58: ...ace for the new images You can see this information in the Flash Space column This screen shows the active and standby if applicable bootflash memory space on each switch and shows the status whether there is enough space for the new images If any switch has insufficient space you cannot proceed Deselect the switch without enough bootflash memory by going back to the first screen and unchecking th...

Page 59: ... Always carefully read the output of install all compatibility check This compatibility check tells you exactly what needs to be upgraded BIOS loader firmware and what modules are not hitless If there are any questions or concerns about the results of the output select n to stop the installation and contact the next level of support ca 9506 install all system scp testuser dino tftpboot rel qa 2_1_...

Page 60: ...graded according to following table Module Image Running Version New Version Upg Required 1 slc 2 0 2b 2 1 1a yes 1 bios v1 1 0 10 24 03 v1 1 0 10 24 03 no 2 slc 2 0 2b 2 1 1a yes 2 bios v1 1 0 10 24 03 v1 1 0 10 24 03 no 3 ips 2 0 2b 2 1 1a yes 3 bios v1 1 0 10 24 03 v1 1 0 10 24 03 no 4 svclc 2 0 2b 2 1 1a yes 4 svcsb 1 3 5m 1 3 5m no 4 svcsb 1 3 5m 1 3 5m no 4 bios v1 1 0 10 24 03 v1 1 0 10 24 ...

Page 61: ...he following topics Power On or Switch Reboot Hangs page 2 13 Corrupted Bootflash Recovery page 2 14 Recovery Using BIOS Setup for Supervisor 1 page 2 16 Recovery from the loader Prompt on Supervisor 2 Modules page 2 19 Recovery from the loader Prompt on Supervisor 1 Modules page 2 20 Recovery from the switch boot Prompt page 2 21 Recovery for Switches with Dual Supervisor Modules page 2 22 Recogn...

Page 62: ... proceed error state you can interrupt the switch boot sequence and recover the image by entering the BIOS configuration utility described in the following section Access this utility only when needed to recover a corrupted internal disk Caution The BIOS changes explained in this section are required only to recover a corrupted bootflash The BIOS is corrupted Replace this module Contact your custo...

Page 63: ...ption Boot loader Starting kickstart loader The boot loader uncompresses loaded software to boot an image using its file name as reference These images are made available through bootflash When the memory test is over press Esc to enter the boot loader prompt Kickstart Uncompressing system switch boot When the boot loader phase is over press Ctrl 3 Control key plus right bracket key to enter the s...

Page 64: ...at you have made a backup of the configuration files before you begin this procedure To recover a corrupted bootflash device no bootable device found message for a switch with a single supervisor 1 module follow these steps Step 1 Connect to the console port of the required switch Step 2 Boot or reboot the switch Step 3 Press Ctrl C to interrupt the BIOS setup during the BIOS memory test You see t...

Page 65: ...he Tab key until you reach the Local IP Address field Step 7 Enter the local IP address for the switch and press the Tab key Step 8 Enter the subnet mask for the IP address and press the Tab key Step 9 Enter the IP address of the default gateway and press the Tab key Step 10 Enter the IP address of the TFTP server and press the Tab key Step 11 Enter the image name kickstart and press the Tab key U...

Page 66: ...and Exit from the main screen to save your changes Note These changes are saved in the CMOS Caution The switch must have IP connectivity to reboot using the newly configured values You see the following prompt switch boot Step 14 Enter the init system command at the switch boot prompt and press Enter to reformat the file system switch boot init system Note The init system command also installs a n...

Page 67: ...help command at the loader prompt to display a list of commands available at this prompt or to obtain more information about a specific command in that list To recover a corrupted kickstart image system error state for a switch with a single supervisor module follow these steps Step 1 Enter the local IP address for the switch at the loader prompt and press Enter loader net ip 172 16 1 2 Step 2 Spe...

Page 68: ...n about a specific command in that list To recover a corrupted kickstart image system error state for a switch with a single supervisor module follow these steps Step 1 Enter the local IP address and the subnet mask for the switch at the loader prompt and press Enter loader ip address 172 16 1 2 255 255 255 0 Found Intel EtherExpressPro100 82559ER at 0xe800 ROM address 0xc000 Probing Intel EtherEx...

Page 69: ... gateway switch boot config ip default gateway 209 165 200 226 b Configure the IP address of the mgmt0 interface switch boot config interface mgmt 0 switch boot config if ip address 209 165 200 227 255 255 255 0 Step 3 Issue the no shutdown command to enable the mgmt0 interface on the switch switch boot config mgmt0 no shutdown Step 4 Enter end to exit to EXEC mode switch boot config mgmt0 end Ste...

Page 70: ... in Step 9 switch login admin Password Cisco Storage Area Networking Operating System SAN OS Software TAC support http www cisco com tac Copyright c 2002 2008 Cisco Systems Inc All rights reserved The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license Some parts of this software may be covered under the GNU Public License or the...

Page 71: ...d where slot is the slot number of the supervisor module with the corrupted bootflash The supervisor module with the corrupted bootflash performs a netboot and checks the bootflash for corruption When the bootup scripts discover that the bootflash is corrupted it generates an init system command which fixes the corrupt bootflash The supervisor boots as the HA Standby Caution If your system has an ...

Page 72: ...eature does not work at this prompt and may result in undesired errors You must type the command exactly as you want the command to appear Tip Use the help command at the loader prompt to display a list of commands available at this prompt or to obtain more information about a specific command in that list Step 2 Specify the local IP address and the subnet mask for the switch loader ip address 172...

Page 73: ...le Kickstart image Step 5 Issue the init system command to repartition and format the bootflash Step 6 Perform the procedure specified in the Recovery from the switch boot Prompt section on page 2 21 Step 7 Perform the procedure specified in the Recovering One Supervisor Module With Corrupted Bootflash section on page 2 23 to recover the other supervisor module Note If you do not issue the reload ...

Page 74: ... 7 Switch or Process Resets Problem Possible Cause Solution The switch or a process on the switch resets A recoverable error occurred on the system or on a process in the system Cisco SAN OS automatically recovered from the problem See the Recoverable System Restarts section on page 2 27 and the Switch or Process Resets section on page 2 26 A nonrecoverable error occurred on the system Cisco SAN O...

Page 75: ...EMON 3 SYSTEM_MSG bind fd 4 family 2 port 123 ad dr 127 0 0 1 in_classd 0 flags 0 fails Address already in use Jan 27 04 08 42 88 LOG_DAEMON 3 SYSTEM_MSG bind fd 4 family 2 port 123 ad dr 127 1 1 1 in_classd 0 flags 1 fails Address already in use Jan 27 04 08 42 88 LOG_DAEMON 3 SYSTEM_MSG bind fd 4 family 2 port 123 ad dr 172 22 93 88 in_classd 0 flags 1 fails Address already in use Jan 27 23 18 5...

Page 76: ...e had abnormal exits and to show if there is a stack trace or core dump switch show process log Process PID Normal exit Stack trace Core Log create time ntp 919 N N N Jan 27 04 08 snsm 972 N Y N Jan 24 20 50 Step 4 Enter the following command to show detailed information about a specific process that has restarted switch show processes log pid 898 Service idehsd Description ide hotswap handler Dae...

Page 77: ...te time 5 fspf 1524 Jan 9 03 11 6 fcc 919 Jan 9 03 09 8 acltcam 285 Jan 9 03 09 8 fib 283 Jan 9 03 08 The output shows all cores that are presently available for upload from the active supervisor The module num column shows the slot number on which the core was generated In the previous example an FSPF core was generated on the active supervisor module in slot 5 An FCC core was generated on the st...

Page 78: ...AB752D 2AC5154C output abbreviated Stack 128 bytes ESP 7FFFF830 TOP 7FFFFCD0 Step 7 Enter the following command to configure the switch to use TFTP to send the core dump to a TFTP server system cores tftp servername path This command causes the switch to enable the automatic copy of core files to a TFTP server For example the following command sends the core files to the TFTP server with the IP ad...

Page 79: ...ither supervisor module is absent the reset reason codes for that supervisor module are not displayed In a Cisco MDS 9200 Series switch the last four reset reason codes for the supervisor module in slot 1 are displayed The show system reset reason module number command displays the last four reset reason codes for a specific module in a given slot If a module is absent then the reset reason codes ...

Page 80: ...d and encrypted in the running configuration Step 3 Click Device Command Line Interface to log into the switch and verify the new password Step 4 Click Admin Save Configuration to save the running configuration to the startup configuration Miscellaneous Software Image Issues This section includes software image issues reported by the relevant release notes and includes the following topics All Por...

Page 81: ...ving the module in an unusable state In some cases the module may reboot Downgrade to a Cisco SAN OS Release 2 0 x version supported by your OSM Upgrade to Cisco SAN OS Release 2 1 2 or 2 1 1b Resetting the module will clear the problem but the problem could reoccur unless you are using a SAN OS version with the bug fix Table 2 10 Switch Reboot after FCIP Reload Symptom Possible Cause Solution Swi...

Page 82: ...ou cannot create modify or delete the admin role Create the admin role before upgrading to Cisco SAN OS Release 2 0 Table 2 13 FC IDs Change After a Link Reset Symptom Possible Cause Solution FC IDs change after a link resets Following an upgrade from Cisco SAN OS Release 1 1 to Cisco SAN OS Release 1 3 or later with persistent FC ID enabled the FC IDs for the storage arrays might change after a l...

Page 83: ... Issues page 3 3 SSM Overview The 32 port Fibre Channel Storage Services Module SSM for the Cisco MDS 9000 Family supports up to 32 Fibre Channel ports and provides distributed intelligent storage services Note Cisco MDS 9500 Series switches running Cisco MDS SAN OS Release 2 0 2b or later support the SSM module The SSI image for the SSM is downloaded from the supervisor module The image for an SS...

Page 84: ...g Fibre Channel switching Fibre Channel switching Intelligent Storage Services Intelligent Storage Services Intelligent Storage Services VSFN VSFN Nondisruptive upgrade for Fibre Channel switching traffic1 1 Requires EPLD version 2 1 2 see Installing EPLD Images on Modules section on page 3 9 and SSI boot image version 2 1 2 When you upgrade or downgrade the SSI boot image on an SSM you might disr...

Page 85: ...ooting PortChannel and trunking issues show boot variables show version install ssi show version module number epld show version epld show ssm provisioning SSM Issues This section describes troubleshooting issues for the SSM and SSI images and it includes the following topics SSM Fails to Boot page 3 4 SSM Upgrade Is Disruptive page 3 9 Checklist Check off Verify that the SSI boot variable is set ...

Page 86: ...annel switching and Intelligent Storage Services Note A newly installed SSM initially operates in Fibre Channel switching mode by default Note If you downgrade to a Cisco MDS SAN OS release that does not support the SSM you must power down the module The boot variables for the SSM are lost Table 3 3 SSM Fails to Boot Symptom Possible Cause Solution SSM fails to boot SSI boot variable is not set Se...

Page 87: ... m9000 ek9 ssi mz 2 1 2 bin in bootflash or slot0 on the active supervisor module Refer to the Cisco MDS SAN OS Release Compatibility Matrix for Storage Service Interface Images at the following URL for more information http www cisco com application pdf en us guest products ps5989 c1683 ccmigration_09186a008021 2dd0 pdf switch dir modflash 4 1 4004128 Sep 26 13 43 02 2005 m9000 ek9 ssi mz 2 1 2 b...

Page 88: ...e shows how to display the available memory for the modflash for the SSM in slot 4 switch dir bootflash 40295206 Aug 05 15 23 51 1980 ilc1 bin 12456448 Jul 30 23 05 28 1980 kickstart image1 12288 Jun 23 14 58 44 1980 lost found 27602159 Jul 30 23 05 16 1980 system image1 12447232 Aug 05 15 08 30 1980 kickstart image2 28364853 Aug 05 15 11 57 1980 system image2 4004128 Sep 26 13 43 02 2005 m9000 ek...

Page 89: ...the SSI boot image that is on the switch by following the procedure described in the Verifying the SSI Boot Image section on page 3 5 Step 4 If the SSM boots then install the SSI image on the SSM switch install ssi modflash 4 1 m9000 ek9 ssi mz 2 1 2 bin module 4 Note If the SSM is configured for Layer 3 Fibre Channel switching or Intelligent Storage Services a warning displays at the command prom...

Page 90: ... Issue the show module command to verify the status of the SSM switch show module Mod Ports Module Type Model Status 4 32 Storage Services Module DS X9032 SSM ok 5 0 Supervisor Fabric 1 DS X9530 SF1 K9 active 6 0 Supervisor Fabric 1 DS X9530 SF1 K9 ha standby Mod Sw Hw World Wide Name s WWN 4 2 1 2 0 30 20 c1 00 05 30 00 06 de to 20 e0 00 05 30 00 06 de 5 2 1 2 4 0 6 2 1 2 4 0 Mod Application Imag...

Page 91: ...sruptive Symptom SSM upgrade disruptive Installing EPLD Images on Modules Tip Refer to the Cisco MDS SAN OS Release Notes for Cisco MDS 9000 EPLD Images to verify whether or not the EPLD has changed for the Cisco SAN OS image version being used Caution Do not insert or remove any modules while an EPLD upgrade or downgrade is in progress Table 3 4 SSM Upgrade Disruptive Symptom Possible Cause Solut...

Page 92: ... slot0 command to verify that the EPLD software image file corresponding to your Cisco MDS SAN OS release is present on the active supervisor module For example if your switch is running Cisco MDS SAN OS Release 2 1 2 you must have m9000 epld 2 1 2 img in bootflash or slot0 on the active supervisor module switch dir bootflash 12288 Jan 01 00 01 07 1980 lost found 2337571 May 31 13 43 02 2005 m9000...

Page 93: ...standby exit switch The following example shows how to display the available memory for the slot0 devices on the active and standby supervisor modules switch dir slot0 12288 Jan 01 00 01 06 1980 lost found 14765056 Mar 21 15 35 06 2005 m9500 sf1ek9 kickstart mz 2 1 1 bin 15944704 Apr 06 16 46 04 2005 m9500 sf1ek9 kickstart mz 2 1 1a bin 48063243 Mar 21 15 34 46 2005 m9500 sf1ek9 mz 2 1 1 bin 48036...

Page 94: ...tem will automatically synchronize the ELPD image to the standby supervisor module if automatic copying is enabled switch config t switch config boot auto copy Step 6 Use the install module number epld url command on the active supervisor module to upgrade EPLD images for a module switch install module 2 epld bootflash m9000 epld 2 1 2 img EPLD Curr Ver New Ver XBUS IO 0x07 0x07 UD Flow Control 0x...

Page 95: ...13 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 3 Managing Storage Services Modules SSM Issues Note When you upgrade the EPLD module on Cisco MDS 9100 Series switches you receive the following message Data traffic on the switch will stop now Do you want to continue y n ...

Page 96: ... c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 3 14 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 3 Managing Storage Services Modules SSM Issues ...

Page 97: ...c system component The first step is to compare what the system is doing to what it should be doing Because a startup problem can usually be attributed to a single component it is more efficient to isolate the problem to a subsystem rather than troubleshoot each separate component in the system Problems with the initial power up are often caused by a module that is not firmly connected to the back...

Page 98: ...mes online and how the software images are handled http www cisco com univercd cc td doc product sn5000 mds9000 index htm Switching module Status LEDs on each module indicate if it has been initialized by the supervisor module A module that is partially installed in the backplane can cause the system to halt SNMP Traps You can set SNMP traps to monitor fans power supplies and temperature settings ...

Page 99: ...r orange after the initialization time see the Troubleshooting Supervisor Issues section on page 4 14 If you have a redundant supervisor module refer to the following website for the latest Cisco MDS 9000 Family configuration guides for descriptions of the supervisor module LEDS how the redundant supervisor module comes online and how the software images are handled http www cisco com univercd cc ...

Page 100: ...efer to power supply documentation in the relevant hardware installation guide to learn more on increasing or decreasing power supply capacity and configuring power supplies Error Message PLATFORM 2 PS_MISMATCH Detected power supply chars This reduces the redundant power available to the system and can cause service disruptions Serial No chars Explanation Detected a new power supply that has reduc...

Page 101: ...tched pair of power supplies and the mode is redundant or if there is a transition from combined to redundant mode If both power supplies are the same capacity or the mode is combined Cisco SAN OS never shuts down a power supply Power supply is not operational Troubleshoot the power supplies See the Troubleshooting the Power Supplies section on page 4 7 Table 4 2 Power Supply INput Ok LED Is Red S...

Page 102: ... command to collect more information Introduced Cisco MDS SAN OS Release 1 3 1 Table 4 3 Power Supply Output Failed LED is On Symptom Possible Causes Solutions Power Supply Output Failed LED is on Power supply is not operational Troubleshoot the power supplies See the Troubleshooting the Power Supplies section on page 4 7 Table 4 4 Power Supply Fan Ok LED is Red Symptom Possible Cause Solution Pow...

Page 103: ...e Input Ok LED lights at this point return the first power cord for replacement c If the Input Ok LED still fails to light when the switch is connected to a different power source with a new power cord the power supply is probably faulty If a second power supply is available install it in the second power supply bay and contact your customer service representative for further instructions Note If ...

Page 104: ...g Fan Is Spinning Fan LED is Red Symptom Fan is spinning but fan LED is red Table 4 5 Fan Is Not Spinning Symptom Possible Cause Solution Fan is not spinning Fan is not correctly seated in the chassis Loosen the captive screws remove the fan module and reinstall it to ensure that the fan module is seated properly Tighten all captive screws and then restart the system Power supply is not operationa...

Page 105: ...an module is removed Cisco SAN OS starts a five minute countdown Caution If the fan module is not reinserted within five minutes the entire switch is shutdown Software reads a byte on the SEEPROM to determine if the fan module is present If the fan module is partially inserted or if software is unable to access the SEEPROM on the fan module for any other reason then Cisco SAN OS cannot distinguish...

Page 106: ...ent the fan module has been removed As soon as the fan module is removed Cisco SAN OS starts a five minute countdown Caution If the fan module is not reinserted within five minutes the entire switch is shut down Software reads a byte on the SEEPROM to determine if the fan module is present If the fan module is partially inserted or if software is unable to access the SEEPROM on the fan module for ...

Page 107: ...k 9 Intake 65 50 40 ok The intake sensor located at the airflow intake on the module is the most critical indicator of module temperature All Cisco SAN OS actions are taken when the major threshold of an intake sensor is exceeded A minor threshold violation or a major threshold violation on an outlet sensor results in the following system message Error Message PLATFORM 0 MOD_TEMPMAJALRM Module dec...

Page 108: ...on Sometimes a temperature sensors fails No explicit action is taken for this condition except generating the following system message Error Message PLATFORM 5 MOD_TEMPFAIL Module dec temperature sensor failed Explanation Module contains a faulty temperature sensor Recommended Action Enter the show environment temperature CLI command or similar Fabric Manager Device Manager command to collect more...

Page 109: ...ice id The device that logged the exception This is interpreted by your customer support representative device errorcode The error code that occurred on the device This is interpreted by your customer support representative error type The severity level of the error Software errors are typically minor or warning All other errors may be hardware problems Number Ports that failed The number of ports...

Page 110: ...ervisors are present in the system at poweredup one of the supervisors will become active and the other standby The active supervisor initialization differs from the standby supervisor If there is no active supervisor in the system the supervisor that boots up first will default to the active supervisor If there is an active supervisor in the system the supervisor that is booting up will default t...

Page 111: ...CE_CRASHED Service xbar PID 2349 hasn t caught signal 9 no core 2005 Sep 27 18 58 06 172 20 150 204 SYSMGR 3 SERVICE_CRASHED Service xbar PID 2352 hasn t caught signal 9 no core Table 4 7 Active Supervisor Reboots Symptom Possible Cause Solution Active supervisor reboots Supervisor process crashed resulting in a supervisor reload Use the show system reset reason CLI command to view the cause of th...

Page 112: ...68 ticks Wed Sep 28 14 17 48 2005 error type FATAL error exception that caused the reboot Number Ports went bad 1 2 3 4 5 6 exception information exception instance 2 device id 12 device errorcode 0x00060a02 system time 1127917067 ticks Wed Sep 28 14 17 47 2005 error type Warning Number Ports went bad 1 2 3 4 5 6 Example 4 9 displays the system messages on the standby supervisor module when a supe...

Page 113: ...Wide Name s WWN 5 2 1 2 1 1 Mod MAC Address es Serial Num 5 00 0b be f7 4d 1c to 00 0b be f7 4d 20 JAB070307XG this terminal session Step 2 Telnet to the standby supervisor console port and verify that it is in standby mode See Example 4 11 Example 4 11 Verify Standby Supervisor Mode runlog telnet sw4 ts 2004 Trying 172 22 22 55 Connected to sw4 ts cisco com 172 22 22 55 Escape character is Table ...

Page 114: ...tion phase with the active supervisor switch show system redundancy status Redundancy mode administrative HA operational None This supervisor sup 1 Redundancy state Active Supervisor state Active Internal state Active with HA standby Other supervisor sup 2 Redundancy state Standby Supervisor state HA standby Internal state HA synchronization in progress The most likely reason for the synchronizati...

Page 115: ... 1 vsan 0x00000029 1436 15 s0009 1 vshd 0x00000028 1408 37 s0009 1 wwn 0x00000030 1435 114 s0009 1 xbar 0x00000017 NA NA s0017 23 xbar_client 0x00000049 1434 917 s0009 1 Looking at the standby supervisor in Example 4 12 shows that the crossbar xbar software component has been restarted 23 times This has probably prevented the standby from initializing properly Step 6 Use the reload module command ...

Page 116: ...e command on the active supervisor to verify that the standby supervisor in the powered up state See Example 4 13 Example 4 13 show module Command Output switch show module Mod Ports Module Type Model Status 5 0 Supervisor Fabric 1 DS X9530 SF1 K9 active 6 0 Supervisor Fabric 1 powered up 8 8 IP Storage Services Module powered dn Mod Sw Hw World Wide Name s WWN 5 2 1 2 1 1 Mod MAC Address es Seria...

Page 117: ...ervisors or to not having current images in Flash memory on the supervisor Entering a copy slot0 bootflash CLI command copied the images anyway Once the images were loaded on the second supervisor and the boot statements were confirmed and saved on the active supervisor the supervisor loaded and came up in standby ha mode Troubleshooting Switching and Services Modules This section describes proble...

Page 118: ...n The module has been powered down because of user configuration or because of an error Use the show running config include poweroff CLI command to determine whether or not the module has been configured as powered down Otherwise the module was powered down because of an error If a module reports a FATAL error the supervisor logs an exception and reboots the module If the supervisor reboots the mo...

Page 119: ...ilure can be obtained using the show platform internal event history errors CLI command See Example 4 15 Example 4 15 Finding Boot Up Failure Codes switch show platform internal event history errors The following error codes are defined No Boot Device 0xF1 Boot Failed 0xC0 Net Boot Failed 0xD0 Unknown Status 0x1B Image Download Once the supervisor receives the registration message it checks the im...

Page 120: ...ule dec reported failure on ports dec dec dec dec chars due to chars in device dec device error hex Explanation The module reported a failure in the runtime diagnostic Module manager is going to power cycle the module Recommended Action Collect information about the module by entering the show module internal all module CLI command In addition this information is stored in the exception log which ...

Page 121: ...tion log is sorted in ascending manner that is the latest state is at the end of the log The error log is sorted in descending manner that is the latest error is at the beginning of the log Use the show module internal event history module CLI command to view the state transition log for a module Use the show module internal event history errors CLI command to view the error log The state transiti...

Page 122: ...see the symptoms listed in this section If you are unable to resolve a problem with the startup gather the information listed under Appendix A Before Contacting Technical Support and contact your technical support representative for assistance as directed in the Obtaining Documentation Obtaining Support and Security Guidelines section on page xxxii Troubleshooting Powered Down Modules Symptom Modu...

Page 123: ...tion The module failed to power up Recommended Action Enter the show platform internal all module dec CLI command to collect more information Introduced Cisco MDS SAN OS Release 1 2 2a Error Message PLATFORM 3 MOD_PWRIDPROMFAIL Module dec failed to power up due to idprom read error Explanation The module cannot be powered up because of an IDPROM read error Recommended Action Enter the show platfor...

Page 124: ... 36 or the Reinitializing a Failed Module Using the CLI section on page 4 37 Module failed to register with the supervisor Use the show module internal event history module CLI command and look for Triggered event LCM_EV_LCP_REGISTRATION_TIMEOUT to verify that the module did not register Right click the module in Device Manager and select Reset or use the reload module CLI command to restart the m...

Page 125: ...tected Serial number JAB064704LH 2005 Sep 27 15 28 15 172 20 150 204 PLATFORM 5 MOD_PWRUP Module 8 powered up Serial number JAB064704LH 2005 Sep 27 15 29 16 172 20 150 204 MODULE 5 MOD_REINIT Re initializing module 8 2005 Sep 27 15 29 22 172 20 150 204 PLATFORM 5 MOD_DETECT Module 8 detected Serial number JAB064704LH Note that module 8 powered up and reinitialized three times This indicates that t...

Page 126: ... after Tue Sep 27 15 30 23 2005 Instance 3 Seq Id 0x3 Ret success E_MTS_TX Dst MTS_SAP_XBAR_MANAGER 48 Opc MTS_OPC_LC_INSERTED 1081 85 Event ESQ_RSP length 32 at 692394 usecs after Tue Sep 27 15 30 23 2005 Instance 3 Seq Id 0x3 Ret null E_MTS_RX Src MTS_SAP_XBAR_MANAGER 48 Opc MTS_OPC_LC_INSERTED 1081 86 FSM ID 3 Slot 8 node 0x0802 Transition at 692410 usecs after Tue Sep 27 15 30 23 2005 Previous...

Page 127: ... of the log and moving backwards in this example you can infer the following Curr state LCM_ST_LC_NOT_PRESENT Indicates that the module is not present Index 112 Triggered event LCM_EV_FAILED_MORE3TIMES Indicates that the module failed repeatedly Index 111 Triggered event LCM_EV_LC_INSERTED_SEQ_FAILED Indicates that the insertion sequence failed Index 86 Previous state LCM_ST_CHECK_INSERT_SEQUENCE ...

Page 128: ...use of a failure in some of the ports Recommended Action Collect module information by entering the show module internal all module CLI command Error Message MODULE 2 MOD_DIAG_FAIL Module dec reported failure on ports dec dec dec dec chars due to chars in device dec device error hex Explanation The module reported a failure in the runtime diagnostic Module manager is going to power cycle the modul...

Page 129: ...ce Manager or use the show logging CLI command to verify bootup problems Use the show module internal event history module CLI command and look for Triggered event LCM_EV_LCP_ALIVE_TIMEOUT to verify that the module did not respond to heartbeat requests Right click the module in Device Manager and select Reset or use the reload module CLI command to restart the module See the Reinitializing a Faile...

Page 130: ...gnose a module in the unknown state follow these steps Step 1 Right click the module and select Module on Device Manager or use the show module CLI command to verify the status of the module Step 2 Choose Logs Switch Resident Syslog Sever Events on Device Manager or use the show logging CLI command to search for common problems Step 3 Use the show platform internal event history errors CLI command...

Page 131: ...Right click the module and select Module on Device Manager or use the show module CLI command to verify the status of the module Step 2 Choose Logs Switch Resident Syslog Server Events on Device Manager or use the show logging CLI command to search for common problems Step 3 Use the show platform internal event history errors CLI command to view possible causes switch show platform internal event ...

Page 132: ...state PLTFRM_STATE_MODULE_START_POWER_UP Triggered event PLTFRM_EVENT_MOD_END_POWER_UP Next state PLTFRM_STATE_MODULE_POWERED_UP 5 FSM Slot 8 Transition at 704067 usecs after Thu Sep 29 17 46 20 2005 Previous state PLTFRM_STATE_MODULE_POWERED_UP Triggered event PLTFRM_EVENT_MODULE_REMOVED Next state PLTFRM_STATE_MODULE_ABSENT When a module is inserted into the switch the supervisor module reads th...

Page 133: ...o reinitialize a failed module using the CLI follow these steps Step 1 Save the running configuration to the startup configuration switch copy running config start config Step 2 Reload the module switch reload module 2 Step 3 If the module is not operating verify the software image on the module switch show module Step 4 If the software image on the module is not the latest download the latest ima...

Page 134: ...ing Cisco SAN OS Software System Reboots section on page 2 13 If you use the module reset reason CLI command and the output has an unknown reset reason this may indicate a hardware problem Some of the conditions that may cause this include the following The switch experienced a power reset This may be because you reset the power supplies or because of a power interruption or failure The front pane...

Page 135: ...entifies the modules supported by the Cisco MDS 9500 Series switches and Cisco MDS 9216A and Cisco MDS 9216i switches as well as the Generation 2 switches Table 5 1 Generation 2 Modules and Switches Part Number Description Modules DS X9148 48 port 4 Gbps Fibre Channel switching module DS 9304 18K9 18 port 1 2 4 Gbps Fibre Channel switching module with 4 Gigabit Ethernet ports DS X9112 12 port 4 Gb...

Page 136: ... and includes the following topics Port Groups page 5 2 Port Speed Mode page 5 3 Dynamic Bandwidth Management page 5 3 Out of Service Interfaces page 5 4 Port Index Availability page 5 4 Port Groups Each module has four groups of one or more ports in port groups that share common resources such as bandwidth and buffer credits Table 5 2 shows the port groups for the Generation 2 Fibre Channel switc...

Page 137: ...Gbps 6 12 8 4 Gbps DS C9222i K9 18 port 4 Gbps 6 12 8 4 Gbps 1 Dedicated bandwidth with no oversubscription 2 Dedicated bandwidth or oversubscribed using shared buffer resources Table 5 2 Bandwidth and Port Groups for Generation 2 Modules continued Module or Switch Description Number of Ports Per Port Group Bandwidth Per Port Group Maximum Bandwidth Per Port Table 5 3 Configurable Port Speeds on G...

Page 138: ...ources from other interfaces Port Index Availability Each chassis in the Cisco MDS 9000 Series has a hardware based maximum port availability based on internally assigned port indexes When the maximum number of port indexes is reached in a chassis any modules remaining or added to the chassis will not boot up The number of physical ports on a Fibre Channel module is equal to its number of port ind...

Page 139: ...his means that no Generation 1 module except a 16 port Fibre Channel switching module can be inserted into slot 1 because some of the port indexes for the slot are already in use Example 5 1 Borrowing Port Indexes from Another Slot switch show port index allocation Module index distribution Slot Allowed Alloted indices info range Total Index values 1 0 31 2 32 63 32 32 63 3 64 95 48 64 95 224 239 ...

Page 140: ...ndexes from slot 1 of the chassis until it has the number of port indexes necessary Note Use the purge module CLI command to free up reserved port indexes after you remove a module Table 5 7 Port Index Requirements Supervisor Module Port Index Requirements Supervisor 1 Generation 1 Indexes must Be contiguous In the range assigned to the given slot Start with the lowest value assigned to that slot ...

Page 141: ...pervisor 2 module to a Supervisor 1 module Initial Troubleshooting Checklist Begin troubleshooting Generation 1 and Generation 2 module issues by checking the following issues Use the show interface transceiver CLI command to view enhanced diagnostics on the X2 transceivers for Generation 2 modules This is supported on 4 Gbps and 10 Gbps ports Use these diagnostics to isolate physical layer proble...

Page 142: ... SUP 3 253 255 In some cases the sequence in which switching modules are inserted into the chassis determines if one or more modules is powered up Table 5 8 Module Does Not Come Online Symptom Possible Cause Solution Module does not come online Not enough port indexes are available See the Verifying Port Index Allocation Using Device Manager section on page 5 8 or the Verifying Port Index Allocati...

Page 143: ...g the CLI To verify port index allocation using the CLI follow these steps Step 1 Use the show port index allocation command to display the allocation of port indexes on the switch switch show port index allocation Module index distribution Slot Allowed Alloted indices info range Total Index values 1 0 255 16 32 47 2 0 255 12 0 11 3 0 255 None 4 0 255 None 7 0 255 None 8 0 255 None 9 0 255 None SU...

Page 144: ...module is powered down because of port index issues use the show module recovery steps command to determine how to correct the problem switch show module 4 recovery steps Failure Reason Contiguous and aligned indices unavailable for Generation 1 modules Check show port index allocation for more details Please follow the steps below 1 Power off module in one of the following slots 12 2 Power on mod...

Page 145: ...bps Total shared bandwidth is 4 8 Gbps Allocated dedicated bandwidth is 8 0 Gbps Interfaces in the Port Group B2B Credit Bandwidth Rate Mode Buffers Gbps fc2 1 16 4 0 shared fc2 2 16 4 0 shared fc2 3 16 4 0 shared fc2 4 16 4 0 shared fc2 5 16 4 0 dedicated fc2 6 16 4 0 dedicated In this example there is not enough available shared bandwidth in Port Group 1 to switch any more ports to 4 Gbps dedica...

Page 146: ...d bandwidth is 4 8 Gbps Allocated dedicated bandwidth is 8 0 Gbps Interfaces in the Port Group B2B Credit Bandwidth Rate Mode Buffers Gbps fc2 1 16 4 0 shared fc2 2 16 4 0 shared fc2 3 16 4 0 shared fc2 4 16 4 0 shared fc2 5 16 4 0 dedicated fc2 6 16 4 0 dedicated In this example there is not enough available shared bandwidth in Port Group 1 to switch any more ports to 4 Gbps dedicated mode Step 2...

Page 147: ...nfigure to see if the port is out of service Using the CLI use the show interface brief command to see if the port is out of service See the Verifying Bandwidth Utilization in a Port Group Using Device Manager section on page 5 11 or the Verifying Bandwidth Utilization in a Port Group Using the CLI section on page 5 12 to free up enough port resources to bring the port in service Not enough bandwi...

Page 148: ...ervisor Module Type Naming Convention 9120 or 9140 Supervisor 1 module Filename begins with m9100 s1ek9 9134 Cisco Fabric Switch for HP c Class BladeSystem Cisco Fabric Switch for IBM BladeCenter Supervisor 2 module Filename begins with m9100 s2ek9 9221i Supervisor 2 module Filename begins with m9200 s2ek9 9216 9216A or 9216i Supervisor 1 module Filename begins with m9200 s1k9 9506 or 9509 Supervi...

Page 149: ...s Feature based licensing Features that are applicable to the entire switch You need to purchase and install a license for each switch that uses the features you are interested in The Enterprise license is an example of a feature based license Module based licensing Features that require additional hardware modules You need to purchase and install a license for each module that uses the features y...

Page 150: ...ay grace period to evaluate the feature You must purchase and install the number of licenses required for that feature before the grace period ends or Cisco SAN OS will disable the feature at the end of the grace period If you try to use an unlicensed feature you may see the following system messages Error Message LICMGR 2 LOG_LIC_GRACE_EXPIRED Grace period expired for feature chars Explanation Th...

Page 151: ...To suspend the grace period countdown for a licensed feature you must disable every feature in that license package Choose Switches Licenses and select the Usage tab in Fabric Manager or use the show license usage CLI command to determine which features are enabled for a license package Initial Troubleshooting Checklist Begin troubleshooting license issues by checking the following issues first Th...

Page 152: ...and any errors such as a missing license Click the Files tab to display information about each License Key file installed on the switch Step 3 Click the Usage tab to see the applications using the feature package on each switch Use this tab to determine which applications depend on each license that you have installed Displaying License Information Using Fabric Manager Web Client Fabric Manager Re...

Page 153: ...License Key Files and Contents switch show license Permanent lic SERVER this_host ANY VENDOR cisco INCREMENT MAINFRAME_PKG cisco 1 0 permanent uncounted HOSTID VDH FOX0646S017 NOTICE LicFileID LicFileID LicLineID 0 LicLineID PAK dummyPak PAK SIGN EE9F91EA4B64 Evaluation lic SERVER this_host ANY VENDOR cisco INCREMENT MAINFRAME_PKG cisco 1 0 30 Dec 2003 uncounted HOSTID VDH FOX0646S017 NOTICE LicFi...

Page 154: ...ches the vendor you purchased your switch from Symptom One click license install fails or cannot connect to the licensing website Serial Number Issues A common problem with licenses stems from not using the correct chassis serial number when ordering your license To obtain the correct chassis serial number using Fabric Manager follow these steps Table 6 1 One Click License Install Fails or Cannot ...

Page 155: ...h using the CLI When entering the chassis serial number during the license ordering process do not use the letter O in place of any zeros in the serial number RMA Chassis Errors or License Transfers Between Switches A license is specific to the switch for which it is issued and is not valid on any other switch If you need to transfer a license from one switch to another contact your customer servi...

Page 156: ...file follow these steps Step 1 Open both license files using WordPad Step 2 Copy both license files to one file Example SERVER this_host ANY VENDOR cisco INCREMENT SAN_EXTN_OVER_IP_IPS2 cisco 1 0 permanent 1 VENDOR_STRING LIC_SOURCE MDS_SWIFT LIC_SOURCE SKU M9500EXT12EK9 SKU HOSTID VDH FOXYYYYYYY NOTICE LicFileID 2005082204514XXXX LicFileID LicLineID 1 LicLineID PAK MDS 1X JAB 0F1A81 PAK SIGN F065...

Page 157: ... disable a feature during the grace period and there are other features in that license package that are still enabled the countdown does not stop for that license package To suspend the grace period countdown for a license package you must disable every feature in that license package To disable the grace period countdown for Fabric Manager Server you must explicitly check in the license using De...

Page 158: ...ER_IP_IPS2 Yes 1 Unused never 1 license s missing SAN_EXTN_OVER_IP_IPS4 No 0 Unused 10G_PORT_ACTIVATION_PKG No 0 Unused SAN_EXTN_OVER_MPS_184_FIPS No 0 Unused STORAGE_SERVICES_ENABLER_PKG Yes 1 Unused never 1 license s missing WARNING License file s missing ips hac1 Checking in the Fabric Manager Server License From Device Manager If you evaluated Fabric Manager Server without a license you can st...

Page 159: ...stalled and operating properly it may show up as missing if you modify your system hardware or encounter a bootflash issue Symptom License listed as missing Table 6 3 License Listed as Missing Symptom Possible Causes Solutions License listed as missing Supervisor module was replaced after license was installed Reinstall the license Supervisor bootflash is corrupted See the Corrupted Bootflash Reco...

Page 160: ... e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 6 12 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 6 Troubleshooting Licensing Licensing Installation Issues ...

Page 161: ...page 7 9 CFS Regions Troubleshooting page 7 10 Overview Many features in the Cisco MDS 9000 Family switches require configuration synchronization in all switches in the fabric It is important to maintain configuration synchronization across a fabric for consistency As of Cisco MDS SAN OS Release 2 0 1b Cisco Fabric Services CFS provides a common infrastructure for automatic configuration synchroni...

Page 162: ...ion database also know as active database or the effective database CFS distribution enabled or disabled on a per application basis The default enable or disable for CFS distribution state differs between applications If CFS distribution is disabled for an application then that application does not distribute any configuration nor does it accept a distribution from other switches in the fabric Exp...

Page 163: ...n the fabric constitute one CFS fabric or a multitude of partitioned CFS fabrics using Device Manager follow these steps a Choose Admin CFS and highlight the application that you want to verify CFS on b Click Details and select the Merge tab in the Details dialog box c If you see multiple rows in the Merge status table then the fabric is partitioned into multiple CFS fabrics Some features enable C...

Page 164: ...how cfs peers name application name vsan vsan id for logical scope applications An example command output for a physical scope application follows Switch show cfs peers name dpvm Scope Physical Switch WWN IP Address 20 00 00 0e d7 0e bf c0 10 76 100 51 Local 20 00 00 0e d7 00 3c 9e 10 76 100 52 Total number of entries 2 Note The show cfs peers name application name command displays the peers for a...

Page 165: ... 00 00 0d ec 0c f1 40 10 76 100 204 Local 20 00 00 05 30 00 4a de 10 76 100 51 Total number of entries 2 If the list of switches in the show cfs merge status name command output is shorter than that of the show cfs peers name command output the fabric is partitioned into multiple CFS fabrics and the merge status may show that the merge has failed is pending or is waiting Merge Failure Troubleshoot...

Page 166: ...fs merge status name application name command Example command output follows Switch show cfs merge status name ntp Physical Merge Status Failure Mon Nov 22 06 49 52 2004 Failure Reason Conflicting entries in the compared databases Local Fabric Switch WWN IP Address 20 00 00 05 30 00 6b 9e 10 76 100 167 Merge Master 20 00 00 0e d7 00 3c 9e 10 76 100 52 Remote Fabric Switch WWN IP Address 20 00 00 0...

Page 167: ...ministrators on the same switch attempt to configure the same application only one administrator is given the lock The other administrator is prevented from making changes to that application until the first administrator commits a change or discards any changes Use the show cfs lock name CLI command to determine the name of the administrator who holds the lock for an application You should check ...

Page 168: ...ws switch show cfs internal session history name ntp detail Time Stamp Source WWN Event User Name Session ID Fri Aug 24 04 30 19 2007 20 00 00 0d ec 04 99 c0 LOCK_REQUEST admin 3848 Fri Aug 24 04 30 19 2007 20 00 00 0d ec 04 99 c0 LOCK_ACQUIRED admin 3848 Fri Aug 24 04 30 19 2007 20 00 00 0d ec 04 99 c0 COMMIT admin 3849 Fri Aug 24 04 30 19 2007 20 00 00 0d ec 04 99 c0 LOCK_RELEASE_REQUEST admin 3...

Page 169: ...ree the CFS lock Clearing Locks Using the CLI When a lock is being held on a remote peer and issuing the application name commit command or the application name abort command does not clear the lock issue the clear application name session command to clear all locks in the fabric After all locks are cleared a new distribution must be started to restore all the switches in the fabric to the same st...

Page 170: ...r merge with switches running SAN OS 3 1x All applications in other regions on the switch running SAN OS 3 2x are ignored by the switch running SAN OS 3 1x CFS Regions configuration is not supported for deregistered applications conditional services or a physical scope application that is currently locked Regions 1 through 200 are available for user configuration Regions 201 through 255 are reserv...

Page 171: ...conditional service is restarted it will automatically be put into the default region To avoid this situation reconfigure the appropriate region information for the conditional service before it starting it again Changing Regions If you move an application from one region to another you may encounter a database mismatch when attempting a merge Follow the steps outlined in the Merge Failure Trouble...

Page 172: ...t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 7 12 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 7 Troubleshooting Cisco Fabric Services CFS Regions Troubleshooting ...

Page 173: ...l port modes E port F port FL port TL port TE port SD port and B port In addition to these modes each interface can be configured in auto or Fx port modes These modes determine the port type during interface initialization Each interface has an associated administrative configuration and operational status The administrative configuration does not change unless you modify it This configuration has...

Page 174: ...edia to ensure there are no damaged parts Verify that the SFP small form factor pluggable devices in use are those authorized by Cisco and that they are not faulty Verify that you have enabled the port by right clicking the port in Device Manager and selecting enable or by using the no shut CLI command Right click the port in Device Manager or use the show interface CLI command to verify the state...

Page 175: ...are failure A hardware failure is detected Error disabled Error conditions require administrative attention Interfaces may be error disabled for various reasons For example Configuration failure Incompatible buffer to buffer credit configuration To make the interface operational you must first fix the error conditions causing this state then administratively shut down and reenable the interface Is...

Page 176: ... state is trunking Overview of the FC MAC Driver and the Port Manager This section describes the internal details of port related components in Cisco SAN OS Use this section to understand the underlying functions that may be causing port related problems The FC MAC driver resides in the module component of the Cisco MDS 9000 Family SAN OS software It performs the following functions Initialization...

Page 177: ...nt software running on the supervisor module The Port Manager handles the following tasks Port configuration management Link events including notifying the registered application on the supervisor module E or TE port initialization SFP validation The FC MAC detects the port is in one of the following states Disable The port is administratively disabled Enable The port is administratively enabled I...

Page 178: ...on This could indicate a disconnected or faulty cable or no active device connection Red box An FSP is present but fabric login FLOGI has failed Typically a mismatch in port or fabric parameters with the neighboring device For example a port parameter mismatch would occur if a node device were connected to a port configured as an E port An example of a fabric parameter mismatch would be differing ...

Page 179: ...ows administrative settings for Mode Speed and Status plus current operational status failure cause and date of the last configuration change Additional tabs include Rx BB Credit Configure and view buffer to buffer credits BB credits Other View PortChannel ID WWN Maximum Transmission Unit MTU configure maximum receive buffer size FLOGI View FC ID pWWN nWWN BB credits and class of service for N por...

Page 180: ...ffic and error statistics including link reset counts offline and non operational sequence errors reset protocol errors and statistics related to buffer to buffer flow control Discards View the number of frames discarded by the port including Class 2 Class 3 and Class F frames EISL frames and totals Link Errors View the number of link errors including link failures signal losses synchronization fa...

Page 181: ...chart Figure 8 5 Device Manager Port Monitoring Line Chart Isolating Port Issues Using Device Manager To isolate port issues using Device Manager follow these steps Step 1 Choose Interfaces FC ALL and verify that the Status Oper field is up to determine if the host HBA and the storage port can provide link level connectivity to their respective switches See Table 8 1 on page 8 3 for details on non...

Page 182: ...e Sets Troubleshooting Port States from the CLI To display complete information for an interface use the show interface command In addition to the state of the port this command displays Port WWN Speed Trunk VSAN status Transmit and receive buffer to buffer credits configured and remaining Maximum receive buffer size Number of frames sent and received Transmission errors including discards errors ...

Page 183: ... FC MAC2 driver in the case of the MDS 9120 MDS 9140 MDS 9216i and the MPS 14 2 module Table 8 2 lists several CLI debugging commands at the FC MAC level Note You must use the attach module CLI command to access these FC MAC show commands Note Use the fcmac2 keyword for the MDS 9120 MDS 9140 MDS 9216i and the MPS 14 2 module Table 8 2 Useful FC MAC Port Commands CLI Command Description show hardwa...

Page 184: ...ther of the ports fails to remain in the online state then you may have a faulty GBIC cabling or HBA subsystem port Step 4 If both ports are online use the show flogi command to verify that the Fibre Channel ports for the host and storage have performed a fabric login FLOGI and are communicating with their respective switches Example 8 2 Using the show flogi command NPI1 sh flogi INTERFACE VSAN FC...

Page 185: ...7e0200 N 21 00 00 e0 8b 08 d3 20 QLogic scsi fcp init At this point the HBA and subsystem ports have successfully established link level connectivity and each one can communicate with its locally attached switch in the fabric The next step is to verify zone membership For a more detailed discussion and description of vsans and zones see Chapter 14 Troubleshooting Zones and Zone Sets Common Problem...

Page 186: ...he connected device on Right click on the port in Device Manager and select disable and then enable or use the shut CLI command followed by the no shut command to disable and enable the port If this does not clear the problem try moving the connection to a different port on the same or another module There is no signal because of a transit fault in the SFP or the SFP may be faulty When this occurs...

Page 187: ... process is completed Checks for error counters 4 Checks whether the port is in the offline state The port goes to the offline state if the FLOGI or ELP in case of auto mode on the port does not succeed 5 Checks for pause state A pause state is in an intermediate state as maintained by the FC MAC driver after the link goes down and before the port is enabled by the Port Manager Note The link reini...

Page 188: ...on Port remains in the initializing state The port is up because the link partner has put itself in a bypass mode Use the show hardware internal fc mac port port statistics command to check whether the Class 3 input counter is increasing after the successful completion of link initialization Note You must use the attach module CLI command to access the FC MAC show commands The FLOGI packet was dro...

Page 189: ...mes input 106008 bytes 0 discards 0 CRC 0 unknown class 0 too long 0 too short 2904 frames output 364744 bytes 0 discards 0 input OLS 0 LRR 0 NOS 0 loop inits 1 output OLS 1 LRR 0 NOS 0 loop inits If the interface is not working correctly check the cabling and the host or storage device interface for faults If the interface is working correctly proceed to the next step Step 2 Verify that the devic...

Page 190: ...nt FLOGI_EV_VALID_FLOGI Next state FLOGI_ST_GET_FCID The hba has sent an FLOGI to the switch 2 FSM 99 21 00 00 e0 8b 07 a4 36 Transition at 322974 usecs after Sun Feb 1 04 18 15 1980 Previous state FLOGI_ST_GET_FCID Triggered event FLOGI_EV_VALID_FCID Next state FLOGI_ST_PERFORM_CONFIG Port Manager Obtains a valid FC_ID from the Domain Mgr 3 FSM 99 21 00 00 e0 8b 07 a4 36 Transition at 323731 usec...

Page 191: ...at you do not disable and then enable a T or TE port This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem Step 6 Use the debug fcns events register vsan command to watch the FLOGI process take place switch debug fcns events register vsan 99 This command enables debug mode for name server registration It generates messages on the switch console related...

Page 192: ...hese will be listed if more name server objects are registered Step 7 If you are managing the switch over a Telnet connection enable terminal monitoring by entering the terminal monitor command in exec mode The system output looks like this switch show fcns database detail vsan 99 VSAN 99 FCID 0x780200 port wwn vendor 21 00 00 e0 8b 07 a4 36 QLogic Port world wide name node wwn 20 00 00 e0 8b 07 a...

Page 193: ...it is important to know the following information Who initiated the link flap The actual link down reason Be sure to check the HBA because a faulty HBA can manifest symptoms on the attached switch port For example if an Nx port is self diagnosed as faulty by the HBA driver or firmware the driver can place the port in optical bypass mode This results in the receive and transmit paths being internal...

Page 194: ... when there is a temporary signal or sync loss condition that lasts for less than 100msec See the Troubleshooting Port Problems section on page 8 15 to verify this condition Right click the port in Device Manager and select disable and then enable or use the shut CLI command followed by the no shut command to disable and enable the port If this does not clear the problem try moving the connection ...

Page 195: ...f 8b 10 encoding in the primary operational states They include AC Active state LR Link recovery state LF Link failure state OF Offline state Figure 8 6 Link Initialization Flow Figure 8 6 shows the link initialization flow It displays the ordered sets transmitted between the ports and the primary operational states of the port during the process They include 1 Active state 2 Link recovery state L...

Page 196: ...s is not necessarily a problem with the physical link but with the way some devices initialize the link Use attach module to connect to the module and then use the show hardware internal debug info interface CLI command See Table 8 2 Loss of signal A signal loss condition persisted for more than 100 milliseconds Look at the Invalid Transmission Word Count to check whether the physical link is real...

Page 197: ...rom the N port and reply with an OLS However because the transmitted OLS never reaches the N port the R_T_TOV timer expires In this scenario the status of the port will also show Link failure or not connected The key difference between this case and the no bit synchronization case is that the input and output counts for OLS and NOS increment as there is bit synchronization but no word synchronizat...

Page 198: ...smit B2B credit remaining Port Bounces Between Initializing and Offline States Symptom Port bounces between the initializing and offline states An ELP failure may result in a port bouncing between the initializing and offline states Table 8 7 lists possible causes and solutions to this problem Table 8 7 Port Bounces Between the Initializing and Offline States Symptom Possible Cause Solution Port b...

Page 199: ... ELP is a frame sent between two switches to negotiate fabric parameters Step 2 Verify that the following parameters match on each switch in the VSAN using the show fctimer command ED_TOV timer RA_TOV timer FS_TOV timer Note Because fabric parameters are configured on a per VSAN basis they are required to be the same for all switches within a VSAN switch show fctimer F_S_TOV 5000 milliseconds D_S_...

Page 200: ...rcable Pkt Size 2112 Hw Capabilities 0xb Connector Type 0x0 SFP info Min Speed 1000 Max Speed 2000 Module Type 8 Connector Type 7 Gigabit Eth Compliance Codes 0 FC Transmitter Type 3 Vendor Name PICOLIGHT Vendor ID 0 4 133 Vendor Part Num PL XPL 00 S23 28 Vendor Revision Level Trunk Info trunk vsans allowed active 1 E Port Bounces Remains Isolated After a Zone Merge Symptom E port remains isolated...

Page 201: ... set with that of the remote switch If the zoning databases between the two switches are overwritten you cannot use the Restore option To work around this you can manually change the content of the zone database on either of the switches using the Edit Local Full Zone Database and then choose Switches Interfaces FC Physical and select down and then up on the Admin Status drop down menu for the iso...

Page 202: ... 21 NOS 25 loop inits Step 2 Verify the zoning information using the following commands show zone vsan vsan id show zoneset vsan vsan id Step 3 Use one of the following two approaches to resolve a zone merge failure Overwrite the zoning configuration of one switch with the other switch s configuration This can be done with the following commands zone copy interface fc slot port import vsan vsan id...

Page 203: ... resolve the ErrDisable state using the CLI follow these steps Step 1 Use the show interface command to verify that the switch detected a problem and disabled the port Check cables SFPs and optics mds show interface fc1 14 fc1 14 is down errDisabled Table 8 9 Port Cycles Through the Up and Down States Symptom Possible Causes Solutions Port cycles through the up and down states One or more packets ...

Page 204: ... the attempt failed mds show logging logfile Jan 4 06 54 04 switch PORT_CHANNEL 5 CREATED port channel 17 created Jan 4 06 54 24 switch PORT 5 IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN Interface port channel 17 is down No operational members Jan 4 06 54 40 switch PORT_CHANNEL 5 PORT_ADDED fc1 8 added to port channel 7 Jan 4 06 54 56 switch PORT 5 IF_DOWN_ADMIN_DOWN Interface fc1 7 is down Admnistratively ...

Page 205: ...the server fail to complete FLOGI to the switch Why does the storage device fail to complete FLOGI to the switch Figure 8 7 illustrates one possible methodology for troubleshooting Fx ports Figure 8 7 Troubleshooting Methodology Begin Server HBA and Storage FLOGI to thier respective switch Troubleshoot physical ports Troubleshoot HBA Check VSAN membership Check zoning config and LUN masking Server...

Page 206: ...e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 8 34 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 8 Troubleshooting Ports Common Problems with Port Interfaces ...

Page 207: ...ic Switch for IBM BladeCenter NPV makes a Fabric or Blade switch appear as a host to the core Fibre Channel switch and as a Fibre Channel switch to the servers in the Fabric or Blade switch NPV aggregates multiple locally connected N ports into one or more external NP links thereby sharing the domain ID of the NPV core switch among multiple NPV switches NPV also allows multiple devices to attach t...

Page 208: ... core switch Only F NP and SD ports are supported in NPV mode CFS and QoS are not supported IVR SDV and FICON are not supported If an NPV link failover occurs servers that are booted over the SAN with NPV will temporarily lose access to their boot LUNs Common CLI Commands for NPV Note Because the output is based on name server database information the show fcns database npv commands can be run fro...

Page 209: ... 80 10 1 96 24 fc1 20 20 00 00 0d ec 2d af 40 fc4 4 20 00 00 0d ec 3d 62 80 10 1 96 24 fc1 19 20 00 00 0d ec 2d af 40 fc4 3 20 00 00 0d ec 3d 62 80 10 1 96 24 fc1 17 20 00 00 0d ec 2d af 40 fc4 1 For additional details about the NPV devices you see in the show fcns database npv output including IP addresses switch names and interface names enter the show fcns database npv detail command switch sho...

Page 210: ...Total number of flogi 4 Common Problems with NPV This section includes common NPV issues and includes the following topics Moving the Login of an End Device page 9 4 NPIV Is Not Enabled page 9 5 VSAN Mismatches page 9 5 Core NPV Device Is Not a Switch page 9 6 NPV Core Switch Port Is Down page 9 6 Server Interface is Down page 9 6 Waiting on FLOGI from the Server or Target page 9 7 Waiting on Exte...

Page 211: ... NPV switch The NPV core switch is defined as an upstream switch on which NPIV is enabled The NPV core switch receives traffic that is passed to it from a downstream switch that has NPV enabled on it A switch that is in NPV mode does not switch traffic instead it passes traffic to the upstream NPV core switch on which NPIV is enabled After NPIV is enabled on the core NPV switch the port should aut...

Page 212: ... 0x000000 State Failed neighbor on the upstream port is not fabric Number of External Interfaces 3 Step 2 If the state is Failed with the reason neighbor on upstream port is not fabric then the external link is connected to non fabric switch NPV Core Switch Port Is Down If the NPV core switch port is in the shutdown state or it is not an F port Step 1 Enter the show npv status command to check the...

Page 213: ...tus of the link switch show npv status npiv is enabled Server Interfaces Interface fc1 6 VSAN 1 NPIV No State Waiting for FLOGI Number of Server Interfaces 7 Step 2 If the State is Waiting for FLOGI then no FLOGI request was received from the server or target Waiting on External Link to Come Up If you are waiting for the external link to come up Step 1 Enter the show npv status command to check th...

Page 214: ...n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 9 8 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 9 Troubleshooting N Port Virtualization Common Problems with NPV ...

Page 215: ...unctional links in the channel Load balances across multiple links and maintains optimum bandwidth utilization Load balancing is based on the source ID destination ID and exchange ID OX ID Provides high availability on an ISL If one link fails traffic previously carried on this link is switched to the remaining links If a link goes down in a PortChannel the upper protocol is not aware of it To the...

Page 216: ...n an E port that E port becomes a TE port A TE port is specific to switches in the Cisco MDS 9000 Family An industry standard E port can link to other vendor switches and is referred to as a nontrunking interface Initial Troubleshooting Checklist Begin troubleshooting Portchannel and trunking issues by verifying that you have completed following actions first Note Use the show running interface CL...

Page 217: ... usage show interface show interface trunk show trunk protocol PortChannel Issues This section describes common PortChannel issues and includes the following topics Cannot Configure a PortChannel page 10 3 Newly Added Interface Does Not Come Online In a PortChannel page 10 4 Cannot Configure a PortChannel Symptom Cannot configure a PortChannel Table 10 1 Cannot Configure a PortChannel Symptom Poss...

Page 218: ...ges Trunking Issues This section describes common trunking issues and includes the following topics Cannot Configure Trunking page 10 5 VSAN Traffic Does Not Traverse Trunk page 10 5 Table 10 2 Newly Added Interface Does Not Come Online in a PortChannel Symptom Possible Cause Solution Newly added interface does not come online in a PortChannel PortChannel mode is on Enable PortChannel manually or ...

Page 219: ...Solution Cannot configure trunking Trunking protocol is disabled Enable trunking In Fabric Manager choose Switches Interfaces FC Logical select the Trunk Config tab and set the Admin drop down menu to trunk Click Apply Changes Use the trunk protocol enable CLI command Table 10 4 VSAN Traffic Does Not Traverse Trunk Symptom Possible Cause Solution VSAN traffic does not traverse trunk VSAN not in al...

Page 220: ...e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 10 6 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 10 Troubleshooting PortChannels and Trunking Trunking Issues ...

Page 221: ...art of different SAN fabrics that do not need to be aware of one another Each VSAN can contain up to 239 switches and has an independent address space that allows identical Fibre Channel IDs FC IDs to be used simultaneously in different VSANs VSANs provide the following capabilities Isolate devices physically connected to the same fabric Reduce the size of a Fibre Channel distributed database Enab...

Page 222: ...se Fabricxx VSANxx to view the VSAN configuration in the Information pane Choose Fabricxx VSANxx and select the Host or Storage tab in the Information pane to view the VSAN members Choose Fabricxx VSANxx Domain Manager to view the FC domain configuration in the Information pane Choose Fabricxx VSANxx FSPF to view the FSPF configuration in the Information pane Choose Fabricxx VSANxx zoneset name to...

Page 223: ...oting Zones and Zone Sets VSAN Issues This section includes the following topics Host Cannot Communicate with Storage page 11 3 E Port Is Isolated in a VSAN page 11 5 Troubleshooting Interop Mode Issues page 11 9 Host Cannot Communicate with Storage Communication problems between a host and storage devices can be caused by port VSAN or zone issues Symptom Host cannot communicate with storage Table...

Page 224: ...nfigured for the VSAN set the Allowed VSANs field to include the VSAN that the host and storage devices are on and click Apply Changes Verifying VSAN Membership Using the CLI To verify VSAN membership for host and storage devices using the CLI follow these steps Step 1 Use the show vsan membership command to see all the ports connected to your host and storage and verify that both devices are in t...

Page 225: ... to add the VSAN to the allowed VSAN list for the interface that connects the host and storage devices E Port Is Isolated in a VSAN Symptom E port is isolated in a VSAN Table 11 2 xE Port Is Isolated in a VSAN Symptom Possible Cause Solution E port is isolated in a VSAN E port connecting to the remote switch is isolated Verify the VSAN See the Resolving an Isolated E Port Using Fabric Manager sect...

Page 226: ... is 20 44 00 05 30 00 63 5e vsan is 4 Beacon is turned off 30 frames input 682 bytes 0 discards 0 runts 0 jabber 0 too long 0 too short 0 input errors 0 CRC 0 invalid transmission words 0 address id 0 delimiter 0 EOF abort 0 fragmented 0 unknown class 30 frames output 583 bytes 0 discards Received 2 OLS 2 LRR 2 NOS 5 loop inits Transmitted 5 OLS 3 LRR 2 NOS 4 loop inits Step 2 Use the show vsan me...

Page 227: ... field to correct the VSAN misconfiguration problems Step 4 Repeat this procedure for all isolated VSANs on this TE port Resolving an Isolated ISL Using the CLI Trunking E ports TE ports are similar to E ports except that they carry traffic for multiple VSANs E ports carry traffic for a single VSAN Because TE ports carry traffic for multiple VSANs ISL isolation can affect one or more VSANs For thi...

Page 228: ...iplier default rxbufsize 2112 encap default user_cfg_flag 0x3 description Hw Capabilities 0xb trunk vsans up 7 trunk vsans isolated 1 8 TE port per vsan information fc2 29 Vsan 1 state down state reason Isolation due to domain other side eport isolated fcid 0x000000 port init flag 0x10000 current state TE_FSM_ST_ISOLATED_DM_ZS fc2 29 Vsan 7 state up state reason None fcid 0x690202 port init flag 0...

Page 229: ...e steps Step 1 Use the show fctimer command to verify that the fabric timers are inconsistent across the VSANs Step 2 Use the fctimer distribute command to enable CFS distribution for the fabric timers Repeat this on all switches in this VSAN Step 3 Use the fctimer command to set each timer Step 4 Use the fctimer commit command to save these changes and distribute them to all switches in the VSAN ...

Page 230: ...abric By default this feature is disabled in all switches in the Cisco MDS 9000 Family For more information on enabling DPVM refer to one of the following guides Cisco MDS 9000 Family Fabric Manager Configuration Guide Cisco MDS 9000 Family CLI Configuration Guide This section contains the following topics Troubleshooting DPVM Using Fabric Manager page 11 10 Troubleshooting DPVM Using the CLI page...

Page 231: ...nfig mode to enable CFS distribution if required Step 2 Use the show dpvm status command in EXEC mode to verify that autolearning is disabled Optionally use the no dpvm auto learn command in config mode if you need to disable autolearning before activating the database Step 3 Use the show dpvm pending diff command in EXEC mode to compare the active and pending databases Optionally use the dpvm com...

Page 232: ... DPVM distribution is enabled you must do an explicit commit for DPVM activate and autolearn to take effect Table 11 4 DPVM Database Not Distributed Symptom Possible Cause Solution DPVM databases are not distributed DPVM distribution has not been enabled on the local switch Choose Fabricxx All VSANs DPVM and select the CFS tab Check the Global field in Fabric Manager or use the show dpvm status CL...

Page 233: ...eck box in Fabric Manager and click Apply Changes or use the dpvm auto learn enable and dpvm commit CLI commands to enable autolearning Port type is not supported Verify that the device you want to autolearn is connected to an F port DPVM does not support FL TE FCIP or PortChannels Table 11 7 VSAN Membership Not Added to Database Symptom Possible Cause Solution The VSAN membership of the port is n...

Page 234: ...he active and config databases Use the dpvm database diff active conf CLI command Override the active database with the config database Choose Fabricxx All VSANs DPVM and select the Actions tab in Fabric Manager Set the Actions drop down menu to forceActivate and click Apply Changes Or use the dpvm activate force and dpvm commit CLI commands to Table 11 9 Cannot Copy Active to Config DPVM Database...

Page 235: ...t was operational goes into suspended or disabled state after DPVM database activation DPVM database maps a connected device to a nonexistent VSAN Choose Switches Interfaces FC Physical in Fabric Manager or use the show interface CLI command to check the interface status for a dynamic VSAN related failure Create the VSAN or map the device to another VSAN Table 11 11 DPVM Merge Failed Symptom Possi...

Page 236: ... this entire process Step 1 Use the no dpvm activate command in config mode to deactivate the DPVM database Step 2 Use the dpvm commit command in config mode to commit the changes to the config database Step 3 Use the no dpvm enable command in config mode to disable DPVM on the switch Table 11 12 DPVM Service Failure Symptom Possible Cause Solution DPVM process may terminate causing a possible swi...

Page 237: ...e two switches The lower the value of the WWN the higher the switch priority When merging two fabrics the administrator can expect the following behavior In Cisco SAN OS Release 2 1 1a and later releases when connecting a single switch fabric to a multi switch fabric a build fabric BF occurs and the switch with the better priority becomes the principal switch In earlier releases when connecting a ...

Page 238: ...ssign a dynamic domain ID after a fabric reconfiguration see the Using Fabric Reconfiguration for Domain ID Assignments section on page 11 20 You may see the following system message in the message log when a domain ID overlap occurs Error Message PORT 5 IF_DOWN_DOMAIN_OVERLAP_ISOLATION Interface chars is down Isolation due to domain overlap Explanation The interface is isolated because of a domai...

Page 239: ...f from the fabric The preferred option has the switch request a specified domain ID If that ID is unavailable it will accept another ID Step 5 Set the Restart drop down menu to disruptive and click Apply Changes to restart the Domain Manager Note While the static option can be applied to runtime after a disruptive or nondisruptive restart the preferred option is applied to runtime only after a dis...

Page 240: ...omain ID for one of the overlapping domain IDs The static option tells the switch to request that particular domain ID If it does not get that particular address it will isolate itself from the fabric The preferred option has the switch request a specified domain ID If that ID is unavailable it will accept another ID Step 5 Use the fcdomain restart disruptive vsan command to restart the Domain Man...

Page 241: ...nges to restart the Domain Manager Using Fabric Reconfiguration for Domain ID Assignments with the CLI To use fabric reconfiguration to reassign domain IDs for a particular VSAN using the CLI follow these steps Step 1 Use the show fcdomain domain list command to determine if you have statically assigned domain IDs on the switches Step 2 If you have statically assigned domain IDs use the no fcdomai...

Page 242: ...ils Symptom Possible Cause Solution CFS distribution of domain ID list fails Configured domain ID in remote switch not present in domain ID list Add all domain IDs in the VSAN to the domain ID list Choose Fabricxx VSANxx Domain Manager Allowed and select the Allowed DomainIDs tab to view the current allowed domain ID list in Fabric Manager Choose Fabricxx VSANxx Domain Manager and select the Confi...

Page 243: ...the Configuration tab to view the existing domain IDs for this VSAN Choose Fabricxx VSANxx Domain Manager Allowed and select the Allowed DomainIDs tab to add any missing domain IDs and then click Apply Changes If CFS is enabled select the CFS tab and select commit from the ConfigAction drop down menu and click Apply Changes Or use the show fcdomain domain list to view the current allowed domain ID...

Page 244: ...y VSANs allows greater control over traffic within the fabric and higher utilization of the deployed fabric resources This section describes how to identify and resolve Fabric Shortest Path First FSFP problems It includes the following topics Troubleshooting FSPF page 11 24 Loss of Two Way Communication page 11 28 Troubleshooting FSPF Figure 11 1 shows a single VSAN topology Figure 11 1 Single VSA...

Page 245: ...ase Step 3 Choose FC Advanced FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and verify that the AdminStatus is up The Cost column shows the cost of the path out of the interface The Intervals column shows the configured FSPF timers for this interface which must match on both sides The State column shows the full or adjacent state if the interf...

Page 246: ...010000 1 1000 238 0x00010001 0x00010003 1 1000 1 The domain 1 view of the fabric topology 2 Domain 1 is owner of the LSR link state record 3 This is a 16 bit counter starting at 0x0000 incremented by one for each switch during flooding and by one for each second held in the database This field is used as a tie breaker if incarnation numbers are the same 4 This is a 32 bit value between 0x80000001 ...

Page 247: ...routes are available Note To issue commands with the internal keyword you must have an account that is a member of the network admin group switch1 show fspf internal route vsan 2 FSPF Unicast Routes VSAN Number Dest Domain Route Cost Next hops 1 0x01 1 1000 fc1 2 1 0xEF 239 1000 fc1 1 1 0xED 238 2000 fc1 1 fc1 2 This shows the total cost of all links The next hop 238 has two interfaces This indica...

Page 248: ...ch must match on both sides The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks The port is now ready to route frames Step 2 Repeat Step 1 to determine the value of the hello interval on the adjacent switch Step 3 Fill in the Hello field to change the hello interval and click Apply Table 11 17 Traffic Is not Being Routed...

Page 249: ... network admin group switch1 show fspf internal route vsan 1 FSPF Unicast Routes VSAN Number Dest Domain Route Cost Next hops 1 0xEF 239 1000 fc1 1 1 1 0xED 238 2000 fc1 1 1 0x01 1 3000 fc1 1 2 1 There is no second path to domain 238 through domain 1 switch 2 2 There is no direct path to domain 1 switch 2 traffic must travel through three ISLs This is based on the route cost column Step 4 Use the ...

Page 250: ...he State column The Intervals column shows the configured FSPF timers for this interface which must match on both sides The State column shows the full or adjacent state if the interface has sent and received all database exchanges and required Acks The port is now ready to route frames Step 2 Repeat Step 1 to determine the value of the retransmit interval on the adjacent switch Step 3 Fill in the...

Page 251: ...sing Fabric Manager follow these steps Step 1 Choose FC Advanced FSPF and select the Interfaces tab to verify that the FSPF parameters are correct for each interface and check the Dead interval column and the State column The Intervals column shows the configured FSPF timers for this interface which must match on both sides The State column shows the full or adjacent state if the interface has sen...

Page 252: ...k the neighbor configuration 2 FSPF is not in full state which indicates a problem Step 4 Use the interface comma nd and then the fspf dead interval command in interface mode to change the dead interval Resolving a Region Mismatch Using Fabric Manager To identify a region mismatch problem on a switch using Fabric Manager follow these steps Step 1 Choose FC Advanced FSPF and select the General tab ...

Page 253: ... change INIT INIT Jan 5 00 39 45 fspf Interface fc1 2 in VSAN 1 Event INACTIVITY State change INIT INIT 2 1 The neighbor switch advertising region is 0 2 FSPF is in init state for each ISL Tip We recommend that you open a second Telnet or SSH session before entering any debug commands If the debug output overwhelms the current session you can use the second session to enter the undebug all command...

Page 254: ... m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 11 34 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 11 Troubleshooting VSANs Domains and FSPF FSPF Issues ...

Page 255: ...er to a replacement disk array and it also minimizes downtime when replacing host bus adapters HBAs or when re hosting an application on a different server Troubleshooting SDV involves checking the configuration of virtual devices domain IDs and zone sets Configuration problems with SDV can prevent devices from communicating properly Note SDV is a distributed service and uses CFS Cisco Fabric Serv...

Page 256: ...ords a SDV virtualized device cannot be part of a IVR zone or zoneset Virtual device names should be unique across VSANs because they are registered with the device alias server which is unaware of VSANs For example if you have enabled SDV and have registered a name vt1 in both VSAN 1 and VSAN 2 then the device alias server cannot store both entries because they have the same name Ensure that the ...

Page 257: ... is being virtualized The Cisco MDS 9124 Fabric Switch is not a rewrite capable switch In other words SDV does not work when real devices and primary virtual devices are connected to the same Cisco MDS 9124 Fabric Switch Caution When restoring a configuration file from an ASCII file for example when you issue the copy bootflash saved config running config command ensure that the pWWNs in the devic...

Page 258: ...h device alias because the device alias database is locked Clear the device alias CFS session and then reconfigure the SDV device Failure to register with device alias because the same name is already in use Use a different name CFS distribution failure Check CFS commands to determine the cause of failure The session was created and is locked and in use by another user Use the same user credential...

Page 259: ... SDV enabled switch Enable SDV on those switches Rewrite entries are not programmed correctly To confirm that the ACL rewrite and capture entries are programmed correctly for the host and virtual device enter the show sdv internal acl cache dump vsan command on switches where the host and virtual devices are connected If the host or virtual device is connected to the Cisco MDS 9124 Fabric Switch c...

Page 260: ...ter the show zone internal sdv table command to view the physical virtual mapping maintained in the zone server Devices should be zoned so that they cannot communicate to both the real device and its virtual component Table 12 5 SDV Merge Fails When ISL Comes Up Symptom Possible Cause Solution SDV merge fails when ISL comes up Configuration mismatch in the merging fabrics To forcefully recover fro...

Page 261: ...e shared across VSANs without compromising other VSAN benefits Troubleshooting IVR involves checking the configuration of domain IDs VSANs border switches and zone sets Configuration problems with IVR can prevent devices from communicating properly Prior to Cisco MDS SAN OS Release 2 1 1a IVR required unique domain IDs for all switches in the fabric As of Cisco MDS SAN OS Release 2 1 1a you can en...

Page 262: ... SAN OS Release 1 3 1 or higher A border switch must be a member of two or more VSANs A border switch that facilities IVR communications must be IVR enabled For redundant paths between active IVR zone members IVR can optionally be enabled on additional border switches The VSAN topology configuration must be updated before a border switch is added or removed Limitations and Restrictions The followi...

Page 263: ...R zone members 20 000 IVR zone members per physical fabric as of Cisco SAN OS Release 3 0 3 10 000 IVR zone members per physical fabric prior to Cisco SAN OS Release 3 0 3 IVR zones 8000 IVR zones per physical fabric as of Cisco SAN OS Release 3 0 3 2000 IVR zones per physical fabric prior to Cisco SAN OS Release 3 0 3 IVR zone sets 32 IVR zone sets per physical fabric Checklist Check off Verify l...

Page 264: ...figuration Table 13 1 CLI Commands for Verification of IVR CLI Command Description show fcdomain domain list Verifies unique domain ID assignment If a domain overlap exists edit and verify the allowed domains list or manually configure static non overlapping domains for each participating switch and VSAN show interface brief Verifies if the ports are operational VSAN membership and other configura...

Page 265: ...stics global data Show ivr global data mem stats Show memory statistics nhvsan change Show ivr fcid rewrite fsm internals plogi captured list Show ivr PLOGI captured pnat Show IVR payload NAT internal information pvm Show IVR PV Master internal information tu fsm Show TU FSM internal debug information vdri fsm Show VDRI FSM internal debug information virtual domains Show IVR capability fsm interna...

Page 266: ...S Session page 13 13 CFS Merge Failed page 13 14 IVR allows device discovery across VSANs IVR also supports FC ping and FC traceroute across VSANs using the following criteria Either FC ID or pWWN can be used Must be initiated from a switch with an active IVR zone member IVR Licensing Issues To use IVR you must obtain the correct licenses for the IVR features you are using and install those licens...

Page 267: ...r IVR IVR Feature Chassis or Module Type License Required Number of Licenses IVR over Fibre Channel and IVR NAT over Fibre Channel All ENTERPRISE_PKG One per IVR enabled chassis IVR over FCIP MDS 9216i 1 1 Cisco MDS 9216i enables the SAN_EXTENSION features without a license for the two Gigabit Ethernet ports on the integrated supervisor card None None MPS 14 2 SAN_EXTN_OVER_IPS2 One per module run...

Page 268: ...VZ_ACTIVATION_FAILED Inter VSAN zoneset chars activation failed Explanation Inter VSAN zone set activation failed Recommended Action No action is required Introduced Cisco MDS SAN OS Release 1 3 1 Table 13 5 IVR NAT Fails Symptom Possible Cause Solution IVR NAT fails Internal message payload uses destination ID IVR NAT modifies the destination ID in the Fibre Channel header If this same destinatio...

Page 269: ...sWWN in the VSAN Only the inter VSAN IVR enabled switch with the lowest sWWN can add the IVR zones to the regular active zone set in a VSAN This switch is waiting until the IVR switch with the lowest sWWN adds the IVR zone and reactivates the zone set Recommended Action No action is required Introduced Cisco MDS SAN OS Release 2 0 1b Symptom IVR zone set activation fails Table 13 6 IVR Activation ...

Page 270: ...o topology Symptom Border switch fails Table 13 7 Border Switch Fails Symptom Possible Causes Solutions Border switch fails IVR topology incorrect Choose Fabricxx All VSANs IVR and select the Action tab in Fabric Manager Check the Auto Discover Topology check box and click Apply Changes Select the CFS tab and set ConfigAction to commit and click Apply Changes Or use the ivr vsan topology auto CLI ...

Page 271: ...anges Select the CFS tab and set ConfigAction to commit and click Apply Changes Or use the ivr virtual fcdomain add vsan ranges CLI command to add existing and future virtual domains to the domain list for the selected VSANs Repeat this on all edge VSANs Internal message payload uses destination ID See the IVR Network Address Translation Fails section on page 13 8 Devices are in different IVR serv...

Page 272: ...main overlap Use the ivr widthdraw domain CLI command to remove the overlapped domain Use persistent FC IDs to reassign the overlapped domain Use the ivr virtual fcdomain add vsan ranges CLI command to add existing and future virtual domains to the domain list for the selected VSANs Repeat this on all edge VSANs Internal message payload uses destination ID See the IVR Network Address Translation F...

Page 273: ...Chapter 2 Troubleshooting Installs Upgrades and Reboots Table 13 12 Host Does Not Have Write Access to Storage Symptom Possible Cause Solution Host does not have write access to storage Host is a member of a read only zone If a host is a member of a read only zone the host has no write access to any IVR zone it may be a member of Remove the host from the read only zone Table 13 13 Locked IVR CFS S...

Page 274: ... the peer merges with this switch The CFS merge may fail if the configuration at the lost peer conflicts with the changes made in this session Also IVR auto topology could be out of sync with this peer We recommend that you discard this CFS session using ivr abort command and then re enter the configuration changes You can alternatively use Fabric Manager and or Device Manager instead of the comma...

Page 275: ...mmits page 13 17 Error Fabric Is Changing Please Retry the Request Later page 13 17 Table 13 14 CFS Merge Failed Symptom Possible Cause Solution CFS merge failed IVR topology incorrect Choose Fabricxx All VSANs IVR and select the Action tab in Fabric Manager Check the Auto Discover Topology check box and click Apply Changes Select the CFS tab and set ConfigAction to commit and click Apply Changes ...

Page 276: ... Troubleshooting Installs Upgrades and Reboots One or more switches in the fabric cannot communicate with Fabric Manager or are not Cisco SAN OS switches Determine if any of the problem switches are required in the IVR topology If not ignore this message and proceed with the IVR configuration If they are required choose Switches and check the Status column to determine the cause and address the pr...

Page 277: ...sco SAN OS Table 13 17 Pending Action Pending Commits Symptom Possible Cause Solution Pending action on pending commit error displays A separate IVR configuration change that was not committed IVR has pending changes that were not committed Choose Fabricxx All VSANS IVR and select the CFS tab in Fabric Manager Set the View Config As drop down menu to pending and verify the pending configuration ch...

Page 278: ... m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 13 18 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 13 Troubleshooting IVR Troubleshooting the IVR Wizard ...

Page 279: ...rity and prevents data loss or corruption Zone sets consist of one or more zones in a VSAN A zone set can be activated or deactivated as a single entity across all switches in the fabric but only one zone set can be activated at any time in a VSAN Zones can be members of more than one zone set A zone consists of multiple zone members Members in a zone can access each other members in different zon...

Page 280: ... Edit Local Full Zone Database dialog box for the selected VSAN Step 3 Click Zones in the left pane The right pane lists the members for each zone Note The active zone set appears in bold If there is no zone set in bold you have not activated a zone set for this VSAN Troubleshooting Zone Configuration Issues with the CLI Much of the information accessed and summarized using Fabric Manager can also...

Page 281: ...t updated by Local CLI Num zonesets 1 Num zones 1 Num aliases 0 Num attribute groups 0 Formattted size 36 bytes 2048 Kb Unassigned Zones 1 zone name z1 vsan 1 Example 14 2 Active Zoning Database Analysis switch show zone analysis active vsan 1 Zoning database analysis vsan 1 Active zoneset zs1 Activated at 08 03 35 UTC Nov 17 2005 Activated by Local GS Default zone policy Deny Number of devices zo...

Page 282: ... a debug command use the no form of the command or use the no debug all command to turn off all debugging For protocol errors use debug zone change errors vsan id For protocol events use debug zone change events vsan id For protocol packets use debug zone change packets vsan id Other useful debug commands include debug zone all change errors events packets database detail errors events gs errors e...

Page 283: ...zone policy to permit if you are not using zoning Got to Step 8 Step 4 Choose Zone Edit Local Full Zone Database and select the VSAN you are interested in Click on the zones folder and verify that the host and storage are both members of the same zone If they are not in the same zone see the Resolving Host and Storage Not in the Same Zone Using Fabric Manager section on page 14 6 Step 5 Choose Zon...

Page 284: ...N by selecting the zone set name that appears in bold If you do not have an active zone set right click on the zone set you want to activate in the Edit Local Full Zone Database dialog box and select Activate to activate the zone set Step 5 Expand the active zone set folder to verify that the zone in Step 3 is in the active zone set If it is not see the Resolving Zone is Not in Active Zone Set Usi...

Page 285: ...w zoneset active command to determine if the zone in Step 4 and the host and disk appear in the active zone set v_188 show zoneset active vsan 2 zoneset name ZoneSet3 vsan 2 zone name Zone5 vsan 2 pwwn 10 00 00 00 77 99 7a 1b Hostalias pwwn 21 21 21 21 21 21 21 21 Diskalias Step 6 If the zone is not in the active zone set see the Resolving Zone is Not in Active Zone Set Using Fabric Manager sectio...

Page 286: ...ctivate the zone set switch config zoneset activate ZoneSet1 vsan 2 Step 6 Verify that the host and storage can now communicate Resolving Zone is Not in Active Zone Set Using the CLI To add a zone to the active zone set using the CLI follow these steps Step 1 Use the show zoneset active command to verify that you have an active zone set If you do not have an active zone set use the zoneset activat...

Page 287: ... is required If this message has the reason FC2 sequence size exceeded then the zone database size has been exceeded You must simplify the zone configuration or if full zone set distribution is enabled then disable full zone set distribution and activate the zone set Error Message ZONE 2 ZS_CHANGE_ACTIVATION_FAILED_RESN_DOM Activation failed reason chars domain dec Explanation The zone server cann...

Page 288: ...VR zones 0 Number of IPS zones 0 Formattted size 38 bytes 2048 Kb Step 2 Use the show zone analysis vsan vsan id command to analyze the full zone set database Verify that the formatted size does not exceed the 2048 KB limit shown If it exceeds the limit you must remove some zones or devices within a zone switch show zone analysis vsan 1 Zoning database analysis vsan 1 Full zoning database Last upd...

Page 289: ... set Resolving Out of Sync Full Zone Database Using Fabric Manager To verify if the full zone database is in sync across switches using Fabric Manager follow these steps Step 1 Choose Fabricxx VSANxx zonesetname and select the Policies tab Step 2 Verify that the Propagation field is set to FullZoneSet If it is not select FullZoneSet from the drop down menu Step 3 Click Apply Changes to save these ...

Page 290: ...ick Apply Changes to save these changes Step 4 If you are using basic zoning Select the same value from the Default Zone Behavior drop down menu for each switch in the VSAN to set the same default zone policy Step 5 If you are using enhanced zoning follow these steps a Choose Fabricxx VSANxx and view the Release field to verify that all switches are capable of working in the enhanced mode All swit...

Page 291: ...efault zone command on each switch in the VSAN to set the same default zone policy Step 3 If you are using enhanced zoning follow these steps a Use the show version command on all switches in the VSAN to verify that all switches are capable of working in the enhanced mode All switches must have Cisco MDS SAN OS Release 2 0 1b or later If one or more switches are not capable of working in enhanced ...

Page 292: ...failure Isolating interface chars Explanation Interface isolated because of a zone merge failure Recommended Action Compare active zoneset with the adjacent switch or enter the zone merge interface CLI command or similar Fabric Manager Device Manager command Introduced Cisco MDS SAN OS Release 1 2 2a Error Message ZONE 2 ZS_MERGE_FULL_DATABASE_MISMATCH Zone merge full database mismatch on interfac...

Page 293: ...ng switch Manually resolve the conflict by editing the full zone set activating the corrected zone set and then bringing up the link If after verifying the Fibre Channel name server you still experience FSPF problems such as discovering remote switches and their attached resources the fabric may have zone configuration problems Examples of zone configuration problems are mismatched active zone set...

Page 294: ...n class 79 frames output 1234 bytes 16777216 discards Received 23 OLS 14 LRR 13 NOS 39 loop inits Transmitted 50 OLS 16 LRR 21 NOS 25 loop inits An E port is segmented isolation due to zone merge failure if the following conditions are true The active zone sets on the two switches differ from each other in terms of zone membership provided there are zones at either side with identical names The ac...

Page 295: ...ve exactly the same members If either of these conditions is violated the E port connecting the two fabrics will appear in an isolated state For example two switches may have the same zone set name and the same zone names but different zone members As a result the VSAN is isolated on the TE port that connects the two switches This issue can be resolved by doing one of the following Modify the zone...

Page 296: ...set active vsan id command to display the active zone set configuration of the first switch Switch1 show zoneset active vsan 99 zoneset name ZoneSet1 vsan 99 zone name VZ1 vsan 99 fcid 0x7800e2 pwwn 22 00 00 20 37 04 ea 2b fcid 0x7800d9 pwwn 22 00 00 20 37 04 f8 a1 Step 2 Use the show zoneset active vsan id command to display the active zone set configuration of the second switch Switch2 show zone...

Page 297: ... init flag 0x38000 current state TE_FSM_ST_E_PORT_UP fc2 29 Vsan 99 state down state reason Isolation due to zone merge failure fcid 0x000000 port init flag 0x0 current state TE_FSM_ST_ISOLATED_VSAN_MISMATCH From this output you can see the VSAN is isolated because of o a zone merge failure Step 5 Do one of the following to resolve the isolation problem Change the membership of one of the zones to...

Page 298: ...ne Step 3 Choose Interfaces FC Physical and select up from the Status Admin drop down menu to enable the connection to the zone to be merged You may see the following system messages Nov 19 10 28 11 switch4 LOG_PORT_CHANNEL 5 FOP_CHAN GED port channel 1 first operational port changed from none to fc1 15 Nov 19 10 28 21 switch4 LOG_PORT 5 IF_UP Interface port channel 1 is up in mode TE Nov 19 10 28...

Page 299: ...OP_CHAN GED port channel 1 first operational port changed from none to fc1 15 Nov 19 10 28 21 switch4 LOG_PORT 5 IF_UP Interface port channel 1 is up in mode TE Nov 19 10 28 21 switch4 LOG_PORT 5 IF_TRUNK_UP Interface fc1 14 vsan 1 is up Nov 19 10 28 21 switch4 LOG_PORT 5 IF_TRUNK_UP Interface fc1 15 vsan 1 is up Nov 19 10 28 21 switch4 LOG_PORT 5 IF_TRUNK_UP Interface fc1 16 vsan 1 is up Nov 19 1...

Page 300: ...ow zone internal change event history show zone status vsan show zone pending diff show zone pending vsan Symptom Cannot configure zoning Table 14 3 Cannot Configure Zoning Symptom Possible Causes Solutions Cannot configure zoning Another user on the same switch is holding the enhanced zoning configuration lock If you are using the CLI you see a message stating that another session is active See t...

Page 301: ...nced Zoning Lock Issues with the CLI To resolve a lock issue using the CLI follow these steps Step 1 Use the show zone status vsan command to determine the lock holder If the lock holder is on this switch the command output shows the user If the lock holder is on a remote switch the command output shows the domain ID of the remote switch switch show zone status vsan 16 VSAN 16 default zone deny di...

Page 302: ... n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 14 24 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 14 Troubleshooting Zones and Zone Sets Enhanced Zoning Issues ...

Page 303: ...curity you must specify the correct device name each time An inaccurate device name can cause unexpected results You can circumvent this problem by defining and using device aliases A device alias is a user friendly name for a port WWN pWWN that can be used in all configuration commands as required All switches in the Cisco MDS 9000 Family support Distributed Device Alias Services device aliases I...

Page 304: ...S 3 MERGE_FAILED Merge failed for app device alias local switch wwn 20 00 00 0d ec 2f c1 40 ip 172 20 150 38 remote switch wwn 20 00 00 0d ec 04 99 40 ip 172 20 150 30 2007 Apr 9 15 52 42 switch 1 DEVICE ALIAS 3 MERGE_FAILED Databases could not be merged due to mismatch Merge Validation Failure Messages When a device alias database fails validation during the merge process review the syslog messag...

Page 305: ...fy the device alias database status Limitations and Restrictions The following limitations and restrictions are associated with the use of device aliases Enhanced mode operation is only supported on switches running SAN OS Release 3 1 1 or later Interop mode VSANs do not accept enhanced mode native alias based configurations Table 15 1 CLI Commands for Troubleshooting Device Alias Issues CLI Comma...

Page 306: ...e the mapping so the pWWN is mapped to the same device alias name on both sides of the fabric See the Resolving Mapping a pWWN to Different Device Alias Names section on page 15 6 The device alias operation mode basic or enhanced is not the same for both fabrics Check the device alias mode Both fabrics must be operating in the same device alias mode See the Resolving Mode Mismatch section on page ...

Page 307: ...ches 1 Remote Fabric Switch WWN IP Address 20 00 00 0d ec 04 99 40 172 20 150 30 Merge Master switch 2 Total number of switches 1 Step 2 Verify that the reason for the merge failure is a database mismatch by using the show device alias merge status command switch 1 show device alias merge status Result Failure Reason Databases could not be merged due to mismatch Step 3 Identify the duplicate devic...

Page 308: ...witch 1 show device alias database device alias name A3 pwwn 21 01 01 01 01 01 01 02 Total number of entries 1 switch 2 show device alias database device alias name A1 pwwn 21 01 01 01 01 01 01 02 Total number of entries 1 Step 4 Make the appropriate changes to the device alias database for one of the fabrics Refer to the Cisco MDS 9000 Family CLI Configuration Guide for details Resolving Mode Mis...

Page 309: ... remove all the native device alias configurations or replace all the device alias members with the corresponding pWWNs Refer to the Cisco MDS 9000 Family CLI Configuration Guide for details Resolving Merge Failures in Mixed Fabric A mixed fabric is a fabric consisting of switches running different software versions Prior to SAN OS Release 3 1 x the limit for device aliases in a merged fabric was ...

Page 310: ...3 Examine syslog messages The syslog for the switch where the validation is rejected and the syslog for the switch managing the merge show relevant error messages Example 15 6 shows a sample message on the switch where the validation is rejected Example 15 6 Validation Rejection Syslog Message 2007 Apr 10 00 00 06 switch 2 DEVICE ALIAS 3 MERGE_VALIDATION_REJECTED Failed SAP 110 Reason inter VSAN z...

Page 311: ... defined on switch 1 Total number of entries 2 switch 2 show device alias database device alias name A1 pwwn 21 01 01 01 01 01 01 01 Pre merge A2 not defined on switch 2 Total number of entries 1 Because IVR is enabled on switch 2 review the IVR zone set switch 2 show ivr zoneset zoneset name s1 zone name z1 pwwn 21 01 01 01 01 01 01 02 vsan 1 autonomous fabric id 1 device alias A2 vsan 2 autonomo...

Page 312: ... up between two fabrics it typically triggers a CFS merge of multiple applications By the time the device alias merge moves to validation stage other applications may be busy handling their own merge This is a transient state and the device alias merge would be backed off and retired after the defined retry time If for any reason the application database remains locked the device alias merge remai...

Page 313: ...remains in progress switch 2 show cfs lock Application ivr Scope Physical fc Switch WWN IP Address User Name User Type 20 00 00 0d ec 04 99 40 172 20 150 30 admin CLI SNMP v3 switch 2 Total number of entries 1 Application device alias Scope Physical fc Switch WWN IP Address User Name User Type 20 00 00 0d ec 2f c1 40 172 20 150 38 CFS Merge Total number of entries 1 Step 4 Resolve the problem by u...

Page 314: ...0 Reason inter VSAN zone Switch and SAP member cannot be in more than one VSAN Expln Reason 2007 Apr 10 11 54 24 switch 1 DEVICE ALIAS 3 COMMIT_FAILED Failed to Commit status commit the pending database inter VSAN zone member cannot be in more Reason than one VSAN Step 3 Review the syslog on the switch where the validation is rejected In this example the following syslog is printed on the switch s...

Page 315: ... id 1 switch 1 conf t Enter configuration commands one per line End with CNTL Z switch 1 config device alias database switch 1 config device alias db device alias name A2 pwwn 21 01 01 01 01 01 01 02 switch 1 config device alias db exit switch 1 config device alias commit inter VSAN zone member cannot be in more than one VSAN Step 5 Correct the conflict by making adjustments to the application con...

Page 316: ...eleased you cannot commit the device alias database changes Resolving Database Size Issues The maximum size of the device alias database increased from 8K to 20K for switches running SAN OS Release 3 1 1 or later If the fabric includes switches running SAN OS Release 3 0 x or earlier and switches running Release 3 1 1 or later the maximum size of the device alias database is limited to 8K This is ...

Page 317: ... attempt to enable device alias enhanced mode fails Unless all of the switches in the fabric can be upgraded to Release 3 1 1 or later you cannot enable device alias enhanced mode If you attempt to enable enhanced mode in a successfully merged mixed fabric or to commit a device alias database that enables enhanced mode to resolve a merge failure in a mixed fabric the configuration or the commit wi...

Page 318: ... c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 15 16 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 15 Troubleshooting Distributed Device Alias Services Validation and Commit Failure Issues ...

Page 319: ...the Fibre Channel FICON iSCSI and FCIP capabilities within a single high availability platform Fibre Channel and FICON are different FC4 protocols and their traffic are independent of each other If required devices using these protocols can be isolated using VSANs The Cisco SAN OS FICON feature supports high availability scalability and SAN extension technologies including VSANs IVR FCIP and PortC...

Page 320: ...er A maximum of 255 port numbers are available You can use the following port numbering schemes Default port numbers based on the chassis type Reserved port numbers The following guidelines apply to FICON port numbers Supervisor modules do not have port number assignments Chassis wide unique port numbers should be reserved for trunking expansion TE ports since TE ports appear in multiple VSANs Eac...

Page 321: ...ot 2 32 through 63 Cisco MDS 9506 Director Slot 1 0 through 31 128 through 153 154 through 253 and port 255 Slot 2 32 through 63 Slot 3 64 through 95 Slot 4 96 through 127 Supervisor modules are not allocated port numbers Slot 5 None Slot 6 None Cisco MDS 9134 Director Slot 1 0 through 33 34 through 59 60 through 253 and port 255 Cisco MDS 9509 Director Slot 1 0 through 31 224 through 249 250 thro...

Page 322: ...ate port numbers if they are not used in the same FICON VSAN For example you can configure port number 1 on interface fc1 1 in FICON VSAN 10 and fc10 1 in FICON VSAN 20 Note A VSAN can have a maximum of 250 port numbers Note FICON port numbers are not changed for ports that are active You must first disable the interfaces using the shutdown CLI command Cisco MDS 9513 Director Slot 1 0 through 15 2...

Page 323: ...IPL This file is created with a default configuration immediately after FICON is enabled in a VSAN Multiple FICON configuration files with the same name can exist in the same switch provided they reside in different VSANs For example you can create a configuration file named XYZ in both VSAN 1 and VSAN 3 Caution When FICON is disabled on a VSAN all FICON configuration files are irretrievably lost ...

Page 324: ...witches from joining the fabric or disrupting current fabric operations It uses the Exchange Fabric Membership Data EFMD protocol to ensure that the list of authorized switches is identical in all switches in the fabric To enforce fabric binding configure the switch world wide name sWWN to specify the xE port connection for each switch Fabric binding policies are enforced on every activation and a...

Page 325: ...icensing requirements See Cisco MDS 9000 Family Fabric Manager Configuration Guide Verify that you enabled in order delivery for the FICON enabled VSAN Verify that you have enabled fabric binding in all switches in the FICON fabric Verify that all switches in the FICON fabric are in the fabric binding database Verify that you have assigned static domain IDs for all switches in the FICON fabric Ver...

Page 326: ...ger check FICON and click Apply to set the Device Manager display to use FICON port numbers Common Troubleshooting Commands in the CLI Use the following CLI commands to troubleshoot FICON issues show ficon show ficon vsan vsan id file ficon file name show ficon vsan vsan id portaddress show ficon vsan vsan id director history show fabric binding status show fabric binding database show fabric bind...

Page 327: ...ew port number after swap noshut command the ports are automatically initialized To swap physical Fibre Channel ports including the port numbers using the CLI follow these steps Step 1 Issue the ficon swap interface old interface new interface command in EXEC mode The specified interfaces are operationally shut down Step 2 Physically swap the front panel port cables between the two ports Step 3 Is...

Page 328: ... com en US products ps5989 produc ts_configuration_guide_chapter09186a0080662c89 ht ml For CLI http www cisco com en US products ps5989 produc ts_configuration_guide_chapter09186a0080664c67 ht ml Table 16 3 Switch ISL Isolated Symptom Possible Cause Solution Switch ISL is isolated Switch WWN is not configured in the fabric binding database Add the sWWN to the fabric binding database for all switch...

Page 329: ...FICON VSAN in the switch Choose Zone Edit Local Full Zone Database in Fabric Manager or use the member CLI command in zone submode to add the CUP to the appropriate zone Table 16 6 Cannot Enable FICON Port Symptom Possible Cause Solution Cannot enable FICON port Port is blocked Unblock the port Choose FICON VSANs in Device Manager highlight the VSAN and click Port Configuration Then unset the Bloc...

Page 330: ...lable for use with FCIP and or PortChannels Choose FICON Port Numbers in Device Manager Alternatively use the ficon logical port assign port numbers CLI command Table 16 8 FCIP Fails for FICON Symptom Possible Cause Solution FCIP fails for FICON FICON port address is not assigned to the FCIP tunnel Assign the FICON port address to FCIP tunnel and restart FCIP tunnel Use the show ficon port numbers...

Page 331: ...n the user ID and password combination provided switches perform local authentication or authorization using the local database or remote authentication or authorization using AAA server s A preshared secret key provides security for communication between the switch and AAA servers This secret key can be configured as a global key for all AAA servers or on a per AAA server basis This security mech...

Page 332: ...he CLI Use the following CLI commands to troubleshoot AAA issues show aaa authentication show user account show radius status show radius server show tacacs status show tacacs server Use the following debug commands to determine the root cause of an issue debug radius aaa request debug radius aaa request lowlevel debug tacacs aaa request and debug tacacs aaa request lowlevel AAA Issues This sectio...

Page 333: ...Fabric Manager section on page 17 4 or the Verifying RADIUS Configuration Using the CLI section on page 17 4 For TACACS servers see the Verifying TACACS Configuration Using Fabric Manager section on page 17 5 or the Verifying TACACS Configuration Using the CLI section on page 17 5 AAA server monitor deadtime set to high Set the deadtime lower to bring AAA servers active more quickly For RADIUS ser...

Page 334: ... the CLI To verify or change the RADIUS configuration using the CLI follow these steps Step 1 Use the show radius server command to display configured RADIUS parameters switch show radius server Global RADIUS shared secret retransmission count 5 timeout value 10 following RADIUS servers are configured myradius cisco users com available for authentication on port 1812 available for accounting on po...

Page 335: ...tab and select commit from the Config Action drop down menu and click Apply Changes to distribute these changes to all switches in the fabric Verifying TACACS Configuration Using the CLI To verify or change the TACACS configuration using the CLI follow these steps Step 1 Use the show tacacs server command to display configured TACACS parameters switch show tacacs server Global TACACS shared secret...

Page 336: ...ig Action drop down menu and click Apply Changes to distribute these changes to all switches in the fabric Step 9 Choose Switches Security AAA and click Create Row to create a server group Step 10 Check the list of switches that you want to configure server groups on Step 11 Set the Server List field to a comma separated list of RADIUS servers Step 12 Set the Deadtime field to configure the time t...

Page 337: ... select commit from the Config Action drop down menu and click Apply Changes to distribute these changes to all switches in the fabric Step 9 Choose Switches Security AAA and click Create Row to create a server group Step 10 Check the list of switches that you want to configure server groups on Step 11 Set the Server List field to a comma separated list of TACACS servers Step 12 Set the Deadtime f...

Page 338: ...n port configured or incorrect server timeout value Reconfigure the authentication port to match those configured on the AAA server or set a higher timeout value For RADIUS servers see the Verifying RADIUS Configuration Using Fabric Manager section on page 17 4 or the Verifying RADIUS Configuration Using the CLI section on page 17 4 For TACACS servers see the Verifying TACACS Configuration Using F...

Page 339: ...CLI follow these steps Step 1 Use the show running config command to view the RADIUS configuration for the server groups switch show running config begin aaa aaa group server radius RadiusGroup server 10 1 1 1 server 10 2 3 4 aaa group server tacacs TacacsGroup server 11 5 4 3 server 11 6 5 4 Step 2 Use the aaa group server radius command to configure the RADIUS servers that you want in this serve...

Page 340: ...5 4 3 server 11 6 5 4 Step 2 Use the aaa group server tacacs command to configure the TACACS servers that you want in this server group Note CFS does not distribute AAA server groups You must copy this configuration to all relevant switches in the fabric User Is Not in Any Configured Role Symptom User is not in any configured role Table 17 3 User Is Not In Any Configured Role Symptom Possible Caus...

Page 341: ...ir exists on the Cisco SAN OS switch Step 4 If the Cisco IOS PIX RADIUS Attributes field is not present follow these steps a Choose Interface RADIUS Cisco IOS PIX b Check the User and Group check boxes for the cisco av pair option and click Submit c Choose User Setup User Data Configuration and add the AV pair to assign the correct role to each user Step 5 Choose System Configuration Logging to ac...

Page 342: ...0 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 17 Troubleshooting RADIUS and TACACS Troubleshooting RADIUS and TACACS With Cisco ACS Refer to the User guide for Cisco Secure ACS at the following website for more information http cisco com en US products sw secursw ps2086 products_user_guide_list html ...

Page 343: ...can access the switch using SNMP for example Fabric Manager or Device Manager and vice versa User Accounts Every Cisco MDS 9000 Family switch user has the account information stored by the system You can add up to 256 users to a switch The authentication information user name user password password expiration date and role membership are stored in the user profile The most important aspect of a us...

Page 344: ...e based authorization limits access to switch operations by assigning users to roles This kind of authentication restricts users to management operations based on the roles to which they have been assigned the user When you execute a command perform command completion or obtain context sensitive help the switch software allows the operation to progress if you have permission to access that switch ...

Page 345: ...min role The rule command specifies operations that can be performed by a specific role Each rule consists of a rule number a rule type permit or deny a command type for example config clear show exec debug and an optional feature name for example FSPF zone VSAN fcping or interface Note In this case exec commands refer to all commands in the EXEC mode that do not fall in the show debug and clear c...

Page 346: ...les for this role Note Rules can only be configured from Device Manager Common Troubleshooting Commands in the CLI Use the following CLI commands to troubleshoot user and role issues show users show user account show role show role status show role session status User and Role Issues This section describes troubleshooting user and role issues and includes the following topics User Cannot Log into ...

Page 347: ...ch follow these messages Step 1 Choose Logs Syslog Setup and select the Severity Levels tab Step 2 Select debug from the Severity Level drop down menu for auth authPriv and aaad Click Apply This sets the switch to log debug information for these facilities Table 18 1 User Cannot Log into Switch Symptom Possible Cause Solution User cannot log into the switch Weak password configured at the AAA serv...

Page 348: ...7 SYSTEM_MSG PAM login updating snmpv3 US M for user testUser 2006 Mar 3 03 08 53 v_190 AUTHPRIV 7 SYSTEM_MSG PAM login snmpv3 attribute v alue null 2006 Mar 3 03 08 53 v_190 AUTHPRIV 7 SYSTEM_MSG PAM login updating snmpv3 US M success for user testUser 2006 Mar 3 03 08 53 v_190 AUTH 6 SYSTEM_MSG login session opened for user t estFoo by uid 0 2006 Mar 3 03 08 53 v_190 AAA 6 AAA_ACCOUNTING_MESSAGE...

Page 349: ... uid 0 2006 Mar 3 03 08 53 v_190 AAA 6 AAA_ACCOUNTING_MESSAGE start dev pts 1_161 4 4 67 125 testUser User Cannot Create Roles Symptom User cannot create roles User Cannot Create Other Users With Fabric Manager or Device Manager Symptom User cannot create other users with Fabric Manager or Device Manager Table 18 2 User Cannot Create Roles Symptom Possible Cause Solution User cannot create roles U...

Page 350: ... Create to create the user Step 6 Choose Security Roles to view the roles Step 7 Right click a role and select Rules to view or modify the rules assigned to a role Step 8 Check the feature check boxes for the features that you want this role to access and click Apply to save these changes Table 18 4 User Cannot Access Certain Features Symptom Possible Cause Solution User cannot access certain feat...

Page 351: ...o expiry date roles sangroup vsan admin no password set local login not allowed Remote login through RADIUS is possible Step 2 Use the username command to modify the roles assigned to a user switch no username user1 role vsan admin Step 3 Use the show role command to view the rules assigned to the role switch show role sangroup Role sangroup Description SAN management group vsan policy permit Rule...

Page 352: ...8 5 User Has Too Much Access Symptom Possible Cause Solution User has too much access User is assigned incorrect role or overlapping roles For RADIUS configure the vendor specific attributes on the server for the role using Cisco AVPair shell roles rolename For TACACS configure the attribute and value pair on the server for the role using roles vsan admin storage admin See the Verifying Roles Usin...

Page 353: ...sangroup vsan admin no password set local login not allowed Remote login through RADIUS is possible Step 2 Use the show role command to view the rules assigned to the role switch show role sangroup Role sangroup Description SAN management group vsan policy deny Permitted vsans 10 30 Rule Type Command type Feature 1 permit config 2 deny config fspf 3 permit debug zone 4 permit exec fcping Step 3 Us...

Page 354: ...field is not present follow these steps a Choose Interface RADIUS Cisco IOS PIX b Check the User and Group check boxes for the cisco av pair option and click Submit c Choose User Setup User Data Configuration and add the AV pair to assign the correct role to each user Step 5 Choose System Configuration Logging to activate logs to look for reasons for failed authentication attempts Step 6 Choose Re...

Page 355: ...C SP Overview page 19 1 Port Security Overview page 19 2 Fabric Binding Overview page 19 2 Initial Troubleshooting Checklist page 19 2 FC SP Issues page 19 4 Port Security Issues page 19 7 Fabric Binding Issues page 19 15 FC SP Overview FC SP capabilities provide switch switch and host switch authentication to overcome security challenges for enterprise wide fabrics Diffie Hellman Challenge Handsh...

Page 356: ... the fabric Domain IDs are mandatory for FICON based fabric binding and optional for non FICON based fabric binding For non FICON based fabric binding not specifying a domain ID means that the switch with the matching WWN can login with any domain ID Initial Troubleshooting Checklist Begin troubleshooting FC SP issues by checking the following issues Begin troubleshooting port security issues by c...

Page 357: ...roubleshoot FC SP issues show fcsp interface show fcsp internal event history errors show fcsp dhchap show fcsp dhchap database Us e the following CLI commands to troubleshoot port security issues show port security status show port security database vsan show port security database active vsan show port security violations show port security internal global show port security internal info vsan s...

Page 358: ... enable CLI command on all switches in your fabric Local switch FC SP password does not match remote password Choose Switches Security FC SP select the General Password tab and set the GenericPassword field in Fabric Manager Or use the fcsp dhchap password CLI command to set the local switch password FC SP DHCHAP configuration does not match remote switch or host See the Verifying FC SP Configurat...

Page 359: ...c sp authentication TOV 30 Step 2 Use the fcsp timeout command to modify the timeout value switch config fcsp timeout 60 Step 3 Use the show fcsp dhchap command to view the hash algorithm and group switch show fcsp dhchap Supported Hash algorithms in order of preference DHCHAP_HASH_MD5 DHCHAP_HASH_SHA_1 Supported Diffie Hellman group ids in order of preference DHCHAP_GROUP_1536 Step 4 Use the fcsp...

Page 360: ... Local Password Non device specific password Password for device with WWN 29 11 bb cc dd 33 11 22 is Password for device with WWN 30 11 bb cc dd 33 11 22 is Other Devices Passwords Password for device with WWN 00 11 22 33 44 aa bb cc is Step 2 Use the show wwn switch command on the switch that you want to add to the FC SP local database to find the sWWN MDS 9216 show wwn switch Switch WWN is 20 00...

Page 361: ...ic page 19 12 Port Security Settings Lost After Reboot page 19 13 Merge Fails page 19 14 Note After correcting a port security configuration issue you do not have to disable the interface and reenable it The port comes up automatically after a port security reactivation if the problem was fixed Table 19 2 Authentication Fails When Using Cisco ACS Symptom Possible Cause Solution Authentication fail...

Page 362: ...Autolearn Using the CLI section on page 19 15 Device is configured for some other port Manually add the device to the configured port security database See the Verifying the Active Port Security Database Using Fabric Manager section on page 19 9 or the Verifying the Active Port Security Database Using the CLI section on page 19 9 Port is shut down because of port security violation Remove the devi...

Page 363: ...ges to copy the configure database to the active database and reactivate port security Step 8 Select the CFS tab if CFS is enabled and select commit from the ConfigAction drop down menu to distribute these changes to all switches in the fabric Verifying the Active Port Security Database Using the CLI To verify the active port security database using the CLI follow these steps Step 1 Use the show p...

Page 364: ...he Actions tab b Check the CopyActive to Config check box and click Apply Changes to copy the active database to the configure database This ensures that no learned entries are lost c Select the CFS tab if CFS is enabled and select commit from the ConfigAction drop down menu to distribute these changes to all switches in the fabric d Select the Config Database tab and click Add Row to add a new en...

Page 365: ... 2 port channel 1 20 00 00 05 30 00 95 de swwn Jul 9 08 32 40 2003 1 Total 2 entries In this example pWWN 21 00 00 e0 8b 06 d9 1d is causing interface fc1 13 to be shut down because of port security violations Step 2 Optionally follow these steps to add the device to the port security database a Use the port security database copy command to copy the active database to the configure database This ...

Page 366: ... Database Using the CLI section on page 19 9 Configure database is empty Choose Fabricxx VSANxx Port Security select the Actions tab heck the CopyActive to Config check box and click Apply Changes in Fabric Manager to copy the active database to the configure database Or use the port security database copy CLI command Not all members of a PortChannel are configured for port security Add the missin...

Page 367: ...g Autolearn Using the CLI To disable autolearn using the CLI follow these steps Step 1 Use the no port security auto learn command to disable autolearn switch no port security auto learn vsan 2 Step 2 Use the port security database copy command to copy the active database to the configure database This ensures that no learned entries are lost switch port security database copy vsan 2 Step 3 If CFS...

Page 368: ... Step 8 Uncheck the AutoLearn check box and click Apply Changes to disable autolearn after all entries are learned Step 9 Select the CFS tab and select commit from the ConfigAction drop down menu to distribute these changes to all switches in the fabric Step 10 Check the CopyActive to Config check box and click Apply Changes to copy the active database to the configure database This ensures that n...

Page 369: ...curity auto learn vsan 2 Step 6 If CFS distribution is enabled use the port security commit command to distribute these changes switch config port security commit vsan 2 Step 7 Use the port security database copy command to copy the active database to the configure database This ensures that no learned entries are lost switch port security database copy vsan 2 Step 8 If CFS distribution is enabled...

Page 370: ...pply Changes to copy the configure database to the active database and activate fabric binding Or use the fabric binding activate CLI command sWWN not present in fabric binding database Add sWWN to fabric binding database See the Verifying Fabric Binding Violations Using Fabric Manager section on page 19 16 or the Verifying Fabric Binding Violations Using the CLI section on page 19 17 Fabric bindi...

Page 371: ... security violations command and search for the interface that is shut down switch show fabric binding violations VSAN Switch WWN domain Last Time Repeat count Reason 2 20 00 00 05 30 00 4a 1e Nov 25 05 44 58 2003 2 sWWN not found 3 20 00 00 05 30 00 4a 1e 0xeb Nov 25 05 46 14 2003 2 Domain mismatch 4 20 00 00 05 30 00 4a 1e Nov 25 05 46 25 2003 1 Database mismatch In VSAN 2 the sWWN itself was no...

Page 372: ...ic Binding Database Using the CLI To verify the config fabric binding database using the CLI follow these steps Step 1 Use the show fabric binding database active command to view the active entries in the database Step 2 Use the fabric binding database copy command to copy the active database to the configure database switch fabric binding database copy vsan 1 Step 3 Use the fabric binding databas...

Page 373: ...5 Select the Actions tab select activate from the Action drop down menu and click Apply Changes to copy the configure database to the active database and reactivate fabric binding Step 6 Copy the running configuration to the startup configuration using the fabric option This saves the port security configure database to the startup configuration on all switches in the fabric Table 19 11 Unauthoriz...

Page 374: ...e Step 2 Use the fabric binding database command to add new entries into the configure database switch config fabric binding database vsan 3 switch config port security swwn 20 00 00 0c 85 90 3e 80 Step 3 Use the fabric binding activate command to activate fabric binding switch config fabric binding activate vsan 2 Step 4 Use the fabric binding database copy command to copy the active database to ...

Page 375: ...onnect separated SAN islands through IP networks using FCIP and allow IP hosts to access Fibre Channel storage using the iSCSI protocol The IPS module allows you to use FCIP and iSCSI features It supports the full range of features available on other switching modules including VSANs security and traffic management The IPS module can be used in any Cisco MDS 9000 Family switch and has eight Gigabi...

Page 376: ...tor mode 500 Maximum concurrent iSCSI session creations per port 5 If more ISCSI sessions try to come up simultaneously on a port the initiator gets a temporary error and then the initiator retries if iSLB CFS is enabled you must use Device Manager to commit any iSCSI global configuration changes made through Fabric Manager iSLB Restrictions iSLB has the following restrictions in Cisco SAN OS Rele...

Page 377: ...se Device Manager which supports iSLB with CFS distribution Initial Troubleshooting Checklist Begin troubleshooting IP storage services issues by checking the following issues Common Troubleshooting Tools in Fabric Manager Use the following Fabric Manager procedures to access IP interfaces FCIP and iSCSI Choose Switches Interfaces to access IP interfaces Choose ISLs FCIP to access FCIP Choose End ...

Page 378: ...ages from all modules Use the following commands as directed by your customer support representative to further troubleshoot iSLB issues Note To issue commands with the internal keyword for troubleshooting purposes you must have a user account that contains the network admin role show ips internal event history errors Displays the errors encountered by the IPS manager show ips internal event histo...

Page 379: ...ng page 20 8 Cannot Assign IP Address to an Interface page 20 9 Note If you configure secondary VRRP IPv6 addresses on an IPFC VSAN interface before a downgrading to a release prior to Cisco Release 3 0 1 you must remove the secondary VRRP IPv6 addresses This is required only when you configure IPv6 addresses Verifying Basic Connectivity Use the procedures in this section to verify that you have I...

Page 380: ...ic Default gateway is 11 18 185 97 C 11 18 185 96 27 is directly connected mgmt0 C 11 18 189 128 26 is directly connected gigabitethernet4 7 Step 4 Use the clear ips arp or clear ipv6 neighbor command to clear the Address Resolution Protocol ARP or neighbor cache to verify that the activity you are viewing is the most current switch clear ips arp interface gigabitethernet 4 7 arp clear successful ...

Page 381: ...nate the domain ID with FFFC to obtain the domain controller address For example if the domain ID is 0xda 218 the concatenated ID is 0xfffcda Step 3 Choose Tools Ping to verify reachability to the destination switch Verifying Switch Connectivity Using the CLI To verify connectivity to a destination switch using the CLI follow these steps Step 1 Use the show fcdomain domain list vsan command to dis...

Page 382: ...to verify the static IP routes Verifying Static IP Routing Using the CLI To verify static IP routes using the CLI follow these steps Step 1 Use the show ip route config or the show ipv6 route command to verify the routes configured switch show ip route config Destination Gateway Mask Metric Interface default 172 17 8 1 0 0 0 0 0 mgmt0 11 2 36 0 11 3 36 1 255 255 252 0 0 11 2 56 0 11 3 56 1 255 255...

Page 383: ... CLI command on that VRRP interface FCIP Issues This section contains information on troubleshooting FCIP tunnels with and without special frames and includes the following topics One to One FCIP Tunnel Creation and Monitoring page 20 10 One to Three FCIP Tunnel Creation and Monitoring page 20 20 FCIP Profile Misconfiguration Examples page 20 21 FCIP Interface Misconfiguration Examples page 20 24 ...

Page 384: ...p 4 Enter no shutdown MDS1 config if no shutdown Step 5 Enter the profile number and profile mode MDS1 config fcip profile 28 The profile number can be any number between 1 255 Step 6 Enter the IP address of the local GE port that will be the endpoint of the FCIP tunnel MDS1 config profile ip address 10 10 10 2 Step 7 Exit profile mode MDS1 config profile exit Step 8 Set the FCIP interface and ent...

Page 385: ...ion fcip profile 28 ip address 10 10 10 2 port 3225 tcp keepalive timeout 60 tcp max retransmissions 4 tcp pmtu enable reset timeout 3600 tcp initial retransmit time 100 tcp window size 64 vsan database vsan 2 name grumpy_02 interface fcip28 no shutdown use profile 28 peer info ipaddr 10 10 11 2 ip route 10 10 11 0 255 255 255 0 10 10 10 1 Setting the Static Route for FCIP Tunnels Using the CLI Th...

Page 386: ...2 23 ips FCIP28 bind with GigabitEthernet2 8 phy GigabitEthernet2 8 Mar 10 21 42 23 ips FCIP28 Queueing bind tunnel to src if event to tunnel FSM resource 0 Mar 10 21 42 23 ips Locked fcip_if_fsm for MTS_OPC_IPS_FCIP_CMI_REQUEST msg id 32480 Mar 10 21 42 23 ips FCIP28 Send bind for GigabitEthernet2 8 to PM phy GigabitEthernet2 8 Mar 10 21 42 23 ips FCIP28 add to run time pss Mar 10 21 42 23 ips FC...

Page 387: ...T_STATE_CHANGE_RANGE mts opc 3114 msg id 32737 Mar 10 21 43 32 ips Hndlr MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE mts_opc 3114 msg_id 32737 Mar 10 21 43 32 ips Dequeued mts msg MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE mts opc 3114 msg id 32778 Mar 10 21 43 32 ips Hndlr MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE mts_opc 3114 msg_id 32778 Mar 10 21 43 32 ips Dequeued mts msg MTS_OPC_PM_LOGICAL_P...

Page 388: ...46 ips Hndlr MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE mts_opc 3114 msg_id 47602 Mar 10 22 59 46 ips fu_fsm_execute_all match_msg_id 0 log_already_open 0 Mar 10 22 59 46 ips fu_fsm_execute_all null fsm_event_list Mar 10 22 59 46 ips fu_fsm_engine mts msg MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE msg_id 47602 dropped Displaying the Debug Output from the FCIP Tunnel IPS Module Using the CLI The fo...

Page 389: ...nimum retransmission timeout is 100 ms Maximum number of re transmissions is 4 Advertised window size is 64 KB Verifying the Establishment of the FCIP Tunnel Using the CLI Use the show interface fcip command to verify that the FCIP tunnel is established and that traffic is passing through MDS1 show interface fcip 28 FCIP28 is trunking Hardware is GigabitEthernet Port WWN is 20 5e 00 05 30 00 59 de...

Page 390: ...his is the local advertised TCP window size and the default is 64 KB Peer receive window Current 64 KB Maximum 64 KB Scale 1 This is the remote endpoint advertised TCP window size Congestion window Current 2 KB This is the minimum window size used during congestion and it is not configurable 5 minutes input rate 136 bits sec 17 bytes sec 0 frames sec 5 minutes output rate 136 bits sec 17 bytes sec...

Page 391: ...0 0 The TCP listen port is ready for new TCP connections You can use the following command to verify that traffic is incrementing on the Gigabit Ethernet port of the FCIP tunnel MDS1 show ips stats mac interface gigabitethernet 2 8 Ethernet MAC statistics for port GigabitEthernet2 8 Hardware Transmit Counters 1074898 frame 1095772436 bytes 0 collisions 0 late collisions 0 excess collisions 0 bad f...

Page 392: ...ity error 0 frames soft queued 0 current Q 0 max Q 0 low memory 0 out of memory drop 0 queue full drop 0 RDL 0 too big RDL drop Flow Control 0 0 0 1 0 2 0 3 Ethereal Screen Captures of the TCP Connection and FCIP Tunnels Figure 20 3 Figure 20 4 and Figure 20 5 are screen captures taken with Ethereal of TCP connections being established and of FCIP tunnels Note that FCIP tunnel activation is the sa...

Page 393: ...000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 20 Troubleshooting IP Storage Services FCIP Issues Figure 20 4 Second Capture of TCP Connection Figure 20 5 shows the FC portion of the EISL initialization over the FCIP tunnel Figure 20 5 Third Capture of TCP Connection exchange link parameter Cisco ...

Page 394: ... MDS1 Configured for Three FCIP Tunnels Displaying the Configuration of the First Switch Using the CLI The following example shows the configuration of switch MDS1 for three tunnels from one Gigabit Ethernet port MDS1 config fcip profile 21 MDS1 config profile ip address 10 10 10 2 MDS1 config profile exit MDS1 config interface fcip 21 MDS1 config if use profile 21 MDS1 config if peer info ipaddr ...

Page 395: ...rations Displaying Incorrect or Nonexistent IP Address for an FCIP Profile Using the CLI MDS22 config fcip profile 21 MDS22 config profile ip addr 1 1 1 1 MDS22 config profile ip addr 34 34 34 34 MDS22 config profile exit MDS22 config exit MDS22 show fcip profile 21 FCIP Profile 21 Internet Address is 34 34 34 34 In this line the interface Gigabit Ethernet port is not shown This means the IP addre...

Page 396: ...rofile end MDS2 show fcip profile 21 FCIP Profile 21 Internet Address is 10 10 11 2 interface GigabitEthernet2 1 Listen Port is 32 This is a new TCP listen port TCP parameters SACK is disabled PMTU discover is enabled reset timeout is 3600 sec Keep alive is 60 sec Minimum retransmission timeout is 300 ms Maximum number of re transmissions is 4 Advertised window size is 64 KB MDS2 show fcip profile...

Page 397: ...a TCP connection on a port that is being used as a TCP listen port MDS1 config if end MDS1 show interface fcip 21 fcip21 is trunking The FCIP tunnel is now up Hardware is GigabitEthernet Port WWN is 20 42 00 05 30 00 59 de Peer port WWN is 20 42 00 0b 5f d5 9f c0 Admin port mode is auto trunk mode is on Port mode is TE vsan is 1 Trunk vsans allowed active 1 2 Trunk vsans operational 1 2 Trunk vsan...

Page 398: ...4 21 37 05 port1 41 FCIP21 SUP Enable tunnel ADMIN UP Mar 14 21 37 05 port1 42 FCIP21 Try to Bring UP the Tunnel Mar 14 21 37 05 port1 43 FCIP21 Bring up tunnel Failed peer ip not set The peer IP address is not set MDS2 show interface fcip 21 fcip21 is down Link failure or not connected Hardware is GigabitEthernet Port WWN is 20 42 00 0b 5f d5 9f c0 Admin port mode is auto trunk mode is on vsan is...

Page 399: ...080000 Mar 14 21 32 27 port1 16 FCIP21 Try to Bring UP the Tunnel Mar 14 21 32 27 port1 17 FCIP21 Tunnel in admin down state Mar 14 21 32 27 port1 18 FCIP21 SUP Switch WWN 0x2000000b5fd59fc0 Mar 14 21 32 27 port1 19 FCIP21 Try to Bring UP the Tunnel Mar 14 21 32 27 port1 20 FCIP21 Tunnel in admin down state Mar 14 21 32 27 port1 21 FCIP21 SUP Response to SB s pull all tunnel info Mar 14 21 32 27 p...

Page 400: ...rt1 1347 FCIP21 Bind the DE 0xd802cdc0 2 to tunnel LEP 0x80111570 Mar 14 23 26 07 port1 1348 FCIP21 Start the active connection 2 to 10 10 10 2 13 The switch is attempting to create a TCP connection on port 13 The creation port must match the TCP listen port on the remote endpoint Mar 14 23 26 07 port1 1349 FCIP21 Active Connect creation FAILED 1 Mar 14 23 26 07 port1 1350 FCIP21 Delete the DE 1 0...

Page 401: ...el Mar 14 23 49 06 port1 1875 FCIP21 Start TCP listener with peer 10 10 10 2 3225 Mar 14 23 49 06 port1 1876 FCIP Create a new listener object for 10 10 11 2 3225 Mar 14 23 49 06 port1 1877 FCIP Create FCIP Listener with local info 10 10 11 2 3225 Mar 14 23 49 06 port1 1878 FCIP21 Passive mode set don t initiate TCP connection A TCP connection will not be established when passive mode is set The G...

Page 402: ...3290 FCIP21 Time stamp tolerance check failed local time 0x3e726d6c2db994b7 tolerance 0x100000000 recv time 0x3e7251ace20db73a Mar 15 00 01 48 port1 3291 FCIP21 fcip_de_rcv Previous partial packet Concatenating Mar 15 00 01 48 port1 3292 FCIP21 Time stamp tolerance check failed local time 0x3e726d6c2db994b7 tolerance 0x100000000 recv time 0x3e7251ace20db73a Mar 15 00 01 48 port1 3293 FCIP21 FCIP f...

Page 403: ...0 Jan 14 14 22 39 port1 854935 FCIP21 Bind DE 1 to TCP hdl 0xd8071000 Jan 14 14 22 39 port1 854936 FCIP21 Bind DE 1 to eport 0x80110550 Jan 14 14 22 39 port1 854937 FCIP21 bind de 1 in eport 0x80110550 hash 1 num conn 2 Jan 14 14 22 39 port1 854938 FCIP21 Received new TCP connection from peer 10 10 10 2 64170 Jan 14 14 22 39 port1 854939 FCIP21 Create a DE 0xd802c900 for this tunnel Jan 14 14 22 3...

Page 404: ...erence Failure Figure 20 8 shows a trace of timestamp difference accepted Figure 20 8 Trace of Time stamp Difference Accepted FCIP Special Frame Tunnel Creation and Monitoring The FCIP tunnel configuration see the One to One FCIP Tunnel Creation and Monitoring section on page 20 10 and the One to Three FCIP Tunnel Creation and Monitoring section on page 20 20 must be completed before adding the FC...

Page 405: ...0 Jan 14 15 25 38 port1 857327 FCIP21 Start the active connection 2 to 10 10 10 2 3225 Jan 14 15 25 38 port1 857328 FCIP21 Active Connect creation SUCCEEDED 1 Jan 14 15 25 38 port1 857329 FCIP21 Bind DE 1 to TCP hdl 0xd8072c00 Jan 14 15 25 38 port1 857330 FCIP21 Setup for Special Frame handling I m Originator This begins the Special Frame setup of the Originator Jan 14 15 25 38 port1 857331 FCIP21...

Page 406: ...enabled acceptable time difference 3000 ms B port mode disabled TCP Connection Information 2 Active TCP connections Control connection Local 10 10 11 2 64792 Remote 10 10 10 2 3225 Data connection Local 10 10 11 2 64794 Remote 10 10 10 2 3225 372 Attempts for active connections 345 close of connections TCP Parameters Path MTU 1500 bytes Current retransmission timeout is 300 ms Round trip time Smoo...

Page 407: ...855299 FCIP21 Setup for Special Frame handling I m Originator Jan 14 15 14 30 port1 855300 FCIP21 Send the SF as Originator wait for response Jan 14 15 14 30 port1 855301 FCIP21 Setup timer to wait for SF Jan 14 15 14 30 port1 855302 FCIP21 TCP Received a close connection 1 reason 1 Jan 14 15 14 30 port1 855303 FCIP21 Delete the DE 1 0xd802d240 Jan 14 15 14 30 port1 855304 FCIP21 DE 670903744 0x00...

Page 408: ...an 14 15 14 37 port1 855347 FCIP21 Set lep operation state to DOWN Jan 14 15 14 37 port1 855348 FCIP21 DE 670902848 0x00000001 terminate tcp connection 0xd8071000 Jan 14 15 14 37 port1 855349 FCIP21 Delete the DE object 1 0xd802d5c0 Jan 14 15 14 37 port1 855350 FCIP21 Received new TCP connection from peer 10 10 10 2 64044 Jan 14 15 14 37 port1 855351 FCIP21 Create a DE 0xd802cac0 for this tunnel J...

Page 409: ...ression mode 1 because the Cisco MDS module could send compressed traffic faster than the IPS module could process iSCSI Issues This section contains information on troubleshooting iSCSI and includes the following topics Troubleshooting iSCSI Authentication page 20 35 Displaying iSCSI Authentication Using Fabric Manager page 20 36 Displaying iSCSI Authentication Using the CLI page 20 36 Troublesho...

Page 410: ...d into target iqn com domainname vrrp 11 gw 21000020374baf02 7 Figure 20 11 shows a failed iSCSI login for the Windows 2000 driver Figure 20 11 Failed iSCSI Login Status Window On Solaris systems a failed login is found in the var adm messages directory and should look similar to the following example Mar 14 11 44 42 ca sun1 iscsid 12561 ID 702911 daemon notice login rejected initiator error 01 Ma...

Page 411: ...Password Configuration Check the client side user name and password with either the switch s local configuration file or the RADIUS server Verifying iSCSI User Account Configuration Using Fabric Manager If iSCSI user authentication is through the switch s local user database choose Switches Security Users and Roles and select the Users tab to verify that the iSCSI users are configured correctly wi...

Page 412: ...key and port for authentication and accounting are an exact match with what is configured on the RADIUS server switch show radius server retransmission count 3 timeout value 5 following RADIUS servers are configured 171 71 49 197 available for authentication on port 1812 available for accounting on port 1813 RADIUS shared secret radius Adjust the RADIUS timeout and retransmission accordingly as th...

Page 413: ... 23 16 25 securityd got back the return value of global radius configuration operation success Mar 4 23 16 25 securityd closing RADIUS pss configuration Mar 4 23 16 25 securityd opening radius configuration for group default Mar 4 23 16 25 securityd opened the configuration successfully Mar 4 23 16 25 securityd GETNEXT request for radius index 0 addr Mar 4 23 16 25 securityd got some reply from 17...

Page 414: ...ponse Displaying the Debug Output for RADIUS Authentication Request Routing Using the CLI The following example shows the output from the debug security radius command switch Mar 5 00 51 13 securityd received CHAP authentication request for user002 Mar 5 00 51 13 securityd RADIUS is enabled hence it will be tried first for CHAP authentication Mar 5 00 51 13 securityd reading RADIUS configuration M...

Page 415: ...e configuring the proper slot or port Ensure that the Gigabit Ethernet interfaces are not shut down Each Gigabit Ethernet interface is partnered with a virtual iSCSI interface For iSCSI to operate on a particular Gigabit Ethernet the virtual iSCSI interface for that port must be in a no shutdown state Choose Switches Interfaces Gigabit Ethernet in Fabric Manager Or use the interface CLI command in...

Page 416: ...tiator iqn 1987 05 com cisco 02 F984BCA7E08C307E2D87A099B2D452F3 FULLMOON FULLMOON Session 1 index 2 Target iqn com domainname IPS TEST 02 07 gw 202300a0b80b14da VSAN 1 ISID 000000000000 TSID 134 Status active no reservation Type Normal ExpCmdSN 44 MaxCmdSN 53 Barrier 0 MaxBurstSize 0 MaxConn 0 DataPDUInOrder No DataSeqInOrder No InitialR2T Yes ImmediateData No Registered LUN 0 Mapped LUN 0 Stats ...

Page 417: ...50102 10 00 00 00 c9 30 ba 06 20 00 00 00 c9 30 ba 06 fc1 9 1 0x750201 50 08 05 f3 00 04 96 71 50 08 05 f3 00 04 96 70 fc1 10 1 0x750301 50 08 05 f3 00 04 96 79 50 08 05 f3 00 04 96 70 iscsi2 7 1 0x750105 20 0d 00 0b be 77 72 42 20 0c 00 0b be 77 72 42 Virtual Target Access Control Use the following guidelines when creating a virtual target Did you specify the correct pWWN If you are creating a vi...

Page 418: ...ate LOGGED_IN StatSN 1356 ExpStatSN 0 MaxRecvDSLength 524288 our_MaxRecvDSLength 1392 CSG 3 NSG 3 min_pdu_size 48 w data 48 AuthMethod none HeaderDigest None len 0 DataDigest None len 0 Version Min 0 Max 0 FC target Up Reorder PDU No Marker send No int 0 Received MaxRecvDSLen key Yes Session 2 index 85 Target iqn com domainname IPS TEST 02 08 gw 2200002037c52e2e VSAN 5 ISID 00023d000055 TSID 135 S...

Page 419: ...ata No Registered LUN 0 Mapped LUN 0 Stats PDU Command 13 Response 13 Bytes TX 1344 RX 0 Number of connection 1 Connection 1 Local IP address 0xa011d64 Peer IP address 0xa011d65 CID 0 State LOGGED_IN StatSN 1356 ExpStatSN 0 MaxRecvDSLength 524288 our_MaxRecvDSLength 1392 CSG 3 NSG 3 min_pdu_size 48 w data 48 AuthMethod none HeaderDigest None len 0 DataDigest None len 0 Version Min 0 Max 0 FC targe...

Page 420: ...eue limit 0 Received PDU in wrong phase 0 FCP Stats Total Sent 4110679 Received 1281518 Error 0 Unknown 0 Sent PLOGI 66367 Rcvd PLOGI_ACC 71 PLOGI_RJT 66296 PRLI 71 Rcvd PRLI_ACC 71 PRLI_RJT 0 Error resp 0 LOGO 0 Rcvd LOGO_ACC 0 LOGO_RJT 0 ABTS 87 Rcvd ABTS_ACC 0 TMF REQ 0 Self orig command 213 Rcvd data 142 resp 213 Rcvd PLOGI 614 Sent PLOGI_ACC 490 LOGO 197 Sent LOGO_ACC 111 PRLI 0 Sent PRLI_ACC...

Page 421: ...235f my kayak iSCSI alias name MY KAYAK Node WWN is 20 0a 00 0b be 77 72 42 dynamic Member of vsans 5 Number of Virtual n_ports 1 Virtual Port WWN is 20 0a 00 0b be 77 72 42 dynamic Interface iSCSI 2 8 Portal group tag is 0x87 VSAN ID 0 FCID 0x0 No of FC sessions 1 No of iSCSI sessions 1 iSCSI session details Target node Statistics PDU Command 0 Response 0 Bytes TX 0 RX 0 Number of connection 1 TC...

Page 422: ...0 1 29 101 1048 Path MTU 1500 bytes Current retransmission timeout is 300 ms Round trip time Smoothed 165 ms Variance 35 Advertized window Current 61 KB Maximum 62 KB Scale 0 Peer receive window Current 63 KB Maximum 63 KB Scale 0 Congestion window Current 63 KB Target node iqn com domainname IPS TEST 02 08 gw 2200002037c5260a Statistics PDU Command 13 Response 13 Bytes TX 1344 RX 0 Number of conn...

Page 423: ...ithin the device This section refers to the scenario in Figure 20 13 Figure 20 13 IPS Window Scaling The IPS module adjusts the Receive Data Field Size that it advertises to its Fibre Channel partner according to the MTU that is configured on the corresponding Gigabit Ethernet port of an iSCSI client If left to the default MTU size the Fibre Channel frame size from the target device is decreased t...

Page 424: ...e IBM ESS Shark had a hardcoded BB_credit value of 64 not configurable The fcrxbbcredit on the corresponding switch port fc1 3 was set to the same value The C4 and C8 represented the corresponding port WWNs pWWN for the IBM Shark storage subsystem The full pWWN is as follows C4 50 05 07 63 00 c4 94 4c in VSAN 778 C8 50 05 07 63 00 c8 94 4c in VSAN 777 Configuring from the Bottom Switch Using the C...

Page 425: ...0 0 0 0 0 0 LISTEN 0 0 MDS_BOTTOM show flogi database vsan 777 INTERFACE VSAN FCID PORT NAME NODE NAME fc1 3 777 0x610000 50 05 07 63 00 c8 94 4c 50 05 07 63 00 c0 94 4c iscsi2 1 777 0x610001 20 05 00 0c 30 6c 24 42 20 00 00 0c 30 57 5e c2 Total number of flogi 2 MDS_BOTTOM show fcns dabase vsan 777 VSAN 777 FCID TYPE PWWN VENDOR FC4 TYPE FEATURE 0x610000 N 50 05 07 63 00 c8 94 4c IBM scsi fcp tar...

Page 426: ...ion 2 Number of TCP connection 2 Configured TCP parameters Local Port is 3260 PMTU discover is enabled default This is especially required if there are devices without jumbo support in the path The initial TCP 3 way handshake will establish a session with a high MSS value provided both the IPS module and the iSCSI client are configured or capable even if there are devices without jumbo frame suppo...

Page 427: ...Advertized window Current 998 KB Maximum 1000 KB Scale 4 Peer receive window Current 1000 KB Maximum 1000 KB Scale 4 Congestion window Current 12 KB VSAN ID 777 FCID 0x610001 No of FC sessions 1 No of iSCSI sessions 1 iSCSI session details Target node shark_nas Statistics PDU Command 392051 Response 392042 Bytes TX 25692593152 RX 0 Number of connection 1 TCP parameters Local 10 48 69 250 3260 Remo...

Page 428: ...ing the registry is a very high risk operation it can render the system unusable requiring a reinstallation of the entire operating system Only advanced users should perform this operation Displaying the Gigabit Ethernet Interface Choose Switches Interfaces Gigabit Ethernet using Fabric Manager to view the Gigabit Ethernet status Or use the show interface CLI command to view the Gigabit Ethernet s...

Page 429: ...thernet2 1 TCP send stats 56252632 segments 76746280484 bytes 56100434 data 152173 ack only packets 1 control SYN FIN RST 0 probes 24 window updates 0 segments retransmitted 0 bytes 0 retransmitted while on ethernet send queue 0 packets split 3 delayed acks sent TCP receive stats 7068115 segments 1061853 data packets in sequence 54245464 bytes in sequence 0 predicted ack 187 predicted data 0 bad c...

Page 430: ...mand Output MDS_BOTTOM show iscsi remote node fcp session detail iSCSI Node name is iqn 1987 05 com cisco 02 75af2f95624c shark nas iSCSI alias name SHARK NAS Node WWN is 20 00 00 0c 30 6c 24 42 dynamic Member of vsans 777 778 Number of Virtual n_ports 1 Virtual Port WWN is 20 00 00 0c 30 6c 24 42 configured Interface iSCSI 2 1 Portal group tag is 0x1001 VSAN ID 0 FCID 0x610001 No of FC sessions 1...

Page 431: ...hich See Example 20 5 Example 20 5 Sample Output for Low Packet Split Count MDS_Top show ips stats tcp interface gigabitethernet 2 1 detail truncated output TCP Statistics for port GigabitEthernet2 1 TCP send stats 10 segments 240 bytes 5 data 5 ack only packets 0 control SYN FIN RST 0 probes 0 window updates 0 segments retransmitted 0 bytes 0 retransmitted while on ethernet send queue 0 packets s...

Page 432: ...meout page 20 61 Session Down pWWN in Use At Remote Switch page 20 62 Redirected Session Does Not Come Up page 20 62 iSLB Zones Not Present in Active Zone Set page 20 63 Traffic Description After iSLB Commit or Activation of Zone Set page 20 63 VRRP Master Overutilized page 20 64 iSLB Zone Set Activation Failed page 20 64 iSLB CFS Commit Fails page 20 65 Resolving an iSLB Merge Failure page 20 65 ...

Page 433: ... to the fabric Normal operation Only the following iSCSI and iSLB configuration is distributed iSLB initiator and iSLB initiator targets iSLB VRRP load balancing configuration iSCSI global authentication parameters authentication algorithm and CHAP user name or password iSCSI dynamic initiator mode iSCSI iSLB or deny Table 20 3 iSLB Configuration Commit or Merge Failed VSAN ID is Not Yet Configure...

Page 434: ...vent history error CLI commands for details on the specific WWN and initiator in error To fix the problem use another WWN or allow the system to assign one for the initiator using the static nWWN pWWN system assign command Table 20 5 iSLB Configuration Commit or Merge Failed Duplicate WWN Found as Symptom Possible Cause Solution iSLB configuration commit or merge failed with error Duplicate WWN fo...

Page 435: ...n failed with error Pending iSLB CFS config has reached its limit The limit of 200 initiators in the pending database has been reached so no more configuration is allowed Use the islb commit CLI command to commit the outstanding changes Table 20 8 iSCSI Disable Failed Cannot Disable Iscsi Large Iscsi Config Present Symptom Possible Cause Solution iSCSI disable failed with error Cannot disable iSCS...

Page 436: ...estroyed To fix the problem use another WWN or allow the system to assign one for the initiator using the static nWWN pWWN system assign CLI command Table 20 11 Redirected Session Does Not Come Up Symptom Possible Cause Solution Redirected session does not come up Connection may be down or initiator to interface mapping may be missing Use the ping CLI command to verify that the connection between ...

Page 437: ...hen create and activate a new zone set for the VSAN in question Then use the islb zoneset activate CLI command to trigger iSLB zoning Zone set activation failed If an active zone set is configured then check for activation failures See the Traffic Description After iSLB Commit or Activation of Zone Set section on page 20 63 Table 20 13 Traffic Disruption After iSLB Commit or Activation of Zone Set...

Page 438: ...Symptom Possible Cause Solution iSLB zone set activation failed iSLB auto zone is enabled but CFS distribution is not enabled Enable CFS distribution for iSLB to share load across multiple switches Use the islb distribute CLI command on each switch in the fabric Zone set activation is not from switch with IVR and iSLB enabled Activate the zone set from a switch that has IVR and iSLB enabled Use th...

Page 439: ...islb commit command Note The iSLB configuration on other switches will be overwritten A commit after a merge failure synchronizes the fabric configuration to the running config of the switch where the commit was performed Table 20 16 iSLB CFS Commit Fails Symptom Possible Cause Solution iSLB CFS commit fails Zone set activation is not from the switch with IVR and iSLB enabled Activate the zone set...

Page 440: ...u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 20 66 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 20 Troubleshooting IP Storage Services iSLB Issues ...

Page 441: ...and if the packet matches the rule also stipulates if the packet should be permitted or denied Each switch in the Cisco MDS 9000 Family can have a maximum of 64 IP ACLs and each IP ACL can have a maximum of 256 filters An IP filter contains rules for matching an IP packet based on the protocol address and port IPv4 filters can also match on an ICMP type and type of service ToS This section include...

Page 442: ... only the first 16 bits of the source Wildcard bits set to one must be contiguous and at the end of the prefix For example a wildcard of 0 255 0 64 would not be valid Use the any option as an abbreviation for a source and source wildcard or destination and destination wildcard 0 0 0 0 255 255 255 255 For IPv6 specify the source or the destination IPv6 addresses in one of two ways Use the 128 bit q...

Page 443: ... 21 2 displays the value for each ICMP type ToS Information IPv4 packets can be filtered based on the ToS conditions delay monetary cost normal service reliability and throughput TCP1 ftp 20 ftp data 21 ssh 22 telnet 23 smtp 25 tasacs ds 65 www 80 sftp 115 http 143 wbem http 5988 wbem https 5989 1 If the TCP connection is already established use the established option to find matches A match occur...

Page 444: ...lter condition to log information about packets that match dropped entries The log output displays the ACL number permit or deny status and port information Use the following CLI commands to ensure that the debug messages are logged to the logfile for the kernel and ipacl facilities logging logfile SyslogFile 7 logging level kernel 7 logging level ipacl 7 IP ACL Issues This section describes troub...

Page 445: ...e interface Choose Switches Security IP ACL in Fabric Manager select the Interfaces tab and remove the ACL name from the ProfileName field Click Apply Changes Or use the no ip access group or the no ipv6 traffic filter CLI command in interface mode A deny filter is too broad Delete the deny filter Choose Security IP ACL in Device Manager right click the access list and click Rules Right click the ...

Page 446: ...TU 2300 bytes Port mode is IPS Speed is 1 Gbps Beacon is turned off Auto Negotiation is turned on ip access group TCPAlow in 5 minutes input rate 0 bits sec 0 bytes sec 0 frames sec 5 minutes output rate 0 bits sec 0 bytes sec 0 frames sec 1916 packets input 114960 bytes 0 multicast frames 0 compressed 0 input errors 0 frame 0 overrun 0 fifo 0 packets output 0 bytes 0 underruns 0 output errors 0 c...

Page 447: ...nterface gigabitethernet 2 1 switch config if ip access group List1 switch config interface gigabitethernet 2 2 switch config if ipv6 traffic filter IPAlow No Packets Are Blocked Symptom No packets are blocked Table 21 4 No Packets Are blocked Symptom Possible Cause Solution No packets are blocked A permit filter is too broad Delete the permit filter Add an appropriate permit filter Choose Securit...

Page 448: ...se Switches ISLs Port Channels to view the Members Admin field to find out which interfaces are part of the PortChannel Choose Switches Security IP ACL on Fabric Manager select the Interfaces tab and add the ACL name to the ProfileName field Click Apply Changes Or use the show port channel database CLI command to find out which interfaces are part of the PortChannel and then use the ip access grou...

Page 449: ...g protecting one or more data flows between a pair of hosts between a pair of security gateways or between a security gateway and a host IPsec is supported for iSCSI and FCIP using IKE and Encapsulated Security Protocol ESP in tunnel mode This section contains the following topics IPsec Compatibility page 22 1 Supported IPsec and IKE Algorithms for Microsoft Windows and Linux Platforms page 22 2 I...

Page 450: ...iation option in a crypto map Security association idle timeout Dynamic crypto maps IPv6 Note Any reference to crypto maps in this document only refers to static crypto maps For IPsec to interoperate effectively with Microsoft iSCSI initiators specify the TCP protocol and the local iSCSI TCP port number default 3260 in the IPv4 ACL This configuration ensures the speedy recovery of encrypted iSCSI ...

Page 451: ...tiator Microsoft IPsec implementation on Microsoft Windows 2000 platform 3DES SHA 1 or MD5 DH group 2 3DES SHA 1 Cisco iSCSI initiator Free Swan IPsec implementation on Linux platform 3DES MD5 DH group 1 3DES MD5 Table 22 2 IKE Transform Configuration Parameters Parameter Accepted Values Default Value Encryption algorithm 56 bit DES CBC 168 bit DES 3DES 128 bit AES 3DES Hash algorithm SHA 1 HMAC v...

Page 452: ...epted Values Encryption algorithm 56 bit DES CBC 168 bit DES 128 bit AES CBC 128 bit AES CTR1 256 bit AES CBC 256 bit AES CTR1 1 If you configure the AES counter CTR mode you must also configure the authentication algorithm Hash authentication algorithm1 optional SHA 1 HMAC variant MD5 HMAC variant AES XCBC MAC Checklist Check off Verify licensing requirements See Cisco MDS 9000 Family Fabric Mana...

Page 453: ...rator interface gigabit 2 1 sad inbound outbound sa index Displays detailed information of an SA from the hardware accelerator show ipsec internal crypto accelerator interface gigabit 2 1 stats Displays detailed information per interface from the hardware accelerator IPsec Issues This section provides the procedures required to troubleshoot IKE and IPsec issues in an FCIP configuration Figure 22 1...

Page 454: ... encryption algorithm hash algorithm and Diffie Hellman DH group is configured on each switch Issue the show crypto ike domain ipsec policy command on both switches Example command outputs for the configuration shown in Figure 22 1 follow MDSA show crypto ike domain ipsec policy Priority 1 auth pre shared lifetime 86300 secs encryption 3des hash md5 DH group 1 MDSC show crypto ike domain ipsec pol...

Page 455: ...100 232 255 255 255 255 10 10 100 231 255 255 255 255 Transform sets tfs 02 Security Association Lifetime 3000 gigabytes 120 seconds PFS Y N Y PFS Group group5 Interface using crypto map set cmap 01 GigabitEthernet1 2 MDSA show crypto transform set domain ipsec Transform set tfs 01 esp 3des null will negotiate tunnel Transform set tfs 02 esp 3des esp md5 hmac will negotiate tunnel Transform set ip...

Page 456: ...255 255 255 10 10 100 232 255 255 255 255 127 deny ip any any Clear test policy for all other traffic MDSC show crypto spd domain ipsec Policy Database for interface GigabitEthernet1 2 direction Both 0 deny udp any port eq 500 any 1 deny udp any any port eq 500 2 permit ip 10 10 100 232 255 255 255 255 10 10 100 231 255 255 255 255 127 deny ip any any Step 2 Issue the show ipsec internal crypto ac...

Page 457: ...Vlan_id 0 0 Action cleartext Inbound Policy 2 Source IP Address 10 10 100 231 255 255 255 255 Destination IP Address 10 10 100 232 255 255 255 255 Source port Destination port Protocol Physical port 1 1 Vlan_id 0 4095 Action ipsec Inbound Policy 127 Source IP Address Destination IP Address Source port Destination port Protocol Physical port 0 0 Vlan_id 0 0 Action cleartext Verifying Interface Stat...

Page 458: ...f Internet address is 10 10 100 232 24 MTU 1500 bytes Port mode is IPS Speed is 1 Gbps Beacon is turned off Auto Negotiation is turned on 5 minutes input rate 7528 bits sec 941 bytes sec 8 frames sec 5 minutes output rate 7288 bits sec 911 bytes sec 8 frames sec 7209 packets input 835518 bytes 0 multicast frames 0 compressed 0 input errors 0 frame 0 overrun 0 fifo 7301 packets output 827630 bytes ...

Page 459: ...nd Buffer Size 0 KB CWM Burst Size 50 KB 5 minutes input rate 2960 bits sec 370 bytes sec 4 frames sec 5 minutes output rate 3184 bits sec 398 bytes sec 4 frames sec 3628 frames input 340644 bytes 3610 Class F frames input 338396 bytes 18 Class 2 3 frames input 2248 bytes 0 Reass frames 0 Error frames timestamp error 0 3624 frames output 359140 bytes 3608 Class F frames output 357332 bytes 16 Clas...

Page 460: ...utput 340828 bytes 3612 Class F frames output 338580 bytes 18 Class 2 3 frames output 2248 bytes 0 Error frames Verifying Security Associations To verify security associations SAs follow these steps Step 1 Issue the show crypto sad domain ipsec command to verify the current peer mode and inbound and outbound index of each switch The example command outputs follow MDSA show crypto sad domain ipsec ...

Page 461: ...ce gigabitethernet 7 1 sad inbound 1 sw172 22 48 91 show ipsec internal crypto accelerator interface gigabitethernet 7 1 sad inbound 1 Inbound SA 1 Mode Tunnel flags 0x492300000000000 IPsec mode is ESP Encrypt algorithm is DES 3DES Auth algorithm is MD5 Source ip address 10 10 100 232 255 255 255 255 Destination ip address 10 10 100 231 255 255 255 255 Physical port 0 mask 0x1 Misc select 0 mask 0...

Page 462: ...nel source ip address 10 10 100 231 Tunnel destination ip address 10 10 100 232 Hard limit 483183820800 bytes Soft limit 376883380224 bytes SA byte count 874544 bytes Elapsed traffic SA user byte count 874544 bytes Elapsed traffic Packet count 7150 Hard limit expiry 1100652419 secs since January 1 1970 remaining 208 9 secs Soft limit expiry 1100652384 secs since January 1 1970 remaining 205 4 secs...

Page 463: ...e configurations are still compatible Clearing Security Associations To clear a specific SA obtain the SA index value and issue the clear crypto sa domain ipsec interface gigabitethernet slot port outbound sa index command To obtain the SA index value issue the show crypto sad domain ipsec command Debugging the IPsec Process Use the following commands to print debug messages to the console debug i...

Page 464: ... map sets 1 IKE transaction stats 0 num 64 max Inbound SA stats 1 num Outbound SA stats 1 num The show crypto global domain ipsec interface gigabitethernet slot port command output displays interface level statistics Example command output follows MDSA show crypto global domain ipsec interface gigabitethernet 7 1 IPSec interface statistics IKE transaction stats 0 num Inbound SA stats 1 num 512 max...

Page 465: ... SAN SANTap has a control path and a data path The control path services requests that create and manipulate replication sessions that are sent by an appliance The control path is implemented using a SCSI based protocol An appliance sends requests to a control virtual target CVT which the SANTap process creates and monitors Responses are sent to the control LUN on the appliance SANTap also allows ...

Page 466: ...y the Cisco VI virtual initiator to a target port on the appliance DVT Data virtual target A DVT is created for every port on a multi ported target that is included in SANTap based services The DVT is created in the host VSAN Once a DVT is created and a host logs into the DVT SANTap installs a DVTLUN for every configured LUN on the target for this host ITL Initiator target LUN tuple Uniquely ident...

Page 467: ...nents Begin your troubleshooting activity as follows DVTs per SSM 32 Cisco SAN OS 3 1 2 with SSI 3 1 2m and later 16 Prior to Cisco SAN OS 3 1 2 with SSI 3 1 2m Sessions per SSM 1024 Cisco SAN OS 3 0 2 with SSI 3 0 2j 2048 Cisco SAN OS 3 1 2b with SSI 3 1 2m and later LUN ID address 32 bits Cisco SAN OS 3 2 16 bits Prior to Cisco SAN OS 3 2 DVT LUNs per SSM 4080 All releases ITLs per DPP 1500 Cisc...

Page 468: ... display information about SANTap Example 23 1 Display SANTap CVT Information switch show santap module 2 cvt CVT Information cvt pwwn 23 4f 00 0d ec 09 3c 02 cvt nwwn 23 9d 00 0d ec 09 3c 02 cvt id 135895180 cvt xmap_id 135895212 cvt vsan 8 cvt name Example 23 2 Display SANTap DVT Information switch show santap module 2 dvt DVT Information dvt pwwn 50 06 0e 80 03 81 32 36 dvt nwwn 50 06 0e 80 03 ...

Page 469: ... 33 33 33 33 33 00 adt lun 0x0 aci pwwn 22 22 22 22 22 22 22 22 cvt pwwn 23 4f 00 0d ec 09 3c 02 num ranges 0 session state 5 redirect mode 0 mrl requested 1 MRL vsan 8 RegionSize 4806720 DiskPWWN 0x234f000dec093c02 DiskLun 0x 1 startLBA 1 pwl requested 1 PWL type 2 UpdatePol 2 RetirePolicy 4 pwl_start 1 iol requested 0 Example 23 5 Display SANTap AVT Information switch show santap module 2 avt AV...

Page 470: ...n 0x0 xmap id 22 rvt id 17 app pwwn 22 00 00 20 37 39 b1 00 app lun 0x0 app vsan 1 Use the following commands to display more advanced troubleshooting information for SANTap show tech support show santap module 2 tech support show isapi tech support show santap vttbl dvt dvt pwwn Messages Logs and Databases The following log files and databases can provide helpful information when troubleshooting ...

Page 471: ...sion of Cisco SAN OS and SSI in use For specific ITL limitations see the Limitations section on page 23 2 To diagnose and resolve ITL problems follow these steps Step 1 Use the show isapi dpp 4 queue command to display DPP queue information Step 2 Verify that the number of ITLs on a DPP is within the limitations for the version of Cisco SAN OS and SSI in use Use the show isapi dpp 4 queue incl LUN...

Page 472: ...n replication is enabled AVT LUNs are created and that can increase the ITL count over the limit See Limitations section on page 23 2 If Reservation Support is not enabled on the RPA 26 AVT LUNs are created at a time The appliance completes recovery of these LUNs and then deletes them before creating more This behavior does not significantly increase the ITL count If Reservation Support is enabled...

Page 473: ...y moved Inaccurate zoning Zoning solutions differ based on the Cisco SAN OS and SSI versions in use With SSI 3 0 2j you must have default zoning in the back end VSAN or zone the target and VIs together in the back end VSAN With SSI 3 1 2 only the host VI and target need to be zoned together in the back end VSAN Adding and removing hosts without performing a purge If you have 16 hosts and you remov...

Page 474: ...d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 23 10 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 23 Troubleshooting SANTap SANTap Issues ...

Page 475: ...vice or user has a key pair containing both a private key and a public key Digital certificates link the digital signature to the remote device A digital certificate contains information to identify a user or device such as the name serial number company department or IP address It also contains a copy of the entity s public key The certificate is itself signed by a certificate authority CA a thir...

Page 476: ...published in a repository Cisco MDS SAN OS allows the manual configuration of pre downloaded CRLs for the trusted CAs and then caches them in the switch bootflash cert store During the verification of a peer certificate by IPsec or SSH the issuing CA s CRL is consulted only if the CRL has already been cached locally and the revocation checking is configured to use CRL Otherwise CRL checking is not...

Page 477: ... and send it to the CA 3 Receive the issued certificate in base64 encoded text form from the CA in an e mail message or in a web browser download 4 Cut and paste the issued certificate to the switch using the certificate import facility Maximum Limits Table 24 1 lists the maximum limits for CAs and digital certificate parameters Initial Troubleshooting Checklist Begin troubleshooting digital certi...

Page 478: ...e and RSA Key Pairs from Backup page 24 10 CA Will Not Generate Identity Certificate Symptom CA will not generate an identity certificate Table 24 2 CA Will Not Generate Identity Certificate Symptom Possible Cause Solution CA will not generate an identity certificate FQDN is not configured Configure the host name and the IP domain name Choose Switches in Fabric Manager and set the LogicalName fiel...

Page 479: ... click Create Table 24 3 Cannot Export Identity Certificate in PKCS 12 Format Symptom Possible Cause Solution Cannot export identity certificate in PKCS 12 format RSA keys not exportable Create exportable RSA keys Choose Switches Security PKI in Fabric Manager and click Create Row Check the Exportable check box and create an RSA key pair Or use the crypto key generate rsa exportable CLI command Ta...

Page 480: ... the CA certificate fingerprint displayed in the IssuerCert FingerPrint column for the trust point row in question Compare the CA certificate fingerprint with the fingerprint already communicated by the CA obtained from the CA web site If the fingerprints match exactly accept the CA by selecting the certconfirm trust point action Otherwise reject the CA by selecting the certnoconfirm trust point a...

Page 481: ...the values of the identity certificate and its related objects like the certificate file name are automatically updated with the appropriate values as per the corresponding attributes in the identity certificate Configuring Certificates on the MDS Switch Using the CLI To configure certificates on an MDS switch using the CLI follow these steps Step 1 Configure the switch FQDN switch config t Enter ...

Page 482: ... 7b3 DXJPANBsIHHzluNccNM87ypyzwuoSNZXOMpeRXXI OzyBAgiXT2ASFuUOwQ1iDM8rO 41jf8RxvYKvysCAwEAAaOBvzCBvDALBgNVHQ8E BAMCAcYwDwYDVR0TAQH BAUwAwEB zAdBgNVHQ4EFgQUJyjyRoMbrCNMRU2OyRhQ GgsWbHEwawYDVR0fBGQwYjAuoCygKoYoaHR0cDovL3NzZS0wOC9DZXJ0RW5yb2xs L0FwYXJuYSUyMENBLmNybDAwoC6gLIYqZmlsZTovL1xcc3NlLTA4XENlcnRFbnJv bGxcQXBhcm5hJTIwQ0EuY3JsMBAGCSsGAQQBgjcVAQQDAgEAMA0GCSqGSIb3DQEB BQUAA0EAHv6UQ 8nE399Tww KaGr0...

Page 483: ...QQIEwlLYXJuYXRha2ExEjAQBgNVBAcTCUJhbmdhbG9yZTEOMAwGA1UEChMFQ2lz Y28xEzARBgNVBAsTCm5ldHN0b3JhZ2UxEjAQBgNVBAMTCUFwYXJuYSBDQTAeFw0w NTExMTIwMzAyNDBaFw0wNjExMTIwMzEyNDBaMBwxGjAYBgNVBAMTEVZlZ2FzLTEu Y2lzY28uY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC GNVACdjQu41C dQ1WkjKjSICdpLfK5eJSmNCQujGpzcuKsZPFXjF2UoiyeCYE8ylncWyw5E08rJ47 glxr42 sI9IRIb 8udU cj9jSSfKK56koa7xWYAu8rDfz8jMCnIM4W1aY q2q4Gb x7RifdV06uF...

Page 484: ...a reboot Certificates not saved to NVRAM Save the running config to startup config to save the trust point to startup Then reimport the certificates See the Configuring Certificates on the MDS Switch Using Fabric Manager section on page 24 5 or the Configuring Certificates on the MDS Switch Using the CLI section on page 24 7 Table 24 6 Cannot Import Certificate and RSA Key Pairs from Backup Sympto...

Page 485: ...Manager choose Switches Security PKI and select the TrustPoint Actions tab Step 5 Select the pkcs12import option from the Command drop down menu to import the key pair identity certificate and the CA certificate or certificate chain in PKCS 12 format to the selected trust point Step 6 Enter the input in bootflash filename format for the PKCS 12 file Step 7 Enter the required password The password ...

Page 486: ...pto ca trustpoint myCA switch config trustpoint delete certificate force Step 4 Optionally use the no rsakeypair command in the trust point config submode to remove the RSA key pairs from the trust point switch config crypto ca trustpoint myCA switch config trustpoint no rsakeypair SwitchA Step 5 Use the copy tftp command to copy the PKCS 12 format file to the switch switch copy tftp adminid p12 b...

Page 487: ...ons make it easy to integrate specific support requirements For those who have service contracts directly with Cisco Systems automatic case generation with the Technical Assistance Center is possible by registering with the AutoNotify service AutoNotify provides fast time to resolution of system problems by providing a direct notification path to Cisco customer support The AutoNotify feature requi...

Page 488: ...ges when certain events occur on the switch You can customize predefined alert groups to execute additional valid show commands when specific events occur The output from these additional show commands is included in the notification message along with that of the predefined show commands Note You can assign a maximum of five user defined show commands to an alert group Only show commands can be a...

Page 489: ...ll Home Level Keyword Used Syslog Level Description Catastrophic 9 Catastrophic N A Network wide catastrophic failure Disaster 8 Disaster N A Significant network impact Fatal 7 Fatal Emergency 0 System is unusable Critical 6 Critical Alert 1 Critical conditions immediate attention needed Major 5 Major Critical 2 Major conditions Minor 4 Minor Error 3 Minor conditions Warning 3 Warning Warning 4 Wa...

Page 490: ...ck an alert group to select it for association You see a check next to that alert group To deselect it and remove the check click it again Table 25 2 Not Receiving Call Home Alerts Symptom Possible Cause Solution Not receiving Call Home alerts The alert is in an alert group that is not configured for the destination profile Add the alert group to the destination profile See the Configuring an Aler...

Page 491: ... these steps Command Purpose Step 1 switch config t Enters configuration mode Step 2 switch config callhome switch config callhome Enters Call Home configuration submode Step 3 switch config callhome destination profile test1 alert group environmental Configures user defined destination message profile test1 to receive Call Home notifications for power fan and temperature related events switch con...

Page 492: ...erts From All Configured Switches Symptom Not receiving Call Home alerts from all configured switches Command Purpose Step 1 switch config t Enters configuration mode Step 2 switch config callhome switch config callhome Enters Call Home configuration submode Step 3 switch config callhome transport email smtp server 192 168 1 1 Configures the DNS IPv4 address or IPv6 address of the SMTP server to r...

Page 493: ...mail address that identifies the source of Call Home notifications Step 3 Click Apply Changes Configuring Call Home Contact Information Using the CLI To assign the contact information follow these steps Command Purpose Step 1 switch config t Enters configuration mode Step 2 switch snmp server contact personname companyname com Configures the SNMP contact name Step 3 switch config callhome switch c...

Page 494: ...Level for a Destination Profile Using the CLI section on page 25 5 Message throttling is disabled Enable Call Home message throttling Choose Switches Events Call Home in Fabric Manager click the General tab check the Duplicate Message Throttle check box and then click Apply Changes Or use the duplicate message throttle CLI command Table 25 5 Not Receiving Syslog based Call Home Alerts Symptom Poss...

Page 495: ... Current Inventory Symptom Periodic inventory notification does not reflect the current inventory Table 25 6 Periodic Inventory Notification Does Not Reflect Current Inventory Symptom Possible Cause Solution Periodic inventory notification does not reflect the current inventory Inventory change occurred after the last system reboot The periodic inventory for Call Home is updated when the switch re...

Page 496: ... c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m 25 10 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Chapter 25 Troubleshooting Call Home Call Home Issues ...

Page 497: ...g Fabric Manager Web Client page 26 7 Troubleshooting Performance Manager page 26 12 Troubleshooting Device Manager page 26 13 Overview Cisco Fabric Manager is a Java and SNMP based network fabric and device management tool with a GUI that displays real time views of your network fabric including Cisco MDS 9000 and third party switches hosts and storage devices In addition to complete configuratio...

Page 498: ...ons See also the Troubleshooting a Nondisruptive Upgrade on a Fabric Switch section on page 2 4 Troubleshooting Fabric Manager Issues This section covers the following topics Cannot Log Into Fabric Manager page 26 3 Cannot Upgrade Fabric Manager page 26 3 The Map Shows Two Switches Where Only One Switch Exists page 26 3 Old Devices Appear on the Map page 26 3 Red Line Through the Switch page 26 4 ...

Page 499: ...s not complete You should open the Java Web Start application on your desktop and disable HTTP proxy If you are using Microsoft Windows open Java Web Start and choose File Preferences to access the HTTP proxy settings Note Starting with SAN OS 3 2 1 Fabric Manager is installed or upgraded from a CD ROM or from Cisco com The Map Shows Two Switches Where Only One Switch Exists If two switches show o...

Page 500: ...fans and power supply monitoring are then received by Fabric Manager If Fabric Manager is not receiving traps verify that your switch is sending traps by using the following CLI commands test pfm test SNMP trap fan test pfm test SNMP trap powersupply test pfm test SNMP trap temp sensor Tips for Using Fabric Manager This section covers the following topics Setting the Map Layout So It Stays After R...

Page 501: ... address in the Switches table and click Apply If the community string is correct the red slash will go away Even if the community string is incorrect double clicking on the Cisco SN5428 will launch the web tool Running Cisco Fabric Manager with Network Multiple Interfaces If your PC has multiple network interfaces NICs the Cisco Fabric Manager applications detect these interfaces automatically ig...

Page 502: ...ses a proxy server for HTTP requests make sure the Java Web Start Application Manager is properly configured with the IP address of your proxy server To configure a proxy server in the Java Web Start Application Manager follow these steps Step 1 Double click the Java Web Start application manager icon on your Windows desktop or choose Program Files Java Web Start Step 2 Select File Preferences fro...

Page 503: ...ovides summary and drill down performance reports These reports are only available if you create a collection using Performance Manager and start the collector This section includes the following topics Cannot Access Fabric Manager Web Client page 26 8 Cannot Launch Fabric Manager Web Client page 26 9 Cannot Log Into Fabric Manager Web Client page 26 11 Note You must log in with a network access r...

Page 504: ... the file Step 4 Restart Fabric Manager Clearing Java Cache If you encounter problems downloading Fabric Manager Web Client the Java cache may contain damaged files or old versions of software To correct this problem clear Java cache Step 1 Navigate to Control Panel Java and double click Java to open the Java Control Panel Step 2 Click View under Temporary Internet Files region to open the Java Ca...

Page 505: ...number in use or if you need to use SSL Cannot Launch Fabric Manager Web Client Symptom Cannot successfully launch Fabric Manager Web Client Table 26 3 Cannot Launch Fabric Manager Web Client Symptom Possible Cause Solution Java Web Start not detected error message displays Appropriate version of Java may not be installed Install the appropriate version of the Java Runtime Environment Cisco recomm...

Page 506: ...tems To allow the web site to run the Java Web Start ActiveX control and launch the Fabric Manager Web Client follow these steps Step 1 Click the here link on the Internet Explorer information bar Step 2 Select Run ActiveX Control from the choices shown Trusting the Remote Fabric Manager Web Site If you are launching the Fabric Manager Web Client and connecting to a remote Fabric Manager Server us...

Page 507: ...e you installed the Web Client If you forget a password you can make a new network admin user locally on the workstation where you installed the Web Client and then log in and delete the old user account under the Admin tab To create a user on the workstation where you installed the Web Client and delete the old user follow these steps Step 1 Go to the Web Client installation directory and cd to t...

Page 508: ...ing a web browser It presents recent statistics in detail and older statistics in summary Performance Manager also integrates with external tools such as Cisco Traffic Analyzer The Performance Manager has three operational stages Definition The Flow Wizard sets up flows in the switches Collection Reads the configuration from the database the flow configuration from the switches and collects the de...

Page 509: ... for multiple VSANs Manage ports PortChannels and trunking Manage SNMPv3 and CLI security access to switches Manage alarms events and notifications Save and copy configuration files and software images View hardware configurations chassis modules port status and statistics Device Manager uses the SNMP user name password combination to communicate with the switch The SNMP user name is automatically...

Page 510: ...ce Manager Wrong Version of Device Manager is Launched Symptom When launching Device Manager from the Fabric Manager Web Client the wrong version is opened Table 26 7 Wrong version of Device Manager is opened Symptom Possible Cause Solution Wrong version of Device Manager is opened Java Web Start cache contains multiple versions of Device Manager software Clear the Java Web Start cache See Clearin...

Page 511: ...at the you should perform prior to contacting your next level of support as this will reduce the amount of time spent resolving the issue Note Do not reload the module or the switch at least until you have completed Step 1 below Some logs and counters are kept in volatile storage and will not survive a reload To prepare for contacting your customer support representative follow these steps Step 1 ...

Page 512: ...og which can be displayed using either the show logging log CLI command or the show logging last number to view the last lines of the log Step 4 Answer the following questions before calling for technical support On which switch host bus adapter HBA or storage port is the problem occurring Which Cisco SAN OS software driver versions operating systems versions and storage device firmware are in you...

Page 513: ...iles to or from the Switch It may be required to move files to or from the switch These files may include log configuration or firmware files Copying Files Using Device Manager To copy the configuration from the switch using Device Manager follow these steps Step 1 Choose Admin Copy Configuration You see the Copy Configuration dialog box Step 2 Set the To field to the server where you want to copy...

Page 514: ...tination scp Select source filesystem sftp Select source filesystem slot0 Select source filesystem startup config Copy startup configuration to destination system Select source filesystem tftp Select source filesystem volatile Select source filesystem Use the following syntax to use secure copy scp as the transfer mechanism scp username server path To copy etc hosts from 172 22 36 10 using the use...

Page 515: ...r switch to generate core dumps under the instruction of your customer support representative Core dumps are decoded by technical support engineers Best practice is to set up cores dumps to go to a TFTP server Then these core dumps can be e mailed directly to your customer support representative Setting Up Core Dumps Using the CLI Use the system cores CLI command to set up core dumps on your switc...

Page 516: ... m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m A 6 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Appendix A Before Contacting Technical Support Using Core Dumps ...

Page 517: ...ily Tools If the server does not see its storage and you cannot use the information available on the host side to determine the root cause of the problem you can obtain additional information from a different viewpoint using the troubleshooting tools provided with the Cisco MDS 9000 Family switches This section introduces these tools and describes the kinds of problems for which you can use each t...

Page 518: ...process than sending the debug output to the console By using the option you can see the options that are available for any switch feature such as FSPF A log entry is created for each entered command in addition to the actual debug output The debug output shows a time stamped account of activity occurring between the local switch and other adjacent switches You can use the debug facility to keep t...

Page 519: ...es Cisco Fabric Services debugging cimserver Enables CIM server debugging core Enables core daemon debugging device alias Enables device alias debugging dstats Enables delta statistics debugging ethport Enables port debugging exceptionlog Enables exception log debugging fc tunnel Enables Fibre Channel tunnel debugging fc2 Enables FC2 debugging fc2d Enables FC2D debugging fcc Enables Fibre Channel ...

Page 520: ... The traceroute utility operates in a similar fashion but can also determine the specific path that a frame takes to its destination on a hop by hop basis license Enables license debugging logfile Directs the debug command output to a logfile module Enables module manager debugging ntp Enables NTP debugging platform Enables platform manager debugging port Enables port debugging port channel Enable...

Page 521: ...s The FC Ping feature verifies reachability of a node by checking its end to end connectivity Choose Tools Ping to access FC ping using Fabric Manager Invoke the FC ping feature using the CLI by providing the FC ID or the destination port WWN information in the following ways switch fcping pwwn 20 00 00 2e c4 91 d4 54 switch fcping fcid 0x123abc Example B 1 FC Ping Command switch fcping fcid 0xef0...

Page 522: ...k across EISL links Example B 2 fctraceroute Command switch fctrace fcid 0xef0000 vsan 1 Route present for 0xef0000 20 00 00 05 30 00 59 de 0xfffcee Latency 0 msec 20 00 00 05 30 00 58 1e 0xfffc6c Timestamp Invalid 20 00 00 05 30 00 59 1e 0xfffcef Latency 0 msec 20 00 00 05 30 00 59 1e 0xfffcef Latency 174860 msec 20 00 00 05 30 00 58 1e 0xfffc6c Note The values rendered by the FC traceroute proce...

Page 523: ...ing the show processes CLI Command page B 8 Viewing CPU Time In Device Manager page B 9 Using the show processes cpu CLI Command page B 9 Using the show system resource CLI Command page B 10 Viewing Running Processes on Device Manager Choose Admin Running Processes on Device Manager to view information about the processes currently running on a switch The Running Processes dialog box See Figure B ...

Page 524: ... the processes that are running and the status of each process See Example B 3 The command output includes PID process ID State process state PC current program counter in hex format Start_cnt how many times a process has been started or restarted TTY terminal that controls the process A usually means a daemon not running on any particular TTY Process name of the process Process states are D unint...

Page 525: ...TTY Process 457 S 2abaa76f 1 portmap 1218 S 2acbac24 1 licmgr 1249 S 2ade633e 1 xbar_client 1250 S 2aca833e 1 wwn 1251 S 2aebbc24 1 vsan 1253 S 2ade433e 1 ttyd 1254 S 2ac51ef4 1 sysinfo 1255 S 2af7333e 1 span Viewing CPU Time In Device Manager The Running Processes dialog display can be sorted based on any column header To sort on CPU utilization click the CPU column header An arrow in the column ...

Page 526: ...ut includes the following Load is defined as number of running processes The average reflects the system load over the past 1 5 and 15 minutes Processes displays the number of processes in the system and how many are actually running when the command is issued CPU states shows the CPU usage percentage in user mode kernel mode and idle time in the last one second Memory usage provides the total mem...

Page 527: ...igure OBFL for all the modules on the switch follow these steps Command Purpose Step 1 switch config terminal switch config Enters configuration mode Step 2 switch config hw module logging onboard Enables all OBFL features switch config hw module logging onboard cpu hog Enables the OBFL CPU hog events switch config hw module logging onboard environmental history Enables the OBFL environmental hist...

Page 528: ...ptime device version obfl history Enabled system health Enabled stack trace Enabled Command Purpose Step 1 switch config terminal switch config Enters configuration mode Step 2 switch config hw module logging onboard module 1 Enables all OBFL features on a module switch config hw module logging onboard module 1 cpu hog Enables the OBFL CPU hog events on a module switch config hw module logging onb...

Page 529: ...MDS 9000 Family Fabric Manager Configuration Guide Command Purpose show logging onboard boot uptime Displays the boot and uptime information show logging onboard cpu hog Displays information for CPU hog events show logging onboard device version Displays device version information show logging onboard endtime Displays OBFL logs to an end time show logging onboard environmental history Displays env...

Page 530: ...of a specific switch and shows the status of each port on the switch From Device Manager you can drill down to get detailed statistics about a specific switch or port Figure B 2 shows the Device Manager Summary View window Figure B 2 Cisco Device Manager Summary View The Summary View window lets you analyze switch performance issues diagnose problems and change parameters to resolve problems or in...

Page 531: ...nd by determining if they are in the same VSAN This option uses versions of the ping and traceroute commands modified for Fibre Channel networks The End to End Connectivity Analysis window displays the selected end points with the switch to which each is attached and the source and target ports used to connect it The output shows all the requests which have failed The possible descriptions are Ign...

Page 532: ... a policy file You can save a switch configuration to a file and then compare all switches against the configuration in the file Figure B 4 Fabric Configuration Analysis Window You use a policy file to define the rules to be applied when running the Fabric Checker When you create a policy file the system saves the rules selected for the selected switch Analyzing the Results of Merging Zones Cisco ...

Page 533: ...Alerts and Alarms You can configure and monitor SNMP RMON Syslog and Call Home alarms and notifications using the different options on the Device Manager Events menu SNMP provides a set of preconfigured traps and informs that are automatically generated and sent to the destinations trap receivers that you identify The RMON Threshold Manager lets you configure thresholds for specific events that tr...

Page 534: ...mGroup and EventGroup The AlarmGroup provides services to set alarms Alarms can be set on one or multiple parameters within a device For example an RMON alarm can be set for a specific level of CPU utilization or crossbar utilization on a switch The EventGroup allows configuration of events actions to be taken based on an alarm condition Supported event types include logging SNMP traps and log and...

Page 535: ... in VSAN 1 PWWN is 21 00 00 20 37 46 78 97 LUN Capacity Status Serial Number Device Id MB 0x0 18210 Online LRA2510000007027 C 1 A 0 T 3 20 00 00 20 37 46 78 97 ST318203FC from SEAGATE Rev 0004 FCID is 0xef02b6 in VSAN 1 PWWN is 21 00 00 20 37 5b cf b9 LUN Capacity Status Serial Number Device Id MB 0x0 18210 Online LR94873000007029 C 1 A 0 T 3 20 00 00 20 37 5b cf b9 ST318203FC from SEAGATE Rev 000...

Page 536: ...lled via a MIB Additionally MIBs are provided to configure enable features within the Cisco MDS 9000 Family There are over 20 new MIBs provided by Cisco for this information and configuration capability IETF IP Storage Working Group MIBs for example ISCSI MIB While many of these MIBs are still work in progress Cisco is helping to draft such MIBs for protocols such as iSCSI and Fibre Channel over I...

Page 537: ...ooting purposes and user accountability Accounting can be implemented locally or remotely using RADIUS The following is an example of an accounting log entries switch show accounting log Sun Dec 15 04 02 27 2002 start dev pts 0_1039924947 admin Sun Dec 15 04 02 28 2002 stop dev pts 0_1039924947 admin vsh exited normally Sun Dec 15 04 02 33 2002 start dev pts 0_1039924953 admin Sun Dec 15 04 02 34 ...

Page 538: ...ergency 1 alert 2 critical 3 error 4 warning 5 notification 6 informational 7 debugging By default the switch logs normal but significant system messages to a log file and sends these messages to the system console Users can specify which system messages should be saved based on the type of facility and the severity level Messages are time stamped to enhance real time debugging and management Enab...

Page 539: ... output I O to and from the device This problem is worse when the point of analysis is on an Inter Switch Link ISL link between two switches In this case the disruption may be significant depending on what devices are downstream from the severed ISL link In Ethernet networks this problem can be solved using the SPAN utility which is provided with the Cisco Catalyst Family of Ethernet switches SPAN...

Page 540: ...isco MDS 9000 Family Port Analyzer Adapter The Cisco MDS 9000 Family Port Analyzer Adapter is a stand alone adapter card that converts Fibre Channel frames to Ethernet frames by encapsulating each Fibre Channel frame into an Ethernet frame This product is meant to be used for analyzing SPAN traffic from a Fibre channel port on a Cisco MDS 9000 Family switch The Cisco MDS 9000 Family Port Analyzer ...

Page 541: ... by frame basis complete with timestamps This kind of information lets you pinpoint a problem with a high degree of accuracy and arrive at a solution more quickly However dedicated protocol analyzers are expensive and they must be placed locally at the point of analysis within the network With the Cisco Fabric Analyzer Cisco has brought Fibre Channel protocol analysis within a storage network to a...

Page 542: ...0000 ff ff fd ff ff fd SW_ILS 1 0x59b7 0xffff 0x7 0xf HLO 0 000089 ff ff fd ff ff fd FC 1 0x59b7 0x59c9 0xff 0x0 Link Ctl ACK1 1 991615 ff ff fd ff ff fd SW_ILS 1 0x59ca 0xffff 0xff 0x0 HLO 1 992024 ff ff fd ff ff fd FC 1 0x59ca 0x59b8 0x7 0xf Link Ctl ACK1 fcanalyer example of fully decoded frame switch2 config fcanalyzer local Capturing on eth2 Frame 1 96 bytes on wire 96 bytes captured Arrival ...

Page 543: ...ation Guide IP Network Simulator Network simulators let you simulate various kinds of IP data network conditions A simulator allows you to troubleshoot IP network problems and can also help you understand the potential impact of additional traffic or specific network conditions to your existing network configuration Network simulator is a generic tool that provides simulation features for all Ethe...

Page 544: ...k instead of at a software re assembly layer like most Ethernet analyzers Fibre Channel protocol analyzers can monitor data from the 8b 10b level all the way to the embedded upper layer protocols Fibre Channel network devices HBAs switches and storage subsystems are not able to monitor many SAN behavior patterns Also management tools that gather data from these devices are not necessarily aware of...

Page 545: ...ures a variety of file operations It has been ported to many systems and is useful for performing a broad range of file system tests and analysis Postmark was designed to create a large pool of continually changing files which simulates the transaction rates of a large Internet mail server PostMark generates an initial pool of random text files in a configurable range of sizes Creation of the pool...

Page 546: ...t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m B 30 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 Appendix B Troubleshooting Tools and Methodology Using Host Diagnostic Tools ...

Page 547: ...vent Traps forward via Email 1 destination Up to 10 destinations ISLB VRRP 20 per switch 20 per switch NPV switches per NPV core switch 100 switches 100 switches VSANs 80 VSANs per physical fabric 4000 VSANs per physical fabric VSANs per NPV device 16 16 Switches in a single MDS physical fabric or VSAN 55 switches 1 239 switches Switches in a mixed or open physical fabric or VSAN 32 switches 239 s...

Page 548: ... limit of 3200 Up to 200 ISLs each with 16 VSANs for a total of 3200 port VSAN instances You can configure more than 200 ISLs with fewer than 16 VSANs or fewer than 200 ISLs with more than 16 VSANs within the total ports per VSAN instance limit of 3200 IP ports per switch No limits No limits Fibre Channel modules vs IPS modules per switch No limits No limits iSCSI and iSLB sessions per IP port 500...

Page 549: ...overy from loader 2 16 recovery using BIOS setup procedure 2 16 recovery with dual supervisors 2 23 SSM 3 5 border switch fails 13 10 buffer to buffer credits See BB_credits C Call Home alert groups 25 2 destination profiles 25 1 Call Home alert groups configuring 25 2 customizing 25 2 Call Home contacts assigning information 25 7 Call Home messages configuring levels 25 2 CFS checking distributio...

Page 550: ...ubleshooting checklist 15 1 using blank commit 15 7 validation fails 15 11 VSAN limitations 15 3 Device Manager password issues 26 13 port oversubscription 1 8 digital certificates configuring using Fabric Manager procedure 24 5 configuring using the CLI procedure 24 7 identity certificate 24 4 importing using Fabric Manager procedure 24 11 initial checklist 24 3 25 3 maximum limits 24 3 overview ...

Page 551: ...status 1 5 Fabric Manager checking in Fabric Manager Server license 6 10 map layout 26 4 problems 26 2 recommened JRE version table 2 5 troubleshooting tools 1 3 using over FCIP 26 5 using with multiple NICs 26 5 using with proxy server 26 6 will not start 2 5 Fabric Manager Web Client passwords 26 11 fans LED is red 4 8 not spinning 4 8 FC ID changes after link reset 2 34 FCIP ISL link failures 2...

Page 552: ...rt groups 5 2 port speeds 5 3 troubleshooting checklist 5 7 grace period See licenses guidelines port swapping 16 8 H hardware overview 4 1 startup issues 4 2 troubleshooting 4 13 I IKE allowed transforms table 22 3 debugging 22 15 overview 22 1 verifying configuration compatibility 22 6 images See software in band management CUP 16 6 ip access lists See IP ACLs IP ACLs creating with Fabric Manage...

Page 553: ...enses 13 6 link isolated 13 12 locked CFS session 13 13 LUN 13 13 NAT fails 12 4 13 8 no write access 13 13 overview 13 1 persistent FC IDs 13 12 release specific support table 13 5 restriction 23 2 restrictions 13 2 SDV limitations 12 2 traffic blocked 13 11 transit VSANs 13 2 troubleshooting checklist 13 3 verifying with CLI 13 4 verifying with Fabric Manager 13 3 zone set activation fails 13 9 ...

Page 554: ...ial checklist 9 2 NPIV 9 5 restrictions 9 2 VSAN mismatches 9 5 O OBFL configuring for a module B 12 configuring for the switch B 11 description B 10 displaying configuration status B 12 displaying logs B 13 on board failure logging See OBFL oversubscription 1 8 P PAA B 24 passwords 26 11 polling interval 26 12 Port Analyzer Adapter See PAA PortChannels description 10 1 initial checklist 10 2 load...

Page 555: ...ng autolearn using Fabric Manager procedure 19 13 disabling autolearn using the CLI procedure 19 13 initial checklist 19 2 overview 19 2 verifying database using CLI procedure 19 9 verifying database using Fabric Manager procedure 19 9 verifying violations using Fabric Manager procedure 19 10 verifying violations using the CLI procedure 19 11 port swapping description 16 8 guidelines 16 8 procedur...

Page 556: ...e restart 2 31 upgrading best practices 2 2 verifying installation 2 6 software images selecting for supervisor modules 5 13 Software Installation Wizard procedure 2 10 SSI boot images configuring with install ssi command 3 6 verifying 3 5 SSI boot variables verifying configuration 3 7 SSM description 3 1 Features per release table 3 2 initial checklist 3 3 modflash 3 5 nondisruptive upgrages tabl...

Page 557: ...iolations 4 11 Threshold Manager B 17 traceroute See FC trace troubleshooting common CLI commands 1 4 common Fabric Manager tools 1 3 domain ID conflicts 11 17 FCIP connections 20 9 flowchart 1 9 FSPF issues 11 24 hardware problems 4 13 IP services 20 5 iSCSI issues 20 35 modules 4 21 overview 1 3 power supplies 4 7 SSM recovery 3 8 switching and services modules 4 20 4 25 symptoms 1 9 VSAN isolat...

Page 558: ...1 enhanced 14 21 enhanced zoning lock issues 14 23 host cannot communicate with storage 14 5 link isolation 14 15 maximum number in a switch C 1 maximum number of members C 1 merge failure 8 3 14 13 merging B 16 mismatched active zone sets 14 17 mismatched default zone policy 14 12 port isolation 8 28 troubleshooting checklist 14 1 troubleshooting with CLI 14 2 troubleshooting with Fabric Manager ...

Page 559: ...Se n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m Index IN 11 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 ...

Page 560: ...Se n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck d o c c i s c o c o m Index IN 12 Cisco MDS 9000 Family Troubleshooting Guide Release 3 x OL 9285 05 ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands