manualshive.com logo in svg

Cisco Nexus 1000V, Troubleshooting Manual, Page: 60 / 280

Share
Download
Cisco Nexus 1000V Troubleshooting Manual Download Page 60

 

5-4

Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(1.1)

OL-31593-01

Chapter 5      Upgrades

Problems with the In-Service Software Upgrade

Error message:

Another install procedure 

may be in progress. 

(0x401E0007)

Another upgrade session is in 
progress from a VSM console 
or SSH/Telnet.

Do one of the following:

•

Continue the first upgrade session in progress.

•

Stop the upgrade and restart one session only using the 
instructions in the 

Cisco Nexus 1000V Installation and 

Upgrade Guide

.

The 

install

 command fails 

with following error 
message:

-- FAIL. Return code 

0x4093001E (Standby 

failed to come online)

Install has failed. Return 

code 0x4093001E (Standby 

failed to come online)

The standby VSM fails to 
boot with the new image.

Do one of the following:

•

Restart the software upgrade using the instructions in 
the 

Cisco Nexus 1000V Installation and Upgrade Guide

.

•

Postpone the upgrade and reset the boot variables to the 
original filenames.

boot kickstart

 

filename

 [

sup-1

] [

sup-2

]

The 

install

 command fails 

with following error 
message:

Install has failed. 

Return code 0x4093001F 

(Standby installer failed 

to take over the 

installation).

Please identify the cause 

of the failure, and try 

“install all” again”

The standby VSM takes more 
than 10 minutes to come up 
and form a stable HA pair 
with the active VSM.

1.

Reset the boot variables to the original filenames.

boot kickstart

 

filename

 [

sup-1

] [

sup-2

]

2.

If the standby is still running the new software version, 
reload it. 

reload

The standby synchronizes with the active, so that both 
are running the original software version.

The 

install

 command fails 

with following error 
message:

Module 2: Waiting for 

module online.

 -- SUCCESS

-- Install has failed. 

Return code 0x40930000 

(Current operation failed 

to complete within 

specified time)

A failure at the standby VSM 
caused it to reload again after 
the 

Continuing with 

installation, please wait

 

message and before the 
switchover.

1.

Inspect the logs.

show logging

2.

Look for standby reloads caused by process failures.

show cores

If a process crash is observed, collect details of the 
upgrade and open a support case.

show system internal log install details

3.

Restart the software upgrade using the instructions in 
the 

Cisco Nexus 1000V Installation and Upgrade Guide

.

The pre-upgrade check 
failed:

Return code 0x40930062 
(free space in the filesystem 
is below threshold).

1.

Table 5-1

 (continued)Problems with the ISSU

Symptom

Possible Causes

Solution

Summary of Contents for Nexus 1000V

Page 1: ... has more than 200 offices worldwide Addresses phone numbers and fax numbers are listed on the Cisco website at www cisco com go offices Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 May 16 2016 Text Part Number OL 31393 01 ...

Page 2: ...UPPLIERS DISCLAIM ALL WARRANTIES EXPRESSED OR IMPLIED INCLUDING WITHOUT LIMITATION THOSE OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING USAGE OR TRADE PRACTICE IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT SPECIAL CONSEQUENTIAL OR INCIDENTAL DAMAGES INCLUDING WITHOUT LIMITATION LOST PROFITS OR LOSS OR DAMAGE TO DATA...

Page 3: ...oting Basics 1 2 Troubleshooting Guidelines 1 2 Gathering Information 1 2 Verifying Ports 1 3 Verifying Layer 2 Connectivity 1 3 Verifying Layer 3 Connectivity 1 3 Overview of Symptoms 1 4 System Messages 1 4 System Message Text 1 4 syslog Server Implementation 1 5 Troubleshooting with Logs 1 6 Viewing Logs 1 6 Cisco Support Communities 1 7 Contacting Cisco or VMware Customer Support 1 7 C H A P T...

Page 4: ... 5 Recovering the Network Administrator Password 3 6 Managing Extension Keys 3 6 Known Extension Problems and Resolutions 3 7 Resolving a Plug In Conflict 3 7 Finding the Extension Key on the Cisco Nexus 1000V 3 7 Finding the Extension Key Tied to a Specific DVS 3 8 Verifying Extension Keys 3 8 Recreating the Cisco Nexus 1000V Installation 3 10 Removing Hosts from the Cisco Nexus 1000V DVS 3 11 Re...

Page 5: ...VSM and VEM Modules 7 1 Information About Modules 7 1 Troubleshooting a Module Not Coming Up on the VSM 7 1 Guidelines for Troubleshooting Modules 7 2 Flowchart for Troubleshooting Modules 7 3 Problems with the VSM 7 4 Verifying the VSM Is Connected to vCenter Server 7 6 Verifying the VSM Is Configured Correctly 7 7 Checking the vCenter Server Configuration 7 10 Checking Network Connectivity Betwe...

Page 6: ...ace 10 4 Port Profile Logs 10 5 Port Profile Troubleshooting Commands 10 5 C H A P T E R 11 Port Channels and Trunking 11 1 Information About Port Channels and Trunking 11 1 Port Channel Overview 11 1 Port Channel Restriction 11 2 Trunking Overview 11 2 Initial Troubleshooting Checklist 11 2 Troubleshooting Asymmetric Port Channels 11 3 Cannot Create Port Channel 11 4 Newly Added Interface Does No...

Page 7: ...n the Same Subnet 12 13 Troubleshooting BPDU Guard 12 14 BPDU Guard Troubleshooting Commands 12 14 C H A P T E R 13 VLANs 13 1 Information About VLANs 13 1 Initial Troubleshooting Checklist 13 2 Cannot Create a VLAN 13 3 C H A P T E R 14 Private VLANs 14 1 Information About Private VLANs 14 1 Private VLAN Domains 14 1 Spanning Multiple Switches 14 1 Private VLAN Ports 14 2 Troubleshooting Guidelin...

Page 8: ...Errors 17 4 Debugging Policy Verification Failures 17 5 Debugging Policing Configuration Errors 17 6 C H A P T E R 18 SPAN 18 1 Information About SPAN 18 1 SPAN Session Guidelines 18 1 Problems with SPAN 18 2 SPAN Troubleshooting Commands 18 3 C H A P T E R 19 Multicast IGMP 19 1 Information About Multicast 19 1 Multicast IGMP Snooping 19 1 Problems with Multicast IGMP Snooping 19 2 Troubleshootin...

Page 9: ... H A P T E R 22 System 22 1 Information About the System 22 1 General Restrictions for vCenter Server 22 2 Extension Key 22 2 Recovering a DVS 22 2 Recovering a DVS With a Saved Copy of the VSM 22 3 Recovering a DVS Without a Saved Copy of the VSM 22 4 Problems Related to VSM and vCenter Server Connectivity 22 5 Connection Failure After ESX Reboot 22 6 Setting the System MTU 22 7 Recovering Lost C...

Page 10: ...5 1 Overview 25 1 VXLAN Tunnel EndPoint 25 2 VXLAN Gateway 25 2 VXLAN Trunks 25 3 VXLAN Border Gateway Protocol Control Plane 25 3 Multi MAC Capability 25 8 Fragmentation 25 8 Scalability 25 8 Supported Features 25 9 VXLAN Troubleshooting Commands 25 9 VSM Commands 25 9 VXLAN Gateway Commands 25 11 VEM Commands 25 14 VEM Packet Path Debugging 25 16 VEM Multicast Debugging 25 17 VXLAN Data Path Deb...

Page 11: ... Cisco TrustSec Troubleshooting Commands 27 1 Debugging Commands 27 2 Host Logging Commands 27 2 show Commands 27 4 Problems with Cisco TrustSec 27 5 C H A P T E R 28 vCenter Plug in 28 1 Information About vCenter Plug in 28 1 Prerequisites for VMware vSphere Web Client 28 1 Generating a Log Bundle 28 2 C H A P T E R 29 Ethanalyzer 29 1 Using Ethanalyzer 29 1 ...

Page 12: ...Contents xii Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 ...

Page 13: ...eway Added a section for troubleshooting commands for VXLAN Gateway Note Starting with Release 5 2 1 SV3 1 15 Cisco Nexus 1000V for VMware vSphere does not support the VXLAN Gateway feature 4 2 1 SV2 2 1 VXLANs Upgrade Added section for problems with VSM VEM Layer 2 to 3 Conversion Tool 4 2 1 SV2 1 1 Upgrades Ethanalyzer Added Ethanalyzer as a Cisco Nexus 1000V protocol analyzer tool content 4 2 1...

Page 14: ...g 4 2 1 SV1 5 1 Troubleshooting ACL Logging NSM Added a new chapter to troubleshoot the Network Segmentation Manager NSM 4 2 1 SV1 5 1 Network Segmentation Manager VXLAN Added a new chapter to troubleshoot the Virtual Extensible Local Area Network VXLAN 4 2 1 SV1 5 1 VXLANs Microsoft NLBUnicast Mode Added a new section for troubleshooting Microsoft Network Load Balancing NLB unicast mode 4 2 1 SV1...

Page 15: ...olve the problems related to storm control 5 2 1 SV3 1 1 Storm Control L3Sec Added information about how to secure the internal control plane communications Control and Packet traffic of Cisco Nexus 1000V in a more robust way than in previous releases It operates only in Layer 3 control mode 5 2 1 SV3 1 1 L3Sec Feature Description Changed in release Where Documented ...

Page 16: ...xvi Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 New and Changed Information ...

Page 17: ...re and maintain a Cisco Nexus 1000V Document Conventions Command descriptions use these conventions Screen examples use these conventions Convention Description boldface font Commands and keywords are in boldface italic font Arguments for which you supply values are in italics Elements in square brackets are optional x y z Optional alternative keywords are grouped in brackets and separated by vert...

Page 18: ...y Information Install and Upgrade Cisco Nexus 1000V Installation and Upgrade Guide Configuration Guides Cisco Nexus 1000V High Availability and Redundancy Configuration Guide Cisco Nexus 1000V Interface Configuration Guide Cisco Nexus 1000V Layer 2 Switching Configuration Guide Cisco Nexus 1000V License Configuration Guide Cisco Nexus 1000V Network Segmentation Manager Configuration Guide Cisco Ne...

Page 19: ...ices Appliance Documentation The Cisco Nexus Virtual Services Appliance VSA documentation is available at http www cisco com en US products ps9902 tsd_products_support_series_home html Virtual Security Gateway Documentation The Cisco Virtual Security Gateway documentation is available at http www cisco com en US products ps13095 tsd_products_support_series_home html Virtual Network Management Cent...

Page 20: ... submitting a service request and gathering additional information see What s New in Cisco Product Documentation at http www cisco com en US docs general whatsnew whatsnew html Subscribe to What s New in Cisco Product Documentation which lists all new and revised Cisco technical documentation as an RSS feed and deliver content directly to your desktop using a reader application The RSS feeds are a...

Page 21: ...tacting Cisco or VMware Customer Support page 1 7 Troubleshooting Process To troubleshoot your network follow these steps Step 1 Gather information that defines the specific symptoms Step 2 Identify all potential problems that could be causing the symptoms Step 3 Systematically eliminate each potential problem from most likely to least likely until the symptoms disappear Best Practices We recommen...

Page 22: ... an existing installation It could be a new host switch or VLAN Has the host ever been able to see the network Are you trying to solve an existing application problem too slow high latency excessively long response time or did the problem show up recently What changed in the configuration or in the overall infrastructure immediately before the applications started to have problems To discover a ne...

Page 23: ...he status should be up Are you checking a physical Ethernet port If so you need to check it by looking at the server or by looking at an upstream switch Check if the network adapters of the Virtual Supervisor Module VSM virtual machine VM are assigned the right port groups and if all of them are connected from vSphere Client Verifying Layer 2 Connectivity Answer the following questions to verify L...

Page 24: ...oubleshooting tools Obtain and analyze protocol traces using SPAN or Ethanalyzer on the CLI Identify or rule out physical port issues Identify or rule out switch module issues Diagnose and correct Layer 2 issues Diagnose and correct Layer 3 issues Obtain core dumps and other diagnostic data for use by the Technical Assistance Center TAC Recover from switch upgrade failures System Messages The syst...

Page 25: ...us 1000V to use the syslog facility on a Solaris platform Although a Solaris host is being used the syslog configuration on all UNIX and Linux systems is very similar Syslog uses the concept of a facility to determine how it should be handled on the syslog server the Solaris system in this example and the message severity Therefore different message severities can be handled differently by the sys...

Page 26: ...he IP address of the switch is listed in brackets tail f var adm nxos_logs Sep 17 11 07 41 172 22 36 142 2 2 2004 Sep 17 11 17 29 pacific PORT 5 IF_DOWN_INITIALIZING VLAN 1 Interface e 1 2 is down Initializing Sep 17 11 07 49 172 22 36 142 2 2 2004 Sep 17 11 17 36 pacific PORT 5 IF_UP VLAN 1 Interface e 1 2 is up in mode access Sep 17 11 07 51 172 22 36 142 2 2 2004 Sep 17 11 17 39 pacific VSHD 5 ...

Page 27: ...u are unable to solve a problem after using the troubleshooting suggestions in this guide contact a customer service representative for assistance and further instructions Before you call have the following information ready to help your service provider assist you as quickly as possible Version of the Cisco Nexus 1000V software that you are running Version of the VMware ESX and vCenter Server sof...

Page 28: ...1 8 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 1 Overview Contacting Cisco or VMware Customer Support ...

Page 29: ...he feature configuration status and performance Additionally you can use the following commands for more information show system Provides information on system level components including cores errors and exceptions Use the show system error id command to find details on error codes switch copy running config startup config 100 2008 Jan 16 09 59 29 zoom VDC 1 BOOTVAR 2 AUTOCOPY_FAILED Autocopy of f...

Page 30: ...mmand to access this feature If the destination cannot be reached the path discovery starts which traces the path up to the point of failure Monitoring Processes and CPUs There CLI enables you to for monitor switch processes CPU status and utilization This section contains the following topics Identifying the Running Processes and their States page 2 2 Displaying CPU Utilization page 2 3 Displayin...

Page 31: ...68 1 init 2 S 0 1 migration 0 3 S 0 1 ksoftirqd 0 4 S 0 1 desched 0 5 S 0 1 migration 1 6 S 0 1 ksoftirqd 1 7 S 0 1 desched 1 8 S 0 1 events 0 9 S 0 1 events 1 10 S 0 1 khelper 15 S 0 1 kthread 24 S 0 1 kacpid 101 S 0 1 kblockd 0 102 S 0 1 kblockd 1 115 S 0 1 khubd 191 S 0 1 pdflush 192 S 0 1 pdflushn Displaying CPU Utilization Use the show processes cpu command to display CPU utilization See Exam...

Page 32: ...d average is defined as the number of running processes The average reflects the system load over the past 1 5 and 15 minutes Processes is the number of processes in the system and how many are actually running when the command is issued CPU states is the CPU usage percentage in user mode kernel mode and idle time in the last one second Memory usage provides the total memory used memory free memor...

Page 33: ...ally Sun Dec 15 04 02 33 2002 start dev pts 0_1039924953 admin Sun Dec 15 04 02 34 2002 stop dev pts 0_1039924953 admin vsh exited normally Sun Dec 15 05 02 08 2002 start snmp_1039928528_172 22 95 167 public Sun Dec 15 05 02 08 2002 update snmp_1039928528_172 22 95 167 public Switchname Note The accounting log shows only the beginning and ending start and stop for each session Syslog The system me...

Page 34: ...onsole command in global CONFIGURATION mode To enable logging for Telnet or SSH use the terminal monitor command in EXEC mode Note Note When logging to a console session is disabled or enabled that state is applied to all future console sessions If a user exits and logs in again to a new session the state is preserved However when logging to a Telnet or SSH session is enabled or disabled that stat...

Page 35: ...the Cisco Nexus 1000V Installation page 3 10 Problems with the Cisco Nexus 1000V Installation Management Center page 3 14 Isolating Installation Problems This section explains how to isolate possible installation problems Verifying Your VMware License Version Before you begin to troubleshoot any installation issues you should verify that your ESX server has the VMware Enterprise Plus license that ...

Page 36: ...y that the following are included in the Licensed Features Enterprise Plus license Distributed Virtual Switch feature Step 4 Do one of the following If your vSphere ESX server has an Enterprise Plus license you have the correct license and visibility to the Cisco Nexus 1000V If your vSphere ESX server does not have an Enterprise Plus license you must upgrade your VMware License to an Enterprise Pl...

Page 37: ...r Use the Verifying Your VMware License Version procedure on page 3 1 to confirm Figure 3 2 Host is Not Visible from the Distributed Virtual Switch Refreshing the vCenter Server Connection You can refresh the connection between the Cisco Nexus 1000V and vCenter Server Step 1 From the Cisco Nexus 1000V Connection Configuration mode on the Virtual Supervisor Module VSM enter the following command se...

Page 38: ...V make certain that you configure a domain ID Without a domain ID the VSM cannot connect to the vCenter Server Follow these guidelines The domain ID should be a value within the range of 1 to 4095 All the control traffic between the VSM and the VEM is carried over the configured control VLAN All the data traffic between the VSM and the VEM is carried over the configured packet VLAN Make sure that ...

Page 39: ...ubleshooting connectivity between the VSM and vCenter Server follow these guidelines Make sure that domain parameters are configured correctly Make sure the Windows VM hosting the vCenter Server has the following ports open Port 80 Port 443 Try reloading the VSM if after verifying the preceding steps the connect still fails Check if the VSM extension is created by the vCenter Server by pointing yo...

Page 40: ...f extensionList Cisco_Nexus_1000v_584325821 is displayed in the value column proceed to connect to the VSM Note The actual value of Cisco_Nexus_1000V_584325821 will vary It should match the extension key from the cisco_nexus_1000v_extension xml file Recovering the Network Administrator Password For information about recovering the network administrator password see the Cisco Nexus 1000V Password R...

Page 41: ...Finding the Extension Key on the Cisco Nexus 1000V You can find the extension key on the Cisco Nexus 1000V BEFORE YOU BEGIN Log in to the Cisco Nexus 1000V VSM CLI in EXEC mode Know that you can use the extension key in the Unregistering the Extension Key in the vCenter Server section on page 3 12 DETAILED STEPS Step 1 From the Cisco Nexus 1000V for the VSM whose extension key you want to view ent...

Page 42: ...d to a Specific DVS You can find the extension key tied to a specific DVS Step 1 From the vSphere Client choose the DVS whose extension key you want to find Step 2 Click the Summary tab The Summary tab opens with the extension key displayed in the Notes section of the Annotations block Verifying Extension Keys You can verify that the Cisco Nexus 1000V and vCenter Server are using the same extensio...

Page 43: ...xtension key used on the Cisco Nexus 1000V using the Finding the Extension Key on the Cisco Nexus 1000V section on page 3 7 Step 2 Find the extension key used on the vCenter Server using the Finding the Extension Key Tied to a Specific DVS section on page 3 8 Step 3 Verify that the two extension keys the one found in Step 1 with that in Step 2 are the same ...

Page 44: ...esolved using any other workaround Flowchart Re creating the Cisco Nexus 1000V Installation Re creating the Cisco Nexus 1000V Installation Removing Hosts from the Cisco Nexus 1000V DVS page 3 11 End Install and set up the Cisco Nexus 1000V VSM using the following documents Cisco Nexus 1000V Installation and Upgrade Guide Removing the Cisco Nexus 1000V from the vCenter Server page 3 11 Unregisterin...

Page 45: ...ight click each host and choose Remove from Distributed Virtual Switch The hosts are now removed from the DVS Removing the Cisco Nexus 1000V from the vCenter Server You can remove the Cisco Nexus 1000V DVS from vCenter Server BEFORE YOU BEGIN Log in to the VSM CLI in EXEC mode DETAILED STEPS Step 1 From the Cisco Nexus 1000V VSM use the following commands to remove the DVS from the vCenter Server ...

Page 46: ...n key name into the vCenter Server MOB You should already have the extension key found in the Finding the Extension Key on the Cisco Nexus 1000V section on page 3 7 After unregistering the extension key in vCenter Server you can start a new installation of the Cisco Nexus 1000V VSM software DETAILED STEPS Step 1 Point your browser to the following URL https vc ip mob moid ExtensionManager The Exte...

Page 47: ...sion key that you found in the Finding the Extension Key on the Cisco Nexus 1000V section on page 3 7 and then click Invoke Method The extension key is unregistered in vCenter Server so that you can start a new installation of the Cisco Nexus 1000V VSM software Step 4 You have completed this procedure Return to Flowchart Re creating the Cisco Nexus 1000V Installation section on page 3 10 ...

Page 48: ... installer application finishes successfully with port migration in Layer 3 mode The VEM is added to the vCenter but does not display when the show module command is entered on the VSM Verify that the Layer 3 control profile VLAN is configured as a system VLAN Verify that the uplink profile is allowing the Layer 3 control VTEP VLAN and that it is a system VLAN From the ESX host VEM enter a vmkping...

Page 49: ...kets on the ESX servers attached as Virtual Ethernet Modules VEM to the VSM A module is either licensed or unlicensed Licensed module A VEM is licensed if it acquires licenses for all of its CPU sockets from the pool of available licenses installed on the VSM Unlicensed module A VEM is unlicensed if it does not acquire licenses for all of its CPU sockets from the pool of available licenses install...

Page 50: ...it the contents of the license file The license is invalidated if its contents are altered If you have already done so contact your Cisco Customer Support Account Team Prerequisites to License Troubleshooting Before you begin troubleshooting licenses verify the information in this checklist Make sure that the name of the license file has fewer than 32 characters by using the show license usage com...

Page 51: ...y viewing the sockets installed on the VEM show module vem license info See Example 4 8 on page 4 7 3 Contact your Cisco Customer Support Account Team to acquire additional licenses You see the following system message PLATFORM 2 PFM_LIC_WARN_EXP Syslog 2008 Dec 19 22 28 30 N1KV PLATFORM 2 PFM_LIC_WARN_EXP WARNING License for VEMs is about to expire in 1 days The VEMs VNICS will be brought down if...

Page 52: ...o activate your purchased licenses click on www cisco com go license The clock has been changed back manually or through NTP which has invalidated evaluation licenses The problem is seen even if there are enough permanent licenses available to license the VEMs as long as evaluation licenses are present You can look for the following syslog message to find the time when the clock changed 2014 Jun 7...

Page 53: ...cket 1 VEM 4 Socket 2 switch show interface veth Displays the messages logged about port profile events within the Cisco Nexus 1000V See Example 4 2 on page 4 6 show license host id Displays the serial number for your Cisco Nexus 1000V license See Example 4 3 on page 4 6 show license file Displays the contents of a named license file See Example 4 4 on page 4 6 svs license transfer src vem vem no ...

Page 54: ...ops Example 4 3 show license host id Command switch show license host id License hostid VDH 8449368321243879080 switch Example 4 4 show license file Command switch show license file sample lic sample lic SERVER this_host ANY VENDOR cisco INCREMENT NEXUS1000V_LAN_SERVICES_PKG cisco 3 0 permanent 16 HOSTID VDH 8449368321243879080 NOTICE LicFileID sample lic LicFileID LicLineID 0 LicLineID PAK dummyP...

Page 55: ...cts enabled Y 1 0 dhcp snooping disabled Y 1 0 vxlan gateway enabled Y 1 0 bgp disabled Y 3 0 bpduguard disabled Y 3 0 License Status Edition Available In Use Expiry Date Advanced 17 0 03 Nov 2014 Scale Support Edition Modules Virtual Ports Advanced 256 12288 Example 4 8 show module vem license info Command n100v show module vem license info Licenses are Sticky Mod Socket Count License Usage Count...

Page 56: ...4 8 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 4 Licenses License Troubleshooting Commands ...

Page 57: ...000V involves upgrading software on both the VSM and the Virtual Ethernet Module VEM An in service software upgrade ISSU is available for a stateful upgrade of the Cisco Nexus 1000V image s running on the VSM A stateful upgrade is one without noticeable interruption of data plane services provided by the switch For detailed information see the Cisco Nexus 1000V Installation and Upgrade Guide Probl...

Page 58: ...ed instructions in the Cisco Nexus 1000V Installation and Upgrade Guide Error message Pre Upgrade check failed Return code 0x807B0002 No such file or directory Error message Pre Upgrade check failed Return code 0x4093000F Failed to copy image The software image files required for the upgrade are not present or were not copied to the bootflash repository There may not be enough room in the bootflas...

Page 59: ...ering the install all command Restart the software upgrade using the correct filenames for the new software images install all kickstart filename1 system filename2 After upgrading the VSMs are not running the new software version The boot variables were not set properly 1 Verify that the running images and boot variables match the upgrade version show version show boot 2 If needed download the req...

Page 60: ...the installation Please identify the cause of the failure and try install all again The standby VSM takes more than 10 minutes to come up and form a stable HA pair with the active VSM 1 Reset the boot variables to the original filenames boot kickstart filename sup 1 sup 2 2 If the standby is still running the new software version reload it reload The standby synchronizes with the active so that bo...

Page 61: ...V Installation and Upgrade Guide After upgrading the host is not added to the VSM An incorrect VEM software version is installed on the host 1 Identify the VEM software version required for the upgrade using the Cisco Nexus 1000V Compatibility Information 2 Proceed with the upgrade using the correct VEM software version and the instructions in the Cisco Nexus 1000V Installation and Upgrade Guide A...

Page 62: ...owing sets of procedures to return your VSM pair to the previous software version Recovering a Secondary VSM with Active Primary section on page 5 7 Recovering a Primary VSM with Active Secondary section on page 5 12 2 Restart the software upgrade using the instructions in the Cisco Nexus 1000V Installation and Upgrade Guide The upgrade GUI stops and times out after 10 minutes and displays the fol...

Page 63: ...grade on the VSM using the Stopping a VSM Upgrade section on page 5 8 Step 2 Change the boot variables back to the previous version using the Changing Boot Variables section on page 5 9 Step 3 From the vCenter Server left hand panel right click the secondary VSM and then choose Delete from Disk The secondary VSM is deleted Step 4 Create a new VSM by reinstalling the software using the vSphere Clie...

Page 64: ...Start Upgrade mgmt0 ipv4 addr 1 1 1 1 Upgrade mgmt0 ipv6 addr Upgrade control0 ipv4 addr Step 2 Stop the upgrade a configure terminal b no svs upgrade start Example switch configure terminal switch config no svs upgrade start WARNING VSM upgrade process is aborted switch config Step 3 Display the upgrade status show svs upgrade status Example switch config show svs upgrade status Upgrade State Abo...

Page 65: ...0v mzg 4 2 1 SV1 4 bin No module boot variable set switch config Step 2 Remove the current system and kickstart boot variables a configure terminal b no boot system c no boot kickstart Example switch configure terminal switch config no boot system switch config no boot kickstart switch config Step 3 Restore the system and kickstart boot variables to the original pre upgrade filenames a boot system...

Page 66: ...e bootflash nexus 1000v mz 4 0 4 SV1 3a bin No module boot variable set switch config Step 6 You have completed this procedure Return to one of these sections Recovering a Secondary VSM with Active Primary section on page 5 7 Recovering a Primary VSM with Active Secondary section on page 5 12 Powering On the VSM You can power on the newly created VSM Step 1 From the vCenter Server left hand panel ...

Page 67: ...ig saved but not pushed to vCenter Server due to inactive connection Step 2 Change the HA role system redundancy role primary secondary standalone Example switch config svs domain system redundancy role secondary Setting will be activated on next reload switch config svs domain Example switch config svs domain system redundancy role primary Setting will be activated on next reload switch config sv...

Page 68: ... page 5 12 Step 6 Power on the newly created VSM by completing the Powering On the VSM procedure on page 5 10 The VSM comes up with the standalone HA role Step 7 Change the HA role of the newly created standalone VSM to primary and save the configuration by completing the Changing the HA Role procedure on page 5 11 Step 8 Power off the newly created VSM by completing the Powering Off the VSM proce...

Page 69: ...r 5 Upgrades Problems with the GUI Upgrade Step 3 Select the Management port group and uncheck the following Device Settings Connected Connect at Power On The connection from the VSM to the host server through the management port is dropped and is not restored when you power on the VSM ...

Page 70: ...Center Server left hand panel right click the VSM and then choose Power Power Off The VSM shuts down Step 2 You have completed this procedure Return to the Recovering a Primary VSM with Active Secondary section on page 5 12 Connecting the Port Groups You can make sure that the port groups to the host connect when you power on the VSM Step 1 In vCenter Server select the VSM and then choose Edit Set...

Page 71: ...h the GUI Upgrade Connect at Power On When you power on the VSM it will connect to the host server through the control port Step 3 Select the Management port group and check the following Device Setting Connect at Power On When you power on the VSM it will connect to the host server through the management port ...

Page 72: ...and line window and run an ssh command on the VSM ssh username vsmIPaddress 2 When prompted Are you sure you want to continue connecting enter yes 3 Rerun the VSM VEM Layer 2 to 3 Conversion Tool by reopening the bat file Ensure that the error does not reappear Command Description show boot Displays boot variable definitions showing the names of software images used to boot the VSM See Example 5 2...

Page 73: ...odule NA ok Mod Sw Hw 1 4 2 1 SV1 4a 0 0 2 4 2 1 SV1 4a 0 0 3 4 2 1 SV1 4 1 9 Mod MAC Address es Serial Num 1 00 19 07 6c 5a a8 to 00 19 07 6c 62 a8 NA 2 00 19 07 6c 5a a8 to 00 19 07 6c 62 a8 NA 3 02 00 0c 00 03 00 to 02 00 0c 00 03 80 NA Mod Server IP Server UUID Server Name 1 10 78 109 43 NA NA 2 10 78 109 43 NA NA 3 10 78 109 51 4220900d 76d3 89c5 17d7 b5a7d1a2487f 10 78 109 51 switch Example ...

Page 74: ...g 4 2 1 SV1 4a bin sup 2 boot system bootflash nexus 1000v mzg 4 2 1 SV1 4a bin sup 2 switch Example 5 6 show startup config include boot Command switch show startup config include boot boot kickstart bootflash nexus 1000v kickstart mzg 4 2 1 SV1 4a bin sup 1 boot system bootflash nexus 1000v mzg 4 2 1 SV1 4a bin sup 1 boot kickstart bootflash nexus 1000v kickstart mzg 4 2 1 SV1 4a bin sup 2 boot ...

Page 75: ...pervisor sup 1 Redundancy state Active Supervisor state Active Internal state Active with HA standby Other supervisor sup 2 Redundancy state Standby Supervisor state HA standby Internal state HA standby switch Example 5 10 show vmware vem upgrade status Command switch show vmware vem upgrade status Upgrade VIBs System VEM Image Upgrade Status Upgrade Notification Sent Time Upgrade Status Time vCen...

Page 76: ...5 20 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 5 Upgrades Upgrade Troubleshooting Commands ...

Page 77: ...e architecture Isolation of processes Isolation between software components to prevent a failure within one process disrupting other processes Restartability Most system functions and services are isolated so that they can be restarted independently after a failure while other services continue to run In addition most system services can perform stateful restarts which allow the service to resume ...

Page 78: ...in a port channel fails the traffic previously carried over the failed link switches to the remaining member ports within the port channel Additionally LACP allows you to configure up to 16 interfaces into a port channel A maximum of eight interfaces can be active and a maximum of eight interfaces can be placed in a standby state For additional information about port channels and LACP see the Cisc...

Page 79: ...t bring up the standby VSM after network connectivity is restored The active VSM does not complete synchronization with the standby VSM Version mismatch between VSMs Check that the primary and secondary VSMs are using the same image version by using the show version command If the active and the standby VSM software versions differ reinstall the secondary VSM with the same version used in the prim...

Page 80: ...nd the standby goes down for 6 seconds the standby VSM transitions to the active state Upon restoration of control and management connectivity both VSMs detect an active active condition 1 Once the system detects active active VSMs one VSM is automatically reloaded based on various parameters such as VEMs attached vCenter connectivity last configuration time and last active time 2 To see any confi...

Page 81: ...red Different domain IDs in the two VSMs Check the domain value by using show system internal redundancy info command If needed update the domain ID and save it to the startup configuration Upgrading the domain ID in a dual VSM system must be done as follows Isolate the VSM with the incorrect domain ID so that it cannot communicate with the other VSM Change the domain ID in the isolated VSM save t...

Page 82: ...V_OPTION_RESTART_STATELESS 23 Death reason SYSMGR_DEATH_REASON_FAILURE_SIGNAL 2 Reason for the process abort Last heartbeat 46 88 secs ago System image name nexus 1000v mzg 4 0 4 SV1 1 bin System image version 4 0 4 SV1 1 S25 PID 3207 Exit code signal 6 core dumped Indicates that a cores for the process was generated CWD var sysmgr work To check redundancy status use the following commands Example...

Page 83: ...M role primary Redundancy role of this VSM status RDN_ST_AC Indicates redundancy state RDN_ST of the this VSM is Active AC state RDN_DRV_ST_AC_SB intr enabled power_off_reqs 0 reset_reqs 0 Other CP slot 1 status RDN_ST_SB Indicates redundancy state RDN_ST of the other VSM is Standby SB active true ver_rcvd true degraded_mode false When true it indicates that communication through the control inter...

Page 84: ... following command Example 6 5 show system internal sysmgr state Command switch show system internal sysmgr state The master System Manager has PID 1988 and UUID 0x1 Last time System Manager was gracefully shutdown The state is SRV_STATE_MASTER_ACTIVE_HOTSTDBY entered at time Tue Apr 28 13 09 13 2009 The b option disable heartbeat is currently disabled The n don t use rlimit option is currently di...

Page 85: ...tatus Redundancy role administrative secondary operational secondary Redundancy mode administrative HA operational HA This supervisor sup 2 Redundancy state Active Supervisor state Active Internal state Active with HA standby Other supervisor sup 1 Redundancy state Standby Supervisor state HA standby Internal state HA standby WARNING Conflicting sup 2 s detected in same domain MAC Latest Collision...

Page 86: ...following command Example 6 8 reload module Command switch reload module 2 This command reloads the secondary VSM Note Entering the reload command without specifying a module will reload the whole system To attach to the standby VSM console use the following command Example 6 9 attach module Command The standby VSM console is not accessible externally but can be accessed from the active VSM throug...

Page 87: ...e VSM Control software of the Cisco Nexus 1000V distributed virtual switch It runs on a virtual machine VM and is based on NX OS software Virtual Ethernet Module VEM Part of the Cisco Nexus 1000V that actually switches data traffic It runs on a VMware ESX host Several VEMs are controlled by one VSM All the VEMs that form a switch domain should be in the same virtual data center as defined by VMwar...

Page 88: ...is added with the same capability then the host connected as VEM modules on VSM in L3 mode will go offline To recover from this scenario we need to remove both the vmknics from Cisco Nexus1000V DVS or migrate them back to vSwitch VMware DVS Once you migrate or removed you can recreate one vmknic on Cisco Nexus1000V DVS or migrate one of the vmknic from vswitch VMware DVS back to Cisco Nexus1000V D...

Page 89: ... troubleshoot modules Flowchart Troubleshooting Modules Troubleshooting Modules End Checking the vCenter Server Configuration page 7 10 Verifying the VSM Is Configured Correctly page 7 7 Checking the VEM Configuration page 7 14 Collecting Logs page 7 16 Verify VSM and VEM Image Versions For more information see the Cisco Nexus 1000V Compatibility Information Checking Network Connectivity Between t...

Page 90: ...n the vCenter Server Following a reboot of the VSM the system stops functioning in one of the following states and does not recover on its own Attempts to debug fail After boot VSM is in loader prompt Corrupt VSM kickstart image 1 Boot the VSM from the CD ROM 2 From the CD Boot menu choose Option 1 Install Nexus1000v and bring up new image Follow the VSM installation procedure Boot variables are n...

Page 91: ...unning config If not reconfigure the VSM using the following section in the Cisco Nexus 1000V Getting Started Guide Setting Up the Software After boot the secondary VSM reboots continuously Control VLAN or control interface down Check control connectivity between the active and the standby VSM Active and standby VSMs fail to synchronize From the active VSM check system manager errors to identify w...

Page 92: ...es section on page 7 3 If not continue with the next step Step 3 Connect to vCenter Server config t svs connection connection_name connect Example switch conf t switch config svs connection HamiltonDC switch config svs conn connect Example switch conf t switch config svs connection HamiltonDC switch config svs conn connect ERROR VMWARE VIM Extension key was not registered before its use Step 4 Do ...

Page 93: ...ured Correctly This section includes the following topics Verifying the Domain Configuration page 7 7 Verifying the System Port Profile Configuration page 7 8 Verifying the Control and Packet VLAN Configuration page 7 8 Verifying the Domain Configuration You can verify the domain configuration BEFORE YOU BEGIN Log in to the CLI in EXEC mode Verify that the output of the show svs domain command ind...

Page 94: ...how port profile name SystemUplink port profile SystemUplink description type ethernet status enabled capability l3control no pinning control vlan pinning packet vlan system vlans 114 115 port group SystemUplink max ports 32 inherit config attributes switchport mode trunk switchport trunk allowed vlan all system mtu 1500 no shutdown evaluated config attributes switchport mode trunk switchport trun...

Page 95: ... 0000 00000000000f Card name Switch name Nexus1000v Switch alias DvsPortset 0 Switch uuid ee 63 3c 50 04 b1 6d d6 58 61 ff ba 56 05 14 fd Card domain 27 Card slot 3 VEM Tunnel Mode L2 Mode VEM Control AIPC MAC 00 02 3d 10 1b 02 VEM Packet inband outband MAC 00 02 3d 20 1b 02 VEM Control Agent DPA MAC 00 02 3d 40 1b 02 VEM SPAN MAC 00 02 3d 30 1b 02 Primary VSM MAC 00 50 56 bc 74 f1 Primary VSM PKT...

Page 96: ...e data center and the Cisco Nexus 1000V DVS in that data center Step 2 Confirm that at least one pnic of the host is added to the DVS and that pnic is assigned to the system uplink profile Step 3 Confirm that the three VSM vnics are assigned to the port groups that contain the control VLAN packet VLAN and management network Checking Network Connectivity Between the VSM and the VEM You can verify L...

Page 97: ...re not reaching the VEM Your uplink configuration is correct Recommended action Check if the VEM s upstream switch has learned the VSM s Control MAC Step 4 Do one of the following If the VEM health check in Step 3 indicates a problem with connectivity to the upstream switch continue with the next step Otherwise go to Step 7 Step 5 On the upstream switch display the MAC address table to verify the ...

Page 98: ...ou have completed this procedure Recovering Management and Control Connectivity of a Host When a VSM is Running on a VEM When the VSM is running on a VEM that it manages but the VSM ports are not configured with system port profiles the control and management connectivity of the VSM can be lost after a host reboot or similar event To recover from the loss you can run the VEM connect script locally...

Page 99: ...ists match all of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking nonallowed VLANs Step 2 Display details about the system VLANs vemcmd show port vlans system Example vemcmd show port vlans system Native VLAN Allowed LTL VSM Port Mode VLAN State Vlans SegID SegID 6 Internal A 1 FWD 1 8 Internal A 3969 FWD 3969 9 Internal A 3969 FWD 3969 10 Internal A 210 FWD 210 11 Internal...

Page 100: ...32 49 Veth1 A 1 FWD 1 50 Veth10 A 232 FWD 232 305 Po2 T 1 FWD 210 211 232 Checking the VEM Configuration You can verify that the ESX host received the VEM configuration and setup Step 1 On the ESX host confirm that the VEM Agent is running and that the correct host uplinks are added to the DVS vem status Example vem status VEM modules are loaded Switch Name Num Ports Used Ports Configured Ports MT...

Page 101: ...ores 4 Processor Sockets 2 Physical Memory 4290351104 Step 4 Verify that the ports of the host added to the DVS are listed and that the ports are correctly configured as access or trunk on the host vemcmd show port Example vemcmd show port LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name 8 0 3969 0 2 2 VIRT UP UP 1 Access l20 9 0 3969 0 2 2 VIRT UP UP 1 Access l21 10 0 3002 0...

Page 102: ...itself would not be visible because the VEM is not yet added to the VSM Step 7 Restore connectivity that is lost due to incorrect port and system VLAN settings vemcmd show port port ltl number vemcmd set system vlan vlan_id ltl port ltl number Example vemcmd show port 48 LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name 48 1b030000 1 0 32 1 VIRT UP DOWN 0 Access vmk1 vemcmd se...

Page 103: ...553 4538 35314e35545a licensed switch Step 3 Using the module number from Step 2 collect the output of the following commands show system internal vem_mgr event history module 13 show module internal event history module 13 show system internal im event history module 13 show system internal vmm event history module 13 show system internal ethpm event history module 13 Note If you need to contact ...

Page 104: ...s running and that the correct host uplinks are added to the DVS See Example 7 9 on page 7 21 vemcmd show card Displays information about cards on the VEM to verify that the domain ID control VLANs and packet VLANs are configured correctly on the host See Example 7 10 on page 7 21 vemcmd show port port LTL number Displays information about ports on the VEM to verify that the ports of the host adde...

Page 105: ...e 7 3 show svs domain Command switch show svs domain SVS domain config Domain id 682 Control vlan 3002 Packet vlan 3003 show module vem mapping Displays information about the VEM that a VSM maps to including the VEM module number status UUID and license status See Example 7 17 on page 7 22 show system internal vem_mgr event history module 13 module number Displays module FSM event information show...

Page 106: ...own evaluated config attributes switchport mode trunk switchport trunk allowed vlan all no shutdown assigned interfaces Example 7 5 show running configuration vlan Command switch show running config vlan 260 261 version 4 0 4 SV1 3 vlan 260 name cp_control vlan 261 name cp_packet switch Example 7 6 vem health check Command vem health check 00 50 56 a3 36 90 VSM Control MAC address 00 50 56 a3 36 9...

Page 107: ...igured Ports Uplinks switch 256 9 256 vmnic1 VEM Agent is running Example 7 10 vemcmd show card Command vemcmd show card Card UUID type 2 58f8afd7 e1e3 3c51 85e2 6e6f2819a7b8 Card name sfish srvr 1 Switch name switch Switch alias DvsPortset 0 Switch uuid 56 e0 36 50 91 1c 32 7a e9 9f 31 59 88 0c 7f 76 Card domain 1024 Card slot 4 VEM Control Control VLAN MAC 00 02 3d 14 00 03 VEM Packet inband out...

Page 108: ...port vlans Native VLAN Allowed LTL VSM Port Mode VLAN State Vlans 17 Eth5 1 T 1 FWD 1 100 119 219 319 18 Eth5 2 T 1 FWD 1 100 119 219 319 49 Veth11 A 119 FWD 119 50 Veth14 A 119 FWD 119 51 Veth15 A 119 FWD 119 305 Po1 T 1 FWD 1 100 119 219 319 Note The output F B The port is blocked on some of the VLANs means that the trunk is not forwarding all VLANs This might be a normal situation depending on ...

Page 109: ...ting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 7 VSM and VEM Modules VSM and VEM Troubleshooting Commands 60 absent 33393935 3234 5553 4538 35314e355400 unlicensed 66 powered up 33393935 3234 5553 4538 35314e35545a licensed switch ...

Page 110: ...7 24 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 7 VSM and VEM Modules VSM and VEM Troubleshooting Commands ...

Page 111: ...L3Sec Possible Causes Solution SVS connection is not up 1 Verify SVS connection Show svs connection 2 If the connection is not connected do connect Key mismatch between VSM VEM 1 Verify key fields mismatch between switch opaque data and vem 2 Do show vms internal info dvs and check the keys present 3 On vem perform vemcmd show sod and check if the fields chunk1 chunk2 and chunk3 are matching 4 If ...

Page 112: ...8 2 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 8 L3Sec Troubleshooting L3Sec ...

Page 113: ...tch can relay frames from one data link to another you must define the characteristics of the interfaces through which the frames are received and sent The configured interfaces can be Ethernet physical interfaces virtual Ethernet interfaces and the management interface Each interface has the following Administrative Configuration The administrative configuration does not change unless you modify ...

Page 114: ...link flapping When a port is flapping it cycles through the following states in this order and then starts over again 1 Initializing The link is initializing 2 Offline The port is offline 3 Link failure or not connected The physical layer is not operational and there is no active device connection To troubleshoot link flapping see the Information About Link Flapping section on page 9 2 Information...

Page 115: ...ping page 9 5 Port ErrDisabled page 9 6 VM Cannot Ping a Secured Port page 9 7 Port Security Violations page 9 8 Port State is Blocked on a VEM page 9 9 Table 9 1 Port Diagnostic Checklist Checklist Example Verify that the module is active show module See Example 9 1 on page 9 11 Verify that the VSM is connected to vCenter Server show svs connections See Example 9 3 on page 9 12 On vSphere Client ...

Page 116: ...e the VLAN as active config t vlan vlan id state active Possible Cause Solution The port connection is bad 1 Verify the port state show system internal ethpm info 2 Disable and then enable the port shut no shut 3 Move the connection to a different port on the same module or a different module 4 Collect the ESX side NIC configuration vss support The link is stuck in initialization state or the link...

Page 117: ...nd then enable the port shut no shut The port should return to the normal state A hardware failure or intermittent hardware error causes a packet drop in the switch A software error causes a packet drop A control frame is erroneously sent to the device An external device might choose to initialize the link again when encountering the error If so the exact method of link initialization varies by de...

Page 118: ...fy the exact configuration error in the list of port state changes show logging logfile 2 Correct the error in the configuration and add the port to the port channel 3 Re enable the port shut no shut A VSM application error has occurred 1 Identify the component that had an error while you were bringing up the port show logging logfile grep interface_number See Example 9 7 on page 9 13 2 Identify t...

Page 119: ...M is set clear the DSM bit on the VSM no port security stop learning The packet VLAN is not allowed on the port 1 Identify the packet VLAN ID show svs domain 2 Verify that the packet VLAN is allowed on VEM uplink ports show port profile na uplink all 3 If the packet VLAN is not allowed on the uplink port profile add it to the allowed VLAN list The packet VLAN is not allowed on the upstream switch ...

Page 120: ...tion Guide Possible Cause Solution The configured maximum number of secured addresses on the port is exceeded 1 Display the secure addresses show port security address vethernet number show port security address interface vethernet number 2 Identify ports with a security violation show logging inc PORT SECURITY 2 ETH_PORT_SEC_SECURITY_VIOLAT ION_MAX_MAC_VLAN 3 Correct the security violation 4 Enab...

Page 121: ...n vlan id On the VEM module do the following 1 Verify that the VLAN is programmed vemcmd show vlan vlan id 2 Verify that the VLAN is allowed on the ports vemcmd show port vlan 3 Create the VLAN on the VSM vlan vlan id The VEM modules are unlicensed 1 Verify that all the modules are in licensed state show module 2 Verify the status of the vEthernet interface It should be up and not VEM Unlicensed s...

Page 122: ...ory interface Displays information about the internal state transitions of the port See Example 9 5 on page 9 12 show logging logfile Displays logged system messages See Example 9 6 on page 9 12 show logging logfile grep interface_number Displays logged system messages for a specified interface See Example 9 7 on page 9 13 show interface brief Displays a table of interface states See Example 9 8 o...

Page 123: ...w interface vethernet Displays the vEthernet interface configuration See Example 9 12 on page 9 14 show interface status Displays the status of the named interface show interface capabilities Displays a tabular view of all configured port profiles See Example 9 13 on page 9 14 show interface virtual port mapping Displays the virtual port mapping for all vEthernet interfaces See Example 9 14 on pag...

Page 124: ... Example 9 4 show cdp neighbors Command switch show cdp neighbors Capability Codes R Router T Trans Bridge B Source Route Bridge S Switch H Host I IGMP r Repeater V VoIP Phone D Remotely Managed Device s Supports STP Dispute Device ID Local Intrfce Hldtme Capability Platform Port ID swordfish 6k 2 Eth3 2 149 R S I WS C6506 E Gig1 38 switch Example 9 5 show port internal event history interface Com...

Page 125: ..._SAP_ACLMGR for opcode MTS_OPC_ETHPM_PORT_PRE_CFG RID_PORT Vethernet3626 2011 Mar 25 11 10 06 n1k bl ETHPORT 2 IF_DOWN_ERROR_DISABLED Interface Vethernet3626 is down Error disabled Reason Client data inconsistency Example 9 8 show interface brief Command switch show int brief Port VRF Status IP Address Speed MTU management interface0 up 172 23 232 141 1000 1500 Ethernet VLAN Type Mode Status Reaso...

Page 126: ...is gentoo1 Network Adapter 1 Hardware is Virtual address is 0050 56bd 42f6 Owner is VM gentoo1 adapter is Network Adapter 1 Active on module 33 VMware DVS port 100 Port Profile is vlan48 Port mode is access Rx 491242 Input Packets 491180 Unicast Packets 7 Multicast Packets 55 Broadcast Packets 29488527 Bytes Tx 504958 Output Packets 491181 Unicast Packets 1 Multicast Packets 13776 Broadcast Packet...

Page 127: ...l unavailable Type unknown Speed 10 100 1000 10000 auto Duplex half full auto Trunk encap type 802 1Q Channel yes Broadcast suppression percentage 0 100 Flowcontrol rx off on desired tx off on desired Rate mode none QOS scheduling rx none tx none CoS rewrite yes ToS rewrite yes SPAN yes UDLD no Link Debounce no Link Debounce Time no MDIX no Port Group Members none port channel12 Model unavailable ...

Page 128: ... Veth2 DVPort3361 static up none switch Example 9 15 module vem execute vemcmd show portsec status Command cyp1 switch module vem 3 execute vemcmd show portsec status LTL if_index Max Aging Aging DSM Sticky VM Secure Time Type Bit Enabled Name Addresses 56 1c0000a0 5 0 Absolute Clr No Ostinato Upgrade VM1 eth1 Example 9 16 show port security Command switch show port security Total Secured Mac Addr...

Page 129: ...9 17 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 9 Ports Port Troubleshooting Commands 50 0050 56a4 38ec STATIC Vethernet11 0 50 0000 0000 0011 DYNAMIC Vethernet11 ...

Page 130: ...9 18 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 9 Ports Port Troubleshooting Commands ...

Page 131: ...re assigned in vCenter Server to a port profile for the following reasons Defining a port configuration by policy Applying a single policy across a large number of ports Supporting both vEthernet and Ethernet ports vEthernet port profiles can be assigned by the server administrator to physical ports a VMNIC or a PNIC Port profiles not configured as vEthernet can be assigned to a VM virtual port No...

Page 132: ... not see the port group on vCenter Server or the following message is displayed Warning Operation succeeded locally but update failed on vCenter server Please check if you are connected to vCenter Server The connection to vCenter server is down 1 Verify that the connection to vCenter Server is Enabled and Connected show svs connections 2 Reconnect to vCenter server For detailed instructions see th...

Page 133: ...rantine_Uplink for ethernet types Unused_Or_Quarantine_Veth for Vethernet types 1 Verify the port profile to interface mapping show port profile virtual usage 2 Reassign the VMNIC or PNIC to a non quarantined port group to enable the interface to be up and forwarding traffic This requires changing the port group on vCenter Server After applying a port profile an online interface is quarantined A s...

Page 134: ... port profile sync status After modifying a port profile an assigned offline interface is quarantined A system message similar to the following is logged PORT PROFILE 2 INTERFACE_QUARAN TINED Interface Ethernet4 3 has been quarantined due to Cache Overrun The interface has been removed from the DVS To bring the interface back online see the Recovering a Quarantined Offline Interface section on pag...

Page 135: ...onfiguration See Example 10 1 on page 10 6 show port profile name name Displays the configuration for a named port profile See Example 10 2 on page 10 7 show port profile brief Displays a tabular view of all configured port profiles See Example 10 3 on page 10 7 show port profile expand interface Displays all configured port profiles expanded to include the interfaces assigned to them See Example ...

Page 136: ...t profile role users Displays the available users and groups See Example 10 8 on page 10 9 show port profile sync status interface if name Displays the interfaces that are not synchronized with the port profile See Example 10 9 on page 10 9 show port profile virtual usage name profile name Displays the port profile usage by interface See Example 10 10 on page 10 9 show msp internal info Displays t...

Page 137: ...e access ip port access group acl1 in capability vxlan no shutdown assigned interfaces port group 1 system vlans none capability l3control no capability iscsi multipath no capability vxlan yes capability l3 vservice no port profile role none port binding static Example 10 3 show port profile brief Command switch show port profile brief VM_PP_NIC8_VLAN_1338 Vethernet 1 3 3 374 0 VM_PP_NIC9_VLAN_133...

Page 138: ...file EthProfile1 Ethernet2 2 switchport mode trunk switchport trunk allowed vlan 110 119 no shutdown switch Example 10 6 show running config port profile Command switch show running config port profile port profile type ethernet UplinkProfile1 description Profile for critical system ports vmware port group switchport mode access switchport access vlan 113 switchport trunk native vlan 113 channel g...

Page 139: ...rt profile virtual usage Command switch show port profile virtual usage Port Profile Port Adapter Owner n1kv uplink0 Po1 Eth3 2 vmnic1 localhost Eth3 3 vmnic2 localhost vlan1767 Veth7 Net Adapter 1 all tool 7 Veth8 Net Adapter 1 all tool 8 aipc1765 Veth4 Net Adapter 1 bl h s inband outband interface 1766 Veth6 Net Adapter 3 bl h s mgmt1764 Veth5 Net Adapter 2 bl h s vpc mac uplink Po7 Eth5 2 vmnic...

Page 140: ...ode system vlans port binding static max ports 32 vmware config information pg name Unused_Or_Quarantine_Uplink dvs ignore port profile role alias information pg id Unused_Or_Quarantine_Uplink dvs uuid type 1 pg id dvportgroup 2444 dvs uuid 44 dc 3b 50 53 11 b7 ac ef ed ef 46 ee df c2 d5 type 2 port profile Unused_Or_Quarantine_Veth id 2 capability 0x0 state 0x1 type 0x1 system vlan mode system vl...

Page 141: ...lan mode system vlans port binding static max ports 32 vmware config information pg name eth break inherit dvs ignore port profile role alias information pg id eth break inherit dvs uuid type 1 pg id dvportgroup 3287 dvs uuid 44 dc 3b 50 53 11 b7 ac ef ed ef 46 ee df c2 d5 type 2 pg id dvportgroup 3294 dvs uuid 44 dc 3b 50 53 11 b7 ac ef ed ef 46 ee df c2 d5 type 2 port profile uplink id 3 capabil...

Page 142: ...lans port binding static max ports 256 vmware config information pg name veth break deinherit dvs ignore port profile role alias information pg id veth break deinherit dvs uuid type 1 pg id dvportgroup 3289 dvs uuid 44 dc 3b 50 53 11 b7 ac ef ed ef 46 ee df c2 d5 type 2 pg id dvportgroup 3296 dvs uuid 44 dc 3b 50 53 11 b7 ac ef ed ef 46 ee df c2 d5 type 2 port profile veth break inherit id 7 capab...

Page 143: ...file fsm Command switch show system internal port profile profile fsm FSM PROFILE_FSM 1 has 4 logged transitions 1 FSM PROFILE_FSM 1 Transition at 856903 usecs after Tue Mar 8 19 11 47 2011 Previous state PPM_PROFILE_ST_SIDLE Triggered event PPM_PROFILE_EV_EIF_STATUS_CHANGE Next state PPM_PROFILE_ST_SIDLE 2 FSM PROFILE_FSM 1 Transition at 858442 usecs after Tue Mar 8 19 11 47 2011 Previous state P...

Page 144: ...e 234 Payload 0x0000 02 00 00 03 00 00 00 00 00 00 03 02 03 02 00 00 3 Event E_MTS_RX length 60 at 624319 usecs after Tue Mar 8 19 12 05 2011 NOT Opc MTS_OPC_PPM_INTERFACE_UPDATE 152601 Id 0X00003908 Ret SUCCESS Src 0x00000101 489 Dst 0x00000101 0 Flags None HA_SEQNO 0X00000000 RRtoken 0x00000000 Sync UNKNOWN Payloadsize 107 Payload 0x0000 00 00 00 02 00 00 00 02 00 00 00 0c 00 00 00 26 4 Event E_...

Page 145: ...ollowing topics Port Channel Overview page 11 1 Trunking Overview page 11 2 Port Channel Overview Port channels aggregate multiple physical interfaces into one logical interface to provide higher bandwidth load balancing and link redundancy A port channel performs the following functions Increases the aggregate bandwidth on a link by distributing traffic among all functional links in the channel L...

Page 146: ...ecklist Use the show port channel compatibility parameters CLI command to determine port channel requirements Ensure that all interfaces in the port channel have the same destination device for Link Aggregation Control Protocol LACP channels By using the Asymmetric Port Channel APC feature in the Cisco Nexus 1000V ports in an ON mode channel can be connected to two different destination devices No...

Page 147: ...R Routed U Up port channel Group Port Type Protocol Member Ports Channel 1 Po1 SU Eth NONE Eth3 4 P 2 Po2 SU Eth NONE Eth3 2 P Eth3 6 P Troubleshooting Asymmetric Port Channels When you are troubleshooting asymmetric port channels follow these guidelines Use APC when you want to configure a port channel whose members are connected to two different upstream switches APC depends on Cisco Discovery P...

Page 148: ...ort channel summary command to verify the number of port channels already configured You can have a maximum of 256 port channels on the Cisco Nexus 1000V Symptom Possible Cause Solution Newly added interface does not come online in a port channel The port channel mode is on 1 Make sure that you have the port channel configuration in the port profile port group used by that interface 2 Check if a p...

Page 149: ...g a Port Channel Configuration You can debug port channels configured through a port profile BEFORE YOUR BEGIN Log in to the CLI in configuration mode DETAILED STEPS Step 1 Verify that you have configured a port channel in the profile switch show port profile name profile name Step 2 Display summary port channel information switch show port channel summary Step 3 Debug the port channel configurati...

Page 150: ...11 6 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 11 Port Channels and Trunking VLAN Traffic Does Not Traverse Trunk ...

Page 151: ...rd page 12 14 Information About Layer 2 Ethernet Switching The Cisco Nexus1000V is a distributed Layer 2 virtual switch that extends across many virtualized hosts It consists of two components The Virtual Supervisor Module VSM which is also known as the control plane CP The VSM acts as the supervisor and contains the Cisco CLI configuration and high level features The Virtual Ethernet Module VEM w...

Page 152: ...S and other network access needed by the kernel This interface carries the IP address of the hypervisor itself and is also bound to a virtual Ethernet port The vswif not shown appears only in CoS based systems and is used as the VMware management port Each type maps to a virtual Ethernet port within the Cisco Nexus1000V Virtual Ethernet Ports VEth A vEth port is a port on the Cisco Nexus 1000V The...

Page 153: ...1 1 between an uplink port and a vmnic Each physical port added to the Cisco Nexus1000V switch appears as a physical Ethernet port just as it would on a hardware based switch The uplink port concept is handled entirely by VMware and is used to associate port configuration with vmnics There is no fixed relationship between the uplink number and vmnic number These can be different on different hosts...

Page 154: ...n includes the following topics Verifying a Connection Between VEM Ports page 12 4 Verifying a Connection Between VEMs page 12 5 Isolating Traffic Interruptions page 12 6 Verifying a Connection Between VEM Ports You can verify a connection between two vEth ports on a VEM Step 1 View the state of the VLANs associated with the port If the VLAN associated with a port is not active the port may be dow...

Page 155: ... two vEth ports are listed in the flood list of the VLAN with which they are trying to communicate switch module vem 3 execute vemcmd show bd Step 5 Verify that the uplink switch to which the VEMs are connected is carrying the VLAN to which the ports belong Step 6 Find out the port on the upstream switch to which the PNIC that is supposed to be carrying the VLAN on the VEM is connected to switch s...

Page 156: ... status enabled system vlans 3002 3003 port group alluplink config attributes switchport mode trunk switchport trunk allowed vlan 1 80 3002 610 620 630 650 no shutdown evaluated config attributes switchport mode trunk switchport trunk allowed vlan 1 80 3002 3003 610 620 630 650 no shutdown assigned interfaces Ethernet2 2 Step 2 Inside the VM verify that the Ethernet interface is up ifconfig a If n...

Page 157: ...ify all MAC addresses on all VEMs controlled by the VSM See Example 12 1 on page 12 8 show mac address table module module number Displays all the MAC addresses on the specified VEM show mac address table static HHHH WWWW HHHH Displays the MAC address table static entries See Example 12 2 on page 12 9 show mac address table address HHHH WWWW HHHH Displays the interface on which the MAC address spe...

Page 158: ...N1KV Internal Port 3 3 12ab 47dd ff89 static 0 Eth3 3 3 342 0002 3d41 5502 static 0 N1KV Internal Port 3 342 0050 568d 5a3f dynamic 0 Eth3 3 3 343 0002 3d21 5502 static 0 N1KV Internal Port 3 show interface brief Displays a table of interface states See Example 12 5 on page 12 10 module vem module number execute vemcmd show port On the VEM displays the port state on a particular VEM This command c...

Page 159: ...3 show mac address table static inc veth Command switch show mac address table static inc veth 460 0050 5678 ed16 static 0 Veth2 3 460 0050 567b 1864 static 0 Veth1 4 switch Example 12 4 show vlan Command Tip This command shows the state of each VLAN created on the VSM switch show vlan VLAN Name Status Ports 1 default active Eth3 3 Eth3 4 Eth4 2 Eth4 3 110 VLAN0110 active 111 VLAN0111 active 112 V...

Page 160: ...AN Type Mode Status Reason Speed Port Interface Ch Eth3 4 1 eth trunk up none 1000 D Eth4 2 1 eth trunk up none 1000 D Eth4 3 1 eth trunk up none 1000 D Example 12 6 module vem module number execute vemcmd show port Command Tip Look for the state of the port module vem 3 execute vemcmd show port LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name 8 0 3969 0 2 2 VIRT UP UP 1 Acce...

Page 161: ...ports Portlist 16 vmnic1 BD 114 vdc 1 vlan 114 1 ports Portlist 16 vmnic1 BD 115 vdc 1 vlan 115 2 ports Portlist 10 l22 16 vmnic1 Example 12 8 module vem module number execute vemcmd show trunk Command Tip If a VLAN is active on a port its CBL state should be 1 If a VLAN is blocked its CBL state is 0 module vem 5 execute vemcmd show trunk Trunk port 16 native_vlan 1 CBL 1 vlan 1 cbl 1 vlan 110 cbl...

Page 162: ...through the website Cisco s End User License Agreement does not apply to the terms and conditions of use of a third party website or any software program or other item accessed through the website Limitations and Restrictions A syslog is generated if one of the following configurations exists when you try to disable automatic static MAC learning for MS NLB because they do not support this feature ...

Page 163: ...the following Confirm that the MS NLB vEths are disabled Confirm that the MS NLB shared MAC starting with 02 BF is not listed in the Layer 2 L2 MAC table Step 1 Generate the VEM status vemcmd show port auto smac learning LTL VSM Port Auto Static MAC Learning 49 Veth4 DISABLED 50 Veth5 DISABLED 51 Veth6 DISABLED Step 2 Generate the Layer 2 MAC address table for VLAN 59 vemcmd show l2 59 Bridge doma...

Page 164: ...ent loops and broadcast radiation We recommend that you enable BPDU guard on access ports so that any end user devices on these ports that have BPDU guard enabled cannot influence the topology Any malfunctioning device connected to a virtual Ethernet port can flood the Layer 2 network with unwanted BPDUs and causes STP to break down When you enable BPDU guard on the access ports it shuts down the ...

Page 165: ...anning tree bpduguard status Command switch config show interface virtual spanning tree bpduguard status 49 Veth36 Enabled 50 Veth68 Enabled 51 Veth73 Enabled 52 Veth77 Enabled Example 12 14 show system internal cdm info port profile name Command switch config if show system internal cdm info port profile name vm port profile vm show interface virtual spanning tree bpduguard status Displays the st...

Page 166: ... vm 4 network none config spanning tree bpduguard enable Example 12 16 vemcmd show card Command switch vemcmd show card Card UUID type 2 35958c78 bce9 11e0 bd1d 30e4dbc2c276 Card name Switch name switch Licensed Yes Global BPDU Guard Disabled Example 12 17 vemcmd show port bpdugard Command switch vemcmd show port bpduguard LTL VSM Port BPDU Guard 49 Veth36 50 Veth68 51 Veth73 Enabled 52 Veth77 Ena...

Page 167: ...he following guidelines for VLANs Keep user traffic off the management VLAN keep the management VLAN separate from user data Note We recommend that you enable sticky Address Resolution Protocol ARP when you configure private VLANs ARP entries are learned on Layer 3 private VLAN interfaces that are sticky ARP entries For security reasons private VLAN port sticky ARP entries do not age out IGMP runs...

Page 168: ... primary VLAN their MAC address tables are merged into one shared MAC table Initial Troubleshooting Checklist Troubleshooting a VLAN problem involves gathering information about the configuration and connectivity of individual devices and the entire network In the case of VLANs begin your troubleshooting activity as follows The following CLI commands are used to display VLAN information show syste...

Page 169: ...lease 5 2 1 SV3 1 1 OL 31593 01 Chapter 13 VLANs Cannot Create a VLAN Cannot Create a VLAN Symptom Possible Cause Solution Cannot create a VLAN Using a reserved VLAN ID VLANs 3968 to 4047 and 4094 are reserved for internal use and cannot be changed ...

Page 170: ...13 4 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 13 VLANs Cannot Create a VLAN ...

Page 171: ... Private VLAN Domains A private VLAN domain consists of one or more pairs of VLANs The primary VLAN makes up the domain and each VLAN pair makes up a subdomain The VLANs in a pair are called the primary VLAN and the secondary VLAN All VLAN pairs within a private VLAN have the same primary VLAN The secondary VLAN ID is what differentiates one subdomain from another Spanning Multiple Switches Privat...

Page 172: ...ured correctly Use the show interface slot port command to verify the interface is up Use the module vem module number execute vemcmd show port command to verify the VEM is configured correctly Private VLAN Troubleshooting Commands Use the commands listed in this section to troubleshoot problems related to private VLANs Example 14 1 show vlan private vlan Command switch show vlan private vlan Prim...

Page 173: ...Packets 7424670 Bytes 5507 Input Packet Drops 0 Output Packet Drops 2 interface resets Example 14 3 show interface veth Command switch show interface v3 Vethernet3 is up Hardware is Virtual address is 0050 56bb 6330 Owner is VM fedora9 adapter is Network Adapter 1 Active on module 3 VMware DVS port 10 Port Profile is pvlancomm153 Port mode is Private vlan host Rx 14802 Input Packets 14539 Unicast ...

Page 174: ...0 1 T 0 2 2 PHYS UP UP 4 Trunk vmnic4 pvlan promiscuous trunk port 153 156 154 156 155 156 157 152 158 152 47 1b020000 154 0 2 0 VIRT UP UP 4 Access fedora9 eth0 pvlan community 156 153 If additional information is required for Cisco Technical Support to troubleshoot a private VLAN issue use the following commands show system internal private vlan info show system internal private vlan event histo...

Page 175: ... to count in the given flow Flows are stored in the NetFlow cache Flow information tells you the following The source address tells you who is originating the traffic The destination address tells you who is receiving the traffic Ports characterize the application using the traffic Class of service CoS examines the priority of the traffic The device interface tells how traffic is being used by the...

Page 176: ...log debug sfnetflow_flowmon all vemlog debug sfnetflow_ager all vemlog debug sfnetflow all Enables packet path debugging for NetFlow policies on the VEM Debug messages are printed for every packet that hits a NetFlow policy Use this command with caution High traffic could result in lot of debug messages vemcmd show netflow monitor Prints the monitor configuration vemcmd show netflow interface Prin...

Page 177: ... the version number of the original command to clear the configuration and then reattempt the command Debugging a Policy Verification Error You can debug a policy verification failure due to some processing on the VSM Step 1 Enter the debug nfm all command Step 2 Save the Telnet SSH session buffer to a file Step 3 Enter the ip flow mon monitor name direction command The command executes once again...

Page 178: ...15 4 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 15 NetFlow Common NetFlow Problems ...

Page 179: ... the packet is permitted or denied If there is no match the device applies a default rule The device processes packets that are permitted and drops packets that are denied ACLs protect networks and specific hosts from unnecessary or unwanted traffic For example ACLs are used to disallow HTTP traffic from a high security network to the Internet ACLs also allow HTTP traffic but only to specific site...

Page 180: ... following commands on the VSM to see run time information of the ACLMGR and ACLCOMP during configuration errors and to collect ACLMGR process run time information configuration errors show system internal aclmgr event history errors show system internal aclmgr event history msgs show system internal aclmgr ppf show system internal aclmgr mem stats to debug memory usage and leaks show system inter...

Page 181: ...p 2 Enter the debug aclmgr all command Step 3 Enter the debug aclcomp all command For the VEMs where the policy exists or is being applied enter the following these steps from the VSM The output goes to the console Step 4 Enter the module vem module number execute vemdpalog debug sfaclagent all command Step 5 Enter the module vem module number execute vemdpalog debug sfpdlagent all command Step 6 ...

Page 182: ...s permit deny If SrcIP DstIP SrcPort DstPort Proto Direction Action Stats Veth4 192 168 1 20 192 168 1 10 5345 8080 6 Ingress permit 1 Veth4 192 168 1 10 192 168 1 20 8080 5769 6 Egress permit 1 Veth4 192 168 1 20 192 168 1 10 6256 8080 6 Ingress permit 1 Veth4 192 168 1 10 192 168 1 20 8080 5801 6 Egress permit 1 Veth4 192 168 1 20 192 168 1 10 5217 8080 6 Ingress permit 1 Veth4 192 168 1 10 192 ...

Page 183: ...the syslog server configuration and check if ACL logging is configured by entering the commands shown in the following procedure BEFORE YOU BEGIN Log in to the VSM and VEM CLI PROCEDURE Troubleshooting an ACL Rule That Does Not Have a Log Keyword If the ACL rule does not have a log keyword any flow that matches the ACL is not reported although the ACL statistics continue to advance You can verify ...

Page 184: ... VEM CLI PROCEDURE Command Description Step 1 show running config aclmg Example switch show running config aclmg switch Verifies that the log keyword is enabled Step 2 show logging ip access list status Example switch show logging ip access list status switch Verifies that ACL logging is configured properly Step 3 vemcmd show acllog config Example switch vemcmd show acllog config switch Verifies A...

Page 185: ... VSM and the VEM and retry the commands BEFORE YOU BEGIN Log in to the CLI in EXEC mode PROCEDURE Step 2 vemcmd show acllog config Example switch vemcmd show acllog config switch Verifies ACL logging on the VEM Step 3 logging ip access list cache max deny flows num Example switch logging ip access list cache max deny flows num switch Increases maximum flows to the desired value Command Description...

Page 186: ...16 8 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 16 ACLs Troubleshooting ACL Logging ...

Page 187: ...lets you classify network traffic so that it can be policed and prioritized in a way that prevents congestion Traffic is processed based on how you classify it and the QoS policies that you put in place Classification marking and policing are the three main features of QoS Traffic Classification Groups network traffic based on defined criteria Traffic Marking Modifies traffic attributes such as DS...

Page 188: ...ch criteria per class map 32 Classes per policy map can be of type qos or queuing 64 Match rules under policy map 200 Command Purpose show policy map policy map name show class map class map name Displays the configured policies and class maps show policy map interface Displays the number of packets hitting the configured policies show policy map interface input output Displays only the installed ...

Page 189: ...nal ipqos event history msgs show system internal ipqos mem stats to debug memory usage and leaks show system internal ipqos status show system internal ipqos log to show aborted plan information show system internal ipqos Troubleshooting the VEM The commands listed in this section can be used to display configured QoS policies on the VEM Example 17 2 module vem module number execute vemcmd show q...

Page 190: ...icy configuration failure caused by processing on the VSM Step 1 Enter the debug aclmgr all command if the policy references an ACL Step 2 Enter the debug ipqos all command Step 3 Enter the policy map and class commands to collect logs for all operations Step 4 Save the Telnet SSH session buffer to a file If you are debugging a policy on a port profile it might be easier to first install it direct...

Page 191: ...lters Adding classmap 3 56 with op 0 and 0 filters Every session should end with the log Debug qosagent Session commit complete and successful Debugging Policy Verification Failures You can debug a policy verification failure on VEM Step 1 Enter the module vem module number execute vemdpalog clear command Step 2 Enter the module vem module number execute vemdpalog sfqosagent all command Step 3 Ent...

Page 192: ...ep 1 Enter the module vem module number execute vemdpalog clear command Step 2 Enter the module vem module number execute vemdpalog sfqosagent all command Step 3 Enter module vem module number execute vemdpalog start command Step 4 Enter the service policy command which will execute the command once again with the DPA debug traces output to vemdpalog Step 5 Enter module vem module number execute v...

Page 193: ...N ERSPAN that can send monitored traffic to an IP destination For detailed information about how to configure local SPAN or ERSPAN see the Cisco Nexus 1000V System Management Configuration Guide SPAN Session Guidelines The following are SPAN session guidelines When a SPAN session contains multiple transmit source ports packets that these ports receive might be replicated even though they are not t...

Page 194: ...uplink switch The SPAN packets might cause problems with the IP tables the MAC tables or both on the uplink switch which can cause problems with the regular traffic A session state is up and the packets are not received at the destination ports Verify that the correct VLANs are allowed on the trunk destination ports The session displays an error 1 Make sure that VSM VEM connectivity is working cor...

Page 195: ...n shut folio Example 18 2 show monitor session Command switch config show monitor session 1 session 1 type erspan source state up source intf rx Eth3 3 tx Eth3 3 both Eth3 3 source VLANs rx tx both filter VLANs filter not specified destination IP 10 54 54 1 ERSPAN ID 999 ERSPAN TTL 64 ERSPAN IP Prec 0 ERSPAN DSCP 0 ERSPAN MTU 1000 Command Purpose show monitor Displays the status of SPAN sessions S...

Page 196: ... SPAN SPAN Troubleshooting Commands Example 18 3 module vem execute vemcmd show span Command switch vemcmd show span RX Ltl Sources 52 TX Ltl Sources 52 RX Vlan Sources TX Vlan Sources Source Filter 2 local 50 RX Ltl Sources 51 TX Ltl Sources 51 RX Vlan Sources TX Vlan Sources Source Filter ...

Page 197: ...s reside Using the port information IGMP snooping can reduce bandwidth consumption in a multi access LAN environment to avoid flooding the entire VLAN The IGMP snooping feature tracks which ports are attached to multicast capable routers to help the routers forward IGMP membership reports The IGMP snooping software responds to topology change notifications In general IGMP snooping works as follows...

Page 198: ...9 n5k sw1 config vlan config ip igmp snooping querier 7 59 59 1 n5k sw1 config vlan config ip igmp snooping query interval 60 n5k sw1 config vlan config ip igmp snooping version 3 n5k sw1 config vlan config Troubleshooting Guidelines Follow these guidelines when troubleshooting multicast IGMP issues Use the show ip igmp snooping command to verify that IGMP snooping is enabled Make sure that the up...

Page 199: ...notify message from DP Jul 15 18 19 27 609459 10 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler SRC_LTL 1039 SWBD 52 pkt_size 56 Jul 15 18 19 27 609470 11 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler Got IGMP Query Jul 15 18 19 27 609479 12 0 99 16 Debug sf_igmp_snoop_handle_query Received v3 query Jul 15 18 19 27 609485 13 0 99 16 Debug sf_igmp_snoop_handle_query Adding v3 router entry in...

Page 200: ...38419 42 14 1 16 Debug Forward report to router port 10347 Jul 15 18 19 34 968621 43 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler IGMP notify message from DP Jul 15 18 19 34 968634 44 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler SRC_LTL 102 SWBD 59 pkt_size 60 Jul 15 18 19 34 968645 45 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler Got IGMP v1 v2 Report Jul 15 18 19 34 968654 46 0 99 ...

Page 201: ...size 68 Jul 15 18 19 37 134483 77 0 99 16 Debug sf_igmp_snoop_start_leave_timers Start leave timer on member 102 for 2 secs Jul 15 18 19 37 134504 78 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler IGMP notify message from DP Jul 15 18 19 37 134511 79 0 99 16 Debug sf_igmp_snoop_v4_pkt_notify_handler SRC_LTL 1039 SWBD 59 pkt_size 56 Jul 15 18 19 37 134518 80 0 99 16 Debug sf_igmp_snoop_v4_pkt_no...

Page 202: ...ism that CDP uses However if you have disabled the CDP protocol on the upstream switch using the no cdp enable command the show cdp neighbor command will not display any information Example 19 1 show cdp neighbor Command switch show cdp neighbor Capability Codes R Router T Trans Bridge B Source Route Bridge S Switch H Host I IGMP r Repeater V VoIP Phone D Remotely Managed Device s Supports STP Dis...

Page 203: ...l 8 23 49 16 748355 igmp 3157 SNOOP vlan 1 clear port Veth43 vlan 1 2014 Jul 8 23 49 16 789832 igmp 3157 SNOOP Switchport interface Veth47 428 has been created obtaining any static mrouter oif configs 2014 Jul 8 23 49 16 797079 igmp 3157 SNOOP Switchport interface Veth38 158 has been created obtaining any static mrouter oif configs 2014 Jul 8 23 49 16 824702 igmp 3157 SNOOP vlan 11 Added Veth43 to...

Page 204: ...tate takes precedence Multicast group table values are as follows Group 227 0 0 1 Multicast LTL 10363 Group Multicast LTL 10358 module vem 3 execute vemcmd show igmp 1784 de In Example 19 2 global IGMP snooping is enabled on VLAN 1784 the disabled global state takes precedence Multicast group table values are as follows Group 227 0 0 1 Multicast LTL 10363 Members 59 1039 Group Multicast LTL 10358 ...

Page 205: ...wing are symptoms possible causes and solutions for problems with multicast IGMP snooping Symptom Possible Causes Solution A VM is interested in the multicast traffic but is not receiving the multicast traffic Use the debug ip igmp snooping vlan command to determine if IGMP snooping is working as expected Examine the output to see if the port is receiving the IGMP report and if the interface has b...

Page 206: ...19 10 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 19 Multicast IGMP Problems with Multicast IGMP Snooping ...

Page 207: ... 4 Problems with IP Source Guard page 20 5 Collecting and Evaluating Logs page 20 5 DHCP DAI and IPSG Troubleshooting Commands page 20 6 Information About DHCP Snooping DHCP snooping acts like a firewall between untrusted hosts and trusted DHCP servers by doing the following Validates DHCP messages received from untrusted sources and filters out invalid response messages from DHCP servers Builds a...

Page 208: ...r that permits IP traffic only when the IP address and MAC address of each packet matches the IP and MAC address bindings of dynamic or static IP source entries in the Dynamic Host Configuration Protocol DHCP snooping binding table For detailed information about configuring IP Source Guard see the Cisco Nexus 1000V Security Configuration Guide Guidelines and Limitations for Troubleshooting The fol...

Page 209: ... that the server is up and running The interface of the DHCP server s connected to the DVS as a VM is not trusted 1 On the Virtual Supervisor Module VSM verify that the interface is trusted show ip dhcp snooping 2 On the VSM verify that the vEthernet interface attached to the server is trusted module vem mod execute vemcmd show dhcps interfaces DHCP requests from the VM are not reaching the server...

Page 210: ...detailed information see the Cisco Nexus 1000V Security Configuration Guide If the configuration appears correct on the VSM but fails on the VEM capture and analyze the error logs from both the VSM and the VEM to identify the reason for the failure If snooping is disabled the binding entry is not statically configured in the binding table 1 On the VSM display the binding table show ip dhcp snoopin...

Page 211: ...ow port profile name profile_name show running interface if_ID show ip verify source For detailed information about configuring IP Source Guard see the Cisco Nexus 1000V Security Configuration Guide The IP address that corresponds to the vEthernet interface is not in the snooping binding table 1 On the VSM display the binding table show ip dhcp snooping binding 2 Configure the missing static entry...

Page 212: ...gent all tmp dpafifo Enables DPA DHCP agent debug logging Logs are output to var log vemdpa log file vemlog debug sfdhcps all Enables data path debug logging and captures logs for the data packets sent between the client and the server vemlog debug sfdhcps_pod all Captures POD Port Opaque Data logging for the feature vemlog debug sfdhcps_config all Enables data path debug logging and captures logs...

Page 213: ...ed vEthernet 3 Yes show feature Displays the features available such as DHCP and whether they are enabled See Example 20 4 on page 20 8 show ip arp inspection Displays the status of DAI See Example 20 5 on page 20 8 show ip arp inspection interface vethernet interface number Displays the trust state and ARP packet rate for a specific interface See Example 20 6 on page 20 8 show ip arp inspection v...

Page 214: ... http server 1 enabled ippool 1 enabled lacp 1 enabled lisp 1 enabled lisphelper 1 enabled netflow 1 disabled port profile roles 1 enabled private vlan 1 disabled sshServer 1 enabled tacacs 1 enabled telnetServer 1 enabled switch Example 20 5 show ip arp inspection Command cyp1 switch config show ip arp inspection Source Mac Validation Disabled Destination Mac Validation Disabled IP Address Valida...

Page 215: ...X00000000 RRtoken 0x00009498 Sync UNKNOWN Payloadsize 132 Payload 0x0000 00 00 00 03 00 00 00 01 00 00 00 64 00 00 00 07 2 Event E_MTS_RX length 60 at 809100 usecs after Mon Oct 8 20 59 08 2012 RSP Opc MTS_OPC_PDL32 148511 Id 0X00E01555 Ret SUCCESS Src 0x00000502 747 Dst 0x00000201 360 Flags None HA_SEQNO 0X00000000 RRtoken 0x00009497 Sync UNKNOWN Payloadsize 132 Payload 0x0000 00 00 00 03 00 00 0...

Page 216: ...44 44 3914 3914 89 r xp isan plugin 0 isan lib libsmm so 3 3 216 216 111 r xp isan plugin 0 isan lib libutils 4 7 69 349 112 r xp isan plugin 0 isan lib libvdc_mg 0 1 0 20 118 r xp isan plugin 2 isan bin dhcp_snoo 0 2 0 64 121 r xp isan plugin 2 isan lib libpdlser 4 29 208 1016 128 r xp lib ld 2 3 3 so 33 33 5363 5371 131 r xp lib tls libc 2 3 3 so 51 51 1347 1637 134 r xp lib tls libpthread 2 3 3...

Page 217: ... storm Troubleshooting Storm Control This section describes the different types of troubleshooting commands to debug Storm Control Troubleshooting VSM Commands page 21 1 Troubleshooting VEM Commands page 21 1 Debugging Storm Control on a VEM page 21 2 Troubleshooting VSM Commands Displays the detailed storm control statistics on an interface show storm control statistics interface interface type m...

Page 218: ...ontrol vemcmd show storm rate ltl ltl Displays the storm control status of whether the port is dropping or allowing traffic on a VEM vemcmd show storm status Debugging Storm Control on a VEM You can debug storm control on a VEM Step 1 vemlog clear Step 2 vemlog start Step 3 vemlog debug sfstormcontrol all Step 4 vemlog show all ...

Page 219: ...0V replaces virtual switches within ESX servers and allows users to configure and monitor the virtual switch using the Cisco NX OS command line interface Nexus 1000V also gives you visibility into the networking components of the ESX servers and access to the virtual switches within the network The Nexus 1000V manages a data center defined by the vCenter Server Each server in the Datacenter is rep...

Page 220: ...nterfaces associated with it A sync operation performed in conjunction with the connect command helps VSM keep in sync with vCenter Server Each VSM uses a unique extension key to communicate with vCenter Server and perform operations on a DVS Extension Key The VSM uses the extension key when communicating with the vCenter Server Each VSM has its own unique extension key such as Cisco_Nexus_1000V_3...

Page 221: ... in Step 1 The extension key allows the VSM to log in to the vCenter server Example switch config t switch config vmware vc extension key Cisco_Nexus_1000V_32943215 Step 3 From the MOB unregister the extension key found in Step 1 For more information see the Unregistering the Extension Key in the vCenter Server procedure on page 3 12 Step 4 From the VC client register the extension plug in for the...

Page 222: ... previously saved a back up copy of the VSM configuration file then you may try recreating the old port profiles before connecting to the VC This procedure has a step for recreating port profiles If you do not recreate these before connecting to VC then all the port groups present on the VC are removed and all ports in use are moved to the quarantine port groups Make sure that the VSM VM switchnam...

Page 223: ...estore the configuration for the vCenter server connection Example switch config t switch config svs connection VC switch config svs conn protocol vmware vim switch config svs conn remote ip address 192 168 0 1 switch config svs conn vmware dvs datacenter name Hamilton DC Step 7 Connect to vCenter Server Example switch config svs conn connect You can now use the old DVS or remove it Problems Relat...

Page 224: ...EM and preserve a non default MTU setting for a physical NIC across reboots of the ESX you must configure a system MTU in the system port profile If you use an MTU other than 1500 the default for a physical NIC attached to the Cisco Nexus 1000V then reboots of the ESX can result in a mismatch with the VMware kernel NIC MTU and failure of the VSM and VEM For example you may manually configure an MT...

Page 225: ... Interface Configuration Guide When you configure a system MTU on a system port profile it takes precedence over an MTU you may have configured on the interface To verify the ESX MTU settings for corresponding PNICs use the ESXcfg nics l command SUMMARY STEPS 1 config t 2 port profile profilename 3 system mtu mtu value 4 show port profile brief expand interface usage name profilename 5 copy runnin...

Page 226: ...on system uplinks SUMMARY STEPS 1 config t 2 module vem module_number execute vemcmd show port port LTL number 3 module vem module_number execute vemcmd set mtu size ltl port LTL number DETAILED STEPS Step 4 show port profile brief expand interface usage name profile name Example switch config port prof show port profile name AccessProf Optional Displays the configuration for verification Step 5 c...

Page 227: ...emcmd show port 48 LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name 17 1a030100 1 T 304 1 32 PHYS UP UP 1 Trunk vmnic1 switch config Step 3 module vem module_number execute vemcmd set mtu size ltl port LTL number Example switch config module vem 3 execute vemcmd set mtu 9000 ltl 17 switch config Designates the MTU size for the port using the LTL number obtained in Step 2 Comm...

Page 228: ...mptom Solution You receive an error message DVS Operation failed for one or more members Issue the vem status v command to verify if the VEM is running on the host Issue the vem unload command to unload the VEM In the vSphere Client remove the stale DVS 1 Go to the Host tab Networking Configuration Distributed Virtual Switch 2 Click Remove The host is visible on the vCenter Server but not the VSM ...

Page 229: ...us information vem support all collects support information vem status collects status information vem version collects version information vemlog show last number of entries displays the circular buffer Example 22 2 vemlog show last Command root ESX cos1 vemlog show last 5 Timestamp Entry CPU Mod Lv Message Oct 13 13 15 52 615416 1095 1 1 4 Warning vssnet_port_pg_data_ Oct 13 13 15 52 620028 1096...

Page 230: ...he vSphere Client you can see error messages under the recent tasks tab You can find detailed description of the error under the Tasks and Events tab The same messages are also propagated to the VSM Table 22 1 lists error messages that you might see on the VSM Table 22 1 Error Messages on the VSM Error Description ERROR VMWARE VIM Extension key was not registered before its use This error indicate...

Page 231: ...ct Spec name This error indicates that a DVS with the same name already exists Warning Operation succeeded locally but update failed on vCenter server VMWARE VIM DVPortgroup test port 0 is in use The resource vim dvs DistributedVirtualPort 0 is in use This warning is displayed when the VSM tries to delete the port profile if the VSM is not aware of the nics attached to the port groups Table 22 1 E...

Page 232: ...22 14 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 22 System Error Messages ...

Page 233: ...ort communities Cisco Support Community for Server Networking Cisco Communities Nexus 1000V Gathering Information for Technical Support At some point you may need to contact your customer support representative or Cisco TAC for some additional assistance This section outlines the steps that the you should perform prior to contacting your next level of support so you can reduce the amount of time t...

Page 234: ...w many devices have this problem Were any traces or debug output captured during the problem time What troubleshooting steps have you attempted Which if any of the following tools were used Ethanalyzer local or remote SPAN CLI debug commands traceroute ping Step 4 Is your problem related to a software upgrade attempt What was the original Cisco Nexus 1000V version What is the new Cisco Nexus 1000V...

Page 235: ...ct source filesystem sftp Select source filesystem slot0 Select source filesystem startup config Copy startup configuration to destination system Select source filesystem tftp Select source filesystem volatile Select source filesystem Use the following syntax to use secure copy scp as the transfer mechanism scp username server path Copy etc hosts from 172 22 36 10 using the user user1 where the de...

Page 236: ...23 4 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 23 Before Contacting Technical Support Copying Files ...

Page 237: ...ith Network Segmentation Manger NSM and includes the following sections Information About Network Segmentation Manager page 24 1 Problems with Network Segmentation Manager page 24 2 Network Segmentation Manager Troubleshooting Commands page 24 7 Information About Network Segmentation Manager See the Cisco Nexus 1000V Network Segmentation Manager Configuration Guide for more information ...

Page 238: ...exus 1000V If not reestablish the Layer 2 or Layer 3 connectivity between vShield Manager and the Cisco Nexus 1000V See the Cisco Nexus 1000V Network Segmentation Manager Configuration Guide for more information vShield Manager is unable to authenticate with NSM Verify if the username and password are accurate by checking the Virtual Supervisor Module system logs The following system log will be d...

Page 239: ...determine the network segment policy the network was attempting to use You will need the information about the tenant organization UUID and the type of network pool the network was being created from VXLAN or VLAN to find the corresponding network segment policy that has these values configured If no network segment policy is configured with these values then use the default network segment policy...

Page 240: ...with these values use the default network segment policy to identify the name of the port profile 2 Check system logs for a port profile failure message reported by NSM See the Cisco NX OS System Messages Reference for more information The network creation triggered from vCloud Director fails A system message similar to the following is logged in vCloud Director Alias ID not found vCloud Director ...

Page 241: ...show svs connection When you enter the command the output must display operational status connected The network creation triggered from vCloud Director fails A system message similar to the following is logged in vCloud Director Operational status is missing vCloud Director is unable to locate the operational status in the SVS connection 1 Verify that the Virtual Supervisor Module VSM has an activ...

Page 242: ... the network Check system logs for a port profile description failure message reported by NSM See the Cisco NX OS System Messages Reference for more information The network deletion triggered from vCloud Director fails A system message similar to the following is logged in vCloud Director Failed to delete interface using the port profile vCloud Director is unable to delete the interfaces inheritin...

Page 243: ...00V configured with NSM show running config port profile Displays the port profile configuration show running config network segment policy Displays the NSM policy configuration show network segment policy usage Displays the network segmentation policy usage by networks show network segment network Displays the networks associated with a network segmentation policy show network segment network id ...

Page 244: ...24 8 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 24 Network Segmentation Manager Network Segmentation Manager Troubleshooting Commands ...

Page 245: ...Gateway page 25 2 VXLAN Trunks page 25 3 VXLAN Border Gateway Protocol Control Plane page 25 3 Multi MAC Capability page 25 8 Fragmentation page 25 8 Scalability page 25 8 Supported Features page 25 9 Overview A Virtual Extensible LAN creates LAN segments by using an overlay approach with MAC in UDP encapsulation and a 24 bit segment identifier in the form of a VXLAN ID The encapsulation carries t...

Page 246: ...ributes those MAC addresses with VXLAN Tunnel Endpoint VTEP IP mappings to other VEMs The VXLAN creates LAN segments by using an overlay approach with MAC in IP encapsulation VXLAN Tunnel EndPoint Each VEM requires at least one IP MAC pair to terminate VXLAN packets This IP MAC address pair is known as the VXLAN Tunnel End Point VTEP IP MAC addresses The VEM supports IPv4 addressing for this purpo...

Page 247: ...unicast only mode to a multi VSM environment using a L2VPN EVPN address family The VTEP information is not exchanged with the VSMs that are running the old version They will continue to work in multicast mode VXLAN 1 0 or unicast only mode in a single Cisco Nexus 1000V VXLAN 1 5 BGP Commands This example shows how to enable BGP switch configure terminal switch config feature bgp Cisco Nexus 1000V ...

Page 248: ...e a BGP peer template and apply it to a BGP peer switch configure terminal switch config router bgp 65536 switch config router template peer BasePeer switch config router neighbor inherit peer session BaseSession switch config router neighbor af inherit peer policy BasePolicy 1 switch config router neighbor af exita switch config router neighbor exit switch config router neighbor 192 168 1 2 remot...

Page 249: ... BGP routing table entry for 3 5000 4 192 168 69 104 88 version 10 Paths 1 available best 1 Flags 0x00001a on xmit list is in l2rib evpn Path type internal path is valid is best path Imported from 172 23 181 68 5000 3 5000 4 192 168 69 104 88 AS Path NONE path sourced internal to AS 172 23 181 68 metric 0 from 172 23 181 68 172 23 181 68 Origin IGP MED not set localpref 100 weight 0 Extcommunity R...

Page 250: ...P I Forwarding Publish Incapable VTEP Note Denotes active gateway module Bridge domain vxlan 5000 VTEP Table Version 13 Port Module VTEP IP Address VTEP Flags Veth5 3 192 168 69 101 D Remote 66 100 0 1 DI Remote 192 168 69 201 DI This example shows how to display the BGP evpn summary switch show bgp l2vpn evpn neighbors 192 168 65 10 BGP summary information for VRF default address family L2VPN EVP...

Page 251: ... seconds Restart time advertised by peer 120 seconds Message statistics Sent Rcvd Opens 2 1 Notifications 0 0 Updates 3 2 Keepalives 4003 4003 Route Refresh 0 0 Capability 0 0 Total 4008 4006 Total bytes 76196 76167 Bytes in queue 0 0 For address family L2VPN EVPN BGP table version 6 neighbor version 6 1 accepted paths consume 60 bytes of memory 1 sent paths Extended community attribute sent to th...

Page 252: ...port profile upstream physical switch port interswitch links and any routers to carry a maximum transmission unit MTU of at least 1550 bytes If that is not possible we recommend that the MTU within the guest VMs you configure to be smaller by 50 bytes If you do not configure a smaller MTU the VEM attempts to notify the VM if it performs Path MTU PMTU Discovery If the VM does not send packets with ...

Page 253: ...You can use the no feature segmentation command to remove all the VXLAN bridge domain configurations on the Cisco Nexus 1000V VXLAN Troubleshooting Commands Use the following commands to display VXLAN attributes This section contains the following topics VSM Commands page 25 9 VXLAN Gateway Commands page 25 11 VSM Commands You can use the commands in this section to troubleshoot problems related t...

Page 254: ...ple 25 4 show system internal seg_bd info port_count switch config show system internal seg_bd info port_count Number of ports 11 Example 25 5 show system internal seg_bd info bd vxlan home switch config show system internal seg_bd info bd vxlan home Bridge domain vxlan home 2 ports in all Segment ID 5555 Manual Active Group IP 235 5 5 5 State UP Mac learning Enabled is_bd_created Yes current stat...

Page 255: ...teway ok To display VXLAN Gateway information that is not attached to the VSM VXLANGW attach vem VXLANGW vem attach vemcmd Execute vem command vemdpa Execute vemdpa command vemdpalog Execute vemdpalog command vemlog Execute vemlog command vempkt Execute vempkt command vemset Execute vemset command switch vem attach To display VXLAN Gateway mappings VXGW switch vem attach vemcmd show vxlan gw mappi...

Page 256: ...ration on the VSM switch show bridge domain Note This command is common for both gateway and VEM Global Configuration Mode Unicast only MAC Distribution Disable Note If you have enabled MAC distribution the above command will display Enable Bridge domain segment cisco 3 ports in all Segment ID 9001 Manual Active Mode Unicast only default MAC Distribution Disable default Group IP NULL State UP Mac ...

Page 257: ... the MACs learned on the VSM through VEM distribution switch show bridge domain mac Bridge domain segment cisco MAC TABLE Version 1 Note You can compare with VEM output using the echo show vxlan version table command MAC Address Module Port VTEP IP Address VM IP Address 0050 5683 014e 5 Veth5 10 106 199 117 0050 5683 0160 4 Veth2 10 106 199 116 0050 5683 0161 4 Veth3 10 106 199 116 To verify the p...

Page 258: ... 168 10 13 0050 5691 01d6 3 Veth177 192 168 10 27 0050 5691 0549 3 Veth695 192 168 10 27 Additional show commands show platform fwm errors show platform fwm info VTEP trace error history show platform fwm info error history show platform fwm event history msgs show platform fwm info vlan all swbd VEM Commands To verify VXLAN vEthernet programming vemcmd show port segments Native Seg LTL VSM Port M...

Page 259: ...display detailed per port per bridge domain statistics for a VXLAN VTEP for all bridge domains vemcmd show vxlan stats ltl vxlan_VTEP_ltl bd all To display detailed per port per bridge domain statistics for a VXLAN VTEP for a specified bridge domain vemcmd show vxlan stats ltl vxlan_VTEP_ltl bd name bd name To verify the bridge domain configuration on the VEM switch vemcmd show bd bd name segment ...

Page 260: ...N 1 Note You can compare the download sequence number against the VTEP download sequence number using the vemcmnd show bd bd name Displays if the MAC address table displays the remote IP learning in the segment cisco bridge domain switch vemcmd show l2 bd name segment cisco Note Use the module command to check the details of VEM and gateway on the VSM Bridge domain 26 brtmax 4096 brtcnt 3 timeout ...

Page 261: ...ets go out on a VXLAN vEthernet interface vempkt capture egress ltl vxlan_veth VEM2 Look at statistics for any failures vemcmd show vxlan stats all vemcmd show vxlan stats ltl veth vxlanVTEP Use the following commands to debug the VXLAN packet path switch module vem 4 execute vemlog debug vssnet all switch module vem 4 execute vemlog debug sfsched all switch module vem 4 execute vemlog debug sfpor...

Page 262: ...g Use the commands listed in this section to troubleshot VXLAN problems This section contains the following topics vemlog Debugging page 25 18 Vempkt page 25 19 Statistics page 25 20 show Commands page 25 20 vemlog Debugging To debug the bridge domain setup or configuration use the following command vemlog debug sfbd all To debug the port configuration CBL vEthernet LTL pinning use the following c...

Page 263: ...863 delete_notif_rx Pending MAC deletes FALSE Jul 1 10 18 20 853876 update timer ticks that pending deletes not sent 0 Jul 1 10 18 20 853890 VxLAN update timer state 1 Jul 1 10 18 20 853906 VSM connected FALSE Jul 1 10 18 20 854021 Last retry slot 0 MAC 00 00 00 00 00 00 Jul 1 10 18 20 854132 Last delete slot 0 MAC 00 00 00 00 00 00 Hash SWBD VTEP Ver MAC Ver Created on DP Need version check 0 409...

Page 264: ...vemcmd show vxlan stats ltl vxlan_VTEP_ltl bd name bd name To display which VXLAN VTEP is used for encapsulation and subsequent pinning to the uplink port channel for static MAC addresses learned on port use the following command vemcmd show vxlan encap ltl vxlan_veth_ltl To display which VXLAN VTEP is used for encapsulation and subsequent pinning to the uplink port channel use the following comma...

Page 265: ... protocol and communicates the presence of end host Virtual Machines VMs to adjacent leaf nodes on the Cisco Dynamic Fabric Automation DFA architecture In addition to detecting the MAC and IP addresses of the end host VMs when a host comes up or during VM mobility events the VDP triggers auto configuration of leaf nodes on the DFA architecture to make them ready for more VM traffic VDP enables net...

Page 266: ...tput of the show interface ethernet command does not contain dynamic VLANs configure the port profile for trunk dynamic mode a switch configure terminal b switch config port profile name c switch config port prof switchport mode trunk d switch config port prof switchport trunk dynamic VM is associated but it is not pinging The encapsulation mode is not native Verify that encapsulation mode is nati...

Page 267: ...nd switch config show evb Edge Virtual Bridging Role VDP Station VDP Mac Address 0180 0000 0000 VDP Resource Wait Delay 22 66 secs VDP Reinit Keep Alive 21 20 secs Example 26 3 show run evb Command switch config show run evb evb resource wait delay 24 evb reinit keep alive 25 ecp retransmission timer exponent 15 ecp max retries 6 Example 26 4 show ecp Command switch config show ecp ECP Max ReTries...

Page 268: ... VM L 14 25 10 eth7 172 VM L 14 25 3 eth7 182 VM L 13 25 9 eth7 192 VM L 14 25 4 eth7 202 VM L 14 25 8 eth7 212 VM L 14 25 7 eth7 222 VM L 14 25 6 eth7 232 VM L 14 25 5 eth7 242 VM L 14 25 9 eth7 252 VM L 15 25 10 eth7 262 VM L 15 25 3 eth7 272 VM L 15 25 2 eth7 282 VM L 15 25 1 eth7 Command Purpose vemcmd show segment segment id Displays a list of VM interfaces that are a part of a segment and in...

Page 269: ...L 14 25 1 eth7 145 VM L 14 25 2 eth7 162 VM L 14 25 10 eth7 172 VM L 14 25 3 eth7 182 VM L 13 25 9 eth7 192 VM L 14 25 4 eth7 202 VM L 14 25 8 eth7 212 VM L 14 25 7 eth7 222 VM L 14 25 6 eth7 232 VM L 14 25 5 eth7 242 VM L 14 25 9 eth7 252 VM L 15 25 10 eth7 262 VM L 15 25 3 eth7 272 VM L 15 25 2 eth7 282 VM L 15 25 1 eth7 294 VM L 15 25 7 eth7 295 VM L 15 25 4 eth7 312 VM L 15 25 5 eth7 322 VM L ...

Page 270: ...th7 162 VM L 14 25 10 eth7 172 VM L 14 25 3 eth7 182 VM L 13 25 9 eth7 192 VM L 14 25 4 eth7 202 VM L 14 25 8 eth7 212 VM L 14 25 7 eth7 222 VM L 14 25 6 eth7 232 VM L 14 25 5 eth7 242 VM L 14 25 9 eth7 252 VM L 15 25 10 eth7 262 VM L 15 25 3 eth7 272 VM L 15 25 2 eth7 282 VM L 15 25 1 eth7 294 VM L 15 25 7 eth7 295 VM L 15 25 4 eth7 312 VM L 15 25 5 eth7 322 VM L 15 25 6 eth7 ...

Page 271: ...y checks and data path replay protection mechanisms Cisco TrustSec also uses the device and user identification information acquired during authentication for classifying or coloring the packets as they enter the network This packet classification is maintained by tagging packets on ingress to the Cisco TrustSec network so that they can be properly identified for the purpose of applying security a...

Page 272: ...to Cisco TrustSec relay functionality debug cts sxp Collects and views logs related to Cisco TrustSec SXP debug cts sap Collects and views logs related to the Cisco TrustSec Security Association Protocol SAP debug cts trace Collects and views logs related to Cisco TrustSec trace functionality show cts internal debug info Displays Cisco TrustSec debug information ESX Host Command Description echo l...

Page 273: ...a path debug logging and captures logs corresponding to the binding database changes To view the logs enable DHCP snooping on the Cisco Nexus 1000V vemlog debug sfipdb all Enables the data path debug logging and captures logs corresponding to the IP database that maintains the IP addresses for all the virtual machines that are being tracked using Cisco TrustSec device tracking To view the logs ena...

Page 274: ...as CTS and whether they are enabled show running configuration cts Displays the running configuration information for Cisco TrustSec show cts device tracking Displays the Cisco TrustSec device tracking configuration show cts ipsgt entries Display the SXP SGT entries for Cisco TrustSec show cts role based sgt map Displays the mapping of the IP address to SGT for Cisco TrustSec show cts sxp connecti...

Page 275: ... on the Cisco Nexus 1000V matches its peer show cts sxp The default source IPv4 address is not configured on the Cisco Nexus 1000V Verify if the default source IPv4 address is not configured on the Cisco Nexus 1000V show cts sxp The SXP peer is not configured as the listener Verify that the SXP peer is configured as the listener show cts sxp connection Cisco TrustSec SXP is unable to learn any IP ...

Page 276: ...27 6 Cisco Nexus 1000V Troubleshooting Guide Release 5 2 1 SV3 1 1 OL 31593 01 Chapter 27 Cisco TrustSec Problems with Cisco TrustSec ...

Page 277: ... Release 4 2 1 SV2 1 1 the Cisco Nexus 1000V Plug in for the VMware vCenter Server vCenter Plug in is supported on the Cisco Nexus 1000V It provides the server administrators with a holistic view of the virtual network and a visibility into the networking aspects of the Cisco Nexus 1000V Starting with Cisco NX OS Release 4 2 1 SV2 1 1 the vCenter Plug in is supported on the vSphere Web Clients onl...

Page 278: ...is installed and configured to a vCenter Generating a Log Bundle You can collect the diagnostic information for VMware vCenter Server by collecting vSphere log files into a single location Step 1 Log in to the Windows server where the VMware vCenter Server is installed Step 2 Choose Start All Programs VMware Generate vSphere Web Client Log Bundle You can use this step to generate the vSphere Web C...

Page 279: ... as tcpdump For more information see the following URL Command Purpose ethanalyzer local interface interface Captures packets sent or received by the supervisor and provides detailed protocol information Note For all commands in this table you can use the control ha primary ha secondary inband outband interface packet interface or management interface ethanalyzer local interface interface limit ca...

Page 280: ...itch ethanalyzer local interface mgmt limit captured frames 4 Capturing on eth1 2012 10 01 19 15 23 794943 10 78 110 241 72 163 145 51 SSH Encrypted response packet len 64 2012 10 01 19 15 23 796142 10 78 110 241 72 163 145 51 SSH Encrypted response packet len 144 2012 10 01 19 15 23 796608 10 78 110 241 72 163 145 51 SSH Encrypted response packet len 144 2012 10 01 19 15 23 797060 10 78 110 241 7...

Reviews:

No comments