background image

 

NetXtreme-UG100

25

NetXtreme-C/NetXtreme-E

 User Guide

 

The supported combination of link speed settings for two ports NetXtreme-E network controller are shown in 

Table 16

.

NOTE:

1 Gb/s link speed for SFP+/SFP28 is currently not support in this release.

P1 – port 1 setting

P2 – port 2 setting

AN – auto-negotiation

No AN – forced speed

{link speed} – expected link speed

AN {link speeds} – advertised supported auto-negotiation link speeds.

Table 16:  Supported Combination of Link Speed Settings

Port1 Link Speed 
Setting

Port 2 Link Setting

Forced 1G

Forced 10G

Forced 25G

AN Enabled {1G}

AN Enabled 
{10G}

AN Enabled 
{25G}

AN Enabled {1/
10G}

AN Enabled {1/
25G}

AN Enabled {10/
25G}

AN Enabled {1/10/
25G}

Forced 1G

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P2: no AN

P2: no AN

P2: no AN

P2: {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1/25G}

P2: AN {10/25G}

P2: AN {1/10/25G}

Forced 10G

P1: no AN

P1: no AN

Not supported

P1: no AN

P1: no AN

Not supported

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P2: no AN

P2: no AN

P2: {1G}

P2: {10G}

P2: AN {1/10G}

P2: AN {1G}

P2: AN {10G}

P2: AN {1/10G}

Forced 25G

P1: no AN

Not supported

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P1: no AN

P2: no AN

P2: no AN

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {1/25G}

P2: AN {25G}

P2: AN {1/25G}

AN Enabled 
{1G}

P1: {1G}

P1: {1G}

P1: {1G}

P1: AN {1G}

P1: AN {1G}

P1: AN {1G}

P1: AN {1G}

P1: AN {1G}

P1: AN {1G}

P1: AN {1G}

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1/25G}

P2: AN {10/25G}

P2: AN {1/10/25G}

AN Enabled 
{10G}

P1: AN {10G}

P1: AN {25G}

Not supported

P1: AN {10G}

P1: AN {10G}

Not supported

P1: AN {25G}

P1: AN {10G}

P1: AN {10G}

P1: AN {10G}

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {1G}

P2: AN {1G}

P2: AN {10G}

P2: AN {1/10G}

AN Enabled 
{25G}

P1: AN {25G}

Not supported

P1: AN {25G}

P1: AN {25G}

Not supported

P1: AN {25G}

P1: AN {1/10G}

P1: AN {25G}

P1: AN {25G}

P1: AN {25G}

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1/25G}

P2: AN {25G}

P2: AN {1/25G}

AN Enabled 
{1/10G}

P1: AN {1/10G} P1: AN {1/10G} P1: AN {1G}

P1: AN {1/10G}

P1: AN {1/10G}

P1: AN {1/10G}

P1: AN {1/25G}

P1: AN {1G}

P1: AN {1/10G}

P1: AN {1/10G}

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1G}

P2: AN {10G}

P2: AN {1/10G}

AN Enabled 
{1/25G}

P1: AN {1/25G} P1: {1G}

P1: AN {1/25G} P1: AN {1/25G}

P1: AN {1G}

P1: AN {1/25G}

P1: AN {10/25G}

P1: AN {1/25G}

P1: AN {1/25G}

P1: AN {1/25G}

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1/25G}

P2: AN {25G}

P2: AN {1/25G}

AN Enabled 
{10/25G}

P1: AN {10/
25G}

P1: {10G}

P1: AN {25G}

P1: AN {10/25G}

P1: AN {10G}

P1: AN {25G}

P1: AN {1/10/25G}

P1: AN {25G}

P1: AN {10/25G}

P1: AN {10/25G}

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {25G}

P2: AN {10/25G}

P2: AN {1/10/25G}

AN Enabled 
{1/10/25G}

P1: AN {1/10/
25G}

P1: {1/10G}

P1: AN {1/25G} P1: AN {1/10/

25G}

P1: AN {1/10G}

P1: AN {1/25G}

P1: AN {1/10/25G}

P1: AN {1/25G}

P1: AN {1/10/25G} P1: AN {1/10/25G}

P2: no AN

P2: no AN

P2: no AN

P2: AN {1G}

P2: AN {10G}

P2: AN {25G}

P2: AN {1/10G}

P2: AN {1/25G}

P2: AN {10/25G}

P2: AN {1/10/25G}

Содержание NetXtreme-E

Страница 1: ...NetXtreme UG100 August 23 2018 NetXtreme C NetXtreme E User Guide...

Страница 2: ...For more information please visit www broadcom com Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability function or design Informati...

Страница 3: ...1 4 5 3 Header Payload Split 11 4 6 UDP Fragmentation Offload 11 4 7 Stateless Transport Tunnel Offload 12 4 8 Multiqueue Support for OS 12 4 8 1 NDIS VMQ 12 4 8 2 VMware NetQueue 12 4 8 3 KVM Xen Mul...

Страница 4: ...ux 22 9 System Level Configuration 23 9 1 UEFI HII Menu 23 9 1 1 Main Configuration Page 23 9 1 2 Firmware Image Properties 23 9 1 3 Device Level Configuration 23 9 1 4 NIC Configuration 23 9 1 5 iSCS...

Страница 5: ...Configuration and Use Case Examples 35 12 SR IOV Configuration and Use Case Examples 36 12 1 Linux Use Case Example 36 12 2 Windows Use Case Example 37 12 3 VMware SRIOV Use Case Example 38 13 NPAR C...

Страница 6: ...Guest OS 48 15 DCBX Data Center Bridging 50 15 1 QoS Profile Default QoS Queue Profile 50 15 2 DCBX Mode Enable IEEE only 51 15 3 DCBX Willing Bit 51 16 DPDK Configuration and Use Case Examples 54 16...

Страница 7: ...CNS14336 Class B Table 2 Safety Approvals Country Certification Type Standard Compliance International CB Scheme ICES 003 Digital Device UL 1977 connector safety UL 796 PCB wiring safety UL 94 flammab...

Страница 8: ...ons not expressly approved by the manufacture responsible for compliance could void the user s authority to operate the equipment 2 Functional Description The Broadcom NetXtreme C BCM573XX and NetXtre...

Страница 9: ...and receive side TCP Checksum Offload Support for transmit and receive side UDP Checksum Offload Support for transmit and receive side NDIS TCP Large Send Offload Support for LSOV1 and LSOV2 NDIS Rece...

Страница 10: ...IEEE 802 1q VLAN Supported Interrupt Moderation Supported MAC VLAN filters Supported Table 7 Virtualization Features Feature Details Linux KVM Multiqueue Supported VMware NetQueue Supported NDIS Virtu...

Страница 11: ...e TPA context is identified by the 4 tuple Source IP destination IP source TCP port and destination TCP port TPA improves system performance by reducing interrupts for network traffic and lessening CP...

Страница 12: ...allows the scaling of virtual machines across multiple processors See Driver Advanced Properties for information on VMQ 4 8 2 VMware NetQueue The VMware NetQueue is a feature that is similar to Micros...

Страница 13: ...ver Converge Ethernet RoCE Remote Direct Memory Access RDMA over Converge Ethernet RoCE is a complete hardware offload feature in the BCM5741X that allows RDMA functionality over an Ethernet network R...

Страница 14: ...pported RoCE on PFs Up to 4 PFs RoCE on VFs Valid for VFs attached to RoCE enabled PFs Host OS Linux Windows ESXi no vRDMA support Guest OS Linux and Windows DCB Up to two COS per port with non shared...

Страница 15: ...dcom NetXtreme E Ethernet adapter verify that the system meets the requirements listed for the operating system 5 2 1 Hardware Requirements Refer to the following list of hardware requirements One ope...

Страница 16: ...em case and disconnect any personal antistatic devices 5 4 Connecting the Network Cables Broadcom Ethernet switches are productized with SFP SFP28 QSFP28 ports that support up to 100 Gb s These 100 Gb...

Страница 17: ...ware ESXi 6 0 U3 or above Table 11 Linux Ethtool Commands Command Description ethtool s ethX speed 25000 autoneg off Set the speed If the link is up on one port the driver does not allow the other por...

Страница 18: ...ethX ntuple on Enable Accelerated RFS Ethtool t ethX Performs various diagnostic self tests echo 32768 proc sys net core rps_sock_flow_entries echo 2048 sys class net ethX queues rx X rps_flow_cnt Ena...

Страница 19: ...mnicX esxcli network nic down n vmnicX Disable vmnicX esxcli network nic up n vmnic6 Enable vmnicX bnxtnetcli s n vmnic6 S 25000 Set the link speed Bnxtnetcli is needed for older ESX versions to suppo...

Страница 20: ...can Auto negotiate speeds whereas 25 Gb s ports are set to forced speeds SR IOV Enable or Disable Default Enabled This parameter works in conjunction with HW configured SR IOV and BIOS configured SR I...

Страница 21: ...Port 1 and Port 2 Reported link speeds are correct and might not match Speed and Duplex setting 0x001B Incompatible speed selection between Port 1 and Port 2 Link configuration became illegal 0x001C...

Страница 22: ...ding driver and adding team members to the bond which would load balance the traffic Use the following steps to setup Linux bonding 1 Execute the following command modprobe bonding mode balance alb Th...

Страница 23: ...Management CCM version numbers 9 1 3 Device Level Configuration Main configuration page The device level configuration allows the user to enable SR IOV mode number of virtual functions per physical f...

Страница 24: ...ers that can be configured using this section are the same as the HII menu NIC configuration 9 2 3 iSCSI Boot Main Menu Parameters that can be configured using this section are the same as the HII men...

Страница 25: ...1 AN 1G P1 AN 1G P1 AN 1G P1 AN 1G P2 no AN P2 no AN P2 no AN P2 AN 1G P2 AN 10G P2 AN 25G P2 AN 1 10G P2 AN 1 25G P2 AN 10 25G P2 AN 1 10 25G AN Enabled 10G P1 AN 10G P1 AN 25G Not supported P1 AN 10...

Страница 26: ...10 25G Forced 1G 1G No link No link No link No link No link No link No link No link No link Forced 10G No link 10G No link No link No link No link No link No link No link No link Forced 25G No link No...

Страница 27: ...speed with the link partner This option must match the AN protocol setting in the link partner port The Broadcom NetXtreme E NIC supports the following auto negotiation protocols IEEE 802 3by 25G 50G...

Страница 28: ...esults in esxcli command failure NOTE VMware does not support 25G 50G speeds in ESX6 0 In this case use the second utility BNXTNETCLI to set 25G 50G speed For ESX6 0U2 the 25G 50G speed is supported e...

Страница 29: ...arameters via DHCP This option is specific to IPv4 Controls whether the iSCSI boot host software acquires the IP address information using DHCP Enabled or use a static IP configuration Disabled IP Aut...

Страница 30: ...Pv4 IP Autoconfiguration Disabled For IPv6 non offload iSCSI parameters via DHCP Disabled CHAP Authentication Disabled DHCP Vendor ID BRCM ISAN Link Up Delay Time 0 Use TCP Timestamp Enabled for some...

Страница 31: ...tions in Configuring the DHCP Server to Support iSCSI Boot on page 33 For IPv4 with the exception of the initiator iSCSI name any settings on the Initiator Parameters 1st Target Parameters or 2nd Targ...

Страница 32: ...P Version IPv6 For IPv6 non offload 2 Select ESC to return to the Main menu NOTE Information on the Initiator Parameters and 1st Target Parameters screens are ignored and do not need to be cleared 3 S...

Страница 33: ...ient The format of the root path as defined in IETC RFC 4173 is iscsi servername protocol port LUN targetname The parameters are defined in Table 19 Table 19 DHCP Option 17 Parameter Definition Parame...

Страница 34: ...iguration menu 10 3 2 DHCP iSCSI Boot Configuration for IPv6 The DHCPv6 server can provide a number of options including stateless or stateful IP configuration as well as information to the DHCPv6 cli...

Страница 35: ...he VXLAN interface is brought up the host present in system 1 can communicate with host present in system 2 The VLXAN format is shown in Table 22 Table 23 provides VXLAN command and configuration exam...

Страница 36: ...fig o boot grub2 grub cfg for legacy mode For UEFI mode edit etc default grub cfg and run grub2 mkconfig o etc grub2 efi cfg Refer to the following example Linuxefi vmlinuz 3 10 0 229 el7 x86_64 root...

Страница 37: ...ization mode to SR IOV c Set the number of virtual functions per physical function d Set the number of MSI X vectors per the VF and Max number of physical function MSI X vectors If the VF is running o...

Страница 38: ...nce the number of MSI X vectors per VM using CCM 2 Enable virtualization in the BIOS a During system boot enter the system BIOS Processor settings Virtualization Technologies and set it to Enabled b D...

Страница 39: ...below For example to enable four VFs on PF 0 and 2 VFs on PF 2 esxcfg module s max_vfs 4 2 bnxtnet reboot required The required VFs of each supported PF are enabled in order during the PF bring up Ref...

Страница 40: ...partitions per physical device In NPAR SRIOV mode only two VFs from each parent physical port can enable RDMA support or total of four VFs RDMA per physical device 13 3 Configuration NPAR can be conf...

Страница 41: ...partitions that should be allocated from the selected physical port Each BCM5741X NIC can support a maximum of 16 partitions on an ARI capable server By default dual port adapters are configured for...

Страница 42: ...ber of RSS queues as fewer cores participate in processing receive packets from that NIC Per processor CPU utilization should be monitored to ensure that there are no hot processors after this change...

Страница 43: ...he HII menu in the BIOS before the RDMA option takes effect in the host or guest OS To enable RDMA in single function mode if Virtualization Mode is None or SR IOV During the system boot access the Sy...

Страница 44: ...S11SP4 Refer to the OFED release notes at the following link and install OFED before compiling bnxt_re driver http downloads openfabrics org downloads OFED release_notes OFED_3 18 2_release_notes To c...

Страница 45: ...using the rpm e kmod bnxt_en command RHEL 7 3 SLES 12 Sp2 has bnxt_en inbox driver older than v1 7 x This driver must be removed and the latest bnxt_en be added before applying the bnxt_re RoCE driver...

Страница 46: ...er of RDMA connections by adding multiple IP addresses each with different a subnet for the same physical port under test Multiple network shares can be created and mapped to each link partner using t...

Страница 47: ...using the following commands cd var log vmware esxcli software vib install no sig check v bnxtroce driver version vib 2 Reboot the machine 3 Verify that the drivers are correctly installed using the...

Страница 48: ...VM To add a PVRDMA device to the VM 1 Select the VM and right click on Edit Settings 2 Add a new Network Adapter 3 Select the network as a Distributed Virtual Switch and Port Group 4 For the Adapter...

Страница 49: ...ewall settings may need to be modified to allow RDMA traffic Ensure the proper firewall settings are in place 5 Add the usr lib in the etc ld so conf file and reload the ldconf by running ldconfig 6 L...

Страница 50: ...llowing settings in the UEFI HII menu are required to enable DCBX support System Setup Device Settings NetXtreme E NIC Device Level Configuration 15 1 QoS Profile Default QoS Queue Profile Quality of...

Страница 51: ...CB specification If the Willing bit on a device is true the device is willing to accept configurations from a remote device through DCBX If the Willing bit on a device is false the device rejects any...

Страница 52: ...wControl priority 4 Priority Enabled 4 False Use the following to create QoS Policy PS C New NetQosPolicy Name SMB policy SMB PriorityValue8021Action 4 Name SMB policy Owner Group Policy Machine Netwo...

Страница 53: ...46BD A0A 6 4EE0E587B95 NetworkProfile All Precedence 127 IPProtocol Both IPDstPrefix 10 240 1 0 24 PriorityValue 7 Name SMB policy Owner 382AFAD 1E73 46BD A0A 6 4EE0E587B95 NetworkProfile All Preceden...

Страница 54: ...huge echo 2048 sys devices system node node0 hugepages hugepages 2048kB nr_hugepages echo 2048 sys devices system node node1 hugepages hugepages 2048kB nr_hugepages modprobe uio insmod build kmod igb_...

Страница 55: ...num The default burst number is 1 when burst_num not presented stop Stops packet forwarding and display accumulated statistics testpmd stop quit Quits to prompt testpmd quit 16 5 Display Functions The...

Страница 56: ...kout cables are available from QSFP to 4xSFP28 ports What are the compatible port speeds For BCM57404AXXXX BCM57414 dual port devices the port speed of each port must be compatible with the port speed...

Страница 57: ...NetXtreme UG100 57 NetXtreme C NetXtreme E User Guide Revision History NetXtreme UG100 August 23 2018 Initial Release...

Страница 58: ......

Отзывы: