manualshive.com logo in svg

Solarflare SFA6902F, User Manual, Page: 114 / 375

Share
Download

Summary of Contents for SFA6902F

Page 1: ...his document is subject to change without notice 2008 2014 Solarflare Communications Inc All rights reserved Trademarks used in this text are registered trademarks of Solarflare Communications Inc Ado...

Page 2: ...2 8 Supported SFP 10G LR Optical Transceivers 27 2 9 QSFP Transceivers and Cables 27 2 10 Supported SFP 1000BASE T Transceivers 29 2 11 Supported 1G Optical Transceivers 30 2 12 Supported Speed and M...

Page 3: ...5 Full Solarflare Package Installation 115 4 6 Install Drivers and Options From a Windows Command Prompt 119 4 7 Unattended Installation 123 4 8 Managing Adapters with SAM 128 4 9 Managing Adapters R...

Page 4: ...nded Installation Solaris 10 266 6 5 Unattended Installation Solaris 11 267 6 6 Configuring the Solarflare Adapter 268 6 7 Setting Up VLANs 270 6 8 Solaris Utilities Package 270 6 9 Configuring the Bo...

Page 5: ...335 9 6 Interface Configuration 338 9 7 Tuning 339 9 8 Driver Properties via sysctl 339 9 9 Firmware Update 340 9 10 Performance 342 Chapter 10 Solarflare Boot ROM Agent 344 10 1 Configuring the Solar...

Page 6: ...I O for virtualized platforms The VNIC architecture can provide a VNIC per Virtual Machine giving over a thousand protected interfaces to the host system granting any virtualized guest operating syste...

Page 7: ...ardware Switch Fabric Full hardware switch fabric in silicon capable of steering any flow based on Layer 2 Layer 3 or application level protocols between physical and virtual interfaces Supporting an...

Page 8: ...s across all guest operating systems Software re programmable on the fly to allow for adjustment as congestion increases on the network Jumbo frame support Support for up to 9216 byte jumbo frames MSI...

Page 9: ...Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes factory enabled PTP and hardware timestamps Enabled by installing AppFlex license 1PPS Optional bracket and cable assembly not...

Page 10: ...rfaces per port Supports OpenOnload Yes factory enabled PTP and hardware timestamps AppFlex license required 1PPS Optional bracket and cable assembly not factory installed SR IOV Yes Network ports 2 x...

Page 11: ...er port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes SR IOV Yes Network ports 2 x 10GBASE T 10G 1G 100M Part number SFN5122F Controller silicon SFC9020 Power 4 9W typical...

Page 12: ...11 Solarflare Communications 2014 7 Solarflare Server Adapter User Guide 1 SR IOV is not supported for Solarflare adapters on IBM System p servers Network ports 2 x SFP 10G 1G Regulatory Product Code...

Page 13: ...r 5 9W typical PCI Express 8 lanes Gen2 5 0GT s 127 SR IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes SR IOV Yes Network ports 2 x SFP 10G 1G...

Page 14: ...apters on IBM System p servers Part number SFN5161T Controller silicon SFL9021 Power 12 9W typical PCI Express 8 lanes Gen2 5 0GT s Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnloa...

Page 15: ...erfaces per port Supports OpenOnload Yes SR IOV Yes Ports 2 x 10GBASE KX4 backplane transmission Part number SFN5814H Controller silicon 2 x SFC9020 Power 7 9W typical PCI Express 8 lanes Gen2 5 0GT s...

Page 16: ...M Onload Server Adapter SR IOV Yes Ports 2 x SFP 10G 1G Regulatory Product Code S6930 Part number SFN6822F Controller silicon SFC9020 Power 5 9W typical PCI Express 8 lanes Gen2 5 0GT s 127 SR IOV vir...

Page 17: ...nterpriseOnload is supported on all Linux variants listed above and is available for all Solarflare Onload network adapters Solarflare are not aware of any issues preventing OpenOnload installation on...

Page 18: ...n source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of c...

Page 19: ...not occur in a particular installation If the equipment does cause harmful interference to radio or television reception which can be determined by turning the equipment off and on the user is encoura...

Page 20: ...staller are solely responsible for ensuring that the correct devices are utilized in the equipment and the equipment with LDMs installed complies with applicable laser safety requirements 1 8 Regulato...

Page 21: ...003 US FCC Part 15 Class B Canada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class B Japan VCCI Regulations V 3 2010 Class B South Korea KCC KN 22 KN 24 Australia AS NZS CISPR 22 2009 Safety1 1 Th...

Page 22: ...ada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class A Japan VCCI Regulations V 3 2010 Class A Australia AS NZS CISPR 22 2009 Safety1 1 The safety assessment has been concluded on this product as...

Page 23: ...ical Transceivers Page 30 Supported Speed and Mode Page 30 Configure QSFP Adapter Page 31 LED States Page 33 Solarflare Mezzanine Adapters SFN5812H and SFN5814H Page 34 Solarflare Mezzanine Adapter SF...

Page 24: ...Performant network adapters Solarflare SFN5161T Dual Port 10GBASE T Server Adapter Solarflare SFN5162F Dual Port 10G SFP Server Adapter Solarflare Mezzanine adapters Solarflare SFN5812H Dual Port 10G...

Page 25: ...cket fitted to the adapter A full height bracket has also been supplied for PCIe slots that require this type of bracket To fit a full height bracket to the Solarflare adapter 1 From the back of the a...

Page 26: ...8 lane or 16 lane PCIe slot refer to the server manual if necessary and insert the Solarflare card 3 Secure the adapter bracket in the slot 4 Replace the cover and restart the server 5 After restarti...

Page 27: ...Depending on the intended use attach a suitable cable For example to achieve 10 Gb s performance use a Category 6 cable To achieve the desired performance the adapter must be connected to a compliant...

Page 28: ...et until it clicks into place Removing the SFP Direct Attach Cable 1 Pull straight back on the release ring to release the cable retention tab Alternatively you can lift the retention clip on the adap...

Page 29: ...artner and vice versa Removing a fiber optic cable 1 Remove the cable from the adapter bracket and replace the fiber optic connector cover 2 Pull the plastic or wire tab to release the adapter bracket...

Page 30: ...SFP Direct Attach Cables Manufacturer Product Code Cable Length Notes Arista CAB SFP SFP 1M 1m Arista CAB SFP SFP 3M 3m Cisco SFP H10GB CU1M 1m Cisco SFP H10GB CU3M 3m Cisco SFP H10GB CU5M 5m HP J928...

Page 31: ...f 10G SR transceivers with Solarflare network adapters However only transceivers in the table below have been fully verified and are therefore supported Table 3 Supported SFP 10G Optical SR Transceive...

Page 32: ...ny issues preventing the use of other brands of QSFP 40G transceivers and cables with Solarflare SFN7000 QSFP adapters However only products listed in the tables below have been fully verified and are...

Page 33: ...to support connectivity flexibility in high speed data center applications These high performance direct attach assemblies support 2 lanes of 10 Gb s per QSFP port and are available in lengths of 1 m...

Page 34: ...transceivers with the Solarflare network adapters However only transceivers in the table below have been fully verified and are therefore supported Table 8 Supported QSFP to SFP Breakout Cables Manuf...

Page 35: ...pters use auto negotiation to automatically select the highest speed supported in common with the link partner OnSFP adaptersthecurrentlyinsertedSFPmodule transceiver determinesthesupportedspeeds typi...

Page 36: ...on The Solarflare 40G breakout cables have only 2 physical cables for details refer to Supported QSFP to SFP Breakout Cables on page 28 Breakout cables from other suppliers may have 4 physical cables...

Page 37: ...module 1G No 1G SFP optical module 10G 1G No 10G or 1G Dual speed modules run at the maximum speed 10G unless explicitly configured to the lower speed 1G SFP 1000BASE T module No 1G These modules sup...

Page 38: ...ates are as follows Table 13 LED States Adapter Type LED Description State QSFP SFP SFP Link Activity Green solid at all speeds Flashing green when network traffic is present LEDs are OFF when there i...

Page 39: ...S chassis HS22 HS22V and HX5 servers The IBM BladeCenter blade supports a single Solarflare mezzanine adapter 1 The blade should be extracted from the BladeCenter in order to install the mezzanine ada...

Page 40: ...block 6 Replace the blade top cover 7 When removing the adapter raise the release handle shown on Figure 3 to ease the adapter upwards until it can be freed from the connector block 2 16 Solarflare M...

Page 41: ...top diagram 3 Fit riser PCB card into the slot as shown in Figure 5 top diagram Note that the riser card only fits one way 4 Offer the adapter to the node and ensure it lies underneath the chassis cov...

Page 42: ...1 The node should be extracted from the rack server in order to install the mezzanine adapter Refer to the PowerEdge rack server manual if necessary Figure 5 SFN6832F C62 Installing into the rack ser...

Page 43: ...deployment should refer to the Solarflare Enhanced PTP User Guide SF 109110 CD 1 Requires an AppFlex license refer to Solarflare AppFlex Technology Licensing on page 12 2 19 Solarflare SFA6902F Applic...

Page 44: ...ring the Solarflare Adapter Page 49 Configuring Receive Transmit Ring Buffer Size Page 50 Setting Up VLANs Page 51 Setting Up Teams Page 52 Running Adapter Diagnostics Page 53 Running Cable Diagnostic...

Page 45: ...es See Upgrading Adapter Firmware with Sfupdate on page 70 Hardware Timestamps Solarflare Flareon SFN7122F1 SFN7142Q1 and SFN7322F adapters support the hardware timestamping of all received packets in...

Page 46: ...only available if the adapter has auxiliary power supplied by a separate cable Task offloads Support for TCP Segmentation Offload TSO Large Receive Offload LRO and TCP UDP IP checksum offload for imp...

Page 47: ...nning kernel version from the source RPM enter the following at the command line rpmbuild rebuild package_name Where package_name is the full path to the source RPM see the note below 3 To build for a...

Page 48: ...rver Adapter a The NetworkManager service and associated GUI tools For more information about his refer to https wiki gnome org NetworkManager b Solarflare recommend using the Network Administration T...

Page 49: ...the last line in etc modprobe d unsupported modules to allow_unsupported_modules 1 2 Install the RPMs root myhost1 rpm ivh kernel module sfc 2 6 5 7 244 smp 2 1 0111 0 sf 1 SLES9 i586 rpm 3 Run YaST t...

Page 50: ...installation kernel Kernel A in the diagram above This is achieved by defining sfc_minimal to rpmbuild This macro disables hardware monitoring MTD support used for access to the adapters flash I2C and...

Page 51: ...ed BIOS PXE code on the adapter runs Adapter must be in PXE boot mode See PXE Support on page 345 SF Boot ROM PXE DHCP request from PXE SF Boot ROM DHCP server filename and next server options SF Boot...

Page 52: ...file can be accessed A Solarflare driver disk Unattended Red Hat Enterprise Linux installations are configured with Kickstart The documentation for Kickstart can be found at http docs redhat com docs...

Page 53: ...T The documentation for AutoYaST can be found at http www suse com ug autoyast_doc index html The prerequisites for a Network AutoYaST installation are SUSE Linux Enterprise installation media A DHCP...

Page 54: ...Linux online manual man ethtool for details of the options that are available for ethtool Configuring Speed and Modes Solarflare adapters by default automatically negotiate the connection speed to th...

Page 55: ...Linux on page 79 To change offload settings for Tx and Rx use the ethtool command ethtool offload ethX rx on off tx on off Configuring Receive Transmit Ring Buffer Size By default receive and transmi...

Page 56: ...Linux Enterprise Server you can use the YaST WOL module to configure Wake on LAN or you can use the ethtool wol g setting In Red Hat Enterprise Linux you can use the ethtool wol g setting 3 12 Setting...

Page 57: ...l adapter or port failure Teaming configuration support provided by the Linux bonding driver includes 802 3ad Dynamic link aggregation Static link aggregation Fault Tolerant To set up an adapter team...

Page 58: ...r to generate an interrupt and verifying that the interrupt has been processed by the network driver tx rx loopback Verifies that the network driver is able to pass packets to and from the network ada...

Page 59: ...the ethtool diagnostic offline tests phy cable pairA length 9 phy cable pairB length 9 phy cable pairC length 9 phy cable pairD length 9 phy cable pairA status 1 phy cable pairB status 1 phy cable pai...

Page 60: ...ported distributions The Solarflare utilities RPM file can be downloaded from the following location https support solarflare com SF 104451 LS is a 32bit binary RPM package SF 107601 LS is a 64bit bin...

Page 61: ...s Sfboot SLES 11 Limitation Due to limitations in SLES 11 using kernel versions prior to 2 6 27 54 it is necessary to reboot the server after running the sfboot utility Sfboot Command Usage The genera...

Page 62: ...h configuration parameters those parameters will be silently ignored i adapter ethX Performs the action on the identified Solarflare network adapter The adapter identifier ethX can be the ifname or MA...

Page 63: ...c 1G 1G bit sec 100M 100M bit sec linkup delay seconds Specifies the delay in seconds the adapter defers its first connection attempt after booting allowing time for the network to come up following a...

Page 64: ...en initiator dhcp is disabled Note that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi initiator dhcp disabled initiator ip 192 168 1 3 netmask ip...

Page 65: ...IQN iSCSI Qualified Name to be used by the adapter when initiator iqn dhcp is disabled The IQN is a symbolic name in the notation form for example iqn 2009 01 com solarflare and is a maximum of 223 c...

Page 66: ...that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi target dhcp disabled target server 192 168 2 2 target port port_number Specifies the Port num...

Page 67: ...Specifies the device vendor ID to be advertised to the DHCP server This must match the vendor id configured at the DHCP server when using DHCP option 43 to obtain the iSCSI target chap enabled disabl...

Page 68: ...t this option is valid if either CHAP or Mutual CHAP is enabled chap enabled mutual chap enabled Note that if there are spaces contained in secret then it must be wrapped in double quotes Example sfbo...

Page 69: ...port adapters where it can be used to establish adapter port priority The range is 1 255 with 1 being the highest priority mpio attempts attempt count Specifies the number of times MPIO will try and u...

Page 70: ...128 256 The maximum number of interrupts a virtual function may use port mode default 10G 40G Configure the port mode to use This is for SFC9140 family adapters only MAC address assignments may change...

Page 71: ...FI Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type PXE MSI X interrupt limit 32 eth2 Boot image Option ROM and UEF...

Page 72: ...10 Level 5 Networks 2002 2005 eth4 Boot image Option ROM and UEFI Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type...

Page 73: ...t Solarflare Communications 2006 2010 Level 5 Networks 2002 2005 eth2 Boot image Option ROM and UEFI Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot s...

Page 74: ...lity v4 0 0 Copyright Solarflare Communications 2006 2013 Level 5 Networks 2002 2005 eth4 Boot image Option ROM only Link speed Negotiated automatically Link up delay time 7 seconds Banner delay time...

Page 75: ...w the current firmware version for all Solarflare adapters and identifies whether the firmware within sfupdate is more up to date To update the firmware for all Solarflare adapters run the command sfu...

Page 76: ...flare interfaces 3 Identify the bus device function for all Solarflare interfaces lspci D d 1924 4 Output similar to the following will be produced 5 NICs installed in this example lspci D d 1924 0000...

Page 77: ...do echo n bdf sys bus pci devices bdf driver unbind done 8 Repeat the process for the other interfaces 0000 04 00 x 0000 83 00 x and 0000 84 00 x doing so in pairs until all the NICs have been upgrade...

Page 78: ...images embedded in the sfupdate tool To re write using an external image specify image filename in the command write fails if the embedded image is the same or a previous version To force a write in t...

Page 79: ...0 Level 5 Networks 2002 2005 eth1 MAC 00 0F 53 01 39 70 Firmware version v3 0 3 PHY type QT2025C PHY version v2 0 2 5 Controller type Solarflare SFC4000 Controller version v3 0 3 2127 BootROM version...

Page 80: ...and installed license keys sfkey adapter eth4 report 2 port adapter eth4 eth5 Product name Solarflare SFN7122F SFP Server Adapter Part number SFN7122F Serial number 712200205071133867100441 MAC addres...

Page 81: ...s installed on an adapter sfkey adapter eth4 inventory keys eth4 eth5 712200205071133867100441 Flareon ONL PTP SCP SSFE PM License information is displayed in Prefix Acronym Suffix format Prefix Facto...

Page 82: ...ys installed sfc driver reload is required after sfkey installs a PTP license To reload the sfc driver modprobe r sfc modprode sfc or when Onload is installed onload_tool reload inventory List by seri...

Page 83: ...above Use with all or with adapter If an installed or active key is reported as An where n is a number it indicates a license unknown to this version of sfkey use an updated sfkey version s silent Sil...

Page 84: ...ning an adapter Throughput Latency CPU utilization Different applications may be more or less affected by improvements in these three metrics For example transactional request response network applica...

Page 85: ...ts by editing the file etc sysconfig network scripts ifcfg ethX and adding MTU mtu on a new line Interrupt Moderation Interrupt Coalescing Interrupt moderation controls the number of interrupts genera...

Page 86: ...t of increased CPU utilization TCP IP Checksum Offload Checksum offload moves calculation and verification of IP Header TCP and UDP packet checksums to the adapter The driver by default has all checks...

Page 87: ...bled by default Therefore there is no opportunity to improve performance from the default TSO is controlled using ethtool sbin ethtool K ethX tso on off Verification of the TSO settings may be perform...

Page 88: ...utilization especially for high bandwidth streaming applications To determine if LRO is enabled on an interface ethtool k ethX If IP forwarding is not required on the server Solarflare recommends eit...

Page 89: ...echo 0 sys class net ethX device lro To enable LRO echo 1 sys class net ethX device lro To show the current value of the per adapter LRO state cat sys class net ethX device lro Modifying this file in...

Page 90: ...defaults minimum 4096 and default 87380 Receive Side Scaling RSS Solarflare adapters support Receive Side Scaling RSS RSS enables packet receive processing to scale with the number of available CPU c...

Page 91: ...number of RSS queues to create A separate MSI X interrupt for a receive queue is affinitized to each CPU packages An RSS queue will be created for each multi core CPU package The first CPU in the pac...

Page 92: ...Linux 2 6 35 that attempts to schedule protocol processing of incoming packets on the same processor as the user thread that will consume the packets Accelerated RFS requires Linux kernel version 2 6...

Page 93: ...n the expected number of active connections at any given time and may be less than the number of open connections The value is rounded up to the nearest power of two Per Queue Flow Count For each adap...

Page 94: ...ntains a bitmap of CPUs that can use the queue to transmit In the following example transmit queue 0 can be used by the first two CPUs and transmit queue 1 can be used by the following two CPUs If hyp...

Page 95: ...ip cat proc irq 139 smp_affinity 00000000 00000000 00000000 00000800 echo 110011 sys class net eth3 queues tx 0 xps_cpus echo 11001100 sys class net eth3 queues tx 1 xps_cpus echo 220022 sys class net...

Page 96: ...alance permanently run sbin chkconfig level 12345 irqbalance off To see whether irqbalanace is currently running run sbin service irqbalance status To disable irqbalance temporarily run sbin service i...

Page 97: ...Q 1 is affinitized to CPU 1 and so on With this configuration the latency and cpu utilization for a particular TCP flow will be dependant on that flow s RSS hash and which CPU that hash resolves onto...

Page 98: ...topology physical_package_id 0 Having determined that cpu0 and cpu10 are on package 1 we can assign each ethX interface s MSI X interrupt to its own CPU on the same package In this case we choose pack...

Page 99: ...ion TX PIO is therefore especially useful for smaller packets The TX PIO feature is enabled by default for packets up to 256 bytes The maximum packet size that can use PIO can be configured with the d...

Page 100: ...th Many chipsets use multiple channels to access main system memory Maximum memory performance is only achieved when the chipset can make use of all channels simultaneously This should be taken into a...

Page 101: ...eave at default for 2 6 16 and later kernels For earlier kernels sysctl net core tcp_rmem 4096 87380 524288 sysctl net core tcp_wmem 4096 87380 524288 Receive Side Scaling RSS Application dependent In...

Page 102: ...sys class net ethX device lro TCP Protocol Tuning Leave at default but changing does not impact latency Receive Side Scaling Application dependent Interrupt affinity irqbalance service Stop irq balanc...

Page 103: ...SS Set to 1 CPU by adding rss_cpus to following line to etc modprobe conf file or add the options line to a file under the etc modprobe d directory options sfc rss_cpus 1 Then reload drivers to use ne...

Page 104: ...PAGE 2 old kernels rx_refill_threshold RX descriptor ring fast slow fill threshold uint 90 lro_table_size1 Size of the LRO hash table Must be a power of 2 uint 128 lro_chain_max1 Maximum length of cha...

Page 105: ...g uint 64 rx_irq_mod_usec Receive interrupt moderation microseconds uint 60 tx_irq_mod_usec Transmit interrupt moderation microseconds uint 150 allow_load_on_failure If set then allow driver load when...

Page 106: ...packets exceeding the maximum frame length tx_bad_bytes Number of bytes transmitted with incorrect FCS tx_packets Number of packets transmitted tx_bad Number of packets transmitted with incorrect FCS...

Page 107: ...being sub ject to multiple collisions tx_excessive_collision Number of packets not transmitted due to excessive colli sions Excessive collisions occur on network under heavy load or when too many dev...

Page 108: ...om the network adapter driver tx_pio_packets Number of packets sent using PIO tx_tso_bursts Number of times when outgoing TCP data is split into pack ets by the adapter driver Refer to TCP Segmentatio...

Page 109: ..._15xx Number of packets received where the length is between 1024 and 1518 bytes 1522 with VLAN tag rx_15xx_to_jumbo Number of packets received where the length is between 1518 bytes 1522 with VLAN ta...

Page 110: ...AN tag bytes not matching the number of actual received bytes rx_internal_error Number of frames that could not be received due to a MAC internal error condition e g frames not received by the MAC due...

Page 111: ...nc_qbb Not currently supported rx_pm_discard_qbb Not currently supported rx_pm_discard_mapping Number of packets dropped because they have an 802 1p priority level configured to be dropped rx_dp_q_dis...

Page 112: ...PIO rx_reset 0 rx_tobe_disc Number of packets marked by the adapter to be discarded because of one of the following Mismatch unicast address and unicast promiscuous mode is not enabled Packet is a pa...

Page 113: ...FIFO at a sufficient rate the MAC will be unable to feed more of the packet to the FIFO In this event the MAC will truncate the frame marking it as such and discard the remainder The driver on seeing...

Page 114: ...ptions From a Windows Command Prompt Page 119 Unattended Installation Page 123 Managing Adapters with SAM Page 128 Managing Adapters Remotely with SAM Page 129 Using SAM Page 130 Configuring Network A...

Page 115: ...ad on page 142 See Configuring Network Adapter Properties in Windows on page 167 Receive Side Scaling RSS Solarflare adapters support RSS multi core load distribution technology See Using SAM to View...

Page 116: ...provide comprehensive adapter and cable fault diagnostics and system reports See Using SAM to Run Adapter and Cable Diagnostics on page 154 See Sfcable Cable Diagnostics Tool on page 200 Firmware upda...

Page 117: ...e_Windows_x86_64 bit_Driver_Package exe This can be renamed e g setup exe before use NOTE The Solarflare adapter should be physically inserted before installing the drivers See Installation on page 18...

Page 118: ...larflare adapter drivers in Windows To install the drivers from the command line see Install Drivers and Options From a Windows Command Prompt on page 119 1 Double click the supplied Setup exe to star...

Page 119: ...14 114 Solarflare Server Adapter User Guide 2 From the Custom Setup screen select the Install Solarflare device drivers option only Figure 8 Solarflare Custom Setup 3 Click Finish to close the wizard...

Page 120: ...ge Installation Procedure Page 116 Repair Remove and Change Drivers and Utilities Page 118 Prerequisites The Solarflare Adapter Manager Utility SAM requires Microsoft NET Framework 3 5 assemblies Thes...

Page 121: ...rivileges to continue installing the drivers 2 Follow the setup instructions in the wizard to complete the driver installation procedure See Figure 10 and Table 28 for a list of setup options 3 Click...

Page 122: ...are NDIS drivers for Windows The Solarflare drivers are installed by default Install Solarflare command line tools Installs the following Solarflare Windows command line tools sfboot exe Boot ROM conf...

Page 123: ...tion options wizards for teaming and VLAN setup adapter statistics and diagnostic tools See Managing Adapters with SAM on page 128 for more details SAM is installed by default Note If this option is g...

Page 124: ...drivers install package is named after the Solarflare document part number e g SF 107785 LS 2_Solarflare_Windows_x86_64 bit_Driver_Package exe This can be renamed e g setup exe before invoking from th...

Page 125: ...omplete the installation as required setup exe Install Quiet Performs a silent installation of the drivers and utilities setup exe Reinstall Reinstalls the drivers and utilities setup exe Uninstall Re...

Page 126: ...es are not allowed ADDLOCAL cannot prevent Launcher from being installed if either NetworkAdapterManager or CommandLineTools are not installed or are still being installed ADDLOCAL Examples Install th...

Page 127: ...tory The Destination Directory will list the following sub directory structure The actual folders files displayed will depend on the Solarflare driver package installed Table 30 lists the drivers supp...

Page 128: ...a network from a WDS server avoiding the need to install each operating system directly from a CD or DVD This guide assumes you have installed and are familiar with WDS For more information on WDS see...

Page 129: ...modify the boot image Solarflare recommends using the ImageX tool supplied with the Windows Automated Installation Kit AIK 1 Within WDS expand the server where the boot image is located and select the...

Page 130: ...ustom Install Image 1 From WDS locate the install image from the Install Images folder on your server 2 Right click the image and select Export Image from the menu 3 Export the image to a location whe...

Page 131: ...ml file is used by the Windows PE boot environment to configure settings including the language credentials for connecting to the WDS server the partitioning of the disk and which image to deploy To a...

Page 132: ...in unattend mode option 4 Click Select File and browse to your AutoUnattend xml file Further Reading Installing and configuring Windows Deployment Services WDS http technet microsoft com en us librar...

Page 133: ...atistics and State Information Page 153 Using SAM to Run Adapter and Cable Diagnostics Page 154 Using SAM for Boot ROM Configuration Page 159 Introduction The Solarflare Adapter Manager SAM is a Micro...

Page 134: ...cover any potential issues which may be affecting adapter performance Also SAM s detailed statistics and state information can be used to view data transfer figures sent and received packet types as w...

Page 135: ...computer OR Click Start All Programs Solarflare Network Adapters Manage network adapters on this computer OR Click Start Administrative Tools Computer Management System Tools Network Adapters Figure 1...

Page 136: ...nufacturer or adapter type Figure 14 Solarflare Adapter Manager SAM For each adapter SAM provides the following details Name and network interface IP address IPv4 and IPv6 if available MAC address Tra...

Page 137: ...formance graphs Right click on an adapter and select Show graphs from the menu By default SAM shows the load transmitted packets and received packets graphs only To view other available graphs Select...

Page 138: ...to change the units used to display data enable separators when displaying large numbers and disable enable warning messages To configure SAM options 1 Start SAM 2 From the Actions pane click Options...

Page 139: ...ptions Description Values Display values using SI units Displays values using international standard units K M G T P E for example 2 3M Enabled by default This can be useful when dealing with the larg...

Page 140: ...load Page 138 RSS and Interrupts Page 139 Segmentation Offload Page 142 Ethernet Link Speed Page 142 Ethernet flow control Page 142 Ethernet Frame Length Page 143 NOTE Changing the value of an Adapter...

Page 141: ...e Settings Use one of the following methods to access the Adapter Features Dialog From SAM right click on an adapter and select Configuration Configure Offload tasks Ethernet and other features From S...

Page 142: ...unications 2014 137 Solarflare Server Adapter User Guide The Adapter Features dialog box will be displayed Figure 16 Solarflare Adapter Manager Adapter Features Click Apply or OK when changes to Adapt...

Page 143: ...es See Configuring Network Adapter Properties in Windows on page 167 for more details Table 32 Checksum Offloads Check box selected Transmit and Receive Transmit checksums are generated and received c...

Page 144: ...nique used to reduce the number of interrupts sent to the CPU With interrupt moderation the adapter will not generate interrupts closer together than the interrupt moderation interval An initial packe...

Page 145: ...the number of processors to be used by RSS If this is greater than or equal to the number of logical processors in the system then all processors are used Interrupt moderation Adaptive adjusts the in...

Page 146: ...The NUMA node id drop down list box is displayed on Windows platforms that support NUMA architectures This constrains the set of CPU cores used for RSS to the specified NUMA node Solarflare recommend...

Page 147: ...etwork Adapter Properties in Windows on page 167 for more details Ethernet Link Speed Generally it is neither necessary or desirable to configure the link speed of the adapter The adapter by default w...

Page 148: ...ption Auto negotiate Flow control is auto negotiated between the devices This is the default setting preferring Generate and respond if they line partner is capable Generate and respond Adapter genera...

Page 149: ...ollowing features Classification of received network traffic in hardware by using the destination MAC address and optionally also the VLAN identifier to route packets to different receive queues dedic...

Page 150: ...ual port Solarflare adapter the first port could be a member of team A and the second port a member of team B or both ports members of the same team This section is only relevant to teams of Solarflar...

Page 151: ...re strongly recommend taking the server offline to avoid disrupting existing services as the team is being configured 2 Start SAM and select a Solarflare adapter in the Network Adapter list 3 From the...

Page 152: ...change team settings 1 Start SAM and from the Network Adapter list select the team you want to reconfigure 2 From the Action menu select Configure this Team The Configure a Team dialog box displays F...

Page 153: ...click the key button Note Before physically removing an adapter from a server first check it is not the key adapter You must reassign the key adapter if you want to remove it from the team to avoid du...

Page 154: ...dd one or more adapters The Available Network Adapters dialog box is displayed Figure 20 Available Adapters To specify the Ethernet frame length MTU Specify a value between 1514 and 9216 bytes Check y...

Page 155: ...To delete a team 1 Start SAM and select a Solarflare adapter team from the Network Adapter list 2 From the Action menu select Delete this team Alternatively to delete all teams and VLANs on the serve...

Page 156: ...VLAN 0 or untagged traffic which allows the interface to communicate with devices that do not support VLAN tagging or that are sending traffic on VLAN 0 To create VLANs 1 Start SAM and select the adap...

Page 157: ...ional name for the VLAN network adapter This option will not be available when remotely administering the server Supports the handling of priority traffic Enables the handling of traffic that is tagge...

Page 158: ...view Solarflare statistics and state information 1 Start SAM and select a Solarflare adapter from the Network Adapter list 2 From the Actions list click Statistics and State The Details from adapter...

Page 159: ...15 Using SAM to Run Adapter and Cable Diagnostics You can verify the Solarflare adapter driver and cable by running SAM s built in diagnostic tools Solarflare 10GBASE T adapter only The tools provide...

Page 160: ...splayed 3 Select the test you want to run no tests are selected by default See Table 38 for a description of the tests that are available 4 To stop as soon as a failure is detected select Stop on firs...

Page 161: ...ver MAC loopback Verifies that the network driver is able to pass packets to and from the network adapter using the MAC loopback layer PHY loopback Verifies that the network driver is able to pass pac...

Page 162: ...rom the Action menu click Diagnostics then Cable Diagnostics The Cable Diagnostics for adapter name dialog box is displayed 3 Click Run offline test or Run online test Offline testing produce more det...

Page 163: ...Cable Pair Diagnostic Results Result Meaning OK Length measured SNR margin Cable is operating correctly The range is 13dB approximately The SNR should be positive Error Pair short at A short circuit...

Page 164: ...or all iSCSI options Using SAM you can access the adapter Boot ROM to configure your firmware settings for adapter booting as described below Configuring the Boot ROM for PXE or iSCSI Booting For more...

Page 165: ...links and should not be changed unless advised The Link Speed options will vary depending on the installed Figure 27 Link tab 4 The Link up delay specifies a wait time before the boot device will atte...

Page 166: ...g DHCP to get the initiator name clear Use DHCP to get the initiator name and enter a iSCSI Qualified Name IQN in the Initiator name field 8 DHCP vendor Id specifies the device vendor ID to be adverti...

Page 167: ...tails for the iSCSI target as described in Table 41 Table 41 iSCSI Target Options Option Description Target server name Target server network address in the form of a dotted quad i e 10 1 2 3 IPv4 add...

Page 168: ...o this LUN on boot up and attempt to load the target operating system from it LUN retry count Specifies the number of times the boot device will attempt to connect to the target LUN logical unit numbe...

Page 169: ...get and the initiator will authenticate the connection If CHAP authentication is configured on the iSCSI target enter the correct settings to allow access to the target Table 42 CHAP Options Option De...

Page 170: ...er entering a failed state 14 When you have finished configuring the iSCSI settings click OK or Apply to save your settings to the Boot ROM Disabling Adapter Booting You can stop the adapter from atte...

Page 171: ...from the Action menu The firmware update window is displayed Figure 32 Solarflare firmware update window If the firmware is up to date the window will contain the OK button If the firmware is out of d...

Page 172: ...n also access the adapter properties using SAM To configure network adapter properties 1 From the Control Panel select System 2 Select Device Manager from the left hand menu 3 Expand the Network adapt...

Page 173: ...pt Moderation is enabled Adaptive Interrupt Moderation allows the adapter to vary it s interrupt moderation automatically according to network traffic demands If Adaptive Interrupt Moderation is disab...

Page 174: ...mance of the network adapter Default setting 60 s IPv4 Checksum Offload Disabled Rx Tx Enabled Rx Enabled Tx Enabled IP checksum offload is a hardware offload technology for reducing the load on a CPU...

Page 175: ...size supported by the adapter Note Devices will drop frames if they are unable to support the specified frame size so ensure the value you set here is supported by other devices on the network Default...

Page 176: ...the adapter hardware Default setting Rx Tx Enabled UDP Checksum Offload IPv4 and IPv6 Disabled Rx Tx Enabled Rx Enabled Tx Enabled UDP checksum offload is a hardware offload technology for reducing th...

Page 177: ...on page 112 Table 44 List Available Command Line Utilities Utility Description sfboot exe A tool for configuring adapter Boot ROM options for PXE and iSCSI booting See Sfboot Boot ROM Configuration To...

Page 178: ...e command tool exe Figure 34 Windows console to run Solarflare command line tools NOTE For all the utilities the options are documented with the forward slash prefix You can also use a single dash or...

Page 179: ...t Command Usage 1 Login with an administrator account 2 Click Start All Programs Solarflare Network Adapters Command Line Interface for Network Adapters 3 From the Command Prompt enter the command usi...

Page 180: ...rors no user interaction You should query the completion code to determine the outcome of commands when operating silently Log Filename Logs output to the specified file in the current folder or an ex...

Page 181: ...sed by the Boot ROM the default is auto On the 10GBASE T adapters auto instructs the adapter to negotiate the highest speed supported in common with it s link partner On QSFP SFP adapters auto instruc...

Page 182: ...les or disables DHCP address discovery for the adapter by the Boot ROM except for the Initiator IQN see initiator iqn dhcp This option is only valid if iSCSI booting is enabled boot type iscsi If init...

Page 183: ...used by the adapter when initiator dhcp is disabled Note that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi initiator dhcp disabled gateway 192...

Page 184: ...adapter 2 lun retry count count Specifies the number of times the adapter attempts to access and login to the Logical Unit Number LUN on the iSCSI Target before failing Note that this option is only v...

Page 185: ...should only be used if your target is using a non standard TCP Port target lun LUN Specifies the Logical Unit Number LUN to be used by the iSCSI target when target dhcp is disabled The default LUN is...

Page 186: ...es the CHAP initiator username maximum 64 characters Note that this option is required if either CHAP or Mutual CHAP is enabled chap enabled mutual chap enabled Note that if there are spaces contained...

Page 187: ...ual chap enabled Note that if there are spaces contained in username then it must be wrapped in double quotes target secret secret Specifies the secret that has been configured on the iSCSi target min...

Page 188: ...ult is 32 Note Using the incorrect setting can impact the performance of the adapter Contact Solarflare technical support before changing this setting sriov enabled disabled Enable SR IOV support for...

Page 189: ...delay time 5 seconds Boot type Disabled PFIOV Disabled Number of Physical Functions 2 MSI X interrupt limit 32 Number of Virtual Functions 0 VF MSI X interrupt limit 8 Firmware variant full feature vi...

Page 190: ...type iscsi Solarflare boot ROM configuration utility v4 1 4 Copyright Solarflare Communications 2006 2014 Level 5 Networks 2002 2005 Network adapters in this computer 1 Solarflare SFN7122F SFP Server...

Page 191: ...ated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type iSCSI Use DHCP for Initiator Enabled Use DHCP for Initiator IQN Enabled LUN busy re...

Page 192: ...Boot image Option ROM only Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type iSCSI Use DHCP for Initiator Disabled...

Page 193: ...B B1 Boot image Option ROM only Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type iSCSI Use DHCP for Initiator Enabl...

Page 194: ...ht click the icon and choose Command line tools instead 3 In the Command Prompt window enter your command using the following syntax sfupdate Adapter Identifier options where Identifier is the name or...

Page 195: ...ght message at startup Verbose Shows extended output information for the command entered Quiet Aliases Silent Suppresses all output including warnings and errors no user interaction You should query t...

Page 196: ...ion This option may be used with the Write and Force options but is not required with the Quiet option Image ImageFileName Sources firmware image from an external file NoWarning Suppress update warnin...

Page 197: ...are Communications 2006 2014 Level 5 Networks 2002 2005 1 Solarflare SFN7122F SFP Server Adapter MAC address 00 0F 53 21 9B B1 Firmware v4 1 0 update to v4 1 4 Boot ROM v4 1 0 6723 update to v4 2 0 10...

Page 198: ...n with an administrator account 2 Click Start All Programs Solarflare Network Adapters Command Line Interface for network adapters If you installed the Solarflare system tray icon you can right click...

Page 199: ...n if required Computer ComputerName Performs the operation on the identified remote computer Administrator rights on the remote host computer is required List Lists all available Solarflare adapters a...

Page 200: ...dapter Adapter_id Specifies the adapter to add to the team Repeat this option for each adapter that you want to include in the team This option must be used when a team is first created It cannot be a...

Page 201: ...the IP address is specified then DHCP is assured You can also use tag priority name DHCP to be explicit Formats tag e g 2 assumes no priority tag priority e g 2 p tag priority name e g 2 p my name ta...

Page 202: ...tolerant default dynamic static See Teaming and VLANs on page 209 for an explanation on the different teaming types Mode mode Specifies how the driver will select adapters to be part of the link aggre...

Page 203: ...tics Display adapter and link aggregation statistics Detailed Display detailed configuration statistics Solarflare teaming configuration utility v4 1 4 Copyright Solarflare Communications 2006 2014 Le...

Page 204: ...right Solarflare Communications 2006 2014 Level 5 Networks 2002 2005 Creating VLAN group done new id 4V Setting VLAN group name using default name Group 4V done Adding adapter 2 done Creating VLAN id...

Page 205: ...enter the following command sfcable Adapter Identifier options where Identifier is the name or ID of the adapter that you want to manage Specifying the adapter is optional if it is not included the co...

Page 206: ...operation on the identified remote computer Administrator rights on the remote host computer is required Adapter Identifier Performs the action on the identified Solarflare network adapter The identif...

Page 207: ...v4 1 4 Copyright Solarflare Communications 2006 2014 Level 5 Networks 2002 2005 1 Solarflare SFN5121T 10GBASE T Server Adapter MAC address 00 0F 53 01 40 8C Link state Up Link speed 10 Gbps Pair 1 OK...

Page 208: ...r your command using the following syntax sfnet Adapter Identifier options where Identifier is the name or ID of the adapter that you want to manage Specifying the adapter is optional if it is not inc...

Page 209: ...Silent Suppresses all output including warnings and errors no user interaction You should query the completion code to determine the outcome of commands when operating silently Log Filename Logs outp...

Page 210: ...on covers the Flow Control and Flow Control Autonegotiation device driver advanced properties speed auto 40g 10g 1g 100m Specify the Ethernet link speed mtu MTU length Specify the maximum Ethernet fra...

Page 211: ...ng stack moderation disabled value Specify interrupt moderation time in micro seconds If a value is given it must be in the range 1 to 1000 NOTE this option covers the device driver advanced propertie...

Page 212: ...able to be added 32 Application failed initialization 33 Access denied Either the remote host refused a connection on the basis of account privileges or a file could not be opened 34 Cannot connect T...

Page 213: ...ady installed 128 User entered an invalid command line 129 Could not open log file 130 A general WMI error occurred Can occur when the connection is lost 131 Missing prerequisite The application needs...

Page 214: ...for supporting load balancing and fault tolerance across a team of network adapters and their associated switch Link aggregation is a partner teaming mode that requires configuration at both ends of...

Page 215: ...led 802 3ad to negotiate the ports that will make up the team LACP must be enabled at both ends of the link for a team to be operational LACP will automatically determine which physical links can be a...

Page 216: ...cations 2014 211 Solarflare Server Adapter User Guide Figure 35 802 1AX Team Figure 36 shows a 802 1AX team with a failed link All traffic is re routed and shared between the other team links Figure 3...

Page 217: ...cally inter operates with common link aggregation schemes such as Intel Link Aggregation Cisco Fast EtherChannel and Cisco Gigabit EtherChannel With static link aggregation all links share the traffic...

Page 218: ...re than one of the adapters will be active with the remainder either in standby or in a fault state In Figure 39 Adapter 1 is active and all data to and from the switch passes through it Figure 39 Fau...

Page 219: ...ure Note that in this example Adapter 3 is now active The order in which the adapters are used is determined by a number of factors including user definable rank VLANs VLANs offer a method of dividing...

Page 220: ...s and Teaming VLANs are supported on all Solarflare adapter teaming configurations VLANs with Fault Tolerant Teams Figure 42 shows a fault tolerant team with two VLANs Figure 42 Fault Tolerant VLANs F...

Page 221: ...Issue 11 Solarflare Communications 2014 216 Solarflare Server Adapter User Guide Figure 43 Failover in Fault Tolerant Team VLAN...

Page 222: ...Ns work in the same way with either Dynamic or Static Link Aggregation teaming configurations Figure 44 shows how VLANs work with these teams Figure 44 VLAN with Dynamic or Static Link Team In case of...

Page 223: ...r It doesn t even need to be in the list of standby adapters but it must physically be within its host The Key Adapter defines the team s RSS support see Receive Side Scaling RSS on page 225 and provi...

Page 224: ...y Adapter Note that although the Key Adapter Adapter 1 has a link failure the integrity of the team is not affected by this failure Dynamic and Static Link Aggregation Teams The assignment of key adap...

Page 225: ...Issue 11 Solarflare Communications 2014 220 Solarflare Server Adapter User Guide Any link failure on the key adapter does not affect the redistribution of traffic to the other links in the team...

Page 226: ...this section is to highlight adapter driver settings that affect the performance metrics described This guide covers the tuning of all Solarflare adapters Latency will be affected by the type of phys...

Page 227: ...compatible with legacy 10 100Mbps Ethernet endpoints However if a larger maximum frame size is used adapter throughput and CPU utilization can be improved CPU utilization is improved because it takes...

Page 228: ...rupt moderation should be disabled This will increase the number of interrupts generated by the network adapter and as such increase CPU utilization Interrupt moderation settings are critical for tuni...

Page 229: ...ation is changed by changing the Offload IP checksum Offload UDP checksum and Offload TCP checksum settings in the Network Adapter s Advanced Properties Page Large Send Offload LSO Large Send offload...

Page 230: ...bled by default and will be used on network adapters that support it Solarflare recommend that RSS is enabled for best networking performance For further information about using RSS on Windows platfor...

Page 231: ...Event Log if it detects that the adapter is plugged into a PCIe slot which electrically has fewer than x8 lanes For SFN5xxx and SFN6xxx Solarflare adapters which require a PCIe Gen 2 x8 slot for opti...

Page 232: ...stems Intel QuickData NetDMA On systems that support Intel I OAT I O Acceleration Technology features such as QuickData a k a NetDMA Solarflare recommend that these are enabled as they are rarely detr...

Page 233: ...hen other CPU cores or devices need to access data currently in the idle CPU core s data cache the CPU core is signaled to return to the fully on power state These changes in CPU core power states cre...

Page 234: ...ewall Depending on the system configuration the built in Windows or any third party Firewall may have a significant impact on throughput and CPU utilization Where high throughput is required on a part...

Page 235: ...ties 3 On server run ntttcpr ntttcpr exe rb 500000 a 24 n 100000 l 524288 m 1 1 server_adapter_IP_interface 4 On client run ntttcps test ntttcps exe rb 500000 a 24 n 100000 l 524288 m 1 1 server_adapt...

Page 236: ...ed Max Frame Size Configure to maximum supported by network in Network Adapter s Advanced Properties Offload Checksums Leave at default Receive Side Scaling RSS Leave at default RSS NUMA Node Leave at...

Page 237: ...ult Enabled TCP IP Checksum Offload Leave at default Receive Side Scaling Application dependent RSS NUMA Node Leave at default All TCP Protocol Tuning Leave at default install with Optimize Windows TC...

Page 238: ...n Notes 0x60000001L BUS_STATUS_DRIVER_VERSION Informational The driver version information No action required 0x60000002L BUS_STATUS_DRIVER_LOAD_FAILU RE Informational The driver failed to load 0xA000...

Page 239: ...DISABLED_FAILURE Error The device could not be initialized with interrupts disabled 0xE0010007L BUS_STATUS_DEVICE_INIT_INTERR UPTS_ENABLED_FAILURE Error The device could not be initialized with interr...

Page 240: ...0010015L BUS_STATUS_DEVICE_VOLTAGE_ER ROR Error One of the device voltage supplies is outside of the critical voltage range The adapter or server maybe faulty 0xE0010016L BUS_STATUS_DEVICE_UNKNOWN_ SE...

Page 241: ...are Feature Set Page 237 Installing Solarflare Drivers and Utilities on VMware Page 238 Configuring Teams Page 239 Configuring VLANs Page 239 Running Adapter Diagnostics Page 241 Configuring the Boot...

Page 242: ...age 260 See TCP Segmentation Offload TSO on page 260 NetQueue Support for NetQueue a performance technology that significantly improves performance in 10 Gigabit Ethernet virtualized environments See...

Page 243: ...uration Guide available from http www vmware com support pubs Installing on VMware ESXi 5 0 and 5 1 To install or update the VIB through the CLI esxcli software vib install v absolute PATH to the vib...

Page 244: ...ch vSwitch The main benefits of creating a team are Increased network capacity for the virtual switch hosting the team Passive failover in the event one of the adapters in the team fails To create a t...

Page 245: ...s for the Virtual Switch you want to create the team for 4 Select a Port Group and click Edit 5 Enter a valid VLAN ID 0 equals no VLAN 6 Click OK Further Reading NIC teaming in VMware ESX Server http...

Page 246: ...troller to generate an interrupt and verifying that the interrupt has been processed by the network driver tx rx loopback Verifies that the network driver is able to pass packets to and from the netwo...

Page 247: ...cX options parameters Note that without adapter the sfboot command applies to all adapters that are present in the target host The format for the parameters are parameter value Sfboot Command Line Opt...

Page 248: ...If this option is used with configuration parameters those parameters will be silently ignored i adapter vmnicX Performs the action on the identified Solarflare network adapter The adapter identifier...

Page 249: ...c 1G 1G bit sec 100M 100M bit sec linkup delay seconds Specifies the delay in seconds the adapter defers its first connection attempt after booting allowing time for the network to come up following a...

Page 250: ...hen initiator dhcp is disabled Note that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi initiator dhcp disabled initiator ip 192 168 1 3 netmask i...

Page 251: ...e IQN iSCSI Qualified Name to be used by the adapter when initiator iqn dhcp is disabled The IQN is a symbolic name in the notation form for example iqn 2009 01 com solarflare and is a maximum of 223...

Page 252: ...that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi target dhcp disabled target server 192 168 2 2 target port port_number Specifies the Port num...

Page 253: ...d Specifies the device vendor ID to be advertised to the DHCP server This must match the vendor id configured at the DHCP server when using DHCP option 43 to obtain the iSCSI target chap enabled disab...

Page 254: ...t this option is valid if either CHAP or Mutual CHAP is enabled chap enabled mutual chap enabled Note that if there are spaces contained in secret then it must be wrapped in double quotes Example sfbo...

Page 255: ...on the adapter mutual chap enabled Note that if there are spaces contained in secret then it must be wrapped in double quotes mpio priority MPIO priority Specifies the Multipath I O MPIO priority for...

Page 256: ...th1 Boot image Option ROM and UEFI Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot skip delay time 5 seconds Boot type Disabled MSI X interrupt limit...

Page 257: ...luded the command is applied to all Solarflare adapters in the machine option is one of the command options listed in Table 62 The format for the options are option parameter Running the command sfupd...

Page 258: ...n sfupdate To use an external image specify image filename in the command write fails if the embedded image is the same or a previous version to that in the adapter To force a write in this case speci...

Page 259: ...unications 2006 2010 Level 5 Networks 2002 2005 eth1 MAC 00 0F 53 01 39 70 Firmware version v3 0 3 PHY type QT2025C PHY version v2 0 2 5 Controller type Solarflare SFC4000 Controller version v3 0 3 21...

Page 260: ...dent on throughput The purpose of this section is to highlight adapter driver settings that affect the performance metrics described This guide covers the tuning of all members of the Solarflare famil...

Page 261: ...you must mount the CD install the tools and configure them For example if the guest is Red Hat mount dev cdrom mnt rpm i mnt VMwareTools rpm vmware tools config pl VMware ESX NetQueue Solarflare adap...

Page 262: ...machine is to be bound on the right hand side of the dialog box To bind a queue s interrupt to a CPU from the VMware ESX console OS enter echo move IRQVEC CPU proc vmware intr tracker Where IRQVEC is...

Page 263: ...does not include the Ethernet preamble or frame CRC Since the MTU should ideally be matched across all endpoints in the same LAN VLAN and since the LAN switch infrastructure must be able to forward su...

Page 264: ...as the interval parameter will turn interrupt moderation off ethtool C vmnicX rx usecs irq interval Verification of the moderation settings may be performed by running ethtool c This parameter is cri...

Page 265: ...TCP segmentation offload benefits applications using TCP Non TCP protocol applications will not benefit but will not suffer from TSO Enabling TCP segmentation offload will reduce CPU utilization on th...

Page 266: ...cated CPU core This allows the receive side TCP processing to be distributed amongst the available CPU cores providing a considerable performance advantage over a conventional adapter architecture in...

Page 267: ...st be enabled on the host if it is to be used in the guests To enable I OAT on the VMware ESX host On some systems the hardware associated with I OAT must first be enabled in the BIOS Log in to the Co...

Page 268: ...266 Unattended Installation Solaris 11 Page 267 Setting Up VLANs Page 270 Solaris Utilities Package Page 270 Configuring the Boot ROM with sfboot Page 270 Upgrading Adapter Firmware with Sfupdate Pag...

Page 269: ...d adapter performance and reduced CPU processing requirements See Configuring Task Offloading on page 269 Receive Side Scaling RSS Support for RSS multi core load distribution technology See Receive S...

Page 270: ...yed Solarflare 10GE NIC Driver i386 DRIVER VERSION LICENSE INFO This package contains scripts which will be executed with super user permission during the process of installing this package Do you wan...

Page 271: ...http docs oracle com cd E19253 01 819 6397 819 6397 pdf To install the Solarflare Solaris package as part of an unattended installation it must be added using the package command to the JumpStart mach...

Page 272: ...an entry migrating_from_jumpstart_to_automated Table 64 Installation Stages In Control Stages of Boot Setup needed BIOS PXE code on the adapter runs Adapter must be in PXE boot mode See PXE Support on...

Page 273: ...sfxge x inet6 plumb ifconfig sfxge x inet6 up Then create an IPv6 interface sfxge x interface with a link local IPv6 address by entering ifconfig sfxge x inet6 addif IPv6 address ipv6 prefix length up...

Page 274: ...v4 TCP and UDP transmit Tx and receive Rx checksum offload as well as TCP segmentation offload To ensure maximum performance from the adapter all task offloads should be enabled which is the default s...

Page 275: ...com cd E19253 01 816 4554 fpdga index html 6 8 Solaris Utilities Package The Solarflare Solaris Utilities package supplied as a 32 bit SVR package and available from https support solarflare com conta...

Page 276: ...Ctrl B to access the Boot Rom agent during server startup See Configuring the Solarflare Boot ROM Agent on page 344 for more information on the Boot Rom agent Sfboot Command Usage The general usage fo...

Page 277: ...ecify silent to suppress simultaneous output to screen if required computer computer_name Performs the operation on a specified remote computer Administrator rights on the remote computer is required...

Page 278: ...instructs the adapter to use the highest link speed supported by the inserted SFP module On 10GBASE T and SFP adapters any other value specified will fix the link at that speed regardless of the capab...

Page 279: ...to be specified initiator ip ip_address netmask subnet The following options may also be needed gateway ip_address primary dns ip_address initiator ip ipv4 address Specifies the IPv4 address in standa...

Page 280: ...mary dns 192 168 0 3 initiator iqn dhcp enabled disabled Enables or disables use of DHCP for the initiator IQN only initiator iqn IQN Specifies the IQN iSCSI Qualified Name to be used by the adapter w...

Page 281: ...that this option is only valid if iSCSI booting is enabled boot type iscsi Example sfboot boot type iscsi target dhcp disabled target server 192 168 2 2 target port port_number Specifies the Port num...

Page 282: ...d Specifies the device vendor ID to be advertised to the DHCP server This must match the vendor id configured at the DHCP server when using DHCP option 43 to obtain the iSCSI target chap enabled disab...

Page 283: ...t this option is valid if either CHAP or Mutual CHAP is enabled chap enabled mutual chap enabled Note that if there are spaces contained in secret then it must be wrapped in double quotes Example sfbo...

Page 284: ...valid for iSCSI booting over multi port adapters where it can be used to establish adapter port priority The range is 1 255 with 1 being the highest priority mpio attempts attempt count Specifies the...

Page 285: ...lhost sfboot list Solarflare boot configuration utility v3 0 5 Copyright Solarflare Communications 2006 2010 Level 5 Networks 2002 2005 sfxge0 Boot image Disabled MSI X interrupt limit 32 sfxge1 Boot...

Page 286: ...ng the command sfupdate with no additional parameters will show the current firmware version for all Solarflare adapters and whether the firmware within sfupdate is more up to date To update the firmw...

Page 287: ...write using an external image specify image filename in the command write fails if the embedded image is the same or a previous version To force a write in this case specify the option force force For...

Page 288: ...v3 0 5 2163 PHY version v2 0 2 5 Controller version v3 0 5 2161 The Boot ROM firmware is up to date The PHY firmware is up to date The image contains a more recent version of the Controller v3 0 5 21...

Page 289: ...n adapter Throughput Latency CPU utilization Different applications may be more or less affected by improvements in these three metrics For example transactional request response network applications...

Page 290: ...the interface If you want to have an MTU configure when the interface is brought up add mtu to the single line of configuration data in etc hostname sfxge X For example IP address mtu size Interrupt M...

Page 291: ...fully utilized LRO should not be enabled if you are using the host to forward packets from one interface to another for example if the host is performing IP routing or acting as a layer2 bridge The dr...

Page 292: ...t in the sfxge driver To limit or disable RSS Uncomment the following line in kernel drv sfxge conf rx_scale_count number of MSI X interrupts requested Limitations of Solaris MSI X interrupt allocatio...

Page 293: ...tilization is low This reduces the power consumption of the CPU For latency sensitive applications where the application switches between having packets to process and having periods of idle time wait...

Page 294: ...ge x mtu size Interrupt moderation Leave at default TCP IP Checksum Offload Leave at default TCP Segmentation Offload Leave at default TCP Large Receive Offload Leave at default TCP Protocol Tuning Le...

Page 295: ...lication dependent Buffer Allocation Method Leave at default PCI Express Lane Configuration Ensure current speed not the supported speed reads back as x8 and 5Gb s Or x8 and Unknown CPU Power Manageme...

Page 296: ...s Lane Configuration Ensure current speed not the supported speed reads back as x8 and 5Gb s Or x8 and Unknown CPU Speed service cpuspeed Leave enabled Memory bandwidth Ensure Memory utilizes all memo...

Page 297: ...maximum of 32 MSI X interrupts across all network devices To use more than 2 MSI X interrupts you need to add e g set ddi_msix_alloc_limit 8 in etc system 128 rx_coalesce_mod e Coalesce RX packets La...

Page 298: ...ink is down during the reset 2 reset the hardware but do not use it again useful for failover mechanisms to ensure this adapter does not become the active link again 0 rx_prealloc_pkt _buffers Number...

Page 299: ...Name sfxge_ndd Page 303 Name sfxge_rss Page 304 Name sfxge_rxq0000 Page 305 Name sfxge_txq0000 Page 305 Name sfxge_vpd Page 305 Table 73 Name mac Parameter Description adv_cap_1000fdx Advertise 1000 M...

Page 300: ...0 false cap_100fdx Capable of 100 Mbps full duplex 1 true 0 false cap_100hdx Capable of 100 Mbps half duplex 1 true 0 false cap_10fdx Capable of 10 Mbps full duplex 1 true 0 false cap_10hdx Capable o...

Page 301: ...ets64 Number of packets received 64 bit counter link_asmpause When adv_ pause_cap and lp_ pause_cap are compared following auto negotiation the flow control mechanism for the link depends on what is m...

Page 302: ...send pause frames when there is receive congestion 0 pause transmission when a pause frame is received If lp_cap_asmpause 0 then 1 send pause frames when there is receive congestion and pause transmis...

Page 303: ...ved that exceed the maximum permitted frame size tx_late_collsions A sending device may detect a collision as it attempts to transmit a frame or before it completes sending the entire frame If a colli...

Page 304: ...meter Description crtime Timestamp when samples were taken link_duplex 0 down 1 half duplex 2 full duplex link_speed 10000 Mbps link_up 1 link is up 0 link is down rx_1024_to_15xx_pkts Number of packe...

Page 305: ...FCS errors these are dropped by the Solarflare driver rx_ge_15xx_pkts Number of packets received with payload size greater than 1518 bytes 1522 bytes VLAN rx_internal_errors Number of frames that cou...

Page 306: ...ctets Total number of octets received rx_pause_pkts Number of pause packets received with valid pause op_code rx_pkts Total number of packets received rx_symbol_errors Count of the number of times the...

Page 307: ...5xx_pkts Number of packets transmitted where length is between 15xx and 9000 bytes 1518 non VLAN 1522 VLAN tx_late_col_pkts A sending device may detect a collision as it attempts to transmit a frame o...

Page 308: ...in the MAC statistics in Table 73 above The adv_cap_ parameters represent a mirror image of the mac adv_ _cap parameter list for an Ethernet device The parameters are also a subset of the cap_ statist...

Page 309: ...device The parameters are also a subset of the cap_ statistics If the cap_ value is 0 the corresponding adv_cap_ must also be 0 except for adv_cap_asmpause and adv_cap_pause parameters 0 lp_cap_1000h...

Page 310: ...es the RX descriptor ring was empty causing a received packet to be discarded Table 79 Name sfxge_txq0000 Parameter Description crtime Timestamp when samples were taken post Number of packets posted t...

Page 311: ...Issue 11 Solarflare Communications 2014 306 Solarflare Server Adapter User Guide SN Solarflare adapter serial number VD Adapter firmware version Table 80 Name sfxge_vpd Parameter Description...

Page 312: ...rted Platforms and Adapters Solarflare KVM SR IOV acceleration currently supports Host Red Hat Enterprise Linux 6 0 6 1 and 6 2 KVM Guest VM Red Hat Enterprise Linux 5 5 5 6 and 5 7 Red Hat Enterprise...

Page 313: ...devices which qemu connects to either emulated hardware e1000 rtl8139 or to virtio devices The guest operating system then uses the appropriate drivers to consume the emulated hardware Figure 48 Gues...

Page 314: ...ossible to support as both the source and destination machine must be identical in every respect to the networking adapter That is not only identical at a hardware level same adapter same slot but als...

Page 315: ...d binds the Solarflare plugin driver to this VF This plugin driver automatically registers with the virtio driver as an accelerated network plugin Once the VF driver has registered subsequent traffic...

Page 316: ...network traffic over multiple queues to scale performance over the available CPUs This is termed receive side scaling RSS and is typically used in non virtualized environments Linux Windows to improv...

Page 317: ...er Solarflare Communications SFL9021 Solarstorm 06 00 1 Ethernet controller Solarflare Communications SFL9021 Solarstorm 06 00 2 Ethernet controller Solarflare Communications Device 1813 06 00 4 Ether...

Page 318: ...t 0 9 4 23 el6 sf3 x86_64 rpm libvirt python 0 9 4 23 el6 sf3 x86_64 rpm libvirt devel 0 9 4 23 el6 sf3 x86_64 rpm libvirt lock sanlock 0 9 4 23 el6 sf3 x86_64 rpm virt manager 0 9 0 7 sf1 el6 x86_64...

Page 319: ...network device to the bridge The instructions to configure bridging can be found at Redhat Bridged networking with libvirt Assign an IP address to the bridge Verify network connectivity between the h...

Page 320: ...eth4 hybrid yes vlan 123 interface Virt manager configuration for SR IOV Adding a new interface A virt manager GUI tool is supplied with the Solarflare driver distribution package To run the GUI tool...

Page 321: ...52 Change SR IOV on Existing Adapter Verifying SR IOV As described above libvirt will manage VFs for interfaces with a vf hotplug attribute If there are spare VFs available when the VM is started the...

Page 322: ...SR IOV support uses a plugin model whereby virtual functions accelerate virtio network adapters These virtio network adapters are attached to the physical network device via a macvtap interface XML Fo...

Page 323: ...64da 4a736edb9a8f uuid forward mode pci passthrough hybrid pf dev eth2 123 vlan 123 forward network Domain XML interface type network source network sfc network mac address 00 50 56 3e 86 29 model typ...

Page 324: ...work 1 In the main virt manager window double click the VM to manage it 2 Open the hardware tab by clicking the i button on the toolbar 3 Select the NIC you wish to use with SR IOV from the list on th...

Page 325: ...erting a line like this options xnap rx_irq_mod_usec value rx_ring value 2 Reload the xnap driver modprobe r xnap modprobe xnap Performance Configuration Equipment Configuration Performance tests were...

Page 326: ...TCP streams per port per direction between each pair of VMs on the two servers message size of 65535 bytes Table 83 Linux KVM Performance Numberof 10G ports Traffic Number of VMs Baseline no SR IOV A...

Page 327: ...ed If the migration fails for whatever reason then the VM will remain on host A and the VFs will be reattached to the VM Figure 53 Migrating a guest KVM Limitations Virt manager will support bridges o...

Page 328: ...stigation Reloading the sfc driver while VFs are in use will result in a kernel panic This is an upstream kernel bug that Solarflare are working to address There is a known issue when using LRO with m...

Page 329: ...eleration currently supports XenServer 6 0 6 1 Intel platforms supporting VT d AMD platforms supporting AMD Vi Guest VM Win2k8r2 and RHEL 6 All Solarflare SFN5xxx and SFN6xxx series adapters The hardw...

Page 330: ...OV solutions PCIe hardware virtual functions VF allow a single network hardware device to appear as multiple virtualized network devices to dom0 These virtualized network devices operate independently...

Page 331: ...network I O solution available in the market Solarflare s SR IOV implementation uses a plug in approach that maintains the traditional software data path through the XenServer paravirtualized drivers...

Page 332: ...hosts For example if a guest is migrated to a host containing a different vendor s NIC then the hot plug event provides the opportunity to bind a new VF driver within the guest and re establish accele...

Page 333: ...ts to the hypervisor line in boot extlinux conf amd_iommu on iommu amd iommu perdev intremap Install XenServer6 with SR IOV supplemental pack 1 Burn the supplemental pack xs SR IOV 1 1 7 sfc iso to a...

Page 334: ...Windows Guests Install the Solarflare acceleration driver into the Windows guest using xnapsetup exe 8 4 Configuration Acceleration The SR IOV supplemental pack manages all VF configuration SR IOV acc...

Page 335: ...ds on DOM0 xl list Name ID Mem VCPUs State Time s Domain 0 0 729 4 r 130 4 dibenchvm080 1 0 1 s d 17 8 dibenchvm020 2 2047 1 b 18 1 xl pci list domain ID Vdev Device 09 0 0000 07 00 6 0a 0 0000 07 00...

Page 336: ...on X5600 48G RAM Host Host Virtualization plan used was XenServer6 Guest A number of guest VMs running Windows Server 2008 R2 64bit were started on each server The benchmarking tool used was IXIA Char...

Page 337: ...idirectional 4 6 64 Gbps 19 3 Gbps Table 87 Linux Guest Bandwidth Numberof 10G ports Traffic Number of VMs Baseline no SR IOV Accelerated SR IOV plug in 1 Unidirectional 1 2 23 Gbps 9 25 Gbps 1 Bidire...

Page 338: ...migrated occasionally the guest kernel fails to notice either the VF unplug before migration or the VF hotplug after migration This issue is believed to be a Red Hat kernel bug and is under investiga...

Page 339: ...9 Driver Properties via sysctl Page 339 Firmware Update Page 340 Performance Page 342 9 1 System Requirements Refer to Software Driver Support on page 12 for supported Mac OS X Distributions Solarflar...

Page 340: ...a PCIe slot Full support for Thunderbolt including plugging and unplugging the Thunderbolt cable is planned for a future release 9 5 Driver Install Uninstall Previous Driver An installed Solarflare n...

Page 341: ...ure 57 Disable Driver Services 3 Repeat above steps for all ports of the driver 4 Double click SF 107120 LS dmg in Finder to mount the disk image Invoke the Solarflare driver uninstall script in Termi...

Page 342: ...d and Install the Mac OS X Driver 1 Download SF 107120 LS dmg into a convenient working directory 2 Double click SF 107120 LS dmg in Finder to mount the disk image 3 Run the Solarflare10GbE pkg instal...

Page 343: ...338 Solarflare Server Adapter User Guide 9 6 Interface Configuration With the adapter driver installed the network interface can be configured using the network interface settings menu Figure 59 Sola...

Page 344: ...nce over a wide range of applications It is recommended to only change the driver settings if advised to do so by Solarflare support 9 8 Driver Properties via sysctl Driver properties are also made vi...

Page 345: ...fupdate is as follows as root sfupdate adapter enX options where enX is the interface name of the Solarflare adapter to be upgraded option is one of the command options listed in Sfupdate Options on p...

Page 346: ...the localhost enX Adapter ifname or MAC address as obtained with list list Shows the adapter ID adapter name and MAC address of each adapter installed in the localhost write Re writes the firmware fr...

Page 347: ...che 8 MB Throughput Netperf TCP_STREAM Results using Netperf IPv4 TCP_STREAM at 1500 MTU Latency Netperf TCP_RR Latency measured using Netperf IPv4 TCP_RR will depend on the interrupt moderation setti...

Page 348: ...ation To configure a ramdisk for the test of size 4500000 x 512k sectors sudo diskutil eraseVolume HFS ramdisk hdiutil attach nomount ram 4500000 Table 92 File System Benchmark Test Results Protocol 1...

Page 349: ...should contact their distributor for further information PXE and iSCSI network boot is not supported for Solarflare adapters on IBM System p servers Configuring the Solarflare Boot ROM Agent Page 344...

Page 350: ...tion PXE requires DHCP and TFTP Servers the configuration of these servers depends on the deployment service used Linux For Red Hat Enterprise and SUSE Linux Enterprise Server please consult your Linu...

Page 351: ...ssue 11 Solarflare Communications 2014 346 Solarflare Server Adapter User Guide 1 On starting or re starting the server press Ctrl B when prompted The Solarflare Boot Configuration Utility is displaye...

Page 352: ...Issue 11 Solarflare Communications 2014 347 Solarflare Server Adapter User Guide 2 Use the arrow keys to highlight the adapter you want to boot via PXE and press Enter The Adapter Menu is displayed...

Page 353: ...n press the arrow keys to change the Boot Image and or the Boot Type 4 From the Boot Type press Space until PXE is selected 5 Solarflare recommend leaving the Adapter Options and BIOS Options at their...

Page 354: ...r details of the Solarflare adapter ID MAC address and other details to validate the connection Refer to the iSCSI target documentation for details on how to configure your target 10 5 Configuring the...

Page 355: ...uration Utility will display 2 Highlight the adapter to configure and Press Enter The Adapter Menu is displayed NOTE If the BIOS supports console redirection and you enable it then Solarflare recommen...

Page 356: ...Issue 11 Solarflare Communications 2014 351 Solarflare Server Adapter User Guide...

Page 357: ...Issue 11 Solarflare Communications 2014 352 Solarflare Server Adapter User Guide 3 From the BootROM Mode option press the space bar to change the Boot Image and or the Boot Type...

Page 358: ...Issue 11 Solarflare Communications 2014 353 Solarflare Server Adapter User Guide From the Boot Type press Space until iSCSI is selected Press Enter The iSCSI Initiator options are displayed...

Page 359: ...iSCSI initiator IQN of the Solarflare adapter if you are not using DHCP to obtain the iSCSI initiator IQN DHCP Vendor Class ID If you are using DHCP to obtain the iSCSI initiator IQN the adapter will...

Page 360: ...ion is required highlight iSCSI CHAP and press Enter Enter User Name and Secret information If Mutual CHAP is required as well as CHAP hIghlight this option and press Enter Enter Target user name and...

Page 361: ...d target configurations from it on startup DHCP Option 17 Root Path The root path option can be used to describe the location of the iSCSI target This information is used in Step 4 on page 354 The iSC...

Page 362: ...to option 17 to describe the location of the iSCSI target The format for the iSCSI target IQN is the same as described for DHCP option 17 Option 43 203 provides a method of completely defining the iSC...

Page 363: ...and Configuring the Boot ROM on page 349 Copy the correct Solarflare driver files to a floppy disk or USB flash drive Refer to Steps to Install 1 Insert the Windows Server 2008 R2 DVD and restart the...

Page 364: ...get and iSCSI initiator You will need to enter these details during the installation process Ensure that the Solarflare Boot ROM is configured for iSCSI boot and can login to the selected iSCSI target...

Page 365: ...es click Back to return to the installer 6 Select the network device To check which is the Solarflare network adapter press Ctrl Alt F4 To return to the Installation screen press Ctrl Alt F1 7 Select...

Page 366: ...ed Targets tab The target should be listed 12 Ensure the Start Up mode is correct for your installation For SUSE Enterprise Linux Server 10 it should be automatic For SUSE Enterprise Linux Server 11 i...

Page 367: ...age of the install is complete the system will reboot Continue to the Configure Boot Device Order to add the iSCSI target and continue the installation process Following the server reboot check that t...

Page 368: ...initiator You will need to enter these details during the installation process Ensure that Solarflare Boot ROM is configured for iSCSI boot and can login to the selected iSCSI target You will need th...

Page 369: ...ll be prompted to load more driver disks Select No 7 A CD Found screen will prompt you to test the CD before installation Select Skip 8 When an Enable network interface screen displays select the Sola...

Page 370: ...From the Disk Partitioning menu select Advanced storage configuration to add the iSCSI target 11 In the Advanced Storage Options window select iSCSI and click Add iSCSI target 12 In the Configure iSCS...

Page 371: ...configured proceed with the rest of the installation If the device configuration displayed is incorrect check your details Following the server reboot check that the iSCSI disk is in an appropriate p...

Page 372: ...o Defaults from the Boot Configuration Utility or click Default from SAM Table 94 Default Adapter Settings Setting Default Value Boot Image Disabled Link speed Auto Link up delay 5 seconds Banner dela...

Page 373: ...on on Linux 80 285 Configure Interrupt moderation on VMware 258 Configure segmentation offload 81 Configuring adapter 48 Configuring checksum offload 81 Running adapter diagnostics on Linux 52 Running...

Page 374: ...tions 22 S SAM see also Configure via Boot ROM agent 345 Boot ROM BIOS settings 159 Boot ROM configuration 158 Boot ROM iSCSI Authentication settings 164 Boot ROM iSCSI Initiator settings 161 Boot ROM...

Page 375: ...ations On Linux 95 288 On Windows 230 U Unattended Installation Driver disks 45 SUSE 48 Windows 121 Unattended Installation Solaris 11 267 V Virtual NIC support 2 VLAN Deleting from SAM 152 Setting up...

Reviews:

No comments