manualshive.com logo in svg

Chelsio Communications Terminator 6, Installation And User Manual, Page: 243 / 432

Share
Download
Chelsio Communications Terminator 6 Installation And User Manual Download Page 243

Chapter XVI. Offload Multi-Adapter Failover (MAFO) 

Chelsio Unified Wire for Linux 

 243

 

 Introduction 

Chelsio’s adapters offer a complete suite of high reliability features, including adapter-to-adapter 
failover. The patented offload Multi-Adapter Failover (MAFO) feature ensures all offloaded traffic 
continue operating seamless in the face of port failure. 

MAFO allows aggregating network interfaces across multiple adapters into a single logical bonded 
interface, providing effective fault tolerance. 

The traffic running over the bonded interface can be fully offloaded to the adapter, thus freeing 
the CPU from TCP/IP overhead. 

 

 

 

 

1.1. 

Hardware Requirements 

 Supported Adapters 

The  following  are  the  currently  shipping  Chelsio  adapters  that  are  compatible  with  the  Offload 
Multi-Adapter Failover driver. 

  T62100-CR 
  T62100-LP-CR 
  T6425-CR 
  T6225-CR 
  T6225-LL-CR 
  T580-CR 
  T580-LP-CR 
  T540-CR 
  T540-BT   
  T520-CR 
  T520-LL-CR 
  T520-BT 
  T420-CR 
  T440-CR 
  T422-CR 
  T420-SO-CR 
  T404-BT 
  T420-BCH 
  T440-LP-CR 
  T420-BT  

•

 

 Portions  of  this  software  are  covered  under  US  Patent, 

Failover  and 

migration for full-offload network interface devices: US 8346919 B1

 

•

 

Use  of  the  covered  technology  is  strictly  limited  to  Chelsio  ASIC-based 
soutions. 

 

Important 

 

Summary of Contents for Terminator 6

Page 1: ...Chelsio Unified Wire for Linux i...

Page 2: ...TIVELY THE SOFTWARE IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS INC Sales For all sales inquiries please send email to sales chelsio com Support For all support related question...

Page 3: ...1 1 0 07 26 2013 1 1 1 08 14 2013 1 1 2 12 06 2013 1 1 3 12 19 2013 1 1 4 03 13 2014 1 1 5 05 02 2014 1 1 6 06 30 2014 1 1 7 10 22 2014 1 1 8 11 04 2014 1 1 9 02 05 2015 1 2 0 03 04 2015 1 2 1 03 25...

Page 4: ...Chelsio Unified Wire for Linux iv 1 3 6 09 29 2017 1 3 7 12 29 2017 1 3 8 02 28 2018 1 3 9 03 30 2018 1 4 0 04 18 2018 1 4 1 07 05 2018 1 4 2 07 18 2018 1 4 3 10 01 2018...

Page 5: ...itramfs 36 Configuring Chelsio Network Interfaces 37 4 1 Configuring Adapters 37 4 2 Configuring network scripts 41 4 3 Creating network scripts 41 4 4 Checking Link 42 Performance Tuning 43 5 1 Gener...

Page 6: ...ual Functions SR IOV 74 3 2 Loading the Driver 74 Software Driver Configuration and Fine tuning 76 4 1 VF Rate Limiting 76 4 2 Bonding 77 4 3 High Capacity VF Configuration 78 Software Driver Unloadin...

Page 7: ...river Installation 110 Software Driver Loading 111 Software Driver Configuration and Fine tuning 112 4 1 Accelerating UDP Socket Communications 112 Software Driver Unloading 118 VII WD TOE 119 Introdu...

Page 8: ...floading LIO iSCSI Connection 144 5 3 Running LIO iSCSI and Network Traffic Concurrently 145 5 4 Performance Tuning 146 Software Driver Unloading 147 6 1 Unloading the LIO iSCSI Target Offload Driver...

Page 9: ...Software Driver Loading 191 Software Driver Configuration and Fine tuning 192 4 1 Accelerating open iSCSI Initiator 192 4 2 HMA 194 4 3 Auto login from cxgb4i initiator at OS bootup 195 4 4 Performan...

Page 10: ...tch 227 4 2 FCoE fabric discovery verification 227 4 3 Formatting the LUNs and Mounting the Filesystem 231 4 4 Creating Filesystem 232 4 5 Mounting the formatted LUN 233 Software Driver Unloading 234...

Page 11: ...2 1 Pre requisites 261 2 2 Installation 261 Software Driver Loading 262 Software Driver Configuration and Fine tuning 263 Software Driver Unloading 264 5 1 Unloading the NIC Driver 264 5 2 Unloading t...

Page 12: ...equirements 298 1 2 Software Requirements 299 Software Driver Installation 300 2 1 Pre requisites 300 2 2 Installation 300 Software Driver Configuration and Fine Tuning 301 3 1 Configuring OVS Machine...

Page 13: ...are Configuration File 333 Software Driver Loading 334 Software Driver Configuration and Fine Tuning 335 5 1 Huge Pages 335 5 2 Binding Network Ports 336 5 3 Unbinding Network Ports 338 5 4 Performanc...

Page 14: ...ot Process 387 7 1 Legacy iSCSI Boot 387 7 2 uEFI iSCSI Boot 395 Creating Driver Update Disk DUD 404 8 1 Creating DUD for RedHat Enterprise Linux 404 8 2 Creating DUD for Suse Enterprise Linux 404 OS...

Page 15: ...Chapter I Chelsio Unified Wire Chelsio Unified Wire for Linux 15 I Chelsio Unified Wire...

Page 16: ...bre Channel and InfiniBand applications over Ethernet at line rate Designed for deployment in virtualized data centers cloud service installations and high performance computing environments Chelsio a...

Page 17: ...ing the drivers tools please refer to their respective sections 1 2 Hardware Requirements The Chelsio Unified Wire software supports Chelsio Terminator series of Unified Wire adapters To know more abo...

Page 18: ...ectory contains various benchmarking tools to measure throughput and latency of various networks chelsio_adapter_config This directory contains scripts and binaries needed to configure Chelsio 40G ada...

Page 19: ...directory contains Unified Boot Option ROM image cubt4 bin uEFI driver ChelsioUD efi default boot configuration file bootcfg and a legacy flash utility cfut4 exe which can be used to flash the option...

Page 20: ...here are two sub directories in the Uboot directory OptionROM and LinuxDUD The OptionROM directory contains Unified Boot Option ROM image cubt4 bin uEFI driver ChelsioUD efi default boot configuration...

Page 21: ...er with a screw or other securing mechanism as described by the system manufacturer s instructions Replace the case cover ix After securing the card ensure that the card is still fully seated in the P...

Page 22: ...s indicate the hardware configuration of the adapter as well as serial number Network device names for Chelsio s physical ports are assigned using the following convention the port farthest from the m...

Page 23: ...with low latency TOE iWARP WD UDP WD TOE IPv6 Bonding MAFO High capacity RDMA Configures adapters to establish a large number of iWARP connections iWARP RDMA Performance Improves iWARP performance iWA...

Page 24: ...G_INFINIBAND y iv Compile the kernel v During kernel compilation please ensure that the following parameters are set as follows CONFIG_CHELSIO_T1 m CONFIG_CHELSIO_T1_1G y CONFIG_CHELSIO_T3 m CONFIG_CH...

Page 25: ...d Wire from source GUI mode with Dialog utility i Download the Unified Wire driver package tarball from Chelsio Download Center ii Untar the tarball using the following command root host tar zxvf driv...

Page 26: ...ad support viii Select the required configuration tuning option ix Under Choose install components select all to install all the related components for the option chosen in step viii or select custom...

Page 27: ...RQ balance daemon c TX Coalasce Write tx_coal 2 to modprobe d conf xi If you already have the required version of OFED software installed select Skip OFED To install OFED 4 8 2 choose the Install OFED...

Page 28: ...tion summary of installed components will be displayed xiv Select View log to view the installation log or Exit to continue xv Select Yes to exit the installer or No to go back xvi Reboot your machine...

Page 29: ...v Enter the number corresponding to the Configuration tuning option in the Input field and press Enter vi If you already have the required version of OFED software installed select Skip OFED To insta...

Page 30: ...m machinefilename iv Select the required configuration tuning option The tuning options may vary depending on the Linux distribution v Select the required Cluster Configuration vi If you already have...

Page 31: ...s individually Provided here are steps to build and install some of them For the complete list view help by running make help Change your current working directory to Chelsio Unified Wire package dire...

Page 32: ...r will remove the Chelsio specific drivers inbox outbox from initramfs To skip this and install drivers root host make SKIP_INIT 1 install The installer will check for the required dependency packages...

Page 33: ...tall all the drivers in the Unified Wire software package This option will not install OFED and drivers built against OFED b Wire Direct Latency Install Wire Direct Latency drivers needed for Low late...

Page 34: ...py i nic_toe all udpso wd crypto ovs Here nic_toe NIC and TOE drivers only all All Chelsio drivers built against inbox OFED udpso UDP segmentation offload capable NIC and TOE drivers only wd Wire Dire...

Page 35: ...IP addresses or hostnames of the nodes in the cluster You can view the sample file sample_machinefile provided in the package to view the format in which the nodes have to be listed iii Now execute th...

Page 36: ...port The firmware version can be verified using ethtool root host ethtool i iface 3 8 Removing Drivers from initramfs Chelsio drivers cxgb4 cxgb4vf iw_cxgb4 cxgb4i csiostor etc might exist in the init...

Page 37: ...th the PEER Setting FEC 100G 50G and 25G speeds support changing Forward Error Correction FEC The existing FEC settings can be viewed using root host cxgbtool ethX fec Below is a sample output on T6 1...

Page 38: ...speeds The supported speeds can be seen using ethtool Below is a sample output for T6 100G port Optics Optics do not support auto negotiation Use the following command to change the speed root host et...

Page 39: ...t ethtool change ethX advertise 0x2000000 o Advertise only 50G root host ethtool change ethX advertise 0x400000000 o Advertise only 25G root host ethtool change ethX advertise 0x80000000 o Advertise 1...

Page 40: ...using existing SFP based cabling The port farthest from the motherboard will appear as the first network interface Port 0 T6 Adapters Chelsio T6 100G adapters can be configured in the following 2 mode...

Page 41: ...OTPROTO static should be changed to BOOTPROTO dhcp The ifcfg ethX files have to be created manually They are required for bringing the interfaces up and down and attribute the desired IP addresses 4 3...

Page 42: ...could look like file etc sysconfig network scripts ifcfg eth1 DEVICE eth1 HWADDR 00 07 43 04 6B E9 ONBOOT no NM_CONTROLLED no BOOTPROTO static IPADDR 10 192 167 112 NETMASK 255 255 240 0 From now on t...

Page 43: ...e 1 intel_idle max_cstate 0 to the kernel command line to prevent the system from entering power saving idle states and avoid CPU frequency changes Turn off irqbalance root host etc init d irqbalance...

Page 44: ...or Linux 44 Disable few services root host t4_latencytune sh interface Set sysctl param net ipv4 tcp_low_latency to 1 root host sysctl w net ipv4 tcp_low_latency 1 To optimize your system for differen...

Page 45: ...te For any distribution specific problems please check README and Release Notes included in the release for possible workaround Please visit Chelsio Download Center for regular updates on various soft...

Page 46: ...ide to use source you can uninstall the package using CLI or GUI mode 7 1 Uninstalling Chelsio Unified Wire from source GUI mode with Dialog utility i Change your current working directory to Chelsio...

Page 47: ...sio Unified Wire for Linux 47 iv The selected components will now be uninstalled v After successful uninstalltion summary of the uninstalled components will be displayed vi Select View log to view uni...

Page 48: ...t install py u target iWARP driver uninstallation on Cluster nodes i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Uninstall iWAR...

Page 49: ...se to uninstall drivers software individually Provided here are steps to uninstall few of them For the complete list view help by running make help Change your current working directory to Chelsio Uni...

Page 50: ...nd Chelsio drivers iWARP driver uninstallation on Cluster nodes i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x OS arch ii Uninstall...

Page 51: ...Chapter II Network NIC TOE Chelsio Unified Wire for Linux 51 II Network NIC TOE...

Page 52: ...a flow processor implementation and a host of features designed for high throughput and low latency in demanding conditions and networking environments TCP offload is fully implemented in the hardware...

Page 53: ...g versions RHEL 7 5 3 10 0 862 el7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 e...

Page 54: ...rectory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x To build and install NIC only driver without offload support root host make nic_install To build and install drivers...

Page 55: ...ll offload support run the following command root host modprobe cxgb4 3 2 Loading in TOE mode with full offload support To enable full offload support run the following command root host modprobe t4_t...

Page 56: ...mber of connections offloaded run the following command root host cat sys kernel debug cxgb4 bus id tids Where TID is the number of offload connections STID is the number of offload servers T6 25G SO...

Page 57: ...Protocol PTP standard defines a protocol for precise synchronization of clock between master and slave devices in a local area network It can provide timing accuracies in nanosecond units The protocol...

Page 58: ...robe cxgb4 ii Assign IP addresses and ensure that master and slave nodes are connected iii Start the ptp4l tool on master using the Chelsio interface root host ptp4l i interface H m iv Start the ptp4l...

Page 59: ...ilable logical networks from 4096 to 16 million thereby addressing the scalability and network segmentation issues associated with large cloud computing deployments Chelsio s Terminator based adapters...

Page 60: ...to the bridge interface root host brctl addif bridge_interface vxlan_interface vi Tx UDP Tunnel Segmentation Offload will be enabled by default on loading the network driver To see the current setting...

Page 61: ...ame vhost queues 8 ii Map the Virtual CPUs of the VM to physical CPUs which will be free Example On a machine with 16 cores VM Virtual CPUs were pinned to physical cores 8 15 leaving cores 0 7 to be u...

Page 62: ...tcp_timestamps 0 root host sysctl w net core netdev_max_backlog 250000 root host sysctl w net core rmem_max 4194304 root host sysctl w net core wmem_max 4194304 root host sysctl w net core rmem_defau...

Page 63: ...t ipv4 tcp_wmem 4096 1048576 4194304 NIC TOE Latency Enable BUSY_POLL feature root host sysctl w net core busy_poll 50 root host sysctl w net core busy_read 50 Receiver Side Scaling RSS Receiver Side...

Page 64: ...ple by using the iperf tool First set up a server on the receiver host root receiver_host iperf s Then on the sender host send data to the server using the iperf client mode To emulate a moderate traf...

Page 65: ...laced into the host memory through DMA operations before an interrupt is thrown reducing the CPU load and thus improving latency It can be changed using the following command root host ethtool C ethX...

Page 66: ...that if your Linux system has IP forwarding enabled i e acting as a bridge or router the LRO needs to be disabled This is due to a known kernel issue Chelsio s card supports both hardware assisted GRO...

Page 67: ...played using the following command root host ethtool S eth6 GROPackets 0 GROMerged 897723 GROPackets is the number of held packets Those are candidate packets held by the kernel to be processed indivi...

Page 68: ...root host modprobe t4_tom unsupported_allow_unload 1 ii Stop all the offloaded traffic servers and connections Check for the reference count root host cat sys module t4_tom refcnt If the reference cou...

Page 69: ...Chapter II Network NIC TOE Chelsio Unified Wire for Linux 69 iii Unload the driver root host rmmod t4_tom root host rmmod toecore root host rmmod cxgb4...

Page 70: ...Chapter III Virtual Function Network vNIC Chelsio Unified Wire for Linux 70 III Virtual Function Network vNIC...

Page 71: ...erver portability cloud computing Virtual Desktop Infrastructure VDI etc Chelsio s Unified Wire family of adapters deliver increased bandwidth lower latency and lower power with virtualization feature...

Page 72: ...ng versions RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 gener...

Page 73: ...2 1 Pre requisites Please make sure that the following requirements are met before installation PCI Express Slot should be ARI capable SR IOV should be enabled in the machine Intel Virtualization Tech...

Page 74: ...om PFs 0 3 of the Chelsio adapter A maximum of 64 virtual functions can be instantiated with 16 virtual functions per physical function Example Instantiating 16 VFs on PF3 of Chelsio adapter Unload th...

Page 75: ...k vNIC Chelsio Unified Wire for Linux 75 The vNIC driver must be loaded on the Guest OS by the root user Any attempt to load the driver as a regular user will fail To load the driver run the following...

Page 76: ...on which rate limiting is applied Value 0 15 ii Run traffic over the VF and the throughput should be rate limited as per the values set in the previous step Example i 4 VFs are instantiated on PF0 roo...

Page 77: ...obe bonding mode bonding mode optional paramters iii Bring up the bonded interface and enslave the VF interfaces to the bond root host ifconfig bond0 up root host ifenslave bond0 ethX ethY iv Assign I...

Page 78: ...0 up root host ifenslave bond0 enp8s1 enp8s1f5d1 root host ifconfig bond0 10 1 1 223 24 The traffic will run over the bond interface in Active Backup mode If the link fails on enp8s1 the traffic will...

Page 79: ...iii Instantiate virtual functions root host modprobe cxgb4 root host echo n sys class net ethX device driver bus_id sriov_numvfs 124 virtual functions can be instantiated on T5 adapter with 31 virtua...

Page 80: ...3 NOARP mtu 0 qdisc noop state DOWN mode DEFAULT qlen 1 link none vf 0 MAC 06 44 3c b1 80 30 link state auto vii To set a VLAN ID on Virtual Function use the following syntax root host ip link set mg...

Page 81: ...or Linux 81 Software Driver Unloading 5 1 Unloading the Driver The vNIC driver must be unloaded on the Guest OS by the root user Any attempt to unload the driver as a regular user will fail To unload...

Page 82: ...Chapter IV iWARP RDMA Chelsio Unified Wire for Linux 82 IV iWARP RDMA...

Page 83: ...per connection or per class basis and to give higher priority to QPs that implement distributed locking mechanisms The iWARP operation also benefits from the high performance and low latency TCP imple...

Page 84: ...aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES...

Page 85: ...s to be installed on all the members If you want to install OFED with NFS RDMA support refer to Setting up NFS RDMA section rdma core devel package should be installed on RHEL 7 4 RHEL 7 5 SLES15 and...

Page 86: ...mpt to load the driver as a regular user will fail To load the iWARP driver we need to load the NIC driver and core RDMA drivers first Run the following commands root host modprobe cxgb4 root host mod...

Page 87: ...command on the client machine root host rping c Vv C10 a server_ip_addr p 9999 You should see ping data like this on the client ping data rdma ping 0 ABCDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqr pi...

Page 88: ...to accept default setup and empty password phrase iv Create authorization file root host cd ssh root host cat pub authorized_keys root host chmod 600 authorized_keys v Copy directory ssh to all system...

Page 89: ...following line adds a DAT version 2 0 device named chelsio2 for that interface chelsio2 u2 0 nonthreadsafe default libdaplofa so 2 dapl 2 0 eth2 0 Open MPI Installation and Setup Open MPI iWARP suppor...

Page 90: ...above step will install openMPI in usr mpi gcc openmpi x y z v Next create a shell script mpivars csh with the following entry path if echo path grep usr mpi gcc openmpi x y z bin then set path usr mp...

Page 91: ...RARY_PATH usr mpi gcc openmpi x y z lib64 LD_LIBRARY_PATH LD_LIBRARY_PATH export LD_LIBRARY_PATH fi MPI_ROOT MPI_ROOT usr mpi gcc openmpi x y z export MPI_ROOT vii Next copy the two files created in s...

Page 92: ...able rdma cm libdir usr mpi gcc mvapich2 x y lib64 root host make root host make install The above step will install MVAPICH2 in usr mpi gcc mvapich2 x y iv Next create a shell script mpivars csh with...

Page 93: ...mvapich2 x y lib64 LD_LIBRARY_PATH LD_LIBRARY_PATH export LD_LIBRARY_PATH fi MPI_ROOT MPI_ROOT usr mpi gcc mvapich2 x y export MPI_ROOT vi Next copy the two files created in steps iv and v to usr mpi...

Page 94: ...u want to build the benchmarks tool against For example in case of openMPI 1 6 4 set the variable as MPI_HOME usr mpi gcc openmpi 1 6 4 iv Next build and install the benchmarks using root host gmake f...

Page 95: ...default is 1000 or 1000ms Increase it to 30000 with this parameter mca btl_openib_connect_rdmacm_resolve_timeout 30000 Run MVAPICH2 application as mpirun_rsh ssh np 8 hostfile mpd hosts MVAPICH2_HOME...

Page 96: ...quash Note that for each directory you export you should have DIFFERENT fsid s ii Load the iwarp modules and make sure peer2peer is set to 1 iii Load xprtrdma and svcrdma modules as root host modprobe...

Page 97: ...hich directories will be exported ii Load the xprtrdma module root host modprobe xprtrdma iii Run the showmount command to show all directories from server as root host showmount e server chelsio ip i...

Page 98: ...Chapter IV iWARP RDMA Chelsio Unified Wire for Linux 98 Software Driver Unloading To unload the iWARP driver run the following command root host rmmod iw_cxgb4...

Page 99: ...Chapter V iSER Chelsio Unified Wire for Linux 99 V iSER...

Page 100: ...P CR T540 CR T540 BT T520 CR T520 LL CR T520 BT Memory free 256 IPv4 128 IPv6 offload connections supported 1 2 Software Requirements Linux Requirements Currently the iSER driver is available for the...

Page 101: ...4 14 67 kernel with iSER components enabled use the following command root host make kernel_install CONFIG_ISCSI_TARGET m CONFIG_INFINIBAND_ISER m CONFIG_INFINIBAND_ISERT m iii Boot into the new kern...

Page 102: ...der version exists v2 7 10 provided in the package will be installed Uninstall any OFED present in the machine rdma core devel package should be installed on RHEL 7 4 RHEL 7 5 SLES15 and SLES 12 SP3 s...

Page 103: ...oot host rmmod iw_cxgb4 ii Load the following modules root host modprobe iw_cxgb4 mpa_rev 2 root host modprobe rdma_ucm iii Start the iWARP Port Mapper Daemon root host iwpmd iv Bring up the Chelsio i...

Page 104: ...lun0 target tpg1 set attribute authentication 0 demo_mode_write_protect 0 generate_node_acls 1 cache_dynamic_acls 1 root host targetcli saveconfig ii Discover LIO target using OpeniSCSI initiator root...

Page 105: ...nections are supported on T6 25G SO adapters The following image shows the HMA reserved memory The following image shows the number of offloaded iSER connections 5 2 Performance Tuning i Apply the per...

Page 106: ...or Linux 106 Software Driver Unloading To unload iSER driver On target run the following commands root host rmmod ib_isert root host rmmod iw_cxgb4 On initiator run the following commands root host rm...

Page 107: ...Chapter VI WD UDP Chelsio Unified Wire for Linux 107 VI WD UDP...

Page 108: ...e bypass of the kernel which results in an ultra low latency Ethernet solution for high frequency trading and other delay sensitive applications 1 1 Hardware Requirements Supported Adapters The follow...

Page 109: ...12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Minimum 4 9...

Page 110: ...Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install WD UDP driver and libraries root host make wdudp_install For...

Page 111: ...mpt to load the driver as a regular user will fail RDMA core modules from the OFED package should be loaded before proceeding To load the WD UDP driver use the following commands which will automatica...

Page 112: ...r High Capacity WD tuning option Application Requirements Certain application behavior is not supported by libcxb4_sock in this release If your application does any of the following it will not work w...

Page 113: ...bcxgb4_sock using one of the methods mentioned below when starting your application Preloading using wdload script root host PROT UDP wdload pathto your_application The above command will generate an...

Page 114: ...our Terminator interface is eth2 To accelerate all applications that preload libcxgb4_sock using eth2 you only need one entry in etc libcxgb4_sock conf endpoint interface eth2 port 0 If you have eth2...

Page 115: ...cxgb4 log ii Restart the service root host etc init d syslog restart iii Finally preload libcxgb4_sock_debug using the command mentioned below when starting your application root host LD_PRELOAD libc...

Page 116: ...64 Scope Link root r9 root r10 ifconfig eth1 grep inet inet addr 192 168 2 112 Bcast 192 168 2 255 Mask 255 255 255 0 inet6 addr fe80 7 4300 104 456a 64 Scope Link root r10 For this benchmark we need...

Page 117: ...the same test with libcxgb4_sock root r9 LD_PRELOAD libcxgb4_sock so usr local src hpcbench udp udptest r 5 a h 192 168 1 112 p 9001 Determining if the application is being offloaded To see if the app...

Page 118: ...Chapter VI WD UDP Chelsio Unified Wire for Linux 118 Software Driver Unloading To unload the WD UDP driver run the following command root host rmmod iw_cxgb4...

Page 119: ...Chapter VII WD TOE Chelsio Unified Wire for Linux 119 VII WD TOE...

Page 120: ...1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the WD TOE driver T62100 CR T62100 LP CR T6425 CR T6225 CR T6225 LL CR...

Page 121: ...king Wire Direct Latency or High Capacity WD configuration tuning Options With any other configuration tuning option the installation needs to be customized 2 2 Installation i Change your current work...

Page 122: ...xgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail Run the following commands to load the driver root host modprobe cxg...

Page 123: ...Netperf application with WD TOE Start netserver at the PEER root host PROT TCP wdload netserver D 4 On the Test machine run netperf application root host PROT TCP wdload netperf H PEER_IP t TCP_RR l...

Page 124: ...Chapter VII WD TOE Chelsio Unified Wire for Linux 124 Software Driver Unloading Unload NIC TOE drivers as mentioned in Software Driver Unloading section of Network NIC TOE chapter...

Page 125: ...Chapter VIII NVMe oF Chelsio Unified Wire for Linux 125 VIII NVMe oF...

Page 126: ...s to achieve fast storage response times irrespective of whether the NVMe SSDs are attached locally or accessed remotely across enterprise or datacenter networks 1 1 Hardware Requirements Supported Ad...

Page 127: ...ault Ubuntu 18 04 1 4 15 0 29 generic Kernel org linux 4 14 67 kernel compiled on RHEL 7 3 and above Kernel org linux 4 9 Minimum 4 9 kernel version supported is 4 9 13 kernel compiled on RHEL 7 3 and...

Page 128: ...s enabled use the following command root host make kernel_install CONFIG_BLK_DEV_NVME m CONFIG_NVME_RDMA m CONFIG_NVME_TARGET m CONFIG_NVME_TARGET_RDMA m CONFIG_NVME_RDMA m CONFIG_BLK_DEV_NULL_BLK m C...

Page 129: ...or above should be installed If this version is not already present in the system or if an older version exists v2 7 10 provided in the package will be installed rdma core devel package should be ins...

Page 130: ...dprobe rdma_ucm ii Bring up the Chelsio interface s root host ifconfig ethX x x x x up iii Mount configfs by running the below command root host mount t configfs none sys kernel config iv On target ru...

Page 131: ...paces cd 1 subsystems n namespaces 1 set device path dev ram1 subsystems n namespaces 1 cd subsystems nvme ram0 set attr allow_any_host 1 subsystems nvme ram0 cd namespaces 1 subsystems n namespaces 1...

Page 132: ...connect all t rdma a target_ip s 4420 iii List the connected targets root host nvme list iv Format and mount the NVMe disks shown with the above command v Disconnect from the target and unmount the di...

Page 133: ...pter before proceeding i Run the performance tuning script to map iWARP queues to different CPUs root host t4_perftune sh n Q rdma N ii Set the inline data size to 8192 before enabling the NVMe port r...

Page 134: ...g Follow the steps mentioned below to unload the drivers On target run the following commands root host rmmod nvmet rdma root host rmmod nvmet root host rmmod iw_cxgb4 On initiator run the following c...

Page 135: ...Chapter IX LIO iSCSI Target Offload Chelsio Unified Wire for Linux 135 IX LIO iSCSI Target Offload...

Page 136: ...culations Offload Speeds at 10 25 40 100Gb Supports Direct Data Placement DDP Supports iSCSI Segmentation Offload and iSCSI PDU recovery 1 1 Hardware Requirements Supported Adapters The following are...

Page 137: ...L 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default Ubuntu 18 04 1 4 15 0 29 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Mi...

Page 138: ...iver installation as directed in the Software Driver Installation section SLES15 SLES 12 SP3 No extra kernel configuration required Kernel org linux 4 14 67 i Change your current working directory to...

Page 139: ...e installation menu root host make menuconfig iv Select Device Drivers Generic Target Core Mod TCM and ConfigFS Infrastructure v Enable Linux iSCSI org iSCSI Target Mode Stack vi Select Save vii Exit...

Page 140: ...e menuconfig iv Select Device Drivers Generic Target Core Mod TCM and ConfigFS Infrastructure v Enable Linux iSCSI org iSCSI Target Mode Stack as a Module if not already enabled vi Select Save vii Exi...

Page 141: ...ckage will be installed along with the kernel 3 2 Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install LIO driver...

Page 142: ...kernel_object_dir Example For RHEL 7 5 root host install py ksrc usr src linux 3 10 0 862 el7 kobj lib modules 3 10 0 862 el7 x86_64 build GUI mode root host install py set kpath Provide the paths fo...

Page 143: ...must be loaded by the root user Any attempt to load the driver as a regular user will fail i Load network driver cxgb4 root host modprobe cxgb4 ii Bring up the interface root host ifconfig ethX IP add...

Page 144: ...www linux iscsi org Doc LIO Admin Manual pdf to do so Sample Configuration Here is a sample iSCSI configuration listing a target configured with 1 RAM disk LUN and ACL not configured 5 2 Offloading LI...

Page 145: ...y load network driver with offload support TOE root host modprobe t4_tom ii Create a new policy file root host cat new_policy_file iii Add the following lines to offload all traffic except LIO iSCSI l...

Page 146: ...Wire chapter before proceeding ii Run the performance tuning script to map LIO Target queues to different CPUs root host t4_perftune sh Q iSCSIT n iii For maximum performance it is recommended to use...

Page 147: ...the steps mentioned below i Log out from the initiator ii Run the following command root host echo 0 sys kernel config target iscsi target_iqn tpgt_1 np target_ip 3260 cxg bit Execute the above comma...

Page 148: ...Chapter X iSCSI PDU Offload Target Chelsio Unified Wire for Linux 148 X iSCSI PDU Offload Target...

Page 149: ...are The performance of this iSCSI stack is outstanding and when used with Chelsio s hardware it is enhanced further Because of the tight integration with Chelsio s TOE adapters this software has a dis...

Page 150: ...upport Persistent Reservation Dynamic LUN Resizing iSCSI Target Redirection Multiple Target device types Block Virtual Block LVM Software RAID EVMS etc Built in RAM Disk Built in zero copy RAM Disk Su...

Page 151: ...ardware storage device is required This device can be any of the device types that are supported block virtual block RAM disk Multiple storage devices are allowed by configuring the devices to one tar...

Page 152: ...versions have not been tested and are not guaranteed to work Requirements for Installing the iSCSI Software When installing the iSCSI software it is required that the system have Linux kernel source o...

Page 153: ...allation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install iSCSI target driver firmware and utilities root host make iscsi_...

Page 154: ...rnel The software is dependent on the underlying NIC adapter driver and thus the limitation on what version of the Linux kernel it can run on is mostly dependent on the NIC driver s limitations The iS...

Page 155: ...ce loads the kernel module and starts all the targets configured by default It can also be used to stop the targets and restart reload configuration Note i While using rpm tar ball for installation a...

Page 156: ...tion settings Retrieve active sessions information of an iSCSI Target Manually flush data to the iSCSI Target disks Reload the iSCSI configuration file Update the iSCSI configuration file Save the cur...

Page 157: ...Basic editing of the iSCSI configuration file Basic commands of the iSCSI control tool including how to start and stop a target A Sample iSCSI Configuration File The default Chelsio iSCSI configuratio...

Page 158: ...the Chelsio iSCSI software is done through iscsictl the command line interface control tool The following are the basic commands needed for effective control of the target Start Target To start all o...

Page 159: ...ngs execute iscsictl with G option root host iscsictl G iscsi_login_complete_time 300 View Help To print help to stdout execute iscsictl with h option root host iscsictl h 4 4 The iSCSI Configuration...

Page 160: ...ead of commas is to list their values as key value pairs repeatedly each on a new line until they are all listed There are three categories of key value parameter the first category belongs to the glo...

Page 161: ...llowed secret must be between 6 and 255 characters Commas are not allowed The target user id and secret are used by the initiator to authenticate the target while doing mutual chap NOTE The double quo...

Page 162: ...ult Value Multiple Values Description MaxConnections 1 to 65535 1 No Initiator and target negotiate the maximum number of connections requested acceptable InitialR2T Yes No Yes No To turn on or off th...

Page 163: ...er nor is it for authentication Target only MaxRecvDataSegmentLengt h 512 to 16777215 224 1 8192 No To declare the maximum data segment length in bytes it can receive in an iSCSI PDU OFMarker Yes No N...

Page 164: ...be less than 256 characters Commas are not allowed secret must be between 6 and 255 characters Commas are not allowed The initiator user id and secret are used by the target to authenticate the initi...

Page 165: ...There can be multiple redirection target portalgroups specified for a particular target portal group and the redirection will happen to these in a round robin manner ShadowMode Yes No No No To turn Sh...

Page 166: ...evice should be created as a RAM Disk size xMB is used with MEM to specify the RamDisk size If not specified the default RamDisk size is 16MB 16 Megabytes The minimum value of x is 1 1MB and the maxim...

Page 167: ...ore initiator names the name must be a fully qualified iSCSI initiator name sip Source IP address specifies one or more IP addresses the initiators are connecting from Dip Destination IP address speci...

Page 168: ...get irrelevant entity block can be removed or commented Chelsio iSCSI Global Settings global iscsi_login_complete_time 300 iscsi_auth_order CHAP DISC_AuthMethod None DISC_Auth_CHAP_Policy Oneway DISC_...

Page 169: ...gth 65535 MaxBurstLength 262144 Local block devices being served up lun 0 is pointed to dev sda lun 1 is pointed to dev sdb TargetDevice dev sda ID aabbccddeeffgghh WWN aaabbbcccdddeeef TargetDevice d...

Page 170: ...AP CHAP is a protocol that is used to authenticate the peer of a connection and uses the notion of a challenge and response i e the peer is challenged to prove its identity The Chelsio iSCSI software...

Page 171: ...cret are configured for use in the case of mutual CHAP AuthMethod and Auth_CHAP_Policy Keys By setting the iSCSI keys AuthMethod and Auth_CHAP_Policy a user can choose whether to enforce CHAP and if m...

Page 172: ...way authentication DISC_Auth_CHAP_Target target CHAP user id and secret DISC_Auth_CHAP_Initiator initiator CHAP user id and secret Sample Chelsio iSCSI Global Settings global DISC_AuthMethod CHAP DIS...

Page 173: ...via the target portal 102 60 60 25 is allowed full read write access to this target ACL dip 102 60 60 25 initiator iqn 2005 09 com chelsio san2 from 102 50 50 22 and connected via the target portal 1...

Page 174: ...tion is useful for measuring network performance The details of the parameters for the key TargetDevice are found in the table of Chelsio Entity Settings section earlier in this document RAM Disk Deta...

Page 175: ...th to the storage device FILE SYNC Where path Is the path to the actual storage device such as dev sdb for a block device or dev md0 for a software RAID The path must exist in the system SYNC When spe...

Page 176: ...umeric string The user should make sure the three values listed above are the same for the target LUNs involved in the multipath 4 8 Target Redirection Support An iSCSI Target can redirect an initiato...

Page 177: ...ortalGroup 1 10 193 184 81 3260 2 PortalGroup 2 10 193 184 85 3261 the PortalGroup 2 IS present on the same system ShadowMode No snip Redirecting to Multiple Portal Groups The Chelsio iSCSI Target Red...

Page 178: ...client This is a simple tool that starts the iSNS client with a client and server parameter iscsictl options Options Mandatory Parameters Optional Parameters Description h Display the help messages v...

Page 179: ...on s will be displayed If target ALL is specified or no parameters are specified the output will be abbreviated Specify specific targets to get detailed configuration data If the target name option is...

Page 180: ...reject only the invalid parameters but will continue on and complete all other valid parameters if any others are specified s target name Stop the specified active iSCSI targets target name parameter...

Page 181: ...pecified targets are not currently active they will be started For Rules 2 3 please note the differences they are not the same The global settings are also reloaded from the configuration file with th...

Page 182: ...05 id isnscln2 query 30 In the first example the minimum command set is given where the IP address of the iSNS server is specified In the second example a fully qualified command is specified by also...

Page 183: ...arted its settings can be modified via reloading of the configuration file i e iscsictl S The following parameters cannot be changed once the target is up and running otherwise the target reload would...

Page 184: ...ommand line iscsictl G The finer points of some of these parameters are described in detail here iscsi_login_complete_time Options An integer value between 0 and 3600 seconds Default value is 300 seco...

Page 185: ...sed to verify the initiator name received and identify if the initiator is running Chelsio drivers if the initiator name contains the same substring as iscsi_chelsio_ini_idstr it is assumed the initia...

Page 186: ...Chapter X iSCSI PDU Offload Target Chelsio Unified Wire for Linux 186 Software Driver Unloading Use the following command to unload the module root host rmmod chiscsi_t4...

Page 187: ...Chapter XI iSCSI PDU Offload Initiator Chelsio Unified Wire for Linux 187 XI iSCSI PDU Offload Initiator...

Page 188: ...Data Out PDUs PDU Transmit and Recovery On transmit side Chelsio hardware accepts the complete PDU header data from the host driver computes and inserts the digests decomposes the PDU into multiple TC...

Page 189: ...lable for the following versions RHEL 7 5 3 10 0 862 el7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8...

Page 190: ...b4i runs on top of NIC module cxgb4 and open iscsi 2 0 872 873 874 only on a Chelsio card openssl devel package should be installed 2 2 Installation i Change your current working directory to Chelsio...

Page 191: ...ail Run the following command to load the driver root host modprobe cxgb4i If loading of cxgb4i displays unkown symbols found error in dmesg follow the steps mentioned below i Kill iSCSI daemon iscsid...

Page 192: ...gb4i in the following format iface iscsi_ifacename iface file name iface hwaddress MAC address iface transport_name cxgb4i iface net_ifacename ethX iface ipaddress iscsi ip address Here iface iscsi_if...

Page 193: ...st p 102 2 2 155 3260 I cxgb4i 00 07 43 04 5b da iii Logging into an iSCSI Target Log into an iSCSI target using the following format root host iscsiadm m node T iqn name of target p target ip address...

Page 194: ...dm m node T iqn name of target p target ip address target port no I cxgb4i iface file name u Example root host iscsiadm m node T iqn 2004 05 com chelsio target1 p 102 2 2 155 3260 1 I cxgb4i 00 07 43...

Page 195: ...ctions 4 3 Auto login from cxgb4i initiator at OS bootup For iSCSI auto login via cxgb4i to work on OS startup please add the following line to start in etc rc d init d iscsid file on RHEL modprobe q...

Page 196: ...nfiguration file and reboot the machine scsi_mod use_blk_mq 1 ii Run the performance tuning script to map iSCSI Initiator queues to different CPUs root host t4_perftune sh Q iSCSI n iii Load initiator...

Page 197: ...Chapter XI iSCSI PDU Offload Initiator Chelsio Unified Wire for Linux 197 Software Driver Unloading To unload the driver execute the following commands root host rmmod cxgb4i root host rmmod libcxgbi...

Page 198: ...Chapter XII Crypto Offload Chelsio Unified Wire for Linux 198 XII Crypto Offload...

Page 199: ...Hardware Requirements Supported adapters The following are the currently shipping Chelsio adapters that are compatible with Chelsio s Crypto Offload driver T62100 CR T62100 LP CR T6425 CR T6225 CR T6...

Page 200: ...CONFIG_LSM_MMAP_MIN_ADDR 65536 CONFIG_SECURITY_SELINUX y CONFIG_SECURITY_SELINUX_BOOTPARAM y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE 1 CONFIG_SECURITY_SELINUX_DISABLE y CONFIG_SECURITY_SELINUX_DEVELOP...

Page 201: ...TO_GHASH y CONFIG_CRYPTO_MD4 m CONFIG_CRYPTO_MD5 y CONFIG_CRYPTO_SHA1 y CONFIG_CRYPTO_SHA256 y CONFIG_CRYPTO_SHA512 y CONFIG_CRYPTO_AES y CONFIG_CRYPTO_AES_X86_64 y CONFIG_CRYPTO_DEFLATE y CONFIG_CRYP...

Page 202: ...kernel patch v Copy the patched kernel source to usr src kernels root host cp r root rpmbuild BUILD kernel 3 10 0 514 el7 linux 3 10 0 514 el7 usr src kernels vi Compile and install the kernel root ho...

Page 203: ...that SELinux and firewall are disabled 3 2 Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Crypto driver and...

Page 204: ...ost modprobe cxgb4 root host modprobe chcr ii Bring up the Chelsio network interface root host ifconfig ethX up Where ethX is the Chelsio interface 4 2 Inline i To load the Crypto Offload driver in In...

Page 205: ...chcr The following applications can be offloaded by Chelsio Co processor Apache Dmcrypt SFTP OpenVPN Strongswan To enable Digests in RHEL 7 3 x86_64 base 3 10 0 514 el7 kernel uncomment the DIGESTS li...

Page 206: ...Follow the steps mentioned below i Create a new policy file and add the following line for each TCP port to be TLS offloaded src or dst port tcp_port offload tls mss 32 bind random nagle all offload...

Page 207: ...installed in usr chssl bin OpenSSL tool i Start TLS offload Server root host cd usr chssl bin root host openssl s_server key key_file cert cert_file accept 443 cipher AES128 GCM SHA256 WWW Example ii...

Page 208: ...r local nginx nginx conf path usr local nginx nginx conf pid path usr local nginx nginx pid with http_ssl_module with ld opt L usr chssl lib L usr local lib with openssl ChelsioUwire x x x x src chope...

Page 209: ...3 Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding Inline TLS i Run the performance tuning script to map TLS...

Page 210: ...nloading To unload Crypto Offload driver in Co processor mode run the following command root host rmmod chcr To unload Crypto Offload driver in Inline mode unload the network driver in TOE mode See So...

Page 211: ...Chapter XIII Data Center Bridging DCB Chelsio Unified Wire for Linux 211 XIII Data Center Bridging DCB...

Page 212: ...working resources reduced heat and noise and less power consumption Administration is simplified since the specifications enable transport of storage and networking traffic over a single unified Ether...

Page 213: ...Wire for Linux 213 Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Minimum 4 9 kernel version supported is 4 9 13 Other kernel versions...

Page 214: ...ware Driver Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Build and install all drivers with DCB support root host...

Page 215: ...itch ports need to be enabled with DCBX configuration Class mapping ETS and PFC Upon loading the network storage driver and interface bringup firmware completes DCBX negotiation with the switch root h...

Page 216: ...E Full Offload Initiator uses the DCBX negotiated parameters ETS PFC etc without any further configuration The network drivers cxgb4 t4_tom and iSCSI drivers cxgb4i chiscsi need further VLAN configura...

Page 217: ...iorities NIC uses priority 0 and FcoE uses priority 3 switch config class map type qos class nic switch config cmap qos match cos 0 switch config cmap qos class map type qos class fcoe switch config c...

Page 218: ...cent 0 vii Configure network qos policy maps and set up the PFC for no drop traffic class switch config policy map type network qos policy test switch config pmap nq class type network qos class nic s...

Page 219: ...switch config vsan database switch config vsan db vsan 2 switch config vsan db vsan 2 interface vfc 13 switch config vsan db exit iii Assign VLAN ID to the Ethernet port on which FCoE service was enab...

Page 220: ...r and the target on that particular VSAN although FLOGI d into the switch will not talk to each other To enable it execute the below command switch config zone default zone permit vsan 2 4 2 Configuri...

Page 221: ...ule 2 proto fip encap ethv2 switch config vlan classifier group 1 add rule 1 switch config vlan classifier group 1 add rule 2 switch config interface vlan 1002 switch conf if vl 1002 fcf forward switc...

Page 222: ...unning NIC and iSCSI traffic and configure corresponding VLAN priority Example Switch is configured with a VLAN priority of 2 and 5 for NIC and iSCSI class of traffic respectively NIC traffic is run o...

Page 223: ...Chapter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 223 XIV FCoE Full Offload Initiator...

Page 224: ...FC0 and FC1 layers of the Fibre Channel stack with Ethernet By retaining the native Fibre Channel constructs FCoE will integrate with existing Fibre Channel networks and management software 1 1 Hardwa...

Page 225: ...tallation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install FCoE full offload initiator driver root host make fcoe_full_off...

Page 226: ...i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver execute the following roo...

Page 227: ...following command to see if the FIP has gone through and a VN_Port MAC address has been assigned Verify if all the FCoE ports are online ready and a successful FIP has taken place using the following...

Page 228: ...w mentioned steps i Determine the WWPN of the initiator local port under sysfs The hosts under fc_host depends on the number of ports on the adapter used root host cat sys class fc_host hostX port_nam...

Page 229: ...t host cat sys kernel debug csiostor pci_id lnodes After finding out the WWPN of the local node to verify the list of discovered targets use the following command root host cat sys kernel debug csiost...

Page 230: ...Chapter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 230...

Page 231: ...apter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 231 4 3 Formatting the LUNs and Mounting the Filesystem Use lsscsi g to list the LUNs discovered by the initiator root host lsscsi...

Page 232: ...covered by the Chelsio FCoE initiators can be accessed via easily identifiable udev path device files like root host ls dev disk by path pci 0000 04 00 0 csio fcoe local_wwpn remote_wwpn lun_wwn 4 4 C...

Page 233: ...FCoE Full Offload Initiator Chelsio Unified Wire for Linux 233 4 5 Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command root host mount...

Page 234: ...helsio Unified Wire for Linux 234 Software Driver Unloading To unload the driver root host modprobe r csiostor If multipath services are running unload of FCoE driver is not possible Stop the multipat...

Page 235: ...Chapter XV Offload Bonding Chelsio Unified Wire for Linux 235 XV Offload Bonding...

Page 236: ...can be fully offloaded to the adapter thus freeing the CPU from TCP IP overhead 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatib...

Page 237: ...693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 ge...

Page 238: ...r Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Chelsio Offload bonding driver root host make bonding_inst...

Page 239: ...chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver with offload support run the following comm...

Page 240: ...the following method i Load the network driver with TOE support root host modprobe t4_tom ii Create a bonded interface root host modprobe bonding mode 1 miimon 100 iii Bring up the bonded interface a...

Page 241: ...Chapter XV Offload Bonding Chelsio Unified Wire for Linux 241 Software Driver Unloading To unload the driver run the following command root host rmmod bonding...

Page 242: ...Chapter XVI Offload Multi Adapter Failover MAFO Chelsio Unified Wire for Linux 242 XVI Offload Multi Adapter Failover MAFO...

Page 243: ...fic running over the bonded interface can be fully offloaded to the adapter thus freeing the CPU from TCP IP overhead 1 1 Hardware Requirements Supported Adapters The following are the currently shipp...

Page 244: ...5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic...

Page 245: ...x 245 Software Driver Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install MAFO feature root host make bonding_in...

Page 246: ...iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver with offload support run the foll...

Page 247: ...modprobe bonding mode 1 miimon 100 iii Bring up the bonded interface and enslave the interfaces to the bond root host ifconfig bond0 up root host ifenslave bond0 ethX ethY iv Assign IPv4 IPv6 address...

Page 248: ...Chapter XVI Offload Multi Adapter Failover MAFO Chelsio Unified Wire for Linux 248 Software Driver Unloading To unload the driver run the following command root host rmmod bonding...

Page 249: ...Chapter XVII UDP Segmentation Offload and Pacing Chelsio Unified Wire for Linux 249 XVII UDP Segmentation Offload and Pacing...

Page 250: ...ividually through the network stack and schedule individual frame transmission resulting in millions of system calls and packet traversals through all protocol layers in the operating system to the ne...

Page 251: ...0 CR T520 LL CR T520 BT T420 CR T440 CR T422 CR T404 BT T420 BCH T440 LP CR T420 BT T420 LL CR T420 CX 1 2 Software Requirements Linux Requirements Currently the UDP Segmentation Offload and Pacing dr...

Page 252: ...ld and install UDP Offload drivers which support large number of offload connections approx 10K i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUw...

Page 253: ...oot user Any attempt to load the driver as a regular user will fail Run the following commands to load the driver root host modprobe cxgb4 root host modprobe t4_tom Though normally associated with the...

Page 254: ...e data stream to This value needs to be in the range of 0 to 14 for T4 series of adapters and a range of 0 to 15 for T5 T6 series of adapters The application will function as per the parameters set fo...

Page 255: ...teps mentioned above i In step ii declare and initialize a new variable in the application int rtp_header_size 16 Here rtp_header_size is the RTP header size in bytes that the application sends ii In...

Page 256: ...t payload size in bytes it should be equal to the value set in the SO_FRAMESIZE socket option in the application in section 4 1 Example The user wants to transfer UDP data on port 0 of the adapter usi...

Page 257: ...entation Offload and Pacing Chelsio Unified Wire for Linux 257 Software Driver Unloading Reboot the system to unload the driver To unload without rebooting refer Unloading the TOE driver section of Ne...

Page 258: ...Chapter XVIII Offload IPv6 Chelsio Unified Wire for Linux 258 XVIII Offload IPv6...

Page 259: ...ort to fully offload IPv6 traffic to the Unified Wire adapter 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with Chelsio Of...

Page 260: ...693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 ge...

Page 261: ...nabled in your system enabled by default Unified Wire must be installed with IPv6 support as explained in the Unified Wire chapter 2 2 Installation i Change your current working directory to Chelsio U...

Page 262: ...xgb After installing Unified Wire package and rebooting the host load the NIC cxgb4 and TOE t4_tom drivers The drivers must be loaded by the root user Any attempt to load the drivers as a regular user...

Page 263: ...d capable drivers root host modprobe t4_tom ii Assign IPv6 address and bring up the interface root host ifconfig ethX IPv6 address up iii All the IPv6 traffic over the Chelsio interface will be offloa...

Page 264: ...oading 5 1 Unloading the NIC Driver To unload the NIC driver run the following command root host rmmod cxgb4 5 2 Unloading the TOE Driver Please reboot the system to unload the TOE driver To unload wi...

Page 265: ...Chapter XIX WD Sniffing and Tracing Chelsio Unified Wire for Linux 265 XIX WD Sniffing and Tracing...

Page 266: ...icast traffic and sending it directly to user space a Get a Queue raw QP idx b Program a filter to redirect specific traffic to the raw QP queue Tracer All tapped traffic is forwarded to user space an...

Page 267: ...Linux 267 Schematic diagram of sniffer and tracer 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the tools T62100 CR T...

Page 268: ...nd Tracing utility is available for the following version RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4...

Page 269: ...tallation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Sniffer tracing filtering tcpdump and iWARP driver root host ma...

Page 270: ...ne having a T5 T4 card The machines should be setup in the following manner Machine A Machine B 192 168 1 100 192 168 1 200 2 Procedure On the Device Under Test DUT start sniffer root host wd_sniffer...

Page 271: ...dure Run wd_tcpdump_trace i iface on the command prompt where iface is one of the interfaces whose traffic you want to trace In the above diagram its port 0 or port 1 root host wd_tcpdump_trace i ifac...

Page 272: ...Chapter XX Classification and Filtering Chelsio Unified Wire for Linux 272 XX Classification and Filtering...

Page 273: ...ccess Control List accept criteria to a particular receive queue Switch proxy ingress packets that meet ACL accept criteria to an output port with optional header rewrite Drop ingress packets that mee...

Page 274: ...7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el...

Page 275: ...will be from 0 to 63 and normal indices will be from 64 to 559 It is mandatory to add prio 1 option when creating high priority filter rules 2 1 Configuration Filter Modes The Classification and Filt...

Page 276: ...ntation mpshittype macmatch ethertype port fcoe 6 fragmentation mpshittype macmatch protocol tos port fcoe 7 fragmentation mpshittype macmatch protocol vlan fcoe 8 fragmentation mpshittype macmatch pr...

Page 277: ...ol vlan port 41 mpshittype macmatch protocol vnic_id port 42 mpshittype macmatch tos vlan port 43 mpshittype macmatch tos vnic_id port 44 mpshittype ethertype vlan port fcoe 45 mpshittype ethertype vn...

Page 278: ...ter rules using cxgbtool root host cxgbtool ethx filter index action pass drop switch prio 1 hitcnts 1 Where ethX Chelsio interface index positive integer set as filter id 0 495 for T5 adapters 0 559...

Page 279: ...on switch iport 0 eport 1 ivlan 3 The above filter rule will route all ingress packets that match VLAN id 3 from port 0 of Chelsio adapter to port 1 Remaining packets will be sent to the host prio opt...

Page 280: ...up root host ifconfig eth0 2 102 1 1 2 24 up root host ifconfig eth0 3 102 1 1 3 24 up root host ifconfig eth0 eth0 Link encap Ethernet HWaddr 00 07 43 04 7D 50 inet addr 102 1 1 1 Bcast 102 1 1 255 M...

Page 281: ...ROMISC MULTICAST MTU 1500 Metric 1 ii Setup a static OR default route towards T6 T5 router to reach 102 1 1 0 24 network root host route add net 102 1 1 0 24 gw 102 1 2 250 Follow these steps on machi...

Page 282: ...cxgbtool eth0 filter 101 lip 102 1 1 0 24 hitcnts 1 action switch eport 0 smac 00 07 43 04 96 40 dmac 00 07 43 04 7D 50 Where smac is the MAC address of eth0 interface on T6 T5 adapter machine and dm...

Page 283: ...1 1 1 root host arp s 102 1 1 1 00 07 43 04 7D 50 Follow these steps on machine with T6 T5 adapter i Update filterMode value with below combination in lib firmware cxgb4 t6 config txt to enable matchi...

Page 284: ...switch it out via eport 0 root host cxgbtool eth0 filter 101 macidx 7 action switch eport 0 hitcnts 1 2 7 Filtering VF traffic To filter VF traffic replace the default filterMode in the firmware conf...

Page 285: ...odprobe cxgb4 root host echo 4 sys class net ethX device driver bus_id sriov_numvfs ii 1 VM was brought up with VF2 cxgb4vf was loaded on the VM and the VF was brought up root host ifconfig enp8s2 enp...

Page 286: ...ure is configured by specifying the filter modes in the firmware configuration file located in lib firmware cxgb4 Adapter initialization will fail if filterMask contains a tuple which is not present i...

Page 287: ...rt fcoe 15 fragmentation macmatch protocol vnic_id port fcoe 16 fragmentation protocol tos vlan port fcoe 17 fragmentation protocol tos vnic_id port fcoe 18 mpshittype macmatch ethertype protocol port...

Page 288: ...lip destination_ip fport source_port lport destination_port proto protocol hitcnts 1 cap maskless Where ethX Chelsio interface index Filter index For LE TCAM filters filter index should be 0 495 for T...

Page 289: ...tioned in filterMask Otherwise filter rule will not honour iii Now to create a hash filter based on the filterMode and filterMask values selected above root host cxgbtool eth18 filter 100 action drop...

Page 290: ...frag 0 matchtype 0 iport 0 fcoe 0 hitcnts 1 cap maskless Hash Filter Index 196568 Examples drop action root host cxgbtool ethX filter 496 action drop lip 102 1 1 1 fip 102 1 1 2 lport 12865 fport 200...

Page 291: ...s parameter and corresponding filter rule index root host cxgbtool ethX filter index delete clear cap maskless 3 5 Filter Priority By default Hash DDR filter has priority over LE TCAM filter To overri...

Page 292: ...parameter enable_mirror can be used to enable mirroring of traffic running on physical ports The mirrored traffic will be received via Mirror PF VF on Mirror Receive queues which will then inject this...

Page 293: ...tered Traffic Mirroring Once mirroring is enabled all the traffic received on a physical port will be duplicated The following example explains the method to filter out the redundant traffic and recei...

Page 294: ...t cxgbtool ethX reg 0x09800 0x13 iv Setup a trace filter root host echo tx1 snaplen 40 sys kernel debug cxgb4 bus_id trace0 Here snaplen is the length in bytes to be captured The above step will trace...

Page 295: ...sys kernel debug cxgb4 bus id sge_qinfo file root host cxgbtool ethX reg 0x09808 Trace Queue0 RspQ ID iv Similarly for other direction and for multiple ports run the follow commands root host echo rx...

Page 296: ...new_destination_port Where ethX Chelsio interface source_ip port Source IP port of incoming packet destination_ip port Destination IP port of incoming packet new_source_ip port Source IP port to be t...

Page 297: ...Chapter XXI OVS Kernel Datapath Offload Chelsio Unified Wire for Linux 297 XXI OVS Kernel Datapath Offload...

Page 298: ...offloading OpenFlow and non OpenFlow network traffic simultaneously including tunnel handling e g VXLAN IPsec NAT IP stack ARP route lookup frag tracking fragment defragment and other kernel function...

Page 299: ...quirements The Chelsio OVS driver has been developed to run on the following 64 bit Linux platforms RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 Kernel org linux 4 9 Minimum 4 9 kernel version supp...

Page 300: ...4 packages should be installed For the complete list of software required visit http docs openvswitch org en latest intro install general 2 2 Installation i Change your current working directory to Ch...

Page 301: ...t L2 Ethernet Type L3 IP Protocol Type L3 IPv4 address L3 IPv6 address L3 IPv4 TOS L3 IP Fragmentation L4 Ports tcp udp src port dst port Tunnel Encapsulation VNI only on T6 Supported Actions The foll...

Page 302: ...adapters and t5 config txt for T5 adapters filterMode fragmentation ethertype protocol tos port filterMask fragmentation ethertype protocol tos port iii Load NIC cxgb4 driver with hash filter support...

Page 303: ...vSwitch manager_options bootstrap ca cert db Open_vSwitch SSL ca_cert pidfile detach log file root host ovs vsctl no wait init root host export DB_SOCK usr local var run openvswitch db sock root host...

Page 304: ...protocol tos port Wild card flow to drop incoming packets on first port root host ovs ofctl add flow br0 in_port 1 dl_type 0x800 action drop Wild card flow to switch ARP packets L2 EtherType 0x0806 o...

Page 305: ...flow to switch 4 tuple IPv4 traffic with TOS 0xE0 root host ovs ofctl add flow br0 in_port 1 dl_type 0x800 nw_proto 6 nw_src 10 1 1 66 tp_src 11000 nw_dst 10 1 1 58 tp_dst 21000 nw_tos 0xE0 action ou...

Page 306: ...he mod_vlan_vid tag will be added with a priority of 0 Modify VLAN priority and switch traffic root host ovs ofctl O OpenFlow11 add flow br0 in_port 1 dl_type 0x800 vlan_pcp 4 action mod_vlan_pcp 3 ou...

Page 307: ...cal var run openvswitch db sock remote db Open_vSwitch Open_vSwitch manager_options bootstrap ca cert db Open_vSwitch SSL ca_cert pidfile detach log file root host ovs vsctl no wait init root host exp...

Page 308: ...rifying OVS Flow Dump OVS flow dump can be verified using root host ovs ofctl dump flows br0 Run traffic between hosts which matches the flow and verify if the n_packet counter is incrementing 3 4 Set...

Page 309: ...pen a web browser and enter the address http localhost 8080 ix Login with admin keyword for both username and password x On the OVS machine add the bridge to the controller and disable in band root ho...

Page 310: ...th Offload Chelsio Unified Wire for Linux 310 Software Driver Uninstallation i Change your working directory to Chelsio Unified Wire directory root host cd ChelsioUwire x x x x ii Uninstall OVS driver...

Page 311: ...Chapter XXII Ring Backbone Chelsio Unified Wire for Linux 311 XXII Ring Backbone...

Page 312: ...o adapters that are compatible with the Chelsio Ring Backbone utility T61100 OCP T62100 CR T62100 LP CR T62100 SO CR T6425 CR T6225 CR T6225 LL CR T6225 OCP T6225 SO CR T580 OCP SO T520 OCP SO T580 CR...

Page 313: ...version supported is 4 9 13 Other versions have not been tested and are not guaranteed to work 1 3 Ring Connectivity Connect the machines in ring backbone Connect port 1 of one machine to the port 0 o...

Page 314: ...tion Install Unified Wire on all the machines in the ring backbone i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install the dr...

Page 315: ...e cxgb4 enable_ringbb 1 ii Check dmesg output and note the mirror VF ID root host dmesg grep i mirror iii Bring up port 0 and port 1 interfaces but assign IP address to only port 0 iv Check the MPS TC...

Page 316: ...nterface_name vi Press any key when prompted to complete the configuration You should be able to run traffic between the machines nodes now Automatic reconfiguration is currently not supported In case...

Page 317: ...Chapter XXIII Traffic Management Chelsio Unified Wire for Linux 317 XXIII Traffic Management...

Page 318: ...ntrol max bandwidth that a connection or a flow a group of connections can use Allocate available bandwidth to several connection or flows based on desired levels of performance Once the offload trans...

Page 319: ...l7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32...

Page 320: ...t can be performed on non offloaded connections as well as on offloaded connections The drivers must be loaded by the root user Any attempt to load the drivers as a regular user will fail Run the foll...

Page 321: ...mode channel Channel No class scheduler class index max rate maximum rate pkt size Packet size Here ethX Chelsio interface scheduler mode specifies whether the rule is configured for individual traff...

Page 322: ...hierarchy Possible values include 1 to 99 Channel Rate Limiting This scheduler hierarchy level can be used to rate limit individual channels Channel rate limiting can be configured using the followin...

Page 323: ...nfig ethX up ii Bind connections to queues root host tc qdisc add dev ethX root handle 1 multiq root host tc filter add dev ethX parent 1 protocol ip prio 1 u32 match ip dst IP address of destination...

Page 324: ...onnections with the given scheduling class Example src host 102 1 1 1 offload class 0 The above example will associate all connections originating from IP address 102 1 1 1 with scheduling class 0 iii...

Page 325: ...function as per the parameters set for that traffic class iii Add socket option definitions In order to use setsockopt to set the options to the TCP socket the following two definitions need to be ma...

Page 326: ...multiq root host tc filter add dev eth0 parent 1 protocol ip prio 1 u32 match ip dst 192 168 5 3 action skbedit queue_mapping 3 iii Bind the NIC TX queue to class 0 root host cxgbtool eth0 sched queu...

Page 327: ...l mode class rate unit bits rate mode absolute channel 0 class 0 max rate 300000 pkt size 1460 4 3 Offloaded Connections with Modified Application The following example demonstrates the method to rate...

Page 328: ...Traffic Management Chelsio Unified Wire for Linux 328 Software Driver Unloading Reboot the system to unload the driver To unload without rebooting refer Unloading the TOE driver section of Network NI...

Page 329: ...Chapter XXIV DPDK Driver Chelsio Unified Wire for Linux 329 XXIV DPDK Driver...

Page 330: ...romiscuous mode All multicast mode Port hardware statistics Jumbo frames only for UIO 1 1 Hardware Requirements The following are the currently shipping Chelsio adapters that are compatible with DPDK...

Page 331: ...e Chelsio DPDK driver has been developed to run on 64 bit Linux platforms Following is the list of supported distributions Linux Distribution Driver Software RHEL 7 5 3 10 0 862 el7 UIO VFIO RHEL 7 4...

Page 332: ...host make dpdk_install The above command will install the driver with the following build configuration arch x86_64 ppc_64 machine native power8 execenv linuxapp toolchain gcc create a target environ...

Page 333: ...mware Configuration File i If not done already load the kernel mode NIC driver cxgb4 root host modprobe cxgb4 ii Flash the firmware configuration file for T6 adapters root host cxgbtool iface loadcfg...

Page 334: ...mentioned below to load DPDK driver with VFIO support i Enable Intel VT d in system BIOS ii Enable Intel iommu by adding the below entry to kernel command line in grub grub2 menu intel_iommu on vfio_i...

Page 335: ...es_hugepages sh Manual i Mount hugetlbfs root host mkdir p mnt huge root host mount t hugetlbfs nodev mnt huge ii Reserve Huge Page memory manually x86_64 root host echo 1024 sys devices system node n...

Page 336: ...devbind py bind igb_uio PCI ID PF4 VFIO support root host dpdk devbind py bind vfio pci PCI ID PF4 Now verify using root host dpdk devbind py status Examples Binding with UIO support Binding with VFI...

Page 337: ...an be assigned to a Virtual Machine VM or can be used in the Hypervisor Host Execute the following commands on the VM Host to bind the VF to DPDK environment UIO support root host dpdk devbind py bind...

Page 338: ...ing the Power Profile to Maximum Performance in BIOS settings Installing the adapter into a PCIe Gen3 x8 x16 slot Allocating required number of mbufs in the application based on the below formula nb_m...

Page 339: ...res to run on Note that core numbering can change between platforms and should be determined beforehand n channels Number of memory channels per processor socket i Enable interactive mode nb cores cor...

Page 340: ...by default and can be disabled via Testpmd as follows testpmd port config all rss none To enable again run testpmd port config all rss ip tcp udp Jumbo Mode There are multiple ways to enable sending a...

Page 341: ...of displaying real time metrics for ports and can handle packets with UDP TCP ARP ICMP GRE MPLS and Queue in Queue The application can run command scripts to set up repeatable test cases Syntax Execut...

Page 342: ...erved Powered by Intel DPDK Flags Port P 0 Link State UP 10000 FD TotalRate Pkts s Rx 0 0 Tx 14880593 14880593 MBits s Rx Tx 0 9999 0 9999 Broadcast 0 Multicast 0 64 Bytes 0 65 127 0 128 255 0 256 511...

Page 343: ...4 Tx and Rx queues mapping cores 1 4 to Rx and 5 8 to Tx queues on port 0 root host pktgen c 3ff n 4 T P m 1 4 5 8 0 N ii Set packet size to 64B on all ports Pktgen set all size 64 iii Start Tx traff...

Page 344: ...stop 0 6 3 Runtime Options The following devargs options can be used while running the applications force_link_up force_link_up devargs option always forces Link as up for VFs This enables VFs on the...

Page 345: ...PDK driver with UIO support root host rmmod igb_uio root host rmmod uio If unloading uio module reports an error unload the following dependent modules and try again root host rmmod bnx2fc root host r...

Page 346: ...fied Wire for Linux 346 Software Driver Uninstallation i Change your working directory to Chelsio Unified Wire directory root host cd ChelsioUwire x x x x ii Uninstall the DPDK driver using the follow...

Page 347: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 347 XXV Unified Boot...

Page 348: ...alities like PXE FCoE and iSCSI SAN boot This section of the guide also describes the use and configuration of Chelsio s DUD for OS installations via PXE server on FC FCoE LUN and iSCSI LUN This solut...

Page 349: ...7124S F Mellanox SX_PPC_M460EX Other platforms switches have not been tested and are not guaranteed to work Supported Adapters Following are the currently shipping Chelsio adapters that are compatibl...

Page 350: ...f Drivers Software and supported Linux distributions Linux Distribution Driver Software DUDs RHEL 7 5 3 10 0 862 el7 PXE FCoE iSCSI RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 PXE iSCSI SLES 15 4...

Page 351: ...ngs vi Select Advanced Secure Boot Options vii Provide the Platform Key PK Key Exchange Key KEK and Allowed Signature Database DB to the respective uEFI NVRAM variables Windows PK Will be generated at...

Page 352: ...SB flash drive as a storage media for the necessary files Follow the steps below to prepare the drive i Create a DOS bootable USB flash drive Click here for instructions ii Create a directory CHELSIO...

Page 353: ...nce the system boots from the USB flash drive change your working directory to CHELSIO directory C cd CHELSIO iii Run the following command to list all Chelsio CNAs present on the system The list disp...

Page 354: ...y previous firmware using the following command C CHELSIO cfut4 d idx xh xf vi Delete any previous Option ROM settings C CHELSIO cfut4 d idx xc vii Run the following command to flash the appropriate f...

Page 355: ...355 viii Flash the unified option ROM onto the Chelsio CNA using the following command C CHELSIO cfut4 d idx ub cubt4 bin Here cubt4 bin is the unified option ROM image file present in the CHELSIO di...

Page 356: ...below command C CHELSIO cfut4 d idx um Hex MAC Address Example C CHELSIO cfut4 d 0 um 000743000123 3 3 uEFI i Reboot the system and go into BIOS setup ii Disable Secure Boot iii Configure the system h...

Page 357: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 357 iv Boot to EFI Shell...

Page 358: ...rivers to determine if Chelsio uEFI driver is loaded If the driver is loaded as shown in the image below continue to step vi If the driver is not loaded load the uEFI driver ChelsioUD efi found in the...

Page 359: ...Linux 359 vii Chelsio HII should be listed as Chelsio T5 T6 Highlight it and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as mentioned here in...

Page 360: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 360 viii Highlight the Chelsio adapter to be configured and press Enter...

Page 361: ...as Flash File Type c Select Update Erase d Press Y to confirm e Reboot system Update firmware a Select Update as Flash Operation b Select FW File as Flash File Type c Enter full path to the firmware f...

Page 362: ...drive as described here ii Boot system to EFI shell and change your working directory to CHELSIO iii Issue command drivers to determine if Chelsio uEFI driver is loaded If loaded you should see a simi...

Page 363: ...363 Upgrading Firmware Using CLI i Use the adapter s device name to update the firmware FS1 CHELSIO fwupdate d device_name f cubt4 bin Example ii Reboot machine for changes to take effect Using FMP i...

Page 364: ...ns Firmware Update Chelsio T6 Controller iii Highlight Select a firmware file option and hit Enter iv Select the USB flash drive which contains the latest Option ROM and hit Enter v Select Option ROM...

Page 365: ...XXV Unified Boot Chelsio Unified Wire for Linux 365 vi Select Start firmware update and hit Enter vii After Firmware update completed successfully prompt appears reboot the machine for changes to take...

Page 366: ...ou wish to restore option ROM settings to their default values i e PXE enabled iSCSI and FCoE disabled use any of the methods mentioned below Using Option ROM boot level Legacy PXE Boot system into Ch...

Page 367: ...Linux 367 Using cxgbtool OS level Change your working directory to OptionROM directory and use cxgbtool to flash the default boot configuration onto the adapter root host cd driver_pakage Uboot Optio...

Page 368: ...x http linux sxs org internet_serving pxeboot html http www howtoforge com ubuntu_pxe_install_server Windows http technet microsoft com en us library cc771670 28WS 10 29 aspx http tftpd32 jounin net U...

Page 369: ...g the PXE server make sure the PXE server works Then reboot the client machine ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will ap...

Page 370: ...on among the supported NIC functions and hit Enter to select viii Enable NIC function bios if not already enabled ix Choose the boot port to try the PXE boot It is recommended to only enable functions...

Page 371: ...he system xii Allow the Chelsio option ROM to initialize and setup PXE devices DO NOT PRESS ALT S to skip Chelsio option ROM xiii In the system setup choose any of the Chelsio PXE devices as the first...

Page 372: ...II should be listed as Chelsio T5 T6 Highlight it and press Enter iv Select the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not...

Page 373: ...nified Wire for Linux 373 vi Enable adapter BIOS if not already enabled vii Select Chelsio Protocol Selection and press Enter It is highly recommended that you use the Save Changes option every time a...

Page 374: ...oose the boot port to try PXE boot It is recommended to enable only those functions and ports which are going to be used Please note that enabling PXE Function 0 will enable port 0 for PXE enabling PX...

Page 375: ...r XXV Unified Boot Chelsio Unified Wire for Linux 375 xi Reboot the system and in BIOS choose any of the available Chelsio PXE devices for PXE boot xii Reboot and hit F12 key when prompted to start PX...

Page 376: ...tion ROM 6 1 Legacy FCoE Boot i Reboot the system ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will appear as below iv Choose the C...

Page 377: ...Unified Wire for Linux 377 vi Choose FCoE from the list to configure and hit Enter vii Choose the first option Configure function parameters from the list of parameter type and hit Enter viii Enable...

Page 378: ...Boot Chelsio Unified Wire for Linux 378 ix Choose the order of the ports to discover FCoE targets x Set discovery timeout to a suitable value Recommended value is 30 xi Hit F10 or Esc and then Y to sa...

Page 379: ...re boot parameters xiii Select the first boot device and hit Enter to discover FC FCoE targets connected to the switch Wait till all reachable targets are discovered xiv List of discovered targets wil...

Page 380: ...Unified Wire for Linux 380 xv From the list of LUNs displayed for the selected target choose one on which operating system has to be installed Hit Enter xvi Hit F10 or Esc and then Y to save the confi...

Page 381: ...Chelsio adapter as the first boot device xx Reboot and boot from the FCoE disk or install the required OS using PXE 6 2 uEFI FCoE Boot i Reboot the system and go into BIOS setup ii Disable Secure Boot...

Page 382: ...lect the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as me...

Page 383: ...not already enabled vii Select Chelsio Protocol Selection and press Enter viii Select FCoE and press Enter ix Under Function Parameters enable FCoE BIOS if not already enabled It is highly recommende...

Page 384: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 384 x Set discovery timeout to a suitable value Recommended value is 30 xi Choose the order of the ports to discover FCoE targets...

Page 385: ...and press Enter to discover FC FCoE targets connected to the switch Wait till all reachable targets are discovered xiii List of discovered targets will be displayed Highlight a target to select it an...

Page 386: ...and press Enter xvi Reboot the system for changes to take effect xvii The discovered LUN should appear in the Boot Configuration section and system BIOS section xviii Select the LUN as the first boot...

Page 387: ...tion ROM 7 1 Legacy iSCSI Boot i Reboot the system ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will appear as below iv Choose the...

Page 388: ...vii Choose the first option Configure Function Parameters from the list of parameter type and hit Enter viii Enable iSCSI BIOS if not already enabled iBFT iSCSI Boot Firmware Table will be selected by...

Page 389: ...ed Boot Chelsio Unified Wire for Linux 389 ix Choose the order of the ports to discover iSCSI targets x Set discovery timeout to a suitable value Recommended value is 30 xi Hit Esc and then Y to save...

Page 390: ...nfigure initiator related properties xiii Initiator properties like IQN Header Digest Data Digest etc will be displayed Change the values appropriately or continue with the default values Hit F10 to s...

Page 391: ...AP Policy field Next choose the CHAP method Finally provide Initiator and Target CHAP credentials as per the authentication method selected Hit F10 to save xvi Go back and choose Configure Network Par...

Page 392: ...igure the port Hit F10 to save xix Go back and choose Configure Target Parameters to configure iSCSI target related properties xx If you want to discover target using DHCP select Yes in the Discover B...

Page 393: ...93 xxi Go back and choose Discover iSCSI Target s to connect to a target xxii Select the portal group on which iSCSI service is provided by the target xxiii A list of available targets will be display...

Page 394: ...A list of LUNs configured on the selected target will be displayed Select the LUN you wish to connect to and hit Enter xxv Hit Esc and then Y to save the configuration xxvi Reboot the machine xxvii Du...

Page 395: ...CSI Target LUN or install the required OS using PXE 7 2 uEFI iSCSI Boot This section describes the method to perform iSCSI boot on uEFI platforms i Reboot the system and go into BIOS setup ii Disable...

Page 396: ...lect the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as me...

Page 397: ...not already enabled vii Select Chelsio Protocol Selection and press Enter viii Select iSCSI and press Enter ix Under Function Parameters enable iSCSI BIOS if not already enabled It is highly recommend...

Page 398: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 398 x Set discovery timeout to a suitable value Recommended value is 30 xi Choose the order of the ports to discover iSCSI targets...

Page 399: ...itor properties like IQN Header Digest Data Digest etc will be displayed Change the values appropriately or continue with the default values xiii Under the first port select Enable DHCP field hit Ente...

Page 400: ...t Chelsio Unified Wire for Linux 400 xiv Under Target Parameters select Enabled for the Boot Target via DHCP parameter to discover target using DHCP To discover target via static IP select Disabled an...

Page 401: ...xv CHAP authentication is disabled by default To enable and configure highlight CHAP Policy and hit Enter Select the policy type from the corresponding pop up and hit Enter again xvi Provide Initiator...

Page 402: ...iSCSI targets connected to the switch Wait till all reachable targets are discovered xviii A list of available targets will be displayed Select the target you wish to connect to and hit Enter xix A l...

Page 403: ...press Enter xxi Reboot the system for changes to take effect xxii The discovered LUN should appear in the Boot Configuration Boot Information section and system BIOS xxiii Select the LUN as the first...

Page 404: ...rrent working directory to LinuxDUD directory root host cd driver_package Uboot LinuxDUD iv Insert a blank formatted USB flash drive v Depending on the distribution to be installed copy the correspond...

Page 405: ...B drive root host mkfs vfat dev sda1 v Depending on the distribution to be installed copy the corresponding image file to the USB stick For example execute the following command for SLES 12 sp3 root h...

Page 406: ...ot PXE Boot media to install the OS and provide the Driver Update Disk as per the detailed instructions for each OS The DUD supports installation of Linux distributions using Chelsio adapters over Net...

Page 407: ...ring the installation ii You will be asked to select the Driver Update Disk device from a list USB drives usually show up as SCSI disks in Linux Enter the index number of the device to be used and hit...

Page 408: ...rocess enter c and hit Enter To install iSCSI Initiator driver the dependent Network driver must also be installed Hence select both cxgb4 and and cxgb4i Hit Enter vi The Driver disk prompt will be di...

Page 409: ...ller will load and prompt you for the driver update disk Select Yes and hit Enter to proceed iii You will be asked to select the Driver Update Disk device from a list USB drives usually show up as SCS...

Page 410: ...Now the installer will search for the appropriate drivers from the driver disk and load them This step may take some time Check on the Alt F3 or Alt F4 screens for log messages v The installer will as...

Page 411: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 411 vi Select the required language from the list vii Select the type of keyboard you have from the list...

Page 412: ...ux 412 viii In this step you can choose the source which contains the OS installation ISO image In this case select NFS directory ix The Chelsio detwork devices will be displayed Select the appropriat...

Page 413: ...ing DHCP or manually using IPv4 IPv6 is currently not supported Hence disable IPv6 before proceeding xi Proceeding with the installation will get NFS FTP HTTP setup page Here provide NFS server detail...

Page 414: ...Unified Boot Chelsio Unified Wire for Linux 414 SLES 15 SLES 12 SPx Installation i Please make sure that the USB drive with DUD image is inserted ii Select the appropriate entry from the PXE menu and...

Page 415: ...pter XXV Unified Boot Chelsio Unified Wire for Linux 415 iii Select Start Installation and then Start Installation or Update iv Select Network as the source of medium to install the SLES Operating Sys...

Page 416: ...the appropriate Chelsio interface from the list to proceed with installation You can view the messages from the Linux kernel and drivers to determine the name of NIC interface by pressing Alt F3 or Al...

Page 417: ...rom CD DVD in the BIOS setup i Insert the OS installation disc into your CD DVD ROM ii On the Grub menu choose Install or upgrade an existing system option if not already selected iii Type e and then...

Page 418: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 418 RHEL 7 X i Choose your installation language and click Continue ii Click INSTALLATION DESTINATION under SYSTEM...

Page 419: ...ction Select the LUN which was saved as boot device in system BIOS iv Under Other Storage Options you can either chose to configure partition automatically or manually Select the appropriate option an...

Page 420: ...transport medium Legacy o For RHEL 6 press Tab and then type dd o For RHEL 7 press Tab and then type ip ibft o For SLES press Tab and then type dd uEFI o For RHEL 6 type e and then type dd o For RHEL...

Page 421: ...hapter XXV Unified Boot Chelsio Unified Wire for Linux 421 RHEL 7 X i On the installer welcome screen choose your installation language and click Continue ii Click Installation Destination under SYSTE...

Page 422: ...io Unified Wire for Linux 422 iii Click Add a disk iv The discovered iSCSI LUNs will appear in the Search tab Select it and click Done Make sure the same LUN discovered at the Option ROM stage is sele...

Page 423: ...you can either chose to configure partition automatically or manually Select the appropriate option and click Done Then proceed with the installation as usual RHEL 6 X i Click Next when the graphical...

Page 424: ...discovered LUNs will appear in the Basic Devices tab Select the LUN which was saved as boot device in system BIOS and click Next iv Proceed with the installation as usual Make sure the same LUN disco...

Page 425: ...ied Wire for Linux 425 SLES 15 SLES 12 SPx Installation i Choose installation language and Keyboard layout type Select the checkbox I Agree to the License terms and click Next ii Click Configure iSCSI...

Page 426: ...discovered LUNs will appear in the Connected Targets tab Select the LUN which was saved as boot device in system BIOS and click OK iv Proceed with the installation as usual Make sure the same LUN disc...

Page 427: ...Chapter XXVI Appendix A Chelsio Unified Wire for Linux 427 XXVI Appendix A...

Page 428: ...g If you get a return value of 0 when doing a cat proc sys net ipv4 icmp_echo_ignore_all on the remote host that means it is configured to reply to incoming pings Change ipv4 to ipv6 in the path if yo...

Page 429: ...WARP traffic One of the reason for this could be port number collisions To fix this use iWARP port mapper iwpmd root host iwpmd Multiple Chelsio adapters Chelsio Option ROM supports upto 4 Chelsio ada...

Page 430: ...oftware contains trade secrets and in order to protect them you may not decompile reverse engineer disassemble or otherwise reduce the Software to a human perceivable form You assume full responsibili...

Page 431: ...ARISING OUT OF OR RELATED TO THE LICENSE OR USE OF THE SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR THE LICENSE GRANTED HEREUNDER THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL...

Page 432: ...covered by Federal law You hereby irrevocably submit to the personal jurisdiction of and irrevocably waive objection to the laying of venue including a waiver of any argument of forum non conveniens...

Reviews:

No comments