manualshive.com logo in svg

Chelsio Communications Terminator 6, Инструкция по установке и эксплуатации, Страница: 87 / 432

Поделиться
Скачать
Chelsio Communications Terminator 6 Скачать руководство пользователя страница 87

Chapter IV. iWARP (RDMA) 

Chelsio Unified Wire for Linux 

 87

 

 Software/Driver Configuration and Fine-tuning 

 

4.1. 

Testing connectivity with ping and rping 

Load the NIC, iWARP & core RDMA modules as mentioned in 

Software/Driver Loading

 section. 

After which, you will see two or four ethernet interfaces for the Terminator device. Configure them 
with an appropriate ip address, netmask, etc. You can use the Linux 

ping

 command to test basic 

connectivity via the Terminator interface. To test RDMA, use the 

rping

 command that is included 

in the librdmacm-utils RPM: 

Run the following command on the server machine:  
 

[root@host~]# rping -s -a server_ip_addr -p 9999  

 

Run the following command on the client machine:  
 

[root@host~]# rping -c 

–

Vv -C10 -a server_ip_addr -p 9999  

 

You should see ping data like this on the client:  
 

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr  
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs  
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst  
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu  
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv  
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw  
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx  
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy  
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz  
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA  
client DISCONNECT EVENT...  
# 

 

 

 

 

Содержание Terminator 6

Страница 1: ...Chelsio Unified Wire for Linux i...

Страница 2: ...TIVELY THE SOFTWARE IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO COMMUNICATIONS INC Sales For all sales inquiries please send email to sales chelsio com Support For all support related question...

Страница 3: ...1 1 0 07 26 2013 1 1 1 08 14 2013 1 1 2 12 06 2013 1 1 3 12 19 2013 1 1 4 03 13 2014 1 1 5 05 02 2014 1 1 6 06 30 2014 1 1 7 10 22 2014 1 1 8 11 04 2014 1 1 9 02 05 2015 1 2 0 03 04 2015 1 2 1 03 25...

Страница 4: ...Chelsio Unified Wire for Linux iv 1 3 6 09 29 2017 1 3 7 12 29 2017 1 3 8 02 28 2018 1 3 9 03 30 2018 1 4 0 04 18 2018 1 4 1 07 05 2018 1 4 2 07 18 2018 1 4 3 10 01 2018...

Страница 5: ...itramfs 36 Configuring Chelsio Network Interfaces 37 4 1 Configuring Adapters 37 4 2 Configuring network scripts 41 4 3 Creating network scripts 41 4 4 Checking Link 42 Performance Tuning 43 5 1 Gener...

Страница 6: ...ual Functions SR IOV 74 3 2 Loading the Driver 74 Software Driver Configuration and Fine tuning 76 4 1 VF Rate Limiting 76 4 2 Bonding 77 4 3 High Capacity VF Configuration 78 Software Driver Unloadin...

Страница 7: ...river Installation 110 Software Driver Loading 111 Software Driver Configuration and Fine tuning 112 4 1 Accelerating UDP Socket Communications 112 Software Driver Unloading 118 VII WD TOE 119 Introdu...

Страница 8: ...floading LIO iSCSI Connection 144 5 3 Running LIO iSCSI and Network Traffic Concurrently 145 5 4 Performance Tuning 146 Software Driver Unloading 147 6 1 Unloading the LIO iSCSI Target Offload Driver...

Страница 9: ...Software Driver Loading 191 Software Driver Configuration and Fine tuning 192 4 1 Accelerating open iSCSI Initiator 192 4 2 HMA 194 4 3 Auto login from cxgb4i initiator at OS bootup 195 4 4 Performan...

Страница 10: ...tch 227 4 2 FCoE fabric discovery verification 227 4 3 Formatting the LUNs and Mounting the Filesystem 231 4 4 Creating Filesystem 232 4 5 Mounting the formatted LUN 233 Software Driver Unloading 234...

Страница 11: ...2 1 Pre requisites 261 2 2 Installation 261 Software Driver Loading 262 Software Driver Configuration and Fine tuning 263 Software Driver Unloading 264 5 1 Unloading the NIC Driver 264 5 2 Unloading t...

Страница 12: ...equirements 298 1 2 Software Requirements 299 Software Driver Installation 300 2 1 Pre requisites 300 2 2 Installation 300 Software Driver Configuration and Fine Tuning 301 3 1 Configuring OVS Machine...

Страница 13: ...are Configuration File 333 Software Driver Loading 334 Software Driver Configuration and Fine Tuning 335 5 1 Huge Pages 335 5 2 Binding Network Ports 336 5 3 Unbinding Network Ports 338 5 4 Performanc...

Страница 14: ...ot Process 387 7 1 Legacy iSCSI Boot 387 7 2 uEFI iSCSI Boot 395 Creating Driver Update Disk DUD 404 8 1 Creating DUD for RedHat Enterprise Linux 404 8 2 Creating DUD for Suse Enterprise Linux 404 OS...

Страница 15: ...Chapter I Chelsio Unified Wire Chelsio Unified Wire for Linux 15 I Chelsio Unified Wire...

Страница 16: ...bre Channel and InfiniBand applications over Ethernet at line rate Designed for deployment in virtualized data centers cloud service installations and high performance computing environments Chelsio a...

Страница 17: ...ing the drivers tools please refer to their respective sections 1 2 Hardware Requirements The Chelsio Unified Wire software supports Chelsio Terminator series of Unified Wire adapters To know more abo...

Страница 18: ...ectory contains various benchmarking tools to measure throughput and latency of various networks chelsio_adapter_config This directory contains scripts and binaries needed to configure Chelsio 40G ada...

Страница 19: ...directory contains Unified Boot Option ROM image cubt4 bin uEFI driver ChelsioUD efi default boot configuration file bootcfg and a legacy flash utility cfut4 exe which can be used to flash the option...

Страница 20: ...here are two sub directories in the Uboot directory OptionROM and LinuxDUD The OptionROM directory contains Unified Boot Option ROM image cubt4 bin uEFI driver ChelsioUD efi default boot configuration...

Страница 21: ...er with a screw or other securing mechanism as described by the system manufacturer s instructions Replace the case cover ix After securing the card ensure that the card is still fully seated in the P...

Страница 22: ...s indicate the hardware configuration of the adapter as well as serial number Network device names for Chelsio s physical ports are assigned using the following convention the port farthest from the m...

Страница 23: ...with low latency TOE iWARP WD UDP WD TOE IPv6 Bonding MAFO High capacity RDMA Configures adapters to establish a large number of iWARP connections iWARP RDMA Performance Improves iWARP performance iWA...

Страница 24: ...G_INFINIBAND y iv Compile the kernel v During kernel compilation please ensure that the following parameters are set as follows CONFIG_CHELSIO_T1 m CONFIG_CHELSIO_T1_1G y CONFIG_CHELSIO_T3 m CONFIG_CH...

Страница 25: ...d Wire from source GUI mode with Dialog utility i Download the Unified Wire driver package tarball from Chelsio Download Center ii Untar the tarball using the following command root host tar zxvf driv...

Страница 26: ...ad support viii Select the required configuration tuning option ix Under Choose install components select all to install all the related components for the option chosen in step viii or select custom...

Страница 27: ...RQ balance daemon c TX Coalasce Write tx_coal 2 to modprobe d conf xi If you already have the required version of OFED software installed select Skip OFED To install OFED 4 8 2 choose the Install OFED...

Страница 28: ...tion summary of installed components will be displayed xiv Select View log to view the installation log or Exit to continue xv Select Yes to exit the installer or No to go back xvi Reboot your machine...

Страница 29: ...v Enter the number corresponding to the Configuration tuning option in the Input field and press Enter vi If you already have the required version of OFED software installed select Skip OFED To insta...

Страница 30: ...m machinefilename iv Select the required configuration tuning option The tuning options may vary depending on the Linux distribution v Select the required Cluster Configuration vi If you already have...

Страница 31: ...s individually Provided here are steps to build and install some of them For the complete list view help by running make help Change your current working directory to Chelsio Unified Wire package dire...

Страница 32: ...r will remove the Chelsio specific drivers inbox outbox from initramfs To skip this and install drivers root host make SKIP_INIT 1 install The installer will check for the required dependency packages...

Страница 33: ...tall all the drivers in the Unified Wire software package This option will not install OFED and drivers built against OFED b Wire Direct Latency Install Wire Direct Latency drivers needed for Low late...

Страница 34: ...py i nic_toe all udpso wd crypto ovs Here nic_toe NIC and TOE drivers only all All Chelsio drivers built against inbox OFED udpso UDP segmentation offload capable NIC and TOE drivers only wd Wire Dire...

Страница 35: ...IP addresses or hostnames of the nodes in the cluster You can view the sample file sample_machinefile provided in the package to view the format in which the nodes have to be listed iii Now execute th...

Страница 36: ...port The firmware version can be verified using ethtool root host ethtool i iface 3 8 Removing Drivers from initramfs Chelsio drivers cxgb4 cxgb4vf iw_cxgb4 cxgb4i csiostor etc might exist in the init...

Страница 37: ...th the PEER Setting FEC 100G 50G and 25G speeds support changing Forward Error Correction FEC The existing FEC settings can be viewed using root host cxgbtool ethX fec Below is a sample output on T6 1...

Страница 38: ...speeds The supported speeds can be seen using ethtool Below is a sample output for T6 100G port Optics Optics do not support auto negotiation Use the following command to change the speed root host et...

Страница 39: ...t ethtool change ethX advertise 0x2000000 o Advertise only 50G root host ethtool change ethX advertise 0x400000000 o Advertise only 25G root host ethtool change ethX advertise 0x80000000 o Advertise 1...

Страница 40: ...using existing SFP based cabling The port farthest from the motherboard will appear as the first network interface Port 0 T6 Adapters Chelsio T6 100G adapters can be configured in the following 2 mode...

Страница 41: ...OTPROTO static should be changed to BOOTPROTO dhcp The ifcfg ethX files have to be created manually They are required for bringing the interfaces up and down and attribute the desired IP addresses 4 3...

Страница 42: ...could look like file etc sysconfig network scripts ifcfg eth1 DEVICE eth1 HWADDR 00 07 43 04 6B E9 ONBOOT no NM_CONTROLLED no BOOTPROTO static IPADDR 10 192 167 112 NETMASK 255 255 240 0 From now on t...

Страница 43: ...e 1 intel_idle max_cstate 0 to the kernel command line to prevent the system from entering power saving idle states and avoid CPU frequency changes Turn off irqbalance root host etc init d irqbalance...

Страница 44: ...or Linux 44 Disable few services root host t4_latencytune sh interface Set sysctl param net ipv4 tcp_low_latency to 1 root host sysctl w net ipv4 tcp_low_latency 1 To optimize your system for differen...

Страница 45: ...te For any distribution specific problems please check README and Release Notes included in the release for possible workaround Please visit Chelsio Download Center for regular updates on various soft...

Страница 46: ...ide to use source you can uninstall the package using CLI or GUI mode 7 1 Uninstalling Chelsio Unified Wire from source GUI mode with Dialog utility i Change your current working directory to Chelsio...

Страница 47: ...sio Unified Wire for Linux 47 iv The selected components will now be uninstalled v After successful uninstalltion summary of the uninstalled components will be displayed vi Select View log to view uni...

Страница 48: ...t install py u target iWARP driver uninstallation on Cluster nodes i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Uninstall iWAR...

Страница 49: ...se to uninstall drivers software individually Provided here are steps to uninstall few of them For the complete list view help by running make help Change your current working directory to Chelsio Uni...

Страница 50: ...nd Chelsio drivers iWARP driver uninstallation on Cluster nodes i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x OS arch ii Uninstall...

Страница 51: ...Chapter II Network NIC TOE Chelsio Unified Wire for Linux 51 II Network NIC TOE...

Страница 52: ...a flow processor implementation and a host of features designed for high throughput and low latency in demanding conditions and networking environments TCP offload is fully implemented in the hardware...

Страница 53: ...g versions RHEL 7 5 3 10 0 862 el7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 e...

Страница 54: ...rectory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x To build and install NIC only driver without offload support root host make nic_install To build and install drivers...

Страница 55: ...ll offload support run the following command root host modprobe cxgb4 3 2 Loading in TOE mode with full offload support To enable full offload support run the following command root host modprobe t4_t...

Страница 56: ...mber of connections offloaded run the following command root host cat sys kernel debug cxgb4 bus id tids Where TID is the number of offload connections STID is the number of offload servers T6 25G SO...

Страница 57: ...Protocol PTP standard defines a protocol for precise synchronization of clock between master and slave devices in a local area network It can provide timing accuracies in nanosecond units The protocol...

Страница 58: ...robe cxgb4 ii Assign IP addresses and ensure that master and slave nodes are connected iii Start the ptp4l tool on master using the Chelsio interface root host ptp4l i interface H m iv Start the ptp4l...

Страница 59: ...ilable logical networks from 4096 to 16 million thereby addressing the scalability and network segmentation issues associated with large cloud computing deployments Chelsio s Terminator based adapters...

Страница 60: ...to the bridge interface root host brctl addif bridge_interface vxlan_interface vi Tx UDP Tunnel Segmentation Offload will be enabled by default on loading the network driver To see the current setting...

Страница 61: ...ame vhost queues 8 ii Map the Virtual CPUs of the VM to physical CPUs which will be free Example On a machine with 16 cores VM Virtual CPUs were pinned to physical cores 8 15 leaving cores 0 7 to be u...

Страница 62: ...tcp_timestamps 0 root host sysctl w net core netdev_max_backlog 250000 root host sysctl w net core rmem_max 4194304 root host sysctl w net core wmem_max 4194304 root host sysctl w net core rmem_defau...

Страница 63: ...t ipv4 tcp_wmem 4096 1048576 4194304 NIC TOE Latency Enable BUSY_POLL feature root host sysctl w net core busy_poll 50 root host sysctl w net core busy_read 50 Receiver Side Scaling RSS Receiver Side...

Страница 64: ...ple by using the iperf tool First set up a server on the receiver host root receiver_host iperf s Then on the sender host send data to the server using the iperf client mode To emulate a moderate traf...

Страница 65: ...laced into the host memory through DMA operations before an interrupt is thrown reducing the CPU load and thus improving latency It can be changed using the following command root host ethtool C ethX...

Страница 66: ...that if your Linux system has IP forwarding enabled i e acting as a bridge or router the LRO needs to be disabled This is due to a known kernel issue Chelsio s card supports both hardware assisted GRO...

Страница 67: ...played using the following command root host ethtool S eth6 GROPackets 0 GROMerged 897723 GROPackets is the number of held packets Those are candidate packets held by the kernel to be processed indivi...

Страница 68: ...root host modprobe t4_tom unsupported_allow_unload 1 ii Stop all the offloaded traffic servers and connections Check for the reference count root host cat sys module t4_tom refcnt If the reference cou...

Страница 69: ...Chapter II Network NIC TOE Chelsio Unified Wire for Linux 69 iii Unload the driver root host rmmod t4_tom root host rmmod toecore root host rmmod cxgb4...

Страница 70: ...Chapter III Virtual Function Network vNIC Chelsio Unified Wire for Linux 70 III Virtual Function Network vNIC...

Страница 71: ...erver portability cloud computing Virtual Desktop Infrastructure VDI etc Chelsio s Unified Wire family of adapters deliver increased bandwidth lower latency and lower power with virtualization feature...

Страница 72: ...ng versions RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 gener...

Страница 73: ...2 1 Pre requisites Please make sure that the following requirements are met before installation PCI Express Slot should be ARI capable SR IOV should be enabled in the machine Intel Virtualization Tech...

Страница 74: ...om PFs 0 3 of the Chelsio adapter A maximum of 64 virtual functions can be instantiated with 16 virtual functions per physical function Example Instantiating 16 VFs on PF3 of Chelsio adapter Unload th...

Страница 75: ...k vNIC Chelsio Unified Wire for Linux 75 The vNIC driver must be loaded on the Guest OS by the root user Any attempt to load the driver as a regular user will fail To load the driver run the following...

Страница 76: ...on which rate limiting is applied Value 0 15 ii Run traffic over the VF and the throughput should be rate limited as per the values set in the previous step Example i 4 VFs are instantiated on PF0 roo...

Страница 77: ...obe bonding mode bonding mode optional paramters iii Bring up the bonded interface and enslave the VF interfaces to the bond root host ifconfig bond0 up root host ifenslave bond0 ethX ethY iv Assign I...

Страница 78: ...0 up root host ifenslave bond0 enp8s1 enp8s1f5d1 root host ifconfig bond0 10 1 1 223 24 The traffic will run over the bond interface in Active Backup mode If the link fails on enp8s1 the traffic will...

Страница 79: ...iii Instantiate virtual functions root host modprobe cxgb4 root host echo n sys class net ethX device driver bus_id sriov_numvfs 124 virtual functions can be instantiated on T5 adapter with 31 virtua...

Страница 80: ...3 NOARP mtu 0 qdisc noop state DOWN mode DEFAULT qlen 1 link none vf 0 MAC 06 44 3c b1 80 30 link state auto vii To set a VLAN ID on Virtual Function use the following syntax root host ip link set mg...

Страница 81: ...or Linux 81 Software Driver Unloading 5 1 Unloading the Driver The vNIC driver must be unloaded on the Guest OS by the root user Any attempt to unload the driver as a regular user will fail To unload...

Страница 82: ...Chapter IV iWARP RDMA Chelsio Unified Wire for Linux 82 IV iWARP RDMA...

Страница 83: ...per connection or per class basis and to give higher priority to QPs that implement distributed locking mechanisms The iWARP operation also benefits from the high performance and low latency TCP imple...

Страница 84: ...aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES...

Страница 85: ...s to be installed on all the members If you want to install OFED with NFS RDMA support refer to Setting up NFS RDMA section rdma core devel package should be installed on RHEL 7 4 RHEL 7 5 SLES15 and...

Страница 86: ...mpt to load the driver as a regular user will fail To load the iWARP driver we need to load the NIC driver and core RDMA drivers first Run the following commands root host modprobe cxgb4 root host mod...

Страница 87: ...command on the client machine root host rping c Vv C10 a server_ip_addr p 9999 You should see ping data like this on the client ping data rdma ping 0 ABCDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqr pi...

Страница 88: ...to accept default setup and empty password phrase iv Create authorization file root host cd ssh root host cat pub authorized_keys root host chmod 600 authorized_keys v Copy directory ssh to all system...

Страница 89: ...following line adds a DAT version 2 0 device named chelsio2 for that interface chelsio2 u2 0 nonthreadsafe default libdaplofa so 2 dapl 2 0 eth2 0 Open MPI Installation and Setup Open MPI iWARP suppor...

Страница 90: ...above step will install openMPI in usr mpi gcc openmpi x y z v Next create a shell script mpivars csh with the following entry path if echo path grep usr mpi gcc openmpi x y z bin then set path usr mp...

Страница 91: ...RARY_PATH usr mpi gcc openmpi x y z lib64 LD_LIBRARY_PATH LD_LIBRARY_PATH export LD_LIBRARY_PATH fi MPI_ROOT MPI_ROOT usr mpi gcc openmpi x y z export MPI_ROOT vii Next copy the two files created in s...

Страница 92: ...able rdma cm libdir usr mpi gcc mvapich2 x y lib64 root host make root host make install The above step will install MVAPICH2 in usr mpi gcc mvapich2 x y iv Next create a shell script mpivars csh with...

Страница 93: ...mvapich2 x y lib64 LD_LIBRARY_PATH LD_LIBRARY_PATH export LD_LIBRARY_PATH fi MPI_ROOT MPI_ROOT usr mpi gcc mvapich2 x y export MPI_ROOT vi Next copy the two files created in steps iv and v to usr mpi...

Страница 94: ...u want to build the benchmarks tool against For example in case of openMPI 1 6 4 set the variable as MPI_HOME usr mpi gcc openmpi 1 6 4 iv Next build and install the benchmarks using root host gmake f...

Страница 95: ...default is 1000 or 1000ms Increase it to 30000 with this parameter mca btl_openib_connect_rdmacm_resolve_timeout 30000 Run MVAPICH2 application as mpirun_rsh ssh np 8 hostfile mpd hosts MVAPICH2_HOME...

Страница 96: ...quash Note that for each directory you export you should have DIFFERENT fsid s ii Load the iwarp modules and make sure peer2peer is set to 1 iii Load xprtrdma and svcrdma modules as root host modprobe...

Страница 97: ...hich directories will be exported ii Load the xprtrdma module root host modprobe xprtrdma iii Run the showmount command to show all directories from server as root host showmount e server chelsio ip i...

Страница 98: ...Chapter IV iWARP RDMA Chelsio Unified Wire for Linux 98 Software Driver Unloading To unload the iWARP driver run the following command root host rmmod iw_cxgb4...

Страница 99: ...Chapter V iSER Chelsio Unified Wire for Linux 99 V iSER...

Страница 100: ...P CR T540 CR T540 BT T520 CR T520 LL CR T520 BT Memory free 256 IPv4 128 IPv6 offload connections supported 1 2 Software Requirements Linux Requirements Currently the iSER driver is available for the...

Страница 101: ...4 14 67 kernel with iSER components enabled use the following command root host make kernel_install CONFIG_ISCSI_TARGET m CONFIG_INFINIBAND_ISER m CONFIG_INFINIBAND_ISERT m iii Boot into the new kern...

Страница 102: ...der version exists v2 7 10 provided in the package will be installed Uninstall any OFED present in the machine rdma core devel package should be installed on RHEL 7 4 RHEL 7 5 SLES15 and SLES 12 SP3 s...

Страница 103: ...oot host rmmod iw_cxgb4 ii Load the following modules root host modprobe iw_cxgb4 mpa_rev 2 root host modprobe rdma_ucm iii Start the iWARP Port Mapper Daemon root host iwpmd iv Bring up the Chelsio i...

Страница 104: ...lun0 target tpg1 set attribute authentication 0 demo_mode_write_protect 0 generate_node_acls 1 cache_dynamic_acls 1 root host targetcli saveconfig ii Discover LIO target using OpeniSCSI initiator root...

Страница 105: ...nections are supported on T6 25G SO adapters The following image shows the HMA reserved memory The following image shows the number of offloaded iSER connections 5 2 Performance Tuning i Apply the per...

Страница 106: ...or Linux 106 Software Driver Unloading To unload iSER driver On target run the following commands root host rmmod ib_isert root host rmmod iw_cxgb4 On initiator run the following commands root host rm...

Страница 107: ...Chapter VI WD UDP Chelsio Unified Wire for Linux 107 VI WD UDP...

Страница 108: ...e bypass of the kernel which results in an ultra low latency Ethernet solution for high frequency trading and other delay sensitive applications 1 1 Hardware Requirements Supported Adapters The follow...

Страница 109: ...12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Minimum 4 9...

Страница 110: ...Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install WD UDP driver and libraries root host make wdudp_install For...

Страница 111: ...mpt to load the driver as a regular user will fail RDMA core modules from the OFED package should be loaded before proceeding To load the WD UDP driver use the following commands which will automatica...

Страница 112: ...r High Capacity WD tuning option Application Requirements Certain application behavior is not supported by libcxb4_sock in this release If your application does any of the following it will not work w...

Страница 113: ...bcxgb4_sock using one of the methods mentioned below when starting your application Preloading using wdload script root host PROT UDP wdload pathto your_application The above command will generate an...

Страница 114: ...our Terminator interface is eth2 To accelerate all applications that preload libcxgb4_sock using eth2 you only need one entry in etc libcxgb4_sock conf endpoint interface eth2 port 0 If you have eth2...

Страница 115: ...cxgb4 log ii Restart the service root host etc init d syslog restart iii Finally preload libcxgb4_sock_debug using the command mentioned below when starting your application root host LD_PRELOAD libc...

Страница 116: ...64 Scope Link root r9 root r10 ifconfig eth1 grep inet inet addr 192 168 2 112 Bcast 192 168 2 255 Mask 255 255 255 0 inet6 addr fe80 7 4300 104 456a 64 Scope Link root r10 For this benchmark we need...

Страница 117: ...the same test with libcxgb4_sock root r9 LD_PRELOAD libcxgb4_sock so usr local src hpcbench udp udptest r 5 a h 192 168 1 112 p 9001 Determining if the application is being offloaded To see if the app...

Страница 118: ...Chapter VI WD UDP Chelsio Unified Wire for Linux 118 Software Driver Unloading To unload the WD UDP driver run the following command root host rmmod iw_cxgb4...

Страница 119: ...Chapter VII WD TOE Chelsio Unified Wire for Linux 119 VII WD TOE...

Страница 120: ...1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the WD TOE driver T62100 CR T62100 LP CR T6425 CR T6225 CR T6225 LL CR...

Страница 121: ...king Wire Direct Latency or High Capacity WD configuration tuning Options With any other configuration tuning option the installation needs to be customized 2 2 Installation i Change your current work...

Страница 122: ...xgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail Run the following commands to load the driver root host modprobe cxg...

Страница 123: ...Netperf application with WD TOE Start netserver at the PEER root host PROT TCP wdload netserver D 4 On the Test machine run netperf application root host PROT TCP wdload netperf H PEER_IP t TCP_RR l...

Страница 124: ...Chapter VII WD TOE Chelsio Unified Wire for Linux 124 Software Driver Unloading Unload NIC TOE drivers as mentioned in Software Driver Unloading section of Network NIC TOE chapter...

Страница 125: ...Chapter VIII NVMe oF Chelsio Unified Wire for Linux 125 VIII NVMe oF...

Страница 126: ...s to achieve fast storage response times irrespective of whether the NVMe SSDs are attached locally or accessed remotely across enterprise or datacenter networks 1 1 Hardware Requirements Supported Ad...

Страница 127: ...ault Ubuntu 18 04 1 4 15 0 29 generic Kernel org linux 4 14 67 kernel compiled on RHEL 7 3 and above Kernel org linux 4 9 Minimum 4 9 kernel version supported is 4 9 13 kernel compiled on RHEL 7 3 and...

Страница 128: ...s enabled use the following command root host make kernel_install CONFIG_BLK_DEV_NVME m CONFIG_NVME_RDMA m CONFIG_NVME_TARGET m CONFIG_NVME_TARGET_RDMA m CONFIG_NVME_RDMA m CONFIG_BLK_DEV_NULL_BLK m C...

Страница 129: ...or above should be installed If this version is not already present in the system or if an older version exists v2 7 10 provided in the package will be installed rdma core devel package should be ins...

Страница 130: ...dprobe rdma_ucm ii Bring up the Chelsio interface s root host ifconfig ethX x x x x up iii Mount configfs by running the below command root host mount t configfs none sys kernel config iv On target ru...

Страница 131: ...paces cd 1 subsystems n namespaces 1 set device path dev ram1 subsystems n namespaces 1 cd subsystems nvme ram0 set attr allow_any_host 1 subsystems nvme ram0 cd namespaces 1 subsystems n namespaces 1...

Страница 132: ...connect all t rdma a target_ip s 4420 iii List the connected targets root host nvme list iv Format and mount the NVMe disks shown with the above command v Disconnect from the target and unmount the di...

Страница 133: ...pter before proceeding i Run the performance tuning script to map iWARP queues to different CPUs root host t4_perftune sh n Q rdma N ii Set the inline data size to 8192 before enabling the NVMe port r...

Страница 134: ...g Follow the steps mentioned below to unload the drivers On target run the following commands root host rmmod nvmet rdma root host rmmod nvmet root host rmmod iw_cxgb4 On initiator run the following c...

Страница 135: ...Chapter IX LIO iSCSI Target Offload Chelsio Unified Wire for Linux 135 IX LIO iSCSI Target Offload...

Страница 136: ...culations Offload Speeds at 10 25 40 100Gb Supports Direct Data Placement DDP Supports iSCSI Segmentation Offload and iSCSI PDU recovery 1 1 Hardware Requirements Supported Adapters The following are...

Страница 137: ...L 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default Ubuntu 18 04 1 4 15 0 29 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Mi...

Страница 138: ...iver installation as directed in the Software Driver Installation section SLES15 SLES 12 SP3 No extra kernel configuration required Kernel org linux 4 14 67 i Change your current working directory to...

Страница 139: ...e installation menu root host make menuconfig iv Select Device Drivers Generic Target Core Mod TCM and ConfigFS Infrastructure v Enable Linux iSCSI org iSCSI Target Mode Stack vi Select Save vii Exit...

Страница 140: ...e menuconfig iv Select Device Drivers Generic Target Core Mod TCM and ConfigFS Infrastructure v Enable Linux iSCSI org iSCSI Target Mode Stack as a Module if not already enabled vi Select Save vii Exi...

Страница 141: ...ckage will be installed along with the kernel 3 2 Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install LIO driver...

Страница 142: ...kernel_object_dir Example For RHEL 7 5 root host install py ksrc usr src linux 3 10 0 862 el7 kobj lib modules 3 10 0 862 el7 x86_64 build GUI mode root host install py set kpath Provide the paths fo...

Страница 143: ...must be loaded by the root user Any attempt to load the driver as a regular user will fail i Load network driver cxgb4 root host modprobe cxgb4 ii Bring up the interface root host ifconfig ethX IP add...

Страница 144: ...www linux iscsi org Doc LIO Admin Manual pdf to do so Sample Configuration Here is a sample iSCSI configuration listing a target configured with 1 RAM disk LUN and ACL not configured 5 2 Offloading LI...

Страница 145: ...y load network driver with offload support TOE root host modprobe t4_tom ii Create a new policy file root host cat new_policy_file iii Add the following lines to offload all traffic except LIO iSCSI l...

Страница 146: ...Wire chapter before proceeding ii Run the performance tuning script to map LIO Target queues to different CPUs root host t4_perftune sh Q iSCSIT n iii For maximum performance it is recommended to use...

Страница 147: ...the steps mentioned below i Log out from the initiator ii Run the following command root host echo 0 sys kernel config target iscsi target_iqn tpgt_1 np target_ip 3260 cxg bit Execute the above comma...

Страница 148: ...Chapter X iSCSI PDU Offload Target Chelsio Unified Wire for Linux 148 X iSCSI PDU Offload Target...

Страница 149: ...are The performance of this iSCSI stack is outstanding and when used with Chelsio s hardware it is enhanced further Because of the tight integration with Chelsio s TOE adapters this software has a dis...

Страница 150: ...upport Persistent Reservation Dynamic LUN Resizing iSCSI Target Redirection Multiple Target device types Block Virtual Block LVM Software RAID EVMS etc Built in RAM Disk Built in zero copy RAM Disk Su...

Страница 151: ...ardware storage device is required This device can be any of the device types that are supported block virtual block RAM disk Multiple storage devices are allowed by configuring the devices to one tar...

Страница 152: ...versions have not been tested and are not guaranteed to work Requirements for Installing the iSCSI Software When installing the iSCSI software it is required that the system have Linux kernel source o...

Страница 153: ...allation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install iSCSI target driver firmware and utilities root host make iscsi_...

Страница 154: ...rnel The software is dependent on the underlying NIC adapter driver and thus the limitation on what version of the Linux kernel it can run on is mostly dependent on the NIC driver s limitations The iS...

Страница 155: ...ce loads the kernel module and starts all the targets configured by default It can also be used to stop the targets and restart reload configuration Note i While using rpm tar ball for installation a...

Страница 156: ...tion settings Retrieve active sessions information of an iSCSI Target Manually flush data to the iSCSI Target disks Reload the iSCSI configuration file Update the iSCSI configuration file Save the cur...

Страница 157: ...Basic editing of the iSCSI configuration file Basic commands of the iSCSI control tool including how to start and stop a target A Sample iSCSI Configuration File The default Chelsio iSCSI configuratio...

Страница 158: ...the Chelsio iSCSI software is done through iscsictl the command line interface control tool The following are the basic commands needed for effective control of the target Start Target To start all o...

Страница 159: ...ngs execute iscsictl with G option root host iscsictl G iscsi_login_complete_time 300 View Help To print help to stdout execute iscsictl with h option root host iscsictl h 4 4 The iSCSI Configuration...

Страница 160: ...ead of commas is to list their values as key value pairs repeatedly each on a new line until they are all listed There are three categories of key value parameter the first category belongs to the glo...

Страница 161: ...llowed secret must be between 6 and 255 characters Commas are not allowed The target user id and secret are used by the initiator to authenticate the target while doing mutual chap NOTE The double quo...

Страница 162: ...ult Value Multiple Values Description MaxConnections 1 to 65535 1 No Initiator and target negotiate the maximum number of connections requested acceptable InitialR2T Yes No Yes No To turn on or off th...

Страница 163: ...er nor is it for authentication Target only MaxRecvDataSegmentLengt h 512 to 16777215 224 1 8192 No To declare the maximum data segment length in bytes it can receive in an iSCSI PDU OFMarker Yes No N...

Страница 164: ...be less than 256 characters Commas are not allowed secret must be between 6 and 255 characters Commas are not allowed The initiator user id and secret are used by the target to authenticate the initi...

Страница 165: ...There can be multiple redirection target portalgroups specified for a particular target portal group and the redirection will happen to these in a round robin manner ShadowMode Yes No No No To turn Sh...

Страница 166: ...evice should be created as a RAM Disk size xMB is used with MEM to specify the RamDisk size If not specified the default RamDisk size is 16MB 16 Megabytes The minimum value of x is 1 1MB and the maxim...

Страница 167: ...ore initiator names the name must be a fully qualified iSCSI initiator name sip Source IP address specifies one or more IP addresses the initiators are connecting from Dip Destination IP address speci...

Страница 168: ...get irrelevant entity block can be removed or commented Chelsio iSCSI Global Settings global iscsi_login_complete_time 300 iscsi_auth_order CHAP DISC_AuthMethod None DISC_Auth_CHAP_Policy Oneway DISC_...

Страница 169: ...gth 65535 MaxBurstLength 262144 Local block devices being served up lun 0 is pointed to dev sda lun 1 is pointed to dev sdb TargetDevice dev sda ID aabbccddeeffgghh WWN aaabbbcccdddeeef TargetDevice d...

Страница 170: ...AP CHAP is a protocol that is used to authenticate the peer of a connection and uses the notion of a challenge and response i e the peer is challenged to prove its identity The Chelsio iSCSI software...

Страница 171: ...cret are configured for use in the case of mutual CHAP AuthMethod and Auth_CHAP_Policy Keys By setting the iSCSI keys AuthMethod and Auth_CHAP_Policy a user can choose whether to enforce CHAP and if m...

Страница 172: ...way authentication DISC_Auth_CHAP_Target target CHAP user id and secret DISC_Auth_CHAP_Initiator initiator CHAP user id and secret Sample Chelsio iSCSI Global Settings global DISC_AuthMethod CHAP DIS...

Страница 173: ...via the target portal 102 60 60 25 is allowed full read write access to this target ACL dip 102 60 60 25 initiator iqn 2005 09 com chelsio san2 from 102 50 50 22 and connected via the target portal 1...

Страница 174: ...tion is useful for measuring network performance The details of the parameters for the key TargetDevice are found in the table of Chelsio Entity Settings section earlier in this document RAM Disk Deta...

Страница 175: ...th to the storage device FILE SYNC Where path Is the path to the actual storage device such as dev sdb for a block device or dev md0 for a software RAID The path must exist in the system SYNC When spe...

Страница 176: ...umeric string The user should make sure the three values listed above are the same for the target LUNs involved in the multipath 4 8 Target Redirection Support An iSCSI Target can redirect an initiato...

Страница 177: ...ortalGroup 1 10 193 184 81 3260 2 PortalGroup 2 10 193 184 85 3261 the PortalGroup 2 IS present on the same system ShadowMode No snip Redirecting to Multiple Portal Groups The Chelsio iSCSI Target Red...

Страница 178: ...client This is a simple tool that starts the iSNS client with a client and server parameter iscsictl options Options Mandatory Parameters Optional Parameters Description h Display the help messages v...

Страница 179: ...on s will be displayed If target ALL is specified or no parameters are specified the output will be abbreviated Specify specific targets to get detailed configuration data If the target name option is...

Страница 180: ...reject only the invalid parameters but will continue on and complete all other valid parameters if any others are specified s target name Stop the specified active iSCSI targets target name parameter...

Страница 181: ...pecified targets are not currently active they will be started For Rules 2 3 please note the differences they are not the same The global settings are also reloaded from the configuration file with th...

Страница 182: ...05 id isnscln2 query 30 In the first example the minimum command set is given where the IP address of the iSNS server is specified In the second example a fully qualified command is specified by also...

Страница 183: ...arted its settings can be modified via reloading of the configuration file i e iscsictl S The following parameters cannot be changed once the target is up and running otherwise the target reload would...

Страница 184: ...ommand line iscsictl G The finer points of some of these parameters are described in detail here iscsi_login_complete_time Options An integer value between 0 and 3600 seconds Default value is 300 seco...

Страница 185: ...sed to verify the initiator name received and identify if the initiator is running Chelsio drivers if the initiator name contains the same substring as iscsi_chelsio_ini_idstr it is assumed the initia...

Страница 186: ...Chapter X iSCSI PDU Offload Target Chelsio Unified Wire for Linux 186 Software Driver Unloading Use the following command to unload the module root host rmmod chiscsi_t4...

Страница 187: ...Chapter XI iSCSI PDU Offload Initiator Chelsio Unified Wire for Linux 187 XI iSCSI PDU Offload Initiator...

Страница 188: ...Data Out PDUs PDU Transmit and Recovery On transmit side Chelsio hardware accepts the complete PDU header data from the host driver computes and inserts the digests decomposes the PDU into multiple TC...

Страница 189: ...lable for the following versions RHEL 7 5 3 10 0 862 el7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8...

Страница 190: ...b4i runs on top of NIC module cxgb4 and open iscsi 2 0 872 873 874 only on a Chelsio card openssl devel package should be installed 2 2 Installation i Change your current working directory to Chelsio...

Страница 191: ...ail Run the following command to load the driver root host modprobe cxgb4i If loading of cxgb4i displays unkown symbols found error in dmesg follow the steps mentioned below i Kill iSCSI daemon iscsid...

Страница 192: ...gb4i in the following format iface iscsi_ifacename iface file name iface hwaddress MAC address iface transport_name cxgb4i iface net_ifacename ethX iface ipaddress iscsi ip address Here iface iscsi_if...

Страница 193: ...st p 102 2 2 155 3260 I cxgb4i 00 07 43 04 5b da iii Logging into an iSCSI Target Log into an iSCSI target using the following format root host iscsiadm m node T iqn name of target p target ip address...

Страница 194: ...dm m node T iqn name of target p target ip address target port no I cxgb4i iface file name u Example root host iscsiadm m node T iqn 2004 05 com chelsio target1 p 102 2 2 155 3260 1 I cxgb4i 00 07 43...

Страница 195: ...ctions 4 3 Auto login from cxgb4i initiator at OS bootup For iSCSI auto login via cxgb4i to work on OS startup please add the following line to start in etc rc d init d iscsid file on RHEL modprobe q...

Страница 196: ...nfiguration file and reboot the machine scsi_mod use_blk_mq 1 ii Run the performance tuning script to map iSCSI Initiator queues to different CPUs root host t4_perftune sh Q iSCSI n iii Load initiator...

Страница 197: ...Chapter XI iSCSI PDU Offload Initiator Chelsio Unified Wire for Linux 197 Software Driver Unloading To unload the driver execute the following commands root host rmmod cxgb4i root host rmmod libcxgbi...

Страница 198: ...Chapter XII Crypto Offload Chelsio Unified Wire for Linux 198 XII Crypto Offload...

Страница 199: ...Hardware Requirements Supported adapters The following are the currently shipping Chelsio adapters that are compatible with Chelsio s Crypto Offload driver T62100 CR T62100 LP CR T6425 CR T6225 CR T6...

Страница 200: ...CONFIG_LSM_MMAP_MIN_ADDR 65536 CONFIG_SECURITY_SELINUX y CONFIG_SECURITY_SELINUX_BOOTPARAM y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE 1 CONFIG_SECURITY_SELINUX_DISABLE y CONFIG_SECURITY_SELINUX_DEVELOP...

Страница 201: ...TO_GHASH y CONFIG_CRYPTO_MD4 m CONFIG_CRYPTO_MD5 y CONFIG_CRYPTO_SHA1 y CONFIG_CRYPTO_SHA256 y CONFIG_CRYPTO_SHA512 y CONFIG_CRYPTO_AES y CONFIG_CRYPTO_AES_X86_64 y CONFIG_CRYPTO_DEFLATE y CONFIG_CRYP...

Страница 202: ...kernel patch v Copy the patched kernel source to usr src kernels root host cp r root rpmbuild BUILD kernel 3 10 0 514 el7 linux 3 10 0 514 el7 usr src kernels vi Compile and install the kernel root ho...

Страница 203: ...that SELinux and firewall are disabled 3 2 Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Crypto driver and...

Страница 204: ...ost modprobe cxgb4 root host modprobe chcr ii Bring up the Chelsio network interface root host ifconfig ethX up Where ethX is the Chelsio interface 4 2 Inline i To load the Crypto Offload driver in In...

Страница 205: ...chcr The following applications can be offloaded by Chelsio Co processor Apache Dmcrypt SFTP OpenVPN Strongswan To enable Digests in RHEL 7 3 x86_64 base 3 10 0 514 el7 kernel uncomment the DIGESTS li...

Страница 206: ...Follow the steps mentioned below i Create a new policy file and add the following line for each TCP port to be TLS offloaded src or dst port tcp_port offload tls mss 32 bind random nagle all offload...

Страница 207: ...installed in usr chssl bin OpenSSL tool i Start TLS offload Server root host cd usr chssl bin root host openssl s_server key key_file cert cert_file accept 443 cipher AES128 GCM SHA256 WWW Example ii...

Страница 208: ...r local nginx nginx conf path usr local nginx nginx conf pid path usr local nginx nginx pid with http_ssl_module with ld opt L usr chssl lib L usr local lib with openssl ChelsioUwire x x x x src chope...

Страница 209: ...3 Performance Tuning Apply the performance settings mentioned in the Performance Tuning section in the Unified Wire chapter before proceeding Inline TLS i Run the performance tuning script to map TLS...

Страница 210: ...nloading To unload Crypto Offload driver in Co processor mode run the following command root host rmmod chcr To unload Crypto Offload driver in Inline mode unload the network driver in TOE mode See So...

Страница 211: ...Chapter XIII Data Center Bridging DCB Chelsio Unified Wire for Linux 211 XIII Data Center Bridging DCB...

Страница 212: ...working resources reduced heat and noise and less power consumption Administration is simplified since the specifications enable transport of storage and networking traffic over a single unified Ether...

Страница 213: ...Wire for Linux 213 Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic Kernel org linux 4 14 67 Kernel org linux 4 9 Minimum 4 9 kernel version supported is 4 9 13 Other kernel versions...

Страница 214: ...ware Driver Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Build and install all drivers with DCB support root host...

Страница 215: ...itch ports need to be enabled with DCBX configuration Class mapping ETS and PFC Upon loading the network storage driver and interface bringup firmware completes DCBX negotiation with the switch root h...

Страница 216: ...E Full Offload Initiator uses the DCBX negotiated parameters ETS PFC etc without any further configuration The network drivers cxgb4 t4_tom and iSCSI drivers cxgb4i chiscsi need further VLAN configura...

Страница 217: ...iorities NIC uses priority 0 and FcoE uses priority 3 switch config class map type qos class nic switch config cmap qos match cos 0 switch config cmap qos class map type qos class fcoe switch config c...

Страница 218: ...cent 0 vii Configure network qos policy maps and set up the PFC for no drop traffic class switch config policy map type network qos policy test switch config pmap nq class type network qos class nic s...

Страница 219: ...switch config vsan database switch config vsan db vsan 2 switch config vsan db vsan 2 interface vfc 13 switch config vsan db exit iii Assign VLAN ID to the Ethernet port on which FCoE service was enab...

Страница 220: ...r and the target on that particular VSAN although FLOGI d into the switch will not talk to each other To enable it execute the below command switch config zone default zone permit vsan 2 4 2 Configuri...

Страница 221: ...ule 2 proto fip encap ethv2 switch config vlan classifier group 1 add rule 1 switch config vlan classifier group 1 add rule 2 switch config interface vlan 1002 switch conf if vl 1002 fcf forward switc...

Страница 222: ...unning NIC and iSCSI traffic and configure corresponding VLAN priority Example Switch is configured with a VLAN priority of 2 and 5 for NIC and iSCSI class of traffic respectively NIC traffic is run o...

Страница 223: ...Chapter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 223 XIV FCoE Full Offload Initiator...

Страница 224: ...FC0 and FC1 layers of the Fibre Channel stack with Ethernet By retaining the native Fibre Channel constructs FCoE will integrate with existing Fibre Channel networks and management software 1 1 Hardwa...

Страница 225: ...tallation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install FCoE full offload initiator driver root host make fcoe_full_off...

Страница 226: ...i cxgbit iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver execute the following roo...

Страница 227: ...following command to see if the FIP has gone through and a VN_Port MAC address has been assigned Verify if all the FCoE ports are online ready and a successful FIP has taken place using the following...

Страница 228: ...w mentioned steps i Determine the WWPN of the initiator local port under sysfs The hosts under fc_host depends on the number of ports on the adapter used root host cat sys class fc_host hostX port_nam...

Страница 229: ...t host cat sys kernel debug csiostor pci_id lnodes After finding out the WWPN of the local node to verify the list of discovered targets use the following command root host cat sys kernel debug csiost...

Страница 230: ...Chapter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 230...

Страница 231: ...apter XIV FCoE Full Offload Initiator Chelsio Unified Wire for Linux 231 4 3 Formatting the LUNs and Mounting the Filesystem Use lsscsi g to list the LUNs discovered by the initiator root host lsscsi...

Страница 232: ...covered by the Chelsio FCoE initiators can be accessed via easily identifiable udev path device files like root host ls dev disk by path pci 0000 04 00 0 csio fcoe local_wwpn remote_wwpn lun_wwn 4 4 C...

Страница 233: ...FCoE Full Offload Initiator Chelsio Unified Wire for Linux 233 4 5 Mounting the formatted LUN The formatted LUN can be mounted on the specified mountpoint using the following command root host mount...

Страница 234: ...helsio Unified Wire for Linux 234 Software Driver Unloading To unload the driver root host modprobe r csiostor If multipath services are running unload of FCoE driver is not possible Stop the multipat...

Страница 235: ...Chapter XV Offload Bonding Chelsio Unified Wire for Linux 235 XV Offload Bonding...

Страница 236: ...can be fully offloaded to the adapter thus freeing the CPU from TCP IP overhead 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatib...

Страница 237: ...693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 ge...

Страница 238: ...r Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Chelsio Offload bonding driver root host make bonding_inst...

Страница 239: ...chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver with offload support run the following comm...

Страница 240: ...the following method i Load the network driver with TOE support root host modprobe t4_tom ii Create a bonded interface root host modprobe bonding mode 1 miimon 100 iii Bring up the bonded interface a...

Страница 241: ...Chapter XV Offload Bonding Chelsio Unified Wire for Linux 241 Software Driver Unloading To unload the driver run the following command root host rmmod bonding...

Страница 242: ...Chapter XVI Offload Multi Adapter Failover MAFO Chelsio Unified Wire for Linux 242 XVI Offload Multi Adapter Failover MAFO...

Страница 243: ...fic running over the bonded interface can be fully offloaded to the adapter thus freeing the CPU from TCP IP overhead 1 1 Hardware Requirements Supported Adapters The following are the currently shipp...

Страница 244: ...5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 generic Ubuntu 16 04 4 4 4 0 116 generic...

Страница 245: ...x 245 Software Driver Installation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install MAFO feature root host make bonding_in...

Страница 246: ...iw_cxgb4 chcr cxgb4vf cxgb4 libcxgbi libcxgb The driver must be loaded by the root user Any attempt to load the driver as a regular user will fail To load the driver with offload support run the foll...

Страница 247: ...modprobe bonding mode 1 miimon 100 iii Bring up the bonded interface and enslave the interfaces to the bond root host ifconfig bond0 up root host ifenslave bond0 ethX ethY iv Assign IPv4 IPv6 address...

Страница 248: ...Chapter XVI Offload Multi Adapter Failover MAFO Chelsio Unified Wire for Linux 248 Software Driver Unloading To unload the driver run the following command root host rmmod bonding...

Страница 249: ...Chapter XVII UDP Segmentation Offload and Pacing Chelsio Unified Wire for Linux 249 XVII UDP Segmentation Offload and Pacing...

Страница 250: ...ividually through the network stack and schedule individual frame transmission resulting in millions of system calls and packet traversals through all protocol layers in the operating system to the ne...

Страница 251: ...0 CR T520 LL CR T520 BT T420 CR T440 CR T422 CR T404 BT T420 BCH T440 LP CR T420 BT T420 LL CR T420 CX 1 2 Software Requirements Linux Requirements Currently the UDP Segmentation Offload and Pacing dr...

Страница 252: ...ld and install UDP Offload drivers which support large number of offload connections approx 10K i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUw...

Страница 253: ...oot user Any attempt to load the driver as a regular user will fail Run the following commands to load the driver root host modprobe cxgb4 root host modprobe t4_tom Though normally associated with the...

Страница 254: ...e data stream to This value needs to be in the range of 0 to 14 for T4 series of adapters and a range of 0 to 15 for T5 T6 series of adapters The application will function as per the parameters set fo...

Страница 255: ...teps mentioned above i In step ii declare and initialize a new variable in the application int rtp_header_size 16 Here rtp_header_size is the RTP header size in bytes that the application sends ii In...

Страница 256: ...t payload size in bytes it should be equal to the value set in the SO_FRAMESIZE socket option in the application in section 4 1 Example The user wants to transfer UDP data on port 0 of the adapter usi...

Страница 257: ...entation Offload and Pacing Chelsio Unified Wire for Linux 257 Software Driver Unloading Reboot the system to unload the driver To unload without rebooting refer Unloading the TOE driver section of Ne...

Страница 258: ...Chapter XVIII Offload IPv6 Chelsio Unified Wire for Linux 258 XVIII Offload IPv6...

Страница 259: ...ort to fully offload IPv6 traffic to the Unified Wire adapter 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with Chelsio Of...

Страница 260: ...693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4 4 21 69 default Ubuntu 18 04 1 4 15 0 29 ge...

Страница 261: ...nabled in your system enabled by default Unified Wire must be installed with IPv6 support as explained in the Unified Wire chapter 2 2 Installation i Change your current working directory to Chelsio U...

Страница 262: ...xgb After installing Unified Wire package and rebooting the host load the NIC cxgb4 and TOE t4_tom drivers The drivers must be loaded by the root user Any attempt to load the drivers as a regular user...

Страница 263: ...d capable drivers root host modprobe t4_tom ii Assign IPv6 address and bring up the interface root host ifconfig ethX IPv6 address up iii All the IPv6 traffic over the Chelsio interface will be offloa...

Страница 264: ...oading 5 1 Unloading the NIC Driver To unload the NIC driver run the following command root host rmmod cxgb4 5 2 Unloading the TOE Driver Please reboot the system to unload the TOE driver To unload wi...

Страница 265: ...Chapter XIX WD Sniffing and Tracing Chelsio Unified Wire for Linux 265 XIX WD Sniffing and Tracing...

Страница 266: ...icast traffic and sending it directly to user space a Get a Queue raw QP idx b Program a filter to redirect specific traffic to the raw QP queue Tracer All tapped traffic is forwarded to user space an...

Страница 267: ...Linux 267 Schematic diagram of sniffer and tracer 1 1 Hardware Requirements Supported Adapters The following are the currently shipping Chelsio adapters that are compatible with the tools T62100 CR T...

Страница 268: ...nd Tracing utility is available for the following version RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 SLES 15 4 12 14 23 default SLES 12 SP3 4 4 73 5 default SLES 12 SP2 4...

Страница 269: ...tallation i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install Sniffer tracing filtering tcpdump and iWARP driver root host ma...

Страница 270: ...ne having a T5 T4 card The machines should be setup in the following manner Machine A Machine B 192 168 1 100 192 168 1 200 2 Procedure On the Device Under Test DUT start sniffer root host wd_sniffer...

Страница 271: ...dure Run wd_tcpdump_trace i iface on the command prompt where iface is one of the interfaces whose traffic you want to trace In the above diagram its port 0 or port 1 root host wd_tcpdump_trace i ifac...

Страница 272: ...Chapter XX Classification and Filtering Chelsio Unified Wire for Linux 272 XX Classification and Filtering...

Страница 273: ...ccess Control List accept criteria to a particular receive queue Switch proxy ingress packets that meet ACL accept criteria to an output port with optional header rewrite Drop ingress packets that mee...

Страница 274: ...7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32 696 el...

Страница 275: ...will be from 0 to 63 and normal indices will be from 64 to 559 It is mandatory to add prio 1 option when creating high priority filter rules 2 1 Configuration Filter Modes The Classification and Filt...

Страница 276: ...ntation mpshittype macmatch ethertype port fcoe 6 fragmentation mpshittype macmatch protocol tos port fcoe 7 fragmentation mpshittype macmatch protocol vlan fcoe 8 fragmentation mpshittype macmatch pr...

Страница 277: ...ol vlan port 41 mpshittype macmatch protocol vnic_id port 42 mpshittype macmatch tos vlan port 43 mpshittype macmatch tos vnic_id port 44 mpshittype ethertype vlan port fcoe 45 mpshittype ethertype vn...

Страница 278: ...ter rules using cxgbtool root host cxgbtool ethx filter index action pass drop switch prio 1 hitcnts 1 Where ethX Chelsio interface index positive integer set as filter id 0 495 for T5 adapters 0 559...

Страница 279: ...on switch iport 0 eport 1 ivlan 3 The above filter rule will route all ingress packets that match VLAN id 3 from port 0 of Chelsio adapter to port 1 Remaining packets will be sent to the host prio opt...

Страница 280: ...up root host ifconfig eth0 2 102 1 1 2 24 up root host ifconfig eth0 3 102 1 1 3 24 up root host ifconfig eth0 eth0 Link encap Ethernet HWaddr 00 07 43 04 7D 50 inet addr 102 1 1 1 Bcast 102 1 1 255 M...

Страница 281: ...ROMISC MULTICAST MTU 1500 Metric 1 ii Setup a static OR default route towards T6 T5 router to reach 102 1 1 0 24 network root host route add net 102 1 1 0 24 gw 102 1 2 250 Follow these steps on machi...

Страница 282: ...cxgbtool eth0 filter 101 lip 102 1 1 0 24 hitcnts 1 action switch eport 0 smac 00 07 43 04 96 40 dmac 00 07 43 04 7D 50 Where smac is the MAC address of eth0 interface on T6 T5 adapter machine and dm...

Страница 283: ...1 1 1 root host arp s 102 1 1 1 00 07 43 04 7D 50 Follow these steps on machine with T6 T5 adapter i Update filterMode value with below combination in lib firmware cxgb4 t6 config txt to enable matchi...

Страница 284: ...switch it out via eport 0 root host cxgbtool eth0 filter 101 macidx 7 action switch eport 0 hitcnts 1 2 7 Filtering VF traffic To filter VF traffic replace the default filterMode in the firmware conf...

Страница 285: ...odprobe cxgb4 root host echo 4 sys class net ethX device driver bus_id sriov_numvfs ii 1 VM was brought up with VF2 cxgb4vf was loaded on the VM and the VF was brought up root host ifconfig enp8s2 enp...

Страница 286: ...ure is configured by specifying the filter modes in the firmware configuration file located in lib firmware cxgb4 Adapter initialization will fail if filterMask contains a tuple which is not present i...

Страница 287: ...rt fcoe 15 fragmentation macmatch protocol vnic_id port fcoe 16 fragmentation protocol tos vlan port fcoe 17 fragmentation protocol tos vnic_id port fcoe 18 mpshittype macmatch ethertype protocol port...

Страница 288: ...lip destination_ip fport source_port lport destination_port proto protocol hitcnts 1 cap maskless Where ethX Chelsio interface index Filter index For LE TCAM filters filter index should be 0 495 for T...

Страница 289: ...tioned in filterMask Otherwise filter rule will not honour iii Now to create a hash filter based on the filterMode and filterMask values selected above root host cxgbtool eth18 filter 100 action drop...

Страница 290: ...frag 0 matchtype 0 iport 0 fcoe 0 hitcnts 1 cap maskless Hash Filter Index 196568 Examples drop action root host cxgbtool ethX filter 496 action drop lip 102 1 1 1 fip 102 1 1 2 lport 12865 fport 200...

Страница 291: ...s parameter and corresponding filter rule index root host cxgbtool ethX filter index delete clear cap maskless 3 5 Filter Priority By default Hash DDR filter has priority over LE TCAM filter To overri...

Страница 292: ...parameter enable_mirror can be used to enable mirroring of traffic running on physical ports The mirrored traffic will be received via Mirror PF VF on Mirror Receive queues which will then inject this...

Страница 293: ...tered Traffic Mirroring Once mirroring is enabled all the traffic received on a physical port will be duplicated The following example explains the method to filter out the redundant traffic and recei...

Страница 294: ...t cxgbtool ethX reg 0x09800 0x13 iv Setup a trace filter root host echo tx1 snaplen 40 sys kernel debug cxgb4 bus_id trace0 Here snaplen is the length in bytes to be captured The above step will trace...

Страница 295: ...sys kernel debug cxgb4 bus id sge_qinfo file root host cxgbtool ethX reg 0x09808 Trace Queue0 RspQ ID iv Similarly for other direction and for multiple ports run the follow commands root host echo rx...

Страница 296: ...new_destination_port Where ethX Chelsio interface source_ip port Source IP port of incoming packet destination_ip port Destination IP port of incoming packet new_source_ip port Source IP port to be t...

Страница 297: ...Chapter XXI OVS Kernel Datapath Offload Chelsio Unified Wire for Linux 297 XXI OVS Kernel Datapath Offload...

Страница 298: ...offloading OpenFlow and non OpenFlow network traffic simultaneously including tunnel handling e g VXLAN IPsec NAT IP stack ARP route lookup frag tracking fragment defragment and other kernel function...

Страница 299: ...quirements The Chelsio OVS driver has been developed to run on the following 64 bit Linux platforms RHEL 7 5 3 10 0 862 el7 RHEL 7 4 3 10 0 693 el7 Kernel org linux 4 9 Minimum 4 9 kernel version supp...

Страница 300: ...4 packages should be installed For the complete list of software required visit http docs openvswitch org en latest intro install general 2 2 Installation i Change your current working directory to Ch...

Страница 301: ...t L2 Ethernet Type L3 IP Protocol Type L3 IPv4 address L3 IPv6 address L3 IPv4 TOS L3 IP Fragmentation L4 Ports tcp udp src port dst port Tunnel Encapsulation VNI only on T6 Supported Actions The foll...

Страница 302: ...adapters and t5 config txt for T5 adapters filterMode fragmentation ethertype protocol tos port filterMask fragmentation ethertype protocol tos port iii Load NIC cxgb4 driver with hash filter support...

Страница 303: ...vSwitch manager_options bootstrap ca cert db Open_vSwitch SSL ca_cert pidfile detach log file root host ovs vsctl no wait init root host export DB_SOCK usr local var run openvswitch db sock root host...

Страница 304: ...protocol tos port Wild card flow to drop incoming packets on first port root host ovs ofctl add flow br0 in_port 1 dl_type 0x800 action drop Wild card flow to switch ARP packets L2 EtherType 0x0806 o...

Страница 305: ...flow to switch 4 tuple IPv4 traffic with TOS 0xE0 root host ovs ofctl add flow br0 in_port 1 dl_type 0x800 nw_proto 6 nw_src 10 1 1 66 tp_src 11000 nw_dst 10 1 1 58 tp_dst 21000 nw_tos 0xE0 action ou...

Страница 306: ...he mod_vlan_vid tag will be added with a priority of 0 Modify VLAN priority and switch traffic root host ovs ofctl O OpenFlow11 add flow br0 in_port 1 dl_type 0x800 vlan_pcp 4 action mod_vlan_pcp 3 ou...

Страница 307: ...cal var run openvswitch db sock remote db Open_vSwitch Open_vSwitch manager_options bootstrap ca cert db Open_vSwitch SSL ca_cert pidfile detach log file root host ovs vsctl no wait init root host exp...

Страница 308: ...rifying OVS Flow Dump OVS flow dump can be verified using root host ovs ofctl dump flows br0 Run traffic between hosts which matches the flow and verify if the n_packet counter is incrementing 3 4 Set...

Страница 309: ...pen a web browser and enter the address http localhost 8080 ix Login with admin keyword for both username and password x On the OVS machine add the bridge to the controller and disable in band root ho...

Страница 310: ...th Offload Chelsio Unified Wire for Linux 310 Software Driver Uninstallation i Change your working directory to Chelsio Unified Wire directory root host cd ChelsioUwire x x x x ii Uninstall OVS driver...

Страница 311: ...Chapter XXII Ring Backbone Chelsio Unified Wire for Linux 311 XXII Ring Backbone...

Страница 312: ...o adapters that are compatible with the Chelsio Ring Backbone utility T61100 OCP T62100 CR T62100 LP CR T62100 SO CR T6425 CR T6225 CR T6225 LL CR T6225 OCP T6225 SO CR T580 OCP SO T520 OCP SO T580 CR...

Страница 313: ...version supported is 4 9 13 Other versions have not been tested and are not guaranteed to work 1 3 Ring Connectivity Connect the machines in ring backbone Connect port 1 of one machine to the port 0 o...

Страница 314: ...tion Install Unified Wire on all the machines in the ring backbone i Change your current working directory to Chelsio Unified Wire package directory root host cd ChelsioUwire x x x x ii Install the dr...

Страница 315: ...e cxgb4 enable_ringbb 1 ii Check dmesg output and note the mirror VF ID root host dmesg grep i mirror iii Bring up port 0 and port 1 interfaces but assign IP address to only port 0 iv Check the MPS TC...

Страница 316: ...nterface_name vi Press any key when prompted to complete the configuration You should be able to run traffic between the machines nodes now Automatic reconfiguration is currently not supported In case...

Страница 317: ...Chapter XXIII Traffic Management Chelsio Unified Wire for Linux 317 XXIII Traffic Management...

Страница 318: ...ntrol max bandwidth that a connection or a flow a group of connections can use Allocate available bandwidth to several connection or flows based on desired levels of performance Once the offload trans...

Страница 319: ...l7 RHEL 7 5 3 10 0 862 el7 ppc64le POWER8 LE RHEL 7 5 4 14 0 49 el7a aarch64 ARM64 RHEL 7 4 3 10 0 693 el7 RHEL 7 4 3 10 0 693 el7 ppc64le POWER8 LE RHEL 7 3 4 5 0 15 el7 aarch64 ARM64 RHEL 6 9 2 6 32...

Страница 320: ...t can be performed on non offloaded connections as well as on offloaded connections The drivers must be loaded by the root user Any attempt to load the drivers as a regular user will fail Run the foll...

Страница 321: ...mode channel Channel No class scheduler class index max rate maximum rate pkt size Packet size Here ethX Chelsio interface scheduler mode specifies whether the rule is configured for individual traff...

Страница 322: ...hierarchy Possible values include 1 to 99 Channel Rate Limiting This scheduler hierarchy level can be used to rate limit individual channels Channel rate limiting can be configured using the followin...

Страница 323: ...nfig ethX up ii Bind connections to queues root host tc qdisc add dev ethX root handle 1 multiq root host tc filter add dev ethX parent 1 protocol ip prio 1 u32 match ip dst IP address of destination...

Страница 324: ...onnections with the given scheduling class Example src host 102 1 1 1 offload class 0 The above example will associate all connections originating from IP address 102 1 1 1 with scheduling class 0 iii...

Страница 325: ...function as per the parameters set for that traffic class iii Add socket option definitions In order to use setsockopt to set the options to the TCP socket the following two definitions need to be ma...

Страница 326: ...multiq root host tc filter add dev eth0 parent 1 protocol ip prio 1 u32 match ip dst 192 168 5 3 action skbedit queue_mapping 3 iii Bind the NIC TX queue to class 0 root host cxgbtool eth0 sched queu...

Страница 327: ...l mode class rate unit bits rate mode absolute channel 0 class 0 max rate 300000 pkt size 1460 4 3 Offloaded Connections with Modified Application The following example demonstrates the method to rate...

Страница 328: ...Traffic Management Chelsio Unified Wire for Linux 328 Software Driver Unloading Reboot the system to unload the driver To unload without rebooting refer Unloading the TOE driver section of Network NI...

Страница 329: ...Chapter XXIV DPDK Driver Chelsio Unified Wire for Linux 329 XXIV DPDK Driver...

Страница 330: ...romiscuous mode All multicast mode Port hardware statistics Jumbo frames only for UIO 1 1 Hardware Requirements The following are the currently shipping Chelsio adapters that are compatible with DPDK...

Страница 331: ...e Chelsio DPDK driver has been developed to run on 64 bit Linux platforms Following is the list of supported distributions Linux Distribution Driver Software RHEL 7 5 3 10 0 862 el7 UIO VFIO RHEL 7 4...

Страница 332: ...host make dpdk_install The above command will install the driver with the following build configuration arch x86_64 ppc_64 machine native power8 execenv linuxapp toolchain gcc create a target environ...

Страница 333: ...mware Configuration File i If not done already load the kernel mode NIC driver cxgb4 root host modprobe cxgb4 ii Flash the firmware configuration file for T6 adapters root host cxgbtool iface loadcfg...

Страница 334: ...mentioned below to load DPDK driver with VFIO support i Enable Intel VT d in system BIOS ii Enable Intel iommu by adding the below entry to kernel command line in grub grub2 menu intel_iommu on vfio_i...

Страница 335: ...es_hugepages sh Manual i Mount hugetlbfs root host mkdir p mnt huge root host mount t hugetlbfs nodev mnt huge ii Reserve Huge Page memory manually x86_64 root host echo 1024 sys devices system node n...

Страница 336: ...devbind py bind igb_uio PCI ID PF4 VFIO support root host dpdk devbind py bind vfio pci PCI ID PF4 Now verify using root host dpdk devbind py status Examples Binding with UIO support Binding with VFI...

Страница 337: ...an be assigned to a Virtual Machine VM or can be used in the Hypervisor Host Execute the following commands on the VM Host to bind the VF to DPDK environment UIO support root host dpdk devbind py bind...

Страница 338: ...ing the Power Profile to Maximum Performance in BIOS settings Installing the adapter into a PCIe Gen3 x8 x16 slot Allocating required number of mbufs in the application based on the below formula nb_m...

Страница 339: ...res to run on Note that core numbering can change between platforms and should be determined beforehand n channels Number of memory channels per processor socket i Enable interactive mode nb cores cor...

Страница 340: ...by default and can be disabled via Testpmd as follows testpmd port config all rss none To enable again run testpmd port config all rss ip tcp udp Jumbo Mode There are multiple ways to enable sending a...

Страница 341: ...of displaying real time metrics for ports and can handle packets with UDP TCP ARP ICMP GRE MPLS and Queue in Queue The application can run command scripts to set up repeatable test cases Syntax Execut...

Страница 342: ...erved Powered by Intel DPDK Flags Port P 0 Link State UP 10000 FD TotalRate Pkts s Rx 0 0 Tx 14880593 14880593 MBits s Rx Tx 0 9999 0 9999 Broadcast 0 Multicast 0 64 Bytes 0 65 127 0 128 255 0 256 511...

Страница 343: ...4 Tx and Rx queues mapping cores 1 4 to Rx and 5 8 to Tx queues on port 0 root host pktgen c 3ff n 4 T P m 1 4 5 8 0 N ii Set packet size to 64B on all ports Pktgen set all size 64 iii Start Tx traff...

Страница 344: ...stop 0 6 3 Runtime Options The following devargs options can be used while running the applications force_link_up force_link_up devargs option always forces Link as up for VFs This enables VFs on the...

Страница 345: ...PDK driver with UIO support root host rmmod igb_uio root host rmmod uio If unloading uio module reports an error unload the following dependent modules and try again root host rmmod bnx2fc root host r...

Страница 346: ...fied Wire for Linux 346 Software Driver Uninstallation i Change your working directory to Chelsio Unified Wire directory root host cd ChelsioUwire x x x x ii Uninstall the DPDK driver using the follow...

Страница 347: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 347 XXV Unified Boot...

Страница 348: ...alities like PXE FCoE and iSCSI SAN boot This section of the guide also describes the use and configuration of Chelsio s DUD for OS installations via PXE server on FC FCoE LUN and iSCSI LUN This solut...

Страница 349: ...7124S F Mellanox SX_PPC_M460EX Other platforms switches have not been tested and are not guaranteed to work Supported Adapters Following are the currently shipping Chelsio adapters that are compatibl...

Страница 350: ...f Drivers Software and supported Linux distributions Linux Distribution Driver Software DUDs RHEL 7 5 3 10 0 862 el7 PXE FCoE iSCSI RHEL 7 4 3 10 0 693 el7 RHEL 6 9 2 6 32 696 el6 PXE iSCSI SLES 15 4...

Страница 351: ...ngs vi Select Advanced Secure Boot Options vii Provide the Platform Key PK Key Exchange Key KEK and Allowed Signature Database DB to the respective uEFI NVRAM variables Windows PK Will be generated at...

Страница 352: ...SB flash drive as a storage media for the necessary files Follow the steps below to prepare the drive i Create a DOS bootable USB flash drive Click here for instructions ii Create a directory CHELSIO...

Страница 353: ...nce the system boots from the USB flash drive change your working directory to CHELSIO directory C cd CHELSIO iii Run the following command to list all Chelsio CNAs present on the system The list disp...

Страница 354: ...y previous firmware using the following command C CHELSIO cfut4 d idx xh xf vi Delete any previous Option ROM settings C CHELSIO cfut4 d idx xc vii Run the following command to flash the appropriate f...

Страница 355: ...355 viii Flash the unified option ROM onto the Chelsio CNA using the following command C CHELSIO cfut4 d idx ub cubt4 bin Here cubt4 bin is the unified option ROM image file present in the CHELSIO di...

Страница 356: ...below command C CHELSIO cfut4 d idx um Hex MAC Address Example C CHELSIO cfut4 d 0 um 000743000123 3 3 uEFI i Reboot the system and go into BIOS setup ii Disable Secure Boot iii Configure the system h...

Страница 357: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 357 iv Boot to EFI Shell...

Страница 358: ...rivers to determine if Chelsio uEFI driver is loaded If the driver is loaded as shown in the image below continue to step vi If the driver is not loaded load the uEFI driver ChelsioUD efi found in the...

Страница 359: ...Linux 359 vii Chelsio HII should be listed as Chelsio T5 T6 Highlight it and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as mentioned here in...

Страница 360: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 360 viii Highlight the Chelsio adapter to be configured and press Enter...

Страница 361: ...as Flash File Type c Select Update Erase d Press Y to confirm e Reboot system Update firmware a Select Update as Flash Operation b Select FW File as Flash File Type c Enter full path to the firmware f...

Страница 362: ...drive as described here ii Boot system to EFI shell and change your working directory to CHELSIO iii Issue command drivers to determine if Chelsio uEFI driver is loaded If loaded you should see a simi...

Страница 363: ...363 Upgrading Firmware Using CLI i Use the adapter s device name to update the firmware FS1 CHELSIO fwupdate d device_name f cubt4 bin Example ii Reboot machine for changes to take effect Using FMP i...

Страница 364: ...ns Firmware Update Chelsio T6 Controller iii Highlight Select a firmware file option and hit Enter iv Select the USB flash drive which contains the latest Option ROM and hit Enter v Select Option ROM...

Страница 365: ...XXV Unified Boot Chelsio Unified Wire for Linux 365 vi Select Start firmware update and hit Enter vii After Firmware update completed successfully prompt appears reboot the machine for changes to take...

Страница 366: ...ou wish to restore option ROM settings to their default values i e PXE enabled iSCSI and FCoE disabled use any of the methods mentioned below Using Option ROM boot level Legacy PXE Boot system into Ch...

Страница 367: ...Linux 367 Using cxgbtool OS level Change your working directory to OptionROM directory and use cxgbtool to flash the default boot configuration onto the adapter root host cd driver_pakage Uboot Optio...

Страница 368: ...x http linux sxs org internet_serving pxeboot html http www howtoforge com ubuntu_pxe_install_server Windows http technet microsoft com en us library cc771670 28WS 10 29 aspx http tftpd32 jounin net U...

Страница 369: ...g the PXE server make sure the PXE server works Then reboot the client machine ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will ap...

Страница 370: ...on among the supported NIC functions and hit Enter to select viii Enable NIC function bios if not already enabled ix Choose the boot port to try the PXE boot It is recommended to only enable functions...

Страница 371: ...he system xii Allow the Chelsio option ROM to initialize and setup PXE devices DO NOT PRESS ALT S to skip Chelsio option ROM xiii In the system setup choose any of the Chelsio PXE devices as the first...

Страница 372: ...II should be listed as Chelsio T5 T6 Highlight it and press Enter iv Select the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not...

Страница 373: ...nified Wire for Linux 373 vi Enable adapter BIOS if not already enabled vii Select Chelsio Protocol Selection and press Enter It is highly recommended that you use the Save Changes option every time a...

Страница 374: ...oose the boot port to try PXE boot It is recommended to enable only those functions and ports which are going to be used Please note that enabling PXE Function 0 will enable port 0 for PXE enabling PX...

Страница 375: ...r XXV Unified Boot Chelsio Unified Wire for Linux 375 xi Reboot the system and in BIOS choose any of the available Chelsio PXE devices for PXE boot xii Reboot and hit F12 key when prompted to start PX...

Страница 376: ...tion ROM 6 1 Legacy FCoE Boot i Reboot the system ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will appear as below iv Choose the C...

Страница 377: ...Unified Wire for Linux 377 vi Choose FCoE from the list to configure and hit Enter vii Choose the first option Configure function parameters from the list of parameter type and hit Enter viii Enable...

Страница 378: ...Boot Chelsio Unified Wire for Linux 378 ix Choose the order of the ports to discover FCoE targets x Set discovery timeout to a suitable value Recommended value is 30 xi Hit F10 or Esc and then Y to sa...

Страница 379: ...re boot parameters xiii Select the first boot device and hit Enter to discover FC FCoE targets connected to the switch Wait till all reachable targets are discovered xiv List of discovered targets wil...

Страница 380: ...Unified Wire for Linux 380 xv From the list of LUNs displayed for the selected target choose one on which operating system has to be installed Hit Enter xvi Hit F10 or Esc and then Y to save the confi...

Страница 381: ...Chelsio adapter as the first boot device xx Reboot and boot from the FCoE disk or install the required OS using PXE 6 2 uEFI FCoE Boot i Reboot the system and go into BIOS setup ii Disable Secure Boot...

Страница 382: ...lect the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as me...

Страница 383: ...not already enabled vii Select Chelsio Protocol Selection and press Enter viii Select FCoE and press Enter ix Under Function Parameters enable FCoE BIOS if not already enabled It is highly recommende...

Страница 384: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 384 x Set discovery timeout to a suitable value Recommended value is 30 xi Choose the order of the ports to discover FCoE targets...

Страница 385: ...and press Enter to discover FC FCoE targets connected to the switch Wait till all reachable targets are discovered xiii List of discovered targets will be displayed Highlight a target to select it an...

Страница 386: ...and press Enter xvi Reboot the system for changes to take effect xvii The discovered LUN should appear in the Boot Configuration section and system BIOS section xviii Select the LUN as the first boot...

Страница 387: ...tion ROM 7 1 Legacy iSCSI Boot i Reboot the system ii Press Alt C when the message to configure Chelsio adapters appears on the screen iii The configuration utility will appear as below iv Choose the...

Страница 388: ...vii Choose the first option Configure Function Parameters from the list of parameter type and hit Enter viii Enable iSCSI BIOS if not already enabled iBFT iSCSI Boot Firmware Table will be selected by...

Страница 389: ...ed Boot Chelsio Unified Wire for Linux 389 ix Choose the order of the ports to discover iSCSI targets x Set discovery timeout to a suitable value Recommended value is 30 xi Hit Esc and then Y to save...

Страница 390: ...nfigure initiator related properties xiii Initiator properties like IQN Header Digest Data Digest etc will be displayed Change the values appropriately or continue with the default values Hit F10 to s...

Страница 391: ...AP Policy field Next choose the CHAP method Finally provide Initiator and Target CHAP credentials as per the authentication method selected Hit F10 to save xvi Go back and choose Configure Network Par...

Страница 392: ...igure the port Hit F10 to save xix Go back and choose Configure Target Parameters to configure iSCSI target related properties xx If you want to discover target using DHCP select Yes in the Discover B...

Страница 393: ...93 xxi Go back and choose Discover iSCSI Target s to connect to a target xxii Select the portal group on which iSCSI service is provided by the target xxiii A list of available targets will be display...

Страница 394: ...A list of LUNs configured on the selected target will be displayed Select the LUN you wish to connect to and hit Enter xxv Hit Esc and then Y to save the configuration xxvi Reboot the machine xxvii Du...

Страница 395: ...CSI Target LUN or install the required OS using PXE 7 2 uEFI iSCSI Boot This section describes the method to perform iSCSI boot on uEFI platforms i Reboot the system and go into BIOS setup ii Disable...

Страница 396: ...lect the Chelsio adapter to be configured and press Enter v Select Configuration Utility and press Enter If Chelsio T5 T6 is not listed please ensure that Chelsio uEFI driver is loaded correctly as me...

Страница 397: ...not already enabled vii Select Chelsio Protocol Selection and press Enter viii Select iSCSI and press Enter ix Under Function Parameters enable iSCSI BIOS if not already enabled It is highly recommend...

Страница 398: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 398 x Set discovery timeout to a suitable value Recommended value is 30 xi Choose the order of the ports to discover iSCSI targets...

Страница 399: ...itor properties like IQN Header Digest Data Digest etc will be displayed Change the values appropriately or continue with the default values xiii Under the first port select Enable DHCP field hit Ente...

Страница 400: ...t Chelsio Unified Wire for Linux 400 xiv Under Target Parameters select Enabled for the Boot Target via DHCP parameter to discover target using DHCP To discover target via static IP select Disabled an...

Страница 401: ...xv CHAP authentication is disabled by default To enable and configure highlight CHAP Policy and hit Enter Select the policy type from the corresponding pop up and hit Enter again xvi Provide Initiator...

Страница 402: ...iSCSI targets connected to the switch Wait till all reachable targets are discovered xviii A list of available targets will be displayed Select the target you wish to connect to and hit Enter xix A l...

Страница 403: ...press Enter xxi Reboot the system for changes to take effect xxii The discovered LUN should appear in the Boot Configuration Boot Information section and system BIOS xxiii Select the LUN as the first...

Страница 404: ...rrent working directory to LinuxDUD directory root host cd driver_package Uboot LinuxDUD iv Insert a blank formatted USB flash drive v Depending on the distribution to be installed copy the correspond...

Страница 405: ...B drive root host mkfs vfat dev sda1 v Depending on the distribution to be installed copy the corresponding image file to the USB stick For example execute the following command for SLES 12 sp3 root h...

Страница 406: ...ot PXE Boot media to install the OS and provide the Driver Update Disk as per the detailed instructions for each OS The DUD supports installation of Linux distributions using Chelsio adapters over Net...

Страница 407: ...ring the installation ii You will be asked to select the Driver Update Disk device from a list USB drives usually show up as SCSI disks in Linux Enter the index number of the device to be used and hit...

Страница 408: ...rocess enter c and hit Enter To install iSCSI Initiator driver the dependent Network driver must also be installed Hence select both cxgb4 and and cxgb4i Hit Enter vi The Driver disk prompt will be di...

Страница 409: ...ller will load and prompt you for the driver update disk Select Yes and hit Enter to proceed iii You will be asked to select the Driver Update Disk device from a list USB drives usually show up as SCS...

Страница 410: ...Now the installer will search for the appropriate drivers from the driver disk and load them This step may take some time Check on the Alt F3 or Alt F4 screens for log messages v The installer will as...

Страница 411: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 411 vi Select the required language from the list vii Select the type of keyboard you have from the list...

Страница 412: ...ux 412 viii In this step you can choose the source which contains the OS installation ISO image In this case select NFS directory ix The Chelsio detwork devices will be displayed Select the appropriat...

Страница 413: ...ing DHCP or manually using IPv4 IPv6 is currently not supported Hence disable IPv6 before proceeding xi Proceeding with the installation will get NFS FTP HTTP setup page Here provide NFS server detail...

Страница 414: ...Unified Boot Chelsio Unified Wire for Linux 414 SLES 15 SLES 12 SPx Installation i Please make sure that the USB drive with DUD image is inserted ii Select the appropriate entry from the PXE menu and...

Страница 415: ...pter XXV Unified Boot Chelsio Unified Wire for Linux 415 iii Select Start Installation and then Start Installation or Update iv Select Network as the source of medium to install the SLES Operating Sys...

Страница 416: ...the appropriate Chelsio interface from the list to proceed with installation You can view the messages from the Linux kernel and drivers to determine the name of NIC interface by pressing Alt F3 or Al...

Страница 417: ...rom CD DVD in the BIOS setup i Insert the OS installation disc into your CD DVD ROM ii On the Grub menu choose Install or upgrade an existing system option if not already selected iii Type e and then...

Страница 418: ...Chapter XXV Unified Boot Chelsio Unified Wire for Linux 418 RHEL 7 X i Choose your installation language and click Continue ii Click INSTALLATION DESTINATION under SYSTEM...

Страница 419: ...ction Select the LUN which was saved as boot device in system BIOS iv Under Other Storage Options you can either chose to configure partition automatically or manually Select the appropriate option an...

Страница 420: ...transport medium Legacy o For RHEL 6 press Tab and then type dd o For RHEL 7 press Tab and then type ip ibft o For SLES press Tab and then type dd uEFI o For RHEL 6 type e and then type dd o For RHEL...

Страница 421: ...hapter XXV Unified Boot Chelsio Unified Wire for Linux 421 RHEL 7 X i On the installer welcome screen choose your installation language and click Continue ii Click Installation Destination under SYSTE...

Страница 422: ...io Unified Wire for Linux 422 iii Click Add a disk iv The discovered iSCSI LUNs will appear in the Search tab Select it and click Done Make sure the same LUN discovered at the Option ROM stage is sele...

Страница 423: ...you can either chose to configure partition automatically or manually Select the appropriate option and click Done Then proceed with the installation as usual RHEL 6 X i Click Next when the graphical...

Страница 424: ...discovered LUNs will appear in the Basic Devices tab Select the LUN which was saved as boot device in system BIOS and click Next iv Proceed with the installation as usual Make sure the same LUN disco...

Страница 425: ...ied Wire for Linux 425 SLES 15 SLES 12 SPx Installation i Choose installation language and Keyboard layout type Select the checkbox I Agree to the License terms and click Next ii Click Configure iSCSI...

Страница 426: ...discovered LUNs will appear in the Connected Targets tab Select the LUN which was saved as boot device in system BIOS and click OK iv Proceed with the installation as usual Make sure the same LUN disc...

Страница 427: ...Chapter XXVI Appendix A Chelsio Unified Wire for Linux 427 XXVI Appendix A...

Страница 428: ...g If you get a return value of 0 when doing a cat proc sys net ipv4 icmp_echo_ignore_all on the remote host that means it is configured to reply to incoming pings Change ipv4 to ipv6 in the path if yo...

Страница 429: ...WARP traffic One of the reason for this could be port number collisions To fix this use iWARP port mapper iwpmd root host iwpmd Multiple Chelsio adapters Chelsio Option ROM supports upto 4 Chelsio ada...

Страница 430: ...oftware contains trade secrets and in order to protect them you may not decompile reverse engineer disassemble or otherwise reduce the Software to a human perceivable form You assume full responsibili...

Страница 431: ...ARISING OUT OF OR RELATED TO THE LICENSE OR USE OF THE SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR THE LICENSE GRANTED HEREUNDER THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL...

Страница 432: ...covered by Federal law You hereby irrevocably submit to the personal jurisdiction of and irrevocably waive objection to the laying of venue including a waiver of any argument of forum non conveniens...

Отзывы:

Нет отзывов