NetXtreme-UG100
36
NetXtreme-C/NetXtreme-E
User Guide
12 SR-IOV: Configuration and Use Case Examples
SR-IOV can be configured, enabled, and used on 10-Gb and 25-Gb Broadcom NetExtreme-E NICs.
12.1 Linux Use Case Example
1. Enable SR-IOV in the NIC cards:
a. SR-IOV in the NIC card can be enabled using the
HII
menu. During system boot, access the system
BIOS ->
NetXtreme-E NIC -> Device Level Configuration
.
b. Set the Virtualization mode to SR-IOV.
c. Set the number of virtual functions per physical function.
d. Set the number of MSI-X vectors per the VF and Max number of physical function MSI-X vectors. If the VF is running
out of resources, balance the number of MSI-X vectors per VM using CCM.
2. Enable virtualization in the BIOS:
a. During system boot, enter the system
BIOS -> Processor settings -> Virtualization Technologies
and set it to
Enabled.
b. During system boot, enter the system
BIOS -> SR-IOV Global
and set it to
Enabled
.
3. Install the desired Linux version with Virtualization enabled (libvirt and Qemu).
4. Enable the iommu kernel parameter.
a. The IOMMU kernel parameter is enabled by editing
/etc/default/grub.cfg
and running
grub2-mkconfig -o /
boot/grub2/grub.cfg
for legacy mode. For UEFI mode, edit
/etc/default/grub.cfg
and run
grub2-mkconfig -
o /etc/grub2-efi.cfg
. Refer to the following example:
Linuxefi /vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/rhel-root ro rd.lvm.lv=rhel/swap
crashkernel=auto rd.lvm.lv=rhel/root rhgb intel_iommu=on quiet LANG=en_US.UTF.8
5. Install bnxt_en driver:
a. Copy the
bnxt_en
driver on to the OS and run
make; make install; modprobe bnxt_en
.
NOTE:
Use
netxtreme-bnxt_en<version>.tar.gz
to install both
bnxt_re
and
bnxt_en
for RDMA functionality on
SRIOV VFs.
6. Enable Virtual Functions through Kernel parameters:
a. Once the driver is installed, lspci displays the NetXtreme-E NICs present in the system. Bus, device, and Function
are needed for activating Virtual functions.
b. To activate Virtual functions, enter the command shown below:
echo X >/sys/bus/pci/device/0000\:Bus\:Dev.Function/sriov_numvfs
NOTE:
Ensure that the PF interfaces are up. VFs are only created if PFs are up. X is the number of VFs that are exported
to the OS.
A typical example would be:
echo 4 > /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs
7. Check the PCIe virtual functions:
The
lspci
command displays the virtual functions with DID set to 16D3 for BCM57402/BCM57404/BCM57406, 16DC
for non-RDMA BCM57412/BCM57414/BCM57416, and 16C1 or RDMA enabled BCM57412/BCM57414/BCM57416.
