13–NVMe-oF Configuration with RDMA
Optimizing Performance
237
BC0154501-00 P
In this example, the target NVMe device latency is 8µsec. The total latency
that results from the use of NVMe-oF is the difference between the initiator
device NVMe-oF latency (30µsec) and the target device NVMe-oF latency
(8µsec), or 22µsec.
4.
Run FIO to measure bandwidth of the local NVMe device on the target
server. Issue the following command:
fio --verify=crc32 --do_verify=1 --bs=8k --numjobs=1
--iodepth=32 --loops=1 --ioengine=libaio --direct=1
--invalidate=1 --fsync_on_close=1 --randrepeat=1
--norandommap --time_based --runtime=60
--filename=/dev/nvme0n1 --name=Write-BW-to-NVMe-Device
--rw=randwrite
Where
--rw
can be
randread
for reads only,
randwrite
for writes only,
or
randrw
for reads and writes.
Optimizing Performance
To optimize performance on both initiator and target servers:
1.
Configure the following system BIOS settings:
Power Profiles = 'Max Performance' or equivalent
ALL C-States = Disabled
Hyperthreading = Disabled
2.
Configure the Linux kernel parameters by editing the
grub
file
(
/etc/default/grub
).
a.
Add parameters to end of line
GRUB_CMDLINE_LINUX
:
GRUB_CMDLINE_LINUX="
nosoftlockup intel_idle.max_cstate=0
processor.max_cstate=1 mce=ignore_ce idle=poll
"
b.
Save the
grub
file.
c.
Rebuild the
grub
file.
To rebuild the
grub
file for a legacy BIOS boot, issue the
following command:
#
grub2-mkconfig -o /boot/grub2/grub.cfg
(Legacy BIOS boot)
To rebuild the
grub
file for an EFI boot, issue the following
command:
#
grub2-mkconfig -o /boot/efi/EFI/
<os>
/grub.cfg
(EFI boot)
d.
Reboot the server to implement the changes.
