Getting

Install required packages

Just use your command line

sudo apt-get update -y
sudo apt-get -y -q install git doxygen hugepages build-essential \
linux-headers-`uname -r` libdpdk-dev libmnl0 libmnl-dev libkmod2 \
libkmod-dev libnuma-dev libelf1 libelf-dev libc6-dev-i386 autoconf \
flex bison libncurses5-dev libreadline-dev python3 graphviz \
python3-pyelftools libboost-all-dev plantuml

pip3 install --user meson ninja

uname -r #Kernel >= 3.16 required
ldd --version #glibc >= 2.7 required

install DPDK

Download from Website
extract: tar xf dpdk.tar.gz
into folder: cd dpdk
build lib, driver, test:

  meson build     # incl all examples: meson -Dexamples=all build
  ninja -C build

include dpdk to path

only for session: export PATH=$PATH:</path/to/file>
permanently add export PATH=$PATH:</path/to/file> to your ~/.bashrc file (at the end).

or with apt sudo apt install dpdk

Hugepages

reserve Hugepages (in runtime):
1. configure Hugepages echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages echo 32 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
2. create directory for mounting mkdir -p /mnt/huge mountpoint -q /mnt/huge mount -t hugetlbfs nodev /mnt/huge
3. Check hugpages grep Huge /proc/meminfo
Run poll mode driver test: build/app/dpdk-testpmd -c7 --vdev=net_pcap0,iface=eth0 --vdev=net_pcap1,iface=eth1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048

in boot time

Modify Linux boot time parameters inside /etc/default/grub. Huge pages will be spread equally between all NUMA sockets. GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=32"

Update the grub configuration file and reboot.

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

Create a folder for a permanent mount point of hugetlbfs mkdir /mnt/huge

Add the following line to the /etc/fstab file: nodev /mnt/huge hugetlbfs defaults 0 0

Persistent huge pages are never swapped by the Linux kernel!

Mac, IP & PCI adress

ifconfig -a to view MAC and IP adress
ethtool -i <interface name> to get PCI adress
Load ibuverbs on each reboot with modprobe -a ib_uverbs

TX side

testpmd \
  -l <core-list> \
  -n <num of mem channels> \
  -w <pci address of the device you plan to use> \
  --vdev="net_vdev_netvsc<id>,iface=<the iface to attach to>" \
  -- --port-topology=chained \
  --nb-cores <number of cores to use for test pmd> \
  --forward-mode=txonly \
  --eth-peer=<port id>,<receiver peer MAC address> \
  --stats-period <display interval in seconds>

RX side

testpmd \
  -l <core-list> \
  -n <num of mem channels> \
  -w <pci address of the device you plan to use> \
  --vdev="net_vdev_netvsc<id>,iface=<the iface to attach to>" \
  -- --port-topology=chained \
  --nb-cores <number of cores to use for test pmd> \
  --forward-mode=rxonly \
  --eth-peer=<port id>,<sender peer MAC address> \
  --stats-period <display interval in seconds>

Problem at Crash

DPDK won't free hugepages and can therefore not be restartet with no hugepages free in system.

Issue could be like you removed /mnt/huge and trying to modify the nr_hugepage

echo 0 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages #Numa case
echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages #non-NUMA case

steps to follow while un-mapping hugepage:

ls -l /mnt/huge/ rm -rf rtemap_* (if there are any "rtemap_*" delete all)
mount | grep huge hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)

ls /dev/hugepages

rtemap_0  rtemap_1202  rtemap_1408  rtemap_1613  rtemap_1819  rtemap_2023  rtemap_387  rtemap_592  rtemap_798
rtemap_1  rtemap_1203  rtemap_1409  rtemap_1614  rtemap_182   rtemap_2024  rtemap_388  rtemap_593  rtemap_799

If abive files are present delete all of them.
1. sudo umount /mnt/huge
2. sudo rm -R /mnt/huge
3. Then write 0 to nr_hugepages mentioned at start

4.1 KiB Raw Permalink Blame History