Optimize Performance with Open vSwitch and DPDK
Focus
Focus
VM-Series

Optimize Performance with Open vSwitch and DPDK

Table of Contents

Optimize Performance with Open vSwitch and DPDK

Learn how to optimize the performance of your VM-Series firewall on KVM using Open vSwitch and DPDK.
Where Can I Use This?What Do I Need?
  • KVM
  • VM-Series Firewall License (BYOL)
  • Panorama
  • VM-Series plugin
By default, KVM uses a linux bridge for VM networking. However, the best performance in a virtual environment is realized with dedicated I/O interfaces (PCI pass-through or SR-IOV). If a virtual switch is required, use a performance-optimized virtual switch (such as Open vSwitch with DPDK).

Install KVM and Open vSwitch on Ubuntu 16.04.1 LTS

For ease of installation, Ubuntu 16.04.1 LTS is recommended for use as the KVM hypervisor platform.
  1. Install KVM and OVS.
    1. Log in to the Ubuntu CLI.
    2. Execute the following commands:
      $ sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils 
      $ sudo apt-get install openvswitch-switch 
  2. Check and compare the versions of relevant packages.
    Execute the following commands:
    $ virsh --version 1.3.1
    $ libvirtd --version
    libvirtd (libvirt) 1.3.1
    $ /usr/bin/qemu-system-x86_64 --version
    QEMU emulator version 2.5.0 (Debian
    1:2.5+dfsg-5ubuntu10.6), Copyright (c) 2003-2008
    Fabrice Bellard
    $ ovs-vsctl --version
    ovs-vsctl (Open vSwitch) 2.5.0
    Compiled Mar 10 2016 14:16:49
    DB Schema 7.12.1

Enable Open vSwitch on KVM

Enable OVS by modifying the guest XML definition network settings.
  1. Modify the guest XML definition as follows.
    [...]
        <interface type='bridge'>
           <mac address='52:54:00:fb:00:01'/>
           <source bridge='ovsbr0'/>
           <virtualport type='openvswitch'/>
           <model type='virtio'/>
           <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
       </interface>
    [...]

Install QEMU, DPDK, and OVS on Ubuntu

Before you can enable DPDK on OVS, you must install QEMU 2.5.0, DPDK 2.2.0, and OVS 2.5.1. Complete the following procedures to install the components.
  1. Log in to the KVM host CLI.
  2. Install QEMU 2.5.0 by executing the following commands:
    apt-get install build-essential gcc pkg-config glib-2.0 libglib2.0-dev libsdl1.2-dev
    libaio-dev libcap-dev libattr1-dev libpixman-1-dev
    apt-get build-dep qemu
    apt-get install qemu-kvm libvirt-bin
    wget http://wiki.qemu.org/download/qemu-2.5.0.tar.bz2
    tar xjvf qemu-2.5.0.tar.bz2
    cd qemu-2.5.0
    ./configure
    make
    make install
  3. Install dpdk-2.2.0.
    1. Execute the following commands:
      wget http://dpdk.org/browse/dpdk/snapshot/dpdk-2.2.0.tar.gz 
      tar xzvf dpdk-2.2.0.tar.gz 
      cd dpdk-2.2.0 
      vi config/common_linuxapp 
    2. Change CONFIG_RTE_APP_TEST=y to CONFIG_RTE_APP_TEST=n
    3. Change CONFIG_RTE_BUILD_COMBINE_LIBS=n to CONFIG_RTE_BUILD_COMBINE_LIBS=y
    4. Execute the following command:
      vi GNUmakefile
    5. Change ROOTDIRS-y := lib drivers app to ROOTDIRS-y := lib drivers
    6. Execute the following command:
      make install T=x86_64-native-linuxapp-gcc
  4. Install OVS 2.5.1 by executing the following commands:
    wget http://openvswitch.org/releases/openvswitch-2.5.1.tar.gz 
    tar xzvf openvswitch-2.5.1.tar.gz 
    cd openvswitch-2.5.1 
    ./configure –with-dpdk=”/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/” 
    make  
    make install 

Configure OVS and DPDK on the Host

After installing the necessary components to support OVS and DPDK, you must configure the host to use OVS and DPDK.
  1. Log in to the KVM host CLI.
  2. If you are replacing or reconfiguring an existing OVS-DPDK setup, execute the following commands to reset any previous configuration. Repeat the command for each interface.
    rm /usr/local/var/run/openvswitch/<interface-name>
  3. Configure initial huge pages for OVS.
    echo 16384 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  4. Mount huge pages for QEMU:
    mkdir /dev/hugepages 
    mkdir /dev/hugepages/libvirt 
    mkdir /dev/hugepages/libvirt/qemu 
    mount -t hugetlbfs hugetlbfs /dev/hugepages/libvirt/qemu 
  5. Use the following command to kill any currently existing OVS daemon.
    killall ovsdb-server ovs-vswitchd
  6. Create directories for the OVS daemon.
    mkdir -p /usr/local/etc/openvswitch 
    mkdir -p /usr/local/var/run/openvswitch 
  7. Clear old directories.
    rm -f /var/run/openvswitch/vhost-user* 
    rm -f /usr/local/etc/openvswitch/conf.db 
  8. Initialize the configuration database.
    ovsdb-tool create /usr/local/etc/openvswitch/conf.db\ 
    /usr/local/share/openvswitch/vswitch.ovsschema 
  9. Create an OVS DB server.
    ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ 
    --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ 
    --private-key=db:Open_vSwitch,SSL,private_key \ 
    --certificate=db:Open_vSwitch,SSL,certificate \ 
    --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ 
    --pidfile --detach 
  10. Initialize OVS.
    ovs-vsctl --no-wait init
  11. Start the database server.
    export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
  12. Install the igb_uio module (network device driver) for DPDK.
    cd ~/dpdk-2.2.0/x86_64-native-linuxapp-gcc/kmod 
    modprobe uio 
    insmod igb_uio.ko 
    cd ~/dpdk-2.2.0/tools/ 
  13. Enable DPDK on interfaces using PCI-ID or interface name.
    ./dpdk_nic_bind.py --bind=igb_uio <your first data interface> 
    ./dpdk_nic_bind.py --bind=igb_uio <your second data interface> 
  14. Start the OVS daemon in DPDK mode. You can change the number of cores for ovs-vswitchd. By changing -c 0x1 to -c 0x3, you can have two core run this daemon.
    ovs-vswitchd --dpdk -c 0x3 -n 4 -- unix:$DB_SOCK --pidfile --detach 
    echo 50000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 
  15. Create the OVS bridge and attach ports to the OVS bridge.
    ovs-vsctl add-br ovs-br0 -- set bridge ovs-br0 datapath_type=netdev 
    ovs-vsctl add-port ovs-br0 dpdk0 -- set Interface dpdk0 type=dpdk 
    ovs-vsctl add-br ovs-br1 -- set bridge ovs-br1 datapath_type=netdev 
    ovs-vsctl add-port ovs-br1 dpdk1 -- set Interface dpdk1 type=dpdk  
  16. Create DPDK vhost user ports for OVS.
    ovs-vsctl add-port ovs-br0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser 
    ovs-vsctl add-port ovs-br1 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser 
  17. Set the number of hardware queues of the NIC used by the host.
    ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=8 
    ovs-vsctl set Open_vSwitch . other_config:n-dpdk-txqs=8 
  18. Set the CPU mask used for OVS.
    ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xffff 
  19. Set the necessary permissions for DPDK vhost user ports. In the example below, 777 is used to give read, write, and executable permissions.
    chmod 777 /usr/local/var/run/openvswitch/vhost-user1 
    chmod 777 /usr/local/var/run/openvswitch/vhost-user2 
    chmod 777 /dev/hugepages/libvirt/qemu  

Edit the VM-Series Firewall Configuration File

Edit the VM-Series firewall XML configuration file to support OVS and DPDK. You can access the XML configuration file or after deploying the VM-Series firewall. If you do this after deploying the firewall, be sure to shut down the firewall before making any changes. The values below are examples. Your values for each parameter will vary based on your VM-Series model.
  1. Log in to the KVM host CLI.
  2. Edit the XML configuration file of your VM-Series firewall.
    1. Open the XML config file using virsh edit $<your-vm-series-name>.
    2. Sets the memory backing for the homepage. Ensure that you provide enough memory to support the VM-Series firewall model you are deploying on the host. See VM-Series System Requirements for more information.
      <memory unit='KiB'>12582912</memory> 
        <currentMemory unit='KiB'>6291456</currentMemory> 
        <memoryBacking> 
          <hugepages/> 
    3. Set the necessary CPU flags for the VM.
      <cpu mode='host-model'> 
    4. Enable memory sharing between the VM and the host.
      <numa> 
            <cell id='0' cpus='0,2,4,6' memory='6291456' unit='KiB' memAccess='shared'/> 
            <cell id='1' cpus='1,3,5,7' memory='6291456' unit='KiB' memAccess='shared'/> 
          </numa> 
    5. Set the DPDK vhost user ports as the VM-Series firewall’s network interfaces. Additionally, set the number of virtio virtual queues provided to the VM-Series firewall by the host.
      <interface type='vhostuser'> 
            <mac address='52:54:00:36:83:70'/> 
            <source type='unix' path='/usr/local/var/run/openvswitch/vhost-user1' mode='client'/> 
      <model type='virtio'/> 
      <driver name=’vhost’ queues=’8’/> 
            <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> 
          </interface> 
          <interface type='vhostuser'> 
            <mac address='52:54:00:30:d7:94'/> 
            <source type='unix' path='/usr/local/var/run/openvswitch/vhost-user2' mode='client'/> 
      <model type='virtio'/> 
      <driver name=’vhost’ qeueus=’8’> 
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> 
          </interface> 

KVM Host-Level Performance Tuning for VM-Series

Execute the following steps on your KVM hosts to enhance performance:
  1. vCPU Pinning - Pin each VM-Series vCPU to a specific physical CPU core. This creates a 1:1 mapping that eliminates scheduling jitter and guarantees dedicated CPU resources for firewall processes.
    # Pin vCPUs 0 and 1 of the firewall VM to physical cores 2 and 3 virsh vcpupin <vm-name> 0 2 --live --config virsh vcpupin <vm-name> 1 3 --live --config
  2. NUMA Alignment - Ensure all the vCPUs and memory allocated to a VM-Series instance reside on the same physical NUMA node. This prevents cross-node memory access latency.
    # Check the server's NUMA hardware layout to plan core allocation numactl --hardware
  3. Emulator Pinning - Isolate the hypervisor emulator threads (which handle I/O and device emulation) to dedicated housekeeping cores, separate from those used by the VM-Series. This prevents hypervisor overhead from impacting packet processing.
    # Pin the emulator for the firewall VM to housekeeping cores 0 and 1 virsh emulatorpin <vm-name> 0-1 --live --config
  4. Kernel Core Isolation - Isolate the physical cores assigned to the VM-Series from the host's operating system scheduler. This ensures the host OS will not run its tasks on cores reserved for the firewall.
    # In /etc/default/grub, add the 'isolcpus' kernel parameter # This example isolates cores 2-20 for VM workloads GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=2-20" # Update GRUB and reboot for the change to take effect sudo update-grub
  5. SR-IOV & NIC Locality - When using SR-IOV, the vCPUs for the VM-Series must be pinned to cores on the same NUMA node as the physical SR-IOV NIC to ensure the lowest latency path for network traffic.
    # 1. Identify the NUMA node of the physical NIC cat /sys/class/net/eth0/device/numa_node # 2. Use the result to guide vCPU pinning # If the result is '0', pin the VM's vCPUs to cores on NUMA Node 0 virsh vcpupin <vm-name> 0 4 --live --config virsh vcpupin <vm-name> 1 5 --live --config
  6. Hyper-Threading - For environments requiring the most predictable, deterministic latency, disabling Hyper-Threading in the server BIOS/UEFI is recommended. This ensures one vCPU has exclusive access to one physical core's full resources.
    # Verify 'Thread(s) per core' is 1 lscpu