Use the following steps to install a v35 virtual monitoring point in your network. The monitoring point runs on a host machine that has the KVM hypervisor installed. KVM runs on one of the following Linux versions: CentOS-7.x, RHEL 7.x or Ubuntu 16.04.3 LTS.

Once KVM is available and the necessary software and configuration information has been downloaded to the KVM host, the VM can be created and configured to connect to APM. After making the physical connection between the KVM host and your network the v35 will connect back to APM and can be used for monitoring. Once the setup is completed, additional features can be set up as required.

  1. Prerequisites
  2. Set up a KVM host
    1. Set up a CentOS or RHEL KVM host
    2. Set up an Ubuntu KVM host
  3. Load install scripts
  4. Load base and config images
  5. Create a v35 virtual machine
  6. Connect to the network
  7. Configure for web proxy if required
  8. Assign licenses
  9. Set the monitoring point location
  10. Set up monitoring
  11. Configure additional features if required
    1. Set up v35 with VLAN
    2. Set up v35 for Usage monitoring
    3. Set up v35 with 10Gbps
    4. Set up v35 with high performance 10Gbps
    5. Set up v35 with 10Gbps and VLAN
    6. Set up v35 with additional interfaces
  12. Helpful commands

Prerequisites

The following are the minimum requirements for the KVM host system:

  • Capable of running a guest virtual machine with at least the v35 requirements.
  • Runs CentOS-7.x, RHEL 7.x or Ubuntu 16.04.3 LTS.
  • Has the following virtualization management packages installed:
    • CentOS-7.x, RHEL 7.x - qemu-kvm qemu-img libvirt libvirt-python libvirt-client virt-install bridge-utils
    • Ubuntu 16.04.3 LTS - qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virtinst
  • Has up to four Ethernet NICs installed on the KVM host. These are mapped to interfaces on the v35 as follows:
v35 interface Required or Optional Supported NIC speeds Function
eth0 Required 1Gbps or 10Gbps Network connectivity, Delivery monitoring, and Experience monitoring
eth1 Optional 1Gbps Usage monitoring
eth2 Optional 1Gbps or 10Gbps Delivery monitoring and Experience monitoring
eth3 Optional 1Gbps or 10Gbps Delivery monitoring and Experience monitoring
  • For 1Gbps: A Linux bridge or an OVS bridge to bind the interface on the v35 to the physical interface on the host.
    • The OVS bridge is required only if you want to use a VLAN. If this is the case, Open vSwitch (OVS) software must be installed. Open a support ticket for any assistance.
  • For 10 Gbps: A 10Gbps NIC with SR-IOV support. Supported 10Gbps NICs include:
    • Intel X540
    • Intel X550
    • Intel 82599

Set up a KVM host

If you have not already done so, set up either a CentOS, a RHEL, or an Ubuntu KVM host.

Set up a CentOS or RHEL KVM host

To set up a CentOS-7.x or RHEL 7.x KVM host:

  1. Install CentOS-7.x or RHEL 7.x.
  2. Install the required packages:

    sudo yum -y install qemu-kvm qemu-img libvirt libvirt-python libvirt-client virt-install bridge-utils
    
  3. Reboot.
  4. Verify the installation to see that all is okay and no guests are running at this point:

    virsh list --all
    

Set up an Ubuntu KVM host

To set up an Ubuntu 16.04.3 LTS KVM host:

  1. Install Ubuntu 16.04.3 LTS.
  2. Install the required packages:

    sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils virtinst
    
  3. Verify the installation to see that all is okay and no guests are running at this point:

    virsh list --all
    

Load install scripts

There are two scripts used for v35 setup that must be downloaded and copied to the KVM host. To download, right-click the links below and select the option to download and save the file (this option is different for different browsers). You must then copy these files to the KVM host.

  • vk35tool.py: Script to create a v35 and configure its interfaces.
  • vk35hook.sh: Hook script to automatically connect a mirror port to an OVS bridge.

Load base and config images

The AppNeta software required for the v35 includes a base image (.qcow2) and a config image (.iso). These must be downloaded and then copied to the KVM host.

To download the base and config images and copy them to your host:

  1. Log in to APM.
  2. Make sure you are using the correct organization
  3. Navigate to > Manage Monitoring Points > + Add Monitoring Points > Virtual Monitoring Point > Download the v35 for KVM Platform.
  4. Click Download KVM Base Image.
    • The KVM base image (.qcow2) is downloaded to your computer.
  5. Click Continue.
  6. Follow the instructions provided to specify the v35 configuration.
    • Note the Hostname as it will be used later.
    • A .zip file containing the config image (.iso) is downloaded to your computer.
  7. Copy the KVM base image (.qcow2) and the config image (.iso) to /var/lib/libvirt/images/ on the host system.
    • It is good practice to rename these files to match the Hostanme but leave the .qcow2 and .iso extensions. This is especially important if you are setting up more than one v35 on the KVM host as the .qcow2 and .iso files cannot be shared by virtual machines.

Create a v35 virtual machine

Once the software and configuration files are available on the host, the v35 virtual machine that uses them can be created.

eth0 on the v35 is mapped to the Linux bridge interface on the KVM host you will be using for network connectivity as well as Delivery and Experience monitoring. By default, the v35 is created with 4096MB RAM and two virtual CPUs.

You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name. For simplicity, use the Hostname noted in a previous step.
BASE_IMAGE Name of the downloaded base image (e.g., <Hostname>.qcow2).
CONFIG_IMAGE Name of the downloaded config image containing initial monitoring point configuration (e.g., <Hostname>.iso).
BRIDGE_NAME Linux bridge interface on the KVM host (determined below).

To create the v35:

  1. Log on to the KVM host.
  2. Go to the directory you copied the install scripts (vk35tool.py and vk35hook.sh) to.
  3. Determine which bridge interfaces are available to the v35.

    ./vk35tool.py scan
    
  4. Install the v35.

    ./vk35tool.py install V35_NAME \
    --image /var/lib/libvirt/images/BASE_IMAGE.qcow2 \
    --config /var/lib/libvirt/images/CONFIG_IMAGE.iso \
    --eth0 type=bridge,source=BRIDGE_NAME
    
  5. Verify that the v35 is “running”.

    virsh list --all
    

Connect to the network

To physically connect the monitoring point to the network:

  1. Verify that the switch you’ll be connecting to has its default port speed and duplex set to auto-negotiate.
  2. Using a standard Ethernet cable, connect the physical interface on the KVM host associated with the bridge port configured above to the switch.

Configure for web proxy if required

If Internet traffic on your network is forwarded by a web proxy, you must configure the monitoring point with the proxy details.

Assign licenses

Every monitoring point is sold with a base license (either Small Office, Large Office, or Data Center, depending on the model) that allows it to monitor a certain number of applications. The base license must be assigned to the monitoring point in APM before it can be used. Optionally, add-on licenses can be assigned.

To license to a monitoring point:

  1. Return to the browser tab where you are running APM.
  2. Navigate to > Manage Monitoring Points.
  3. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
    • The monitoring point status should show “Connection Established” ().
    • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.
  4. Assign a base license to the monitoring point.
  5. Optionally, assign add-on licenses as required.

Set the monitoring point location

When adding a new monitoring point to APM, you need to specify its location. This setting provides essential geographical context for the data that the monitoring point collects.

To set the location of the monitoring point:

  1. Enter the monitoring point location if prompted during the setup procedure.

Set up monitoring

At this point the monitoring point setup is complete. Continue the APM setup procedure at Set up monitoring.

Configure additional features if required

Once the basic v35 is set up and is connecting back to APM, you can install additional features as required. These include:

Set up v35 with VLAN

This procedure assumes that the v35 is already set up and running and that Open vSwitch (OVS) software is installed on your KVM host and an OVS bridge has been created. You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
V35_INTERFACE v35 interface to use (either eth0, eth2, or eth3).
OVS_NET_LAN Virtual network based on the OVS bridge configured to forward VLAN traffic.
VLAN_PORTGROUP Port group that defines a set of tagged and untagged VLANs the traffic is received from.

To update the v35 to use VLAN:

  1. Shutdown the v35.

    virsh shutdown V35_NAME
    
  2. Update the v35 configuration to use VLAN.

    ./vk35tool.py update V35_NAME \
    --V35_INTERFACE type=network,source=OVS_NET_LAN, \
    portgroup=VLAN_PORTGROUP
    
  3. Start the v35.

    virsh start V35_NAME
    
  4. Verify that the v35 is “running”.

    virsh list --all
    
  5. Verify that the v35 is connected to APM.

    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait a couple minutes for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is ready to monitor VLAN traffic.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Set up v35 for Usage monitoring

There are a few steps required in order to set up the v35 for Usage monitoring:

  1. Configure the v35 for Usage monitoring.
  2. Connect to a mirror port.
  3. Set up the vk35hook.sh script.
    • This step only needs to be done once.

Configure the v35 for Usage monitoring

This procedure assumes that the v35 is already set up and running and that Open vSwitch (OVS) software is installed on your KVM host and an OVS bridge has been created. You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
OVS_NET_SPAN Virtual network based on the OVS bridge configured for mirroring traffic

To update the v35 configuration for Usage monitoring:

  1. Shutdown the v35.

    virsh shutdown V35_NAME
    
  2. Update the v35 configuration for Usage monitoring. Map eth1 to the OVS port configured for capturing mirrored traffic.

    ./vk35tool.py update V35_NAME \
    --eth1 type=network,source=OVS_NET_SPAN
    
  3. Start the v35.

    virsh start V35_NAME
    
  4. Verify that the v35 is “running”.

    virsh list --all
    
  5. Verify that the v35 is connected to APM.

    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” ().
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Connect to a mirror port

If you have not already done so, see Deploying Your Monitoring Point for information on where to deploy your monitoring point in you network for Usage monitoring.

To connect the v35 to a mirror port:

  1. Using a standard Ethernet cable, connect the OVS source port on the KVM host configured for capturing mirrored traffic to the port on your switch configured to mirror uplink traffic.

Set up the vk35hook.sh script

The vk35hook.sh script is used to map the physical port that is capturing mirrored traffic to the OVS bridge (which is mapped to eth1 on the v35). Whenever the v35 is shut down this mapping is removed so the vk35hook.sh script needs to be run whenever the v35 is started.

To set up the vk35hook.sh script and have it run automatically:

  1. Edit vk35hook.sh and set the following parameters.
    • PORT - the source physical interface name on the KVM host (e.g., eno2)
    • BRIDGE - the name of the OVS bridge (OVS_NET_SPAN set above)
    • DOMAIN - the v35 name (V35_NAME set above)
    • MAC - the MAC address of eth1 on the v35. To find this, use:

       virsh -q domiflist V35_NAME | grep "52:54:00:e1" | awk ' {print $1}' | xargs ip -o link show dev | awk '{print $(NF-2)}'
      
  2. If /etc/libvirt/hooks/qemu does not already exist on the KVM host:

    cp vk35hook.sh /etc/libvirt/hooks/qemu
    chmod 755 /etc/libvirt/hooks/qemu
    
  3. If /etc/libvirt/hooks/qemu already exists on the KVM host:
    1. Edit it.
    2. Confirm that the first line in this file is: #!/usr/bin/env bash
    3. Add the following to the end of file: bash /PATH/vk35hook.sh $@ <&0
      • Where PATH is the file path to vk35hook.sh.
  4. Restart the virtualization infrastructure on the KVM host:

    systemctl restart libvirtd
    
  5. Verify that the v35 is “running”.

    virsh list --all
    
  6. Verify that the v35 is connected to APM.
    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is configured to capture mirrored traffic.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Set up v35 with 10Gbps

10Gbps support is only available for network connectivity (access to APM), Delivery monitoring, and Experience monitoring. It is not available for Usage monitoring.

There are a few steps required in order to set up the v35 to use a 10Gbps interface:

  1. Configure host for virtual functions support.
  2. Configure host for SR-IOV support.
  3. Apply changes and verify them.
  4. Create a virtual function network.
  5. Configure the v35 for 10Gbps.
  6. Connect to the network via 10Gbps.

Configure host for virtual functions support

To configure the KVM host for virtual functions support:

  1. Edit /etc/modprobe.d/ixgbe.conf on the KVM host.
    1. Add the following lines:

      options ixgbe max_vfs=8
      blacklist ixgbevf
      

Configure host for SR-IOV support

To configure the KVM host for SR-IOV support:

  1. Edit /etc/default/grub.
    1. Locate GRUB_CMDLINE_LINUX and add intel_iommu=on to it.
      • For example, “GRUB_CMDLINE_LINUX … intel_iommu=on …”.
      • Use amd_iommu=on if the KVM host has an AMD processor.
  2. Regenerate the grub.cfg file.
    • For CentOS or RHEL, use:

       sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      
    • For Ubuntu, use:

       sudo update-grub
      
  3. Rebuild the boot initramfs.
    • For CentOS or RHEL, use:

       sudo dracut --regenerate-all -f
      
    • For Ubuntu, use:

       sudo update-initramfs -k all -u
      

Apply changes and verify them

To apply the changes made in the previous two steps and verify them:

  1. Reboot the KVM host to apply the changes.
  2. Verify that SR-IOV support is configured.

    cat /proc/cmdline
    
    • Look for intel_iommu=on (or amd_iommu=on).
  3. Verify that virtual functions (vf) have been created.

    ip link show
    

Create a virtual function network

Once the KVM host is configured with virtual function (vf) support and SR-IOV you can create a virtual function network for the v35 to use. In this example we create enp5s0f1-vf based on interface enp5s0f1.

To create a virtual function network:

  1. Create an .xml network definition file. For example:

    $ cat enp5s0f1-vf.xml
    <network>
      <name>enp5s0f1-vf</name>
      <forward mode='hostdev' managed='yes'>
        <pf dev='enp5s0f1'/>
      </forward>
    </network>
    
  2. Define the network.

     virsh net-define enp5s0f1-vf.xml
    
  3. Start the network.

     virsh net-start enp5s0f1-vf
    
  4. Configure the network to start each time the system is started.

     virsh net-autostart enp5s0f1-vf
    

Configure v35 to use 10Gbps interface

This procedure assumes that the v35 is already set up and running and that a 10Gbps NIC is already installed in the KVM host. It also assumes that virtual functions support, SR-IOV support, and a virtual function network have been configured.

You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
V35_INTERFACE v35 interface to use (either eth0, eth2, or eth3).
10G_INTERFACE The name of a 10Gbps Virtual Function (-vf) network on the KVM host (e.g., enp5s0f1-vf).

To update the v35 configuration for 10Gbps:

  1. Find the name of the 10Gbps interface you want to use (e.g., enp5s0f1-vf).

    ./vk35tool.py scan
    
  2. Shutdown the v35.

    virsh shutdown V35_NAME
    
  3. Update the v35 configuration for 10Gbps support. Map an interface on the v35 to the 10Gbps interface on the KVM host.

    ./vk35tool.py update V35_NAME \
    --V35_INTERFACE type=network,source=10G_INTERFACE
    
  4. Start the v35.

    virsh start V35_NAME
    
  5. Verify that the v35 is “running”.

    virsh list --all
    

Connect to the network via 10Gbps

To physically connect the monitoring point to the network via 10Gbps:

  1. Verify that the switch you’ll be connecting to is 10Gbps capable has its default port speed and duplex set to auto-negotiate.
  2. Using the appropriate cable, connect the 10Gbps port to the switch.
  3. Verify that the v35 is connected to APM.
    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is configured to use the 10Gbps interface.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Set up v35 with high performance 10Gbps

This procedure assumes that the v35 is already set up and running and that it has been configured for 10Gbps.

When using 10Gbps ports for monitoring, you can improve performance by running the v35 on at least two dedicated CPU cores. Doing this involves three steps:

  1. Determine which CPU cores to use for the v35.
  2. Isolate the CPU cores.
  3. Pin the v35 to the isolated cores

Determine which CPU cores to use for the v35

The following points should be used to determine which CPU cores on the KVM host to isolate for use with the v35:

  • If you have a multi-socket (NUMA) system:
    • The isolated cores should be on the same node so they access the same local memory.
    • Use cores on the node that is connected to the PCI slot containing the 10Gbps NIC.
  • If you have a hyper-threaded system (where a single physical processor core can act like two logical cores), make sure the isolated cores are not logical cores on the same physical core.

Isolate the CPU cores

You’ll need the following information to complete the setup:

Parameter Description
CPUS A list of the CPUs you want to isolate. For example, 4-6,8-10 to isolate cores 4, 5, 6, 8, 9, and 10.

To configure the KVM host to isolate CPU cores:

  1. Edit /etc/default/grub.
    1. Locate GRUB_CMDLINE_LINUX and add isolcpus=CPUS and nohz_full=CPUS to it.
      • For example, “GRUB_CMDLINE_LINUX … isolcpus=4-6,8-10 nohz_full=4-6,8-10 …”.
  2. Regenerate the grub.cfg file.
    • For CentOS or RHEL, use:

       sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      
    • For Ubuntu, use:

       sudo update-grub
      
  3. Rebuild the boot initramfs.
    • For CentOS or RHEL, use:

       sudo dracut --regenerate-all -f
      
    • For Ubuntu, use:

       sudo update-initramfs -k all -u
      
  4. Reboot the KVM host to apply the changes.
  5. Verify the changes.

    cat /proc/cmdline
    
    • Look for isolcpus=CPUS and nohz_full=CPUS.

Pin the v35 to the isolated cores

You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
CPUS A list of the isolated CPU cores.

To update the v35 configuration so it is pinned to the isolated CPU cores:

  1. Shutdown the v35.

    virsh shutdown V35_NAME
    
  2. Update the v35 configuration to pin it to the CPUs that have been isolated.

    ./vk35tool.py update V35_NAME --cpuset CPUS
    
  3. Start the v35.

    virsh start V35_NAME
    
  4. Verify that the v35 is “running”.

    virsh list --all
    
  5. Verify that the v35 is connected to APM.

    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is configured for high performance 10Gbps.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Set up v35 with 10Gbps and VLAN

This procedure assumes that the v35 is already set up and running and that it has been configured for 10Gbps or high performance 10Gbps.

You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
V35_INTERFACE v35 interface to use (either eth0, eth2, or eth3).
10G_INTERFACE The name of a 10Gbps Virtual Function (-vf) interface on the KVM host (e.g., enp5s0f1-vf).
VLAN_ID VLAN ID to use.

To update the v35 configuration to use VLAN:

  1. Shutdown the v35.

    virsh shutdown V35_NAME
    
  2. Update the v35 configuration to specify the VLAN to use on the 10Gbps interface.

    ./vk35tool.py update V35_NAME \
    --V35_INTERFACE type=network,source=10G_INTERFACE,vlan=VLAN_ID
    
  3. Start the v35.

    virsh start V35_NAME
    
  4. Verify that the v35 is “running”.

    virsh list --all
    
  5. Verify that the v35 is connected to APM.

    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is ready to monitor VLAN traffic on the 10Gbps interface.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Set up v35 with additional interfaces

In addition to eth0, you can configure up to two additional interfaces for Delivery and Experience monitoring (eth2 and/or eth3). This procedure assumes that the v35 is already set up and running.

You’ll need the following information to complete the setup:

Parameter Description
V35_NAME v35 virtual machine name.
V35_INTERFACE v35 interface to use (either eth2 or eth3).
BRIDGE_NAME Additional Linux bridge interface on the KVM host (determined below).

To update the v35 to use an additional interface:

  1. Determine which bridge interfaces are available to the v35.

    ./vk35tool.py scan
    
  2. Shutdown the v35.

    virsh shutdown V35_NAME
    
  3. Update the v35 configuration to use an additional interface.

    ./vk35tool.py update V35_NAME \
    --V35_INTERFACE type=bridge,source=BRIDGE_NAME
    
  4. Start the v35.

    virsh start V35_NAME
    
  5. Verify that the v35 is “running”.

    virsh list --all
    
  6. Verify that the v35 is connected to APM.

    1. In APM, navigate to > Manage Monitoring Points.
    2. Wait a couple minutes for the new monitoring point to show up in the list (you may need to refresh your screen).
      • The monitoring point status should show “Connection Established” (). At this point it is ready to monitor traffic.
      • If it doesn’t appear after a few minutes, you’ll need to troubleshoot the problem.

Helpful commands

The following commands can be used on the host to manage your KVM virtual machines.

Command Description
virsh list –all List all virtual machines on your host.
virsh start V35_NAME Start a v35 instance that has been shut down.
virsh shutdown V35_NAME Shutdown a v35 instance (graceful).
virsh destroy V35_NAME Shutdown a v35 instance (force).
virsh undefine V35_NAME Delete a v35 instance.
virsh -h virsh help.

The following commands can be used to create and manage v35 virtual machines.

Command Description
vk35tool.py install V35_NAME Create a v35 VM.
vk35tool.py update V35_NAME Update a v35 VM configuration.
vk35tool.py show V35_NAME Show network interface on a v35 VM.
vk35tool.py scan List Linux interfaces available to a v35 VM.
vk35tool.py -h vk35tool help.