Analyze Results

Learn about dashboards, reports, tests, and interpreting monitoring results.

Voice Delivery

AppNeta Performance Manager (APM) has monitoring capability designed specifically for ensuring good voice quality. It enables you to assess your network’s ability to handle voice traffic with three tools: Basic voice assessments, Advanced voice assessments, and Voice tests. In order to use these tools, the source Monitoring Point needs to be licensed for voice.

Basic voice assessments provide a way to assess the ability of multiple network paths to carry voice traffic. This testing happens at a point in time and relies on diagnostic techniques to infer voice quality. Basic voice assessments do not require a Monitoring Point as a target and are typically used to:

  • Provide a quick voice assessment of your network in advance of a VOIP deployment
  • Check an existing voice deployment for issues
  • Assess network paths using a large number of concurrent calls (up to 250)
    • Assess endpoints (e.g., VOIP handsets) on a LAN
    • Assess a voice service provider across a WAN (no MP required at provider)
  • Validate QoS

Advanced voice assessments also provide a way to assess the ability of multiple network paths to carry voice traffic. The difference from Basic assessments is that this testing happens continually over a specified period of time and relies on both diagnostic and continuous monitoring techniques to infer voice quality. Like basic voice assessments, Advanced voice assessments do not require a Monitoring Point as a target. They are typically used to:

  • Provide a quick voice assessment of your network in advance of a VOIP deployment
  • Check an existing voice deployment for issues
  • Assess network paths (up to 25) over a period of time in order to capture transient conditions
    • Assess endpoints (e.g., VOIP handsets) on a LAN
    • Assess a voice service provider across a WAN (no MP required at provider)
  • Determine how varying call loads affect the ability to handle voice traffic using the call ramp-up feature

Voice tests provide a way to simulate multiple simultaneous voice calls using the actual voice codecs and protocols. They allow you to more accurately measure of how your network would treat voice traffic, but at the expense of greater bandwidth consumption. A voice test is comprised of one or more sessions; each session specifies a path, a number of concurrent calls, and QoS settings. Tests are structured this way so that you can evaluate a variety of call scenarios, all within the same test. Voice tests require a Monitoring Point as the target (though the target Monitoring Point does not require a voice license) and are typically used to:

  • Create voice traffic load to expose network problems. Other tools are used to diagnose the issues found.
  • Test a network’s ability to handle actual voice traffic (for example, prior to deploying additional voice endpoints)
  • Test a network’s ability to handle inter-office voice traffic (for example, HQ to branch)

Note: If you do not have a voice license, you can use PathTest as a replacement for Voice test, though it does not use voice protocols (SIP/RTP/RTCP) to simulate voice traffic so results are less accurate than using Voice test. PathTest is also useful when larger call loads are required.

Features include:

Feature Basic voice
assessments		 Advanced voice
assessments		 Voice tests
Concurrent calls 250 25 Depends on monitoring
point capacity
Target Monitoring Point required No No Yes
(target does not require
a voice license)
Protocols used ICMP ICMP UDP (SIP/RTP/RTCP)
Network load Light Light True call loads
Targets assessed/tested One at a time One at a time,
round robin,
for period specified		 Simultaneously
Compares QoS and non-QoS Yes
(if QoS selected)		 No No
Uses Diagnostics Yes Yes No
Uses CPA No Yes No
Individual test results available Yes No No
Schedule assessment/test No Yes Yes
Call ramp-up No Yes No
Downloadable report Yes Yes Yes
Statistics generated      
- Readiness Yes Yes No
- MOS Yes Yes Yes
- Voice packet loss Yes Yes Yes
- Voice jitter Yes Yes Yes
- Latency No Yes Yes
- Availability No Yes No
- Total capacity No Yes No
- Packet discards No No Yes
- Packet reordering No No Yes

Voice monitoring metrics

The following describes the metrics available on voice assessment and voice test reports:

  • Readiness - A representation of MOS that helps you understand how well a path is handling voice traffic:
    • 4.2 - 5.00 - Excellent
    • 3.8 - 4.19 - Good
    • 3.4 - 3.79 - Marginal
    • 2.8 - 3.39 - Poor
    • 1.0 - 2.79 - Very poor
  • MOS - An estimate of the rating that a typical user would give to the sound quality of a call. It is expressed on a scale of 1 to 5, where 5 is perfect. It is a function of loss, latency, and jitter. It also varies with voice codec and call load. If audio codec G.722.1 is selected for a session, a MOS score will not appear.
  • Voice packet loss - See Data and voice loss.
  • Voice jitter - See Data and voice jitter.
  • Latency - See Round-trip time and latency.
  • Availability - The percentage of time a connection exists between the Monitoring Point and the target during the period of the assessment.
  • Total capacity - See Capacity
  • Packet discards - Packets arriving at the destination too late, as determined by the jitter buffer size, are discarded. For example, when the default jitter buffer size of 40ms is used, if a packet arrives more than 40ms later than expected, it is discarded.
  • Packet reordering - Packets within a burst that arrived out of their original sequence are considered reordered. Because packets travel through a network path independently, they may not arrive in the order in which they were sent. In multi-link, load-balanced networks, data can become reordered, especially during high-traffic periods.

Assessing a voice service provider

If the voice quality you are receiving from a cloud-based VOIP service is not acceptable, there are a few steps you can take to determine where the problem is:

  1. Test the WAN link - Create a dual-ended path from the location experiencing the problem to an AppNeta Global Monitoring Target and let it run for a while. If all looks good, continue.
  2. Test the connection to the VOIP server - Create a single-ended path from the location experiencing the problem to the service provider’s PBX (you may need to get in touch with the service provider to determine its IP address). Apply the default voice WAN alert profile to capture any events and initiate diagnostics tests. Let this run for a while. If all looks good, continue.
  3. Run a basic voice assessment - Use the single-ended path to create and run a basic voice assessment against the service provider’s PBX. Review the report generated. If all looks good, continue.
  4. Run an advanced voice assessment - Use the single-ended path to create and run an advanced voice assessment against the service provider’s PBX. Let it run for a while and review the report generated. If all looks good, continue.
  5. Contact AppNeta Support - If you are still unable to determine where the voice quality issues are originating from, contact AppNeta Support.

Assessing endpoints on a LAN

When you are assessing voice endpoints on a LAN, make sure you are targeting the correct devices. Devices such as printers will fail a voice assessment. This issue is particularly evident when using the “Discover My Network” method to create single-ended paths. To avoid the issue, narrow the address range to just your VOIP subnet.

Also note that not all VOIP handsets will respond to test packets. This results in a low MOS even though the audio sounds fine. If you need help, contact AppNeta Support.

Voice quality assessments

Voice quality assessments are based on four metrics that affect voice quality: bandwidth utilization, packet loss, latency, and jitter. When human voice is converted from analog to digital, it is sampled thousands of times per second, using one of several techniques called ‘codecs’ that not only converts the sampled voice, but also compresses it. The amount of compression, the number of samples taken, and the number of samples packed into each IP packet all directly affected how much bandwidth is consumed by the call. When your call is traversing a network with insufficient bandwidth for the VOIP configuration, the call will experience higher latency and possibly packet loss. Packet loss is important because VOIP traffic uses the UDP protocol. With UDP, lost packets are not re-transmitted, resulting in broken audio on the listener’s end.

Basic voice assessments

The actions available for Basic voice assessments include:

Create a Basic voice assessment

To create a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. Click + New Basic Voice Assessment.
  3. In the Assessment Name field, enter a name for the assessment.
  4. In the Select Paths section, specify the paths to include in the assessment.
    • Data only target types will not be available for selection.
    • Paths with a target type of Auto are converted to target type Voice Handset.
  5. Update the remaining fields as appropriate.
    • Call Load - If the call load is >1 and you specify a call load target type that matches the path target type, the path will be tested up to five times with different call loads. Otherwise, it will be tested once.
    • Quality of Service - If this is set, two tests will be performed for each call load - one with QoS set and one without.
  6. Click Save Assessment.
    • The assessment is saved.

Note: For paths targeting a macOS NMP, note that macOS rate limits ICMP responses by default. There are two ways to disable this behavior:

  • To disable it temporarily, enter sysctl –w net.inet.icmp.icmplim=0 from the command line. The limit will be reset when the system restarts.
  • To permanently remove it, create /etc/sysctl.conf and add the following line: net.inet.icmp.icmplim=0.

Run a Basic voice assessment

To run a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to run, select > Start (or Run Assessment Again).
    • The assessment starts (or you will be given a chance to edit the assessment before starting it). The status shows: .

View Basic voice assessment results

To view Basic voice assessment results:

  1. Navigate to Delivery > Voice Assessments.
  2. Click the completed assessment you are interested in (status shows: ).
    • The assessment results are displayed.
  3. To view all tests on a path, click to the left of the path.
    • The individual tests are listed.
  4. To view test details, click the test you are interested in.

Note: You can also share a link to the page or download the content as a PDF.

Edit a Basic voice assessment

To edit a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to edit, select > Edit.
  3. Update fields as appropriate.
  4. Click Save Assessment.
    • The assessment is saved.

Delete a Basic voice assessment

To delete a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to delete, select > Delete.
  3. Click OK.
    • The assessment is deleted.

Pause a Basic voice assessment

To pause a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to pause (status shows: ), select > Pause.
    • The assessment is paused. The status shows: .

Resume a paused Basic voice assessment

To resume a paused Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the paused assessment you want to resume (status shows: ), select > Resume.
    • The assessment resumes. The status shows: .

Stop a Basic voice assessment

To stop a Basic voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the currently running assessment you want to stop (status shows: ), select > Stop.
    • The assessment stops. The status shows: .

Advanced voice assessments

The actions available for Advanced voice assessments include:

Create an Advanced voice assessment

To create an Advanced voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. Click + New Advanced Voice Assessment.
  3. In the Assessment Name field, enter a name for the assessment.
  4. In the Select Paths section, specify the paths to include in the assessment.
    • Data only target types will not be available for selection.
    • Paths with a target type of Auto are converted to target type Voice Handset.
  5. Update the remaining fields as appropriate.
    • Assessment Schedule - Schedule the start and end times for testing.
    • Call Load Analysis - If the call load is >1 and you specify a call load target type that matches the path target type, the path will be tested up to five times with different call loads. Otherwise, it will be tested once. You can also ramp-up the number of concurrent calls as the test proceeds and have the ramp-up repeat daily.
  6. Click Start Assessment.
    • The assessment is saved and run.

View an Advanced voice assessment

To view Advanced voice assessment results:

  1. Navigate to Delivery > Voice Assessments.
  2. Click the completed assessment you are interested in (status shows: ).
    • The assessment results are displayed.

Note: You can also share a link to the page or download the content as a PDF.

Edit an Advanced voice assessment

To edit an Advanced voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to edit, select > Edit.
  3. Update fields as appropriate.
  4. Click Save Assessment.
    • The assessment is saved.

Delete an Advanced voice assessment

To delete an Advanced voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to delete, select > Delete.
  3. Click OK.
    • The assessment is deleted.

Stop an Advanced voice assessment

To stop an Advanced voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the currently running assessment you want to stop (status shows: ), select > Stop.
    • The assessment stops. The status shows: .

Re-run an Advanced voice assessment

To re-run an Advanced voice assessment:

  1. Navigate to Delivery > Voice Assessments.
  2. For the assessment you want to run, select > Run Assessment Again.
    • You will be given a chance to edit the assessment before starting it.
  3. Click Start Assessment.
    • The assessment is run. The status shows: .

Voice tests

You can create a Voice test directly or you can create it from a template. The test can be run once or it can be scheduled to run repeatedly.

The actions available for Voice tests include:

The actions available for Voice test templates include:

The actions available for Voice test schedules include:

Create a Voice test

To create a Voice test:

  1. Navigate to Delivery > Voice Tests.
  2. Click + New Voice Test.
  3. Update fields as appropriate.
  4. Click Add new Session….
  5. Select the network path to run the test on.
  6. Update fields as appropriate.
  7. Click OK.
    • The source and destination are checked for applicability. Four checkmarks are required in each row to continue.
  8. Add more test sessions if required.
  9. Click Start Voice Test.
    • The Voice test starts.

Note: The maximum number of calls or sessions you can have in a Voice test is subject to device load. APM will reject any test configuration that will exceed device load for either the source or the target. An error message indicates which device would be overloaded and by how much.

Create a Voice test from a template

To create a Voice test from a template:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Templates tab.
  3. For the template you want to use, select > Create Test.
  4. Update fields as appropriate.
  5. Click OK.
    • The source and destination are checked for applicability. Four checkmarks are required in each row to continue.
  6. Click Start Voice Test.
    • The Voice test starts.

Run a Voice test

To run a Voice test:

  1. Navigate to Delivery > Voice Tests.
  2. For the Voice test you want to run, select > Run Test Again.
  3. Click Start Voice Test.
    • The Voice test starts.

View Voice test results

To view Voice test results:

  1. Navigate to Delivery > Voice Tests.
  2. Click the completed test you are interested in (status shows: ).
    • The test results are displayed.

Note: You can also share a link to the page or download the content as a PDF.

Edit a Voice test

To edit a Voice test:

  1. Navigate to Delivery > Voice Tests.
  2. For the Voice test you want to edit, select > Edit.
  3. Update fields as appropriate.
  4. Click Start Voice Test.
    • The Voice test starts.

Delete a Voice test

To delete a Voice test:

  1. Navigate to Delivery > Voice Tests.
  2. For the Voice test you want to delete, select > Delete.
  3. Click OK.
    • The Voice test is deleted.

Create a Voice test template

Voice test templates are used to create Voice tests with similar criteria.

To create a Voice test template:

  1. Navigate to Delivery > Voice Tests.
  2. Click + New Voice Test.
  3. Update fields as appropriate.
  4. Click Add new Session….
  5. Select the network path to run the test on.
  6. Update fields as appropriate.
  7. Click OK.
    • The source and destination are checked for applicability. Four checkmarks are required in each row to continue.
  8. Add more test sessions if required.
  9. Click Save as Template.
  10. Click Submit.
  11. Update fields as appropriate.
  12. Click Save Template.
    • The configuration is saved as a template.
  13. Click the Templates tab to confirm the template was created.

View a Voice test template

To view a Voice test template:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Templates tab.
  3. Click the template you want to view.
    • The template is displayed.

Edit a Voice test template

To edit a Voice test template:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Templates tab.
  3. Click the template you want to edit.
  4. Update fields as appropriate.
  5. Click Save Template.
    • The updated template is saved.

Delete a Voice test template

To delete a Voice test template:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Templates tab.
  3. For the template you want to delete, select > Delete.
  4. Click OK.
    • The template is deleted.

Create a Voice test schedule

Voice test schedules are used to repeatedly execute a Voice test at a given interval for a specified period of time.

To create a Voice test schedule:

  1. Navigate to Delivery > Voice Tests.
  2. Click + New Voice Test.
  3. Update fields as appropriate.
  4. Click Add new Session….
  5. Select the network path to run the test on.
  6. Update fields as appropriate.
  7. Click OK.
    • The source and destination are checked for applicability. Four checkmarks are required in each row to continue.
  8. Add more test sessions if required.
  9. Click Schedule Test.
  10. In the Schedule Name field, enter a schedule name.
  11. Update the remaining fields as appropriate.
  12. Click Create Schedule.
    • The schedule is saved and a test is scheduled.
  13. Click the Schedules tab to confirm the schedule was created.

View a Voice test schedule

To view a Voice test schedule:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Schedules tab.
  3. Click the schedule you want to view.
    • The schedule is displayed.

Stop a scheduled Voice test

To stop a scheduled Voice test:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Schedules tab.
  3. For the schedule you want to stop, select > Stop.
    • The schedule is stopped.

Copy a Voice test schedule

To copy a Voice test schedule:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Schedules tab.
  3. For the schedule you want to copy, select > Copy.
  4. Click Schedule Test.
  5. Change the Schedule Name field.
  6. Update the remaining schedule fields as appropriate.
  7. Click Create Schedule.
    • The new schedule is created.

Delete a Voice test schedule

To delete a Voice test schedule:

  1. Navigate to Delivery > Voice Tests.
  2. Click the Schedules tab.
  3. For the schedule you want to delete, select > Delete.