Voice Delivery

AppNeta Performance Manager (APM) has monitoring capability designed specifically for ensuring good voice quality. It enables you to assess your network’s ability to handle voice traffic with two tools: voice assessment and voice test.

Assessment vs. test

Assessments differ from tests both in methodology and purpose. They rely on continuous monitoring and diagnostic techniques to infer voice quality. They’re good tools for testing the suitability of your entire network in advance of a voip deployment, or checking an existing deployment for issues. Many paths can be tested simultaneously, and the testing is light on bandwidth consumption compared to a voice test. A typical use for a voice assessment might be to test the quality of the network paths between the server room and the voip handsets on end-users’ desks. This is as opposed to performing the assessment over a WAN link.

In contrast to assessments, voice tests simulate voice calls using the same application layer protocols and codecs that would be used in an actual voice call. As a result, voice tests provide a more accurate measure of how your network would treat voice traffic, but at the expense of greater bandwidth consumption. A voice test can test 100+ concurrent voice calls depending on monitoring point model and as such, it consumes the same amount of bandwidth that one would expect if there were that many actual calls being made on the network. Voice tests should be used to measure voice performance between two sites—perhaps across an MPLS WAN link, or a trunk between two buildings. As it uses the same signaling and codecs as a voice call, it’s able to gather more in-depth metrics for analysis, such as packet reorder and discards, which is not possible with voice assessments. If you are experiencing voice issues between sites, or want to load test a link for voice performance, then a voice test is the correct tool to use.

Prerequisite: Voice delivery monitoring requires path targets to be a monitoring point.

Basic vs. advanced

Basic voice assessment runs a diagnostic on each selected path but returns path-level—rather than hop-level—readiness, MOS, loss, and jitter.

Advanced voice assessment is designed to be executed for longer periods in order to capture transient conditions. To that end, they consist of periodic diagnostics plus continuous monitoring. Because of this combination of analysis techniques advanced assessments are able to additionally return latency, total capacity, and the percentage of time the path was available during the test. If the net MOS score is less than 3.8 an additional diagnostic is triggered for the path. Advanced assessment config options also include call load ramp-up and scheduled start/stop time.

Voice test vs. PathTest

Voice test is similar to PathTest: they’re both short-duration voice testing tools and they offer mostly the same options. The difference is that PathTest uses layer 3 to simulate voice traffic, like continuous monitoring and voice assessments; and it returns only packet loss statistics at the selected bit rate. You would use these results primarily to corroborate the voice loss results returned from continuous monitoring, which is in part why PathPlus is included in the basic license. Generally voice tests and assessments are better options, with one exception which is that PathTest can generate higher call loads than voice test.

Simulated vs. real

When measuring call quality, it doesn’t matter whether you are using test traffic containing real encoded human voice. Test traffic is comprised of two components, one is SIP and the other is RTP. SIP is the control protocol that is responsible for coordinating the two endpoints of the call. SIP doesn’t care whether those endpoints are monitoring points or actual humans, the nature and flow of that traffic is the same. The second component, RTP, is the application layer protocol that wraps the encoded human voice sample. The resulting RTP packet is encapsulated in UDP, and then again in IP, so ultimately all the network layer sees, or needs to see for that matter, is the IP header.

So how can APM assess call quality when the RTP payload is meaningless? The answer is that we know which network characteristics have an impact on call quality, and we know how each one impacts call quality. The characteristics we’re interested in are bandwidth utilization, loss, latency, and jitter.

Bandwidth
To convert human voice from analog to digital, it’s sampled thousands of times per second, using one of several techniques called ‘codecs’ that results in not only conversion, but also compression. The amount of compression, the number of samples taken, and the number of samples packed into each IP packet all directly affected how much bandwidth is consumed by the call. When your call is traversing a network with insufficient bandwidth for the voip configuration, the call will experience higher latency and possibly packet loss.
Packet loss
Packet loss is important because when you use UDP, lost packets are not re-transmitted, and so lost packets results in broken audio on the listener’s end.
Latency
See latency
Jitter
See jitter

Issues with cloud-based providers

Suppose you have a monitoring point deployed at small office and the cloud-based voip service is choppy and echoey. In a situation like this, the first step would be to verify the integrity of the network connection itself. Create a dual-ended path to one of our AppNeta WAN targets, and add the default WAN alert profile to it. Let it run for a while, and if everything with that path looks good, the next step is to verify the connection to the voip server. Create a single-ended path to the service provider’s PBX; you might have to get in touch with the service provider to get its IP. Use the default voice WAN alert profile to capture any events and kick off diagnostics. Again, let it run for a while, and once you’ve established that your network paths are in decent shape, run a basic voice assessment. It has the advantage of being lightest in weight of all the tools in APM; pay attention to MOS, loss, and jitter. If the basic assessment doesn’t capture the issues you’re experiencing, try a longer running advanced voice assessment.

Choosing good voice targets

All handsets are not created equal, some have a limited ability to respond to our test packets. This would manifest as low MOS even though the audio sounds fine. When in doubt, open a support ticket.

Pay attention to the kind of network device you are targeting for an assessment. If running an assessment on a path that is targeting a printer, that assessment is going to return terrible results—which is expected because it’s not related to voice in any way. When you’re using ‘discover my network’. Narrow the address range to just your voip subnet. This shouldn’t be an issue with ‘add monitored paths’ or ‘add new path’ since data only target types will not be available for selection.

Voice monitoring terms

Packet discards
Packets arriving at the destination too late, as determined by the jitter buffer size. For example, if a packet arrives more than 40ms later than expected, it is discarded when the default jitter buffer size of 40ms is used.
Total capacity
See what is capacity?
Availability
The percentage of time a connection exists between the monitoring point and the target during the length of the assessment.
Reordering
Packets within a burst that arrived out of their original sequence. Because packets travel through a network path independently, they may not arrive in the order in which they were sent. In multi-link, load-balanced networks, data can become reordered, especially during high-traffic periods
Readiness
A 5-tier representation of MOS that helps you understand well a path is handling voice traffic: 4.2 - 5 is excellent, 3.8 - 4.19 is good, 3.4 - 3.79 is marginal, 2.8 - 3.39 is poor, and 1.0 - 2.79 is very poor.
MOS
An estimate of the rating that a typical user would give to the sound quality of a call. It is expressed on a scale of 1 to 5, where 5 is perfect. It is a function of loss, latency, and jitter. It also varies with voice codec and call load. If audio codec G.722.1 is selected for a session, a MOS score will not appear.
Packet loss
See simulated vs. real.
Latency
See simulated vs. real.
Jitter
See simulated vs. real.

Voice test

In contrast to assessments and continuous monitoring, voice test uses higher layer protocols SIP and RTP simulate voice calls. This better approximates actual calls and offers more voice-related statistics, but comes at the expense of greater bandwidth consumption. It returns to you a couple more voice-related statistics than continuous monitoring, but also requires the path target to be a monitoring point.

A voice test is comprised of one or more sessions; each session specifies a path, a number of concurrent calls, and QoS settings. Tests are structured this way so that you can evaluate a variety of call scenarios, all within the same test.

  1. You must add at least one session: click ‘Add new Session…’.
  2. All of the options have defaults, so the least you have to do is select a path from the drop-down.
  3. Click OK to accept your settings. APM will then verify: the source and destination monitoring point software version number; that the source monitoring point is connected to APM; and that the destination monitoring point is reachable at the ports specified in the sessions advanced setting. If all of these checks must pass in order to include the session in the test. In any case, you’ll notice that the ‘Save Voice Test’ and ‘Save as Template’ buttons become available.

  4. Select one or more valid sessions, and then one of the following: ‘Start Voice Test’, ‘Save Voice Test’, ‘Schedule Test’, or ‘Save as Template’.

    Save as template
    Save the test so that you can later clone it to make new tests; templates are listed on the ‘templates’ tab of the manage voice tests page, where there are additional options under in the action menu. Click a row edit the template.
    Start voice test
    Start the voice test immediately.
    Save voice test
    Save your voice test settings so that you can run the test later. Saved voice tests are listed on the ‘tests’ tab of the manage voice tests page. Rows display a diskette icon, and additional options are available under in the action menu.
    Schedule test
    Start the voice test at a later date and time. Scheduled voice tests are listed on the ‘schedules’ tab of the manage voice tests page, where there are additional options under in the action menu.
  5. Voice test results:
    • ‘Run test again’ takes you to the New Voice Test page and loads the exact same session configurations. From there you can start, save, schedule, etc.
    • Download a copy of the report by selecting ‘download pdf’ from this dropdown.
    • Tests can be in one of these states: initializing, running, completed, saved, stopped, and failed.
    • Links are provided to view advanced settings and metrics.
    • You can display up to 10 sessions at a time on the same set of charts.
    • Moving your cursor along plotted data displays a the time the measurement was taken; the chart is updated with the value.

The maximum number of calls or sessions you can have in a voice test is subject device load. APM will reject any test configuration that will exceed device load for either the source or the target. An error message indicates which device would be overloaded and by how much.

Basic voice assessment

Assessments enable you to evaluate the voice capability of your entire network at once. Whereas voice test assesses the voice capability on a per-path basis, i.e., each path has its own test settings, assessment is a shotgun approach in which you add multiple paths and apply the same settings to each. Basic voice assessment doesn’t offer scheduling, call load ramp-up, MOS-triggered diagnostics, or a path readiness score; advanced voice assessment does.

If any path is targeting an OS X sequencer: Note that OS X rate limits ICMP responses by default. There are two ways to disable this behavior:
  • To disable it temporarily: enter sysctl –w net.inet.icmp.icmplim=0 from the command line; the limit will be reset when the system restarts.
  • To permanently remove it: create _/etc/sysctl.conf_ and add the following line: net.inet.icmp.icmplim=0
  1. Navigate to Delivery > Voice Delivery, and click ‘+new basic voice assessment’.

  2. Choose from the applicable existing paths, or create new paths.

    Add from monitored paths
    Data only target types will not be available for selection. All targets other than ‘AppNeta monitoring point’ are converted to target type ‘voice handset’, this just controls some under-the-hood mechanics of testing.
    Discover targets
    See Discover my network.
Add new path
Source monitoring points must have a voice delivery license.

  1. Configure your basic voice assessment’s settings.

    Quality of service
    If QoS testing is enabled, the each path is tested twice: once with the QoS on, and once with QoS off. After a new QoS setting is created, you can manage it from > Manage Path QoS Templates.
    Target type
    When a selected path has a target type that is not selected for call load analysis, that path is tested with only one call.
    Call load
    The default number of concurrent calls is 5, the maximum is 250. When the call load is greater than one, the test is performed multiple times starting with one call and incrementing by one each time.

  2. Basic voice assessment results:

    • Download a copy of the report by selecting ‘download pdf’ from this dropdown.
    • ‘Run assessment again’ takes you to the New Voice Test page and loads the exact same session configurations. From there you can start, save, schedule, etc.
    • The bar chart shows the number of tested devices at each readiness level.

Advanced voice assessment

Assessments enable you to evaluate the voice capability of your entire network at once. Whereas voice test assesses the voice capability on a per-path basis, i.e., each path has its own test settings, assessment is a shotgun approach in which you add multiple paths and apply the same settings to each. Advanced voice assessment is very similar to continuous monitoring: it uses layer 3 using ICMP and UDP packets to simulate voice traffic and returns to you mostly the same measurements; the difference is that it allows you to vary the call load and ramp-up, and returns to you a snapshot of multiple paths in a nicely formatted report. The report includes a color-coded readiness score for each path so that you know at-a-glance where any problems lie.

  1. Navigate to Delivery > Voice Delivery > and click the button shown.

  2. Choose from the applicable existing paths, or create new paths.

    Add from monitored paths
    Paths with a target type of ‘auto’ are converted to target type ‘voice handset’.
    Discover targets
    See Discover my network
Add new path
Source monitoring points must have a voice delivery license.
  1. When you schedule a voice assessment for a future date, the paths involved in the assessment count towards your path count, not just on the date the assessment runs, but from the date you make the schedule until the assessment completes.

  2. Adjust the slider to update the calculated duration for cycle, ramp-up, and steady-state.

    Call load ramp-up
    Call load ramp-up simulates call load fluctuations that could occur over a 24-hour period. One ‘call load cycle’ consists of a period during which call load increases from 1 to the specified number of concurrent calls, and a period of steady state at the specified concurrent call number.
    Concurrent calls
    The default number of concurrent calls is 5, the maximum is 25. When the call load is greater than one, the test is performed multiple times starting with one call and incrementing by one each time.
    Target types
    When a selected path has a target type that is not selected for call load analysis, that path is tested with only one call.
  3. Advanced voice assessment results:
    • Download a copy of the report by selecting ‘download pdf’ from this dropdown.
    • ‘Run assessment again’ takes you to the New Voice Test page and loads the exact same session configurations. From there you can start, save, schedule, etc.
    • The bar chart shows the number of tested devices at each readiness level.