When you receive an APM web path alert notification, use this workflow to troubleshoot the problem.

Step 1: Record web path alert notification details.

Make note of the alert notification details to use during investigation:

  • Event Time - the time of the violation
  • Monitoring Point - the source Monitoring Point
  • Web App Group - the web app group involved
  • Workflow - the workflow involved
  • Target - the target web app
  • Details - the violation details (e.g., HTTP Error)
Step 2: Determine how widespread the problem is.

To determine the problem scope we compare the number of violating paths from the same web app group, from the same source Monitoring Point, or to the same target to see which is largest. The procedure to use depends on when the violation occurred.

Violation is current

For current violations:

  1. In APM, navigate to Experience > Web Paths.
  2. Filter by violating web paths.
    GIF of creating a Status = Error/Violated filter.
  3. Filter by the web app group listed in the alert notification (e.g., ‘Gsuite’). Note the number of matching paths (e.g., 5).
    GIF of creating a web app group filter.
  4. Remove the web app group filter (e.g., ‘Gsuite’).
  5. Filter by the source Monitoring Point listed in the alert notification (e.g., ‘Boston-MA-r90’). Note the number of matching paths (e.g., 10).
    GIF of creating a source Monitoring Point filter.
  6. Remove the source Monitoring Point filter (e.g., ‘Boston-MA-r90’).
  7. Filter by the target listed in the alert notification (e.g., ‘accounts.google.com’). Note the number of matching paths (e.g., 5).
    GIF of creating a target filter.
  8. Repeat the search using the filter that returned the most violating web paths (in this example, the filter by source Monitoring Point) to focus on the issue causing the most path violations.
  9. Confirm that the web paths are violating for the same reason.
    1. Click the web path identified in the alert notification (same Monitoring Point, Target, Web App Group, and Workflow).
    2. Make note of the violation type(s) in the Web Path Details pane on the right.
    3. Make note of the error message(s) in the Latest Transaction Details pane.
      Screen shot of Test Timeline page showing the violation types on the right and the error messages within the Latest Transaction Details pane.
    4. Repeat for several other violating web paths in the list to confirm that they are violating for the same reason.
      • This will confirm how widespread the problem is (e.g., affecting 10 web paths related to the source Monitoring Point).
Violation happened this week

For violations that happened this week:

  1. In APM, navigate to Experience > Events.
  2. Find violation events of the same type at the same time and note the number of matching events.
    1. Filter by ‘Event Type = Web Alert Profile’ (show events triggered when an alert threshold is violated or cleared).
    2. Filter by ‘violated’ (show only violation events).
    3. Filter by the source of the path known to have had a problem (e.g., the string ‘boston’ was in the source web path that was alerted on. We could have used ‘Boston-MA-r90’ to be more specific.).
    4. Sort events by ‘Event Time’.
    5. Note the number of violation events of the same type (e.g., Page Load Time) that occurred around the time of the alert by reviewing the ‘Description’ column and the time stamps.
      GIF of the Experience Events: Past 7 Days page with filters created for 'Event Type = Web Alert Profile', 'violated', then ordering by Event Time.
  3. Repeat the search for the target then web app group and note the number of violation events that occurred at the time of interest.
    • This will provide an indication of how widespread the problem is based on source, target, and web app group.
Violation happened more than a week ago

For violations that happened more than a week ago:

  1. In APM, navigate to Experience > Web Paths
  2. In the Search field, enter search terms to filter web paths suspected of having the same problem.
    Screen shot of the Web Paths page with 'test-mp' highlighted in the search box.
  3. Ctrl-click (or Cmd-click) several of the web paths listed.
    • The web path performance charts for each web path are displayed in separate tabs.
  4. For the web path performance charts in each tab, select a time range and look for similar issues at the same time.
    • This will provide an indication of how widespread the problem is.
Step 3: Investigate web path alerts.
Connectivity alerts

Web path Connectivity alerts are caused when the script running on the Monitoring Point cannot connect to the target web app.

  1. Determine whether, in addition to the web path Connectivity alert, you are also receiving a network path Connectivity alert with the same source and target.
  2. If so, troubleshoot the network problem.
  3. Otherwise, try accessing the web app from your browser (to confirm that the target web app is available and the credentials used by your script are valid). Typical issues include:
    • There is a problem resolving the web app IP address using its hostname (see DNS chart).
    • The web app is not running.
    • There is a web app infrastructure problem.
    • There is a script issue (for example, invalid credentials).
HTTP Error alerts

HTTP Error alerts occur when the HTTP Status returned is 4xx (client errors) or 5xx (server errors).

Review the script and the web app.

  • Has the script changed?
    • If so, review script changes and update as appropriate.
  • Has the web app changed?
    • If so, review changes and update script as appropriate.
  • Is the web app available?
    • Is it out of service due to scheduled maintenance?
      • If so, wait for maintenance to complete.
      • If not, have the team responsible for the web app get it working.
Script Error alerts

Script Errors are typically caused by a problem with the script or the web app.

  1. Check the Test Drill-down page at the time of the error.
    • The error message and snapshot will provide clues to the cause of the problem.
  2. For script issues, test the Selenium script or test the AppNeta script running on the web path and make changes as necessary.
  3. For Selenium scripts, try using the “captureEntirePageScreenshot” command to help with debugging.
Apdex Score alerts

Apdex Score alerts occur when the Apdex score dips below the alert threshold specified.

  1. Review the performance charts for the web path that generated the alert.
  2. Adjust the chart timeline to display the time range of interest.
  3. Check the DNS chart for issues with DNS server availability or slow response.
  4. Check the End User Experience and Milestone Breakdown charts looking for end user experience time components (network, server, browser) or milestones taking more time than expected and causing the unacceptable Apdex score.
  5. If the network time is longer than expected, troubleshoot the network problem.
  6. Otherwise, review individual web tests during the issue to identify the resource(s) causing the component or milestone to take longer than expected.
Page Load Time alerts

Page Load Time alerts occur when the time for a page to load is longer than the threshold specified.

  1. Review the performance charts for the web path that generated the alert.
  2. Adjust the chart timeline to display the time range of interest.
  3. Check the DNS chart for issues with DNS server availability or slow response.
  4. Check the End User Experience and Milestone Breakdown charts looking for end user experience time components (network, server, browser) or milestones taking more time than expected.
  5. If the network time is longer than expected, troubleshoot the network problem.
  6. Otherwise, review individual web tests during the issue to identify the resource(s) causing the component or milestone to take longer than expected.
HTTP Status alerts

HTTP Status alerts occur when the expected HTTP status is not returned by the target.

Review the script and the web app.

  • Is the expected HTTP status correct?
    • If not, determine why the wrong HTTP status is being returned.
  • Has the script changed?
    • If so, review script changes and update as appropriate.
  • Has the web app changed?
    • If so, review changes and update script as appropriate.
  • Is the web app available?
    • If not, have the team responsible for the web app get it working.
Step 4: Document your findings.

If you need to pass the issue to another team for resolution, record information about the problem:

  1. Make a copy of the alert notification.
  2. Copy and save a deep link to relevant web paths. The time range is captured in the deep link.
  3. Download PDFs or take screenshots of relevant web path performance charts.
  4. Attach these and any additional information to your issue tracking ticket. For example, information gathered in Step 3: Respond to web path alerts.

Related Topics: