KB110 - Packet Capture Basics

Background: Why are packet captures necessary?

When troubleshooting any kind of technical issue it is necessary that we have hard data to analyze, especially given the fact that when SIP trunks are provided over the greater Internet Clearfly only has visibility of and control over packets until they leave our network.

For example, consider a situation in which a customer is unable to receive incoming calls. Clearfly can perform a capture from the edge of its network which will show what packets are being sent to the customer’s IP address as well as what packets are received from the customer’s system. However, just because we see the packet leaving our network does not mean that it arrives at the customer’s phone system.

Although between Clearfly’s SBC and the customer’s phone system there are a large and ever-changing number of places where packets can be lost or mangled, 95% of the time the problem can be isolated to a customer’s last-mile Internet connection or a router or firewall between the customer’s phone system and their ISP's hand-off to the customer’s network. One of the fastest and most effective ways to flush out a problem is with a packet capture that will not only show if packets are arriving at the customer’s phone system (which helps troubleshoot signaling problems), but it will also help determine if there is packet loss or jitter in the path (which helps diagnose downstream voice quality problems). If there is a problem, then we can start moving hop-by-hop from the customer’s system towards Clearfly’s network.

Given the importance of packet captures as troubleshooting tools, we wanted to share some quick and basic tips that should help get you up and running with our favorite packet capture tool, Wireshark.

Where should I capture?

Unfortunately, there’s no "one size fits all" answer to this question. Various factors will play into this decision:

  • What type of problem are you having?

    • Signaling - SIP problems, such as call setup or teardown, calls dropping, etc.

    • Media - RTP problems, such as calls cutting out, garbled or tinny-sounding audio, echo, etc.

  • Is there a firewall or router between your phone system and your Internet connection?

  • What type of Internet connection do you have?

    • Especially for quality problems, is the connection symmetric (same bandwidth up and down), or asymmetric?

    • This refers to the first interface you have access to. Ethernet is common these days for cable, DSL, EFM, FTTH (FIOS), etc.

    • If your equipment is directly terminating a T1, DS3, etc you will likely want to capture on the first available Ethernet hop

  • What systems do you already have in place that may be able to capture packets?

What hardware do I need?

Packet captures can be obtained from many devices on your network:

  • The best, and most reliable source for packet capture data is via a dumb network hub or a dedicated network tap appliance

    • Very reliable method because the L2 data comes directly from the network

    • Clearfly uses and recommends Dualcomm’s network taps

  • Some managed switches have the ability to set up mirror or monitor ports

    • Can often provide accurate L2 or L2 captures

    • Capabilities can vary by make, model and configuration

    • Configuration can be quite complex

  • Some phone systems are capable of capturing signaling (and sometimes media) packets

  • Some routers or firewalls offer remote or local packet capture capabilities

Captured packets may have already been manipulated, and thus don’t necessarily provide a clear, accurate picture of what is happening on your network

What software do I need?

Although there are many different open-source and commercial products available to capture and analyze packets, Clearfly recommends Wireshark, a free, open-source program that is available for Windows, Mac, Linux, BSD, Solaris and OS X. Wireshark’s user-friendly GUI includes built-in VoIP analysis tools that make it the de facto standard for troubleshooting SIP and RTP problems.

Wireshark: VoIP Tools

Note: The information below is accurate as of Wireshark version 1.12.3. Menus and features may look slightly different in older or newer versions.

Wireshark has a number of built-in tools that can be invaluable when troubleshooting VoIP media or signaling problems. Namely:

  • Capture & Display Filters

  • VoIP Calls Dissector

    • In the menus: TelephonyVoIP Calls

    • VoIP Calls Window: Wireshark VoIP Calls

  • SIP statistics

    • In the menus: TelephonySIP

    • Enter sip (case-sensitive) as the filter value and click Create Stat

    • Statistics for specific calls are available with more specific filters, such as by call-id: sip.Call-ID == "12013223@192.168.7.195"

    • Useful for a quick statistical signaling overview: Wireshark SIP Statistics

  • SIP ladder diagrams

    • In the menus: TelephonyVoIP Calls

      • From the VoIP Calls window, select a call (or calls) and click the Flow button

    • Ladder diagrams are a useful way to quickly visualize a call’s signaling flow

    • In addition to SIP, RTP and RTP events (DTMF) are shown: Wireshark SIP Ladder Diagram

  • RTP stream analysis

    • In the menus: TelephonyRTPShow All Streams

      • From the RTP Streams window, select a stream (or streams) and click the Analyze button

      • For both directions in a call, select a stream and click the Find Reverse button before clicking Analyze

    • Excellent tool for analyzing jitter and packet loss: Wireshark RTP Stream Analysis

      • A call consists of two unidirectional flows, and the stats may be very different for each flow

      • Direction is important, so make sure your call examples detail the user’s experience

  • RTP decoder for playing back media streams

    • In the menus: TelephonyVoIP Calls

      • From the VoIP Calls window, select a call (or calls) and click the Player button

    • Also an option from within the RTP stream analysis window

    • G.711 playback is supported directly in Wireshark

    • G.729 playback is possible, but requires a bit more work. Details here.

    • Great tool for troubleshooting call quality issues Wireshark RTP Player

      • If the stats in the player window show jitter or packet loss, look for network problems

      • If the stats are clean but the audio is poor quality, look for pre-packetization problems

Wireshark: Capture & Display Filters

Wireshark supports filtering at two different levels: the capture level and the display level:

Capture Filters
  • Determines what packets will actually be copied to memory or disk. Packets that do not match this filter will not be saved to the capture file.

  • Details on syntax are available here

  • Specified before the capture begins and cannot be changed while a capture is running

  • In the menus: CaptureOptions

  • Important filter to maintain manageable capture sizes

  • From a VoIP troubleshooting perspective, host or net filters will likely be common

    • Example 1: If capturing inside your firewall and your phone system has an IP address of 192.168.1.200, your capture filter might be
      host 192.168.1.200

    • Example 2: If you’re unsure of the IP of your phone system or just want to see all traffic to Clearfly’s voice network, a good filter might be
      net 208.85.62.0/24

  • Capture Options Window: Wireshark Capture Options

Display Filters
  • Display filters are useful for filtering and analyzing packets that have already been captured

  • Details on syntax are available here

  • Simply type your filter into the "Filter" input field in Wireshark. The background will turn green when your syntax is correct: Wireshark Display Filter

  • Can be changed real-time while a capture is still in progress

  • Examples:

    • Display all SIP packets: sip

    • Display SIP & RTP packets: sip or rtp

    • Display SIP packets containing an SDP: sdp

    • Display all SIP packets for a single call: sip.Call-ID == "12013223@192.168.7.195"

Wireshark: Rolling Packet Captures

When troubleshooting VoIP problems, especially intermittent issues, it may sometimes be necessary to leave a capture running for a long period of time. To ensure that you don’t run out of memory on your capture PC and to minimize the chances of Wireshark crashing it is possible to set up a rolling packet capture, which will write a series of capture files to disk.

Setting up a rolling packet capture is pretty straightforward:

  1. Open the Capture Options window: CaptureOptions

  2. Define a Capture Filter

  3. Set a file name in the "File:" field

  4. Check the "Use Multiple Files" checkbox

  5. Check "Next File Every" and enter a value for the size of each individual file in the capture

    • Clearfly usually recommends a file size of 50 megabytes

  6. If disk space is limited or you’re not sure how long you’ll have to leave the capture running, you may want to check the "Ring buffer with" checkbox

    • The value you enter in the adjacent field will tell Wireshark how many files you want to write. Once the maximum number of files is reached Wireshark will begin overwriting the oldest files

    • Clearfly typically writes 20 files. At 50 MB, this will result in 1 gigabyte of call history Wireshark Rolling Packet Capture

Recommendations For Sending Captures to Clearfly

If you’re not able to isolate your phone system for your capture then the resultant files may contain a substantial amount of irrelevant data. Once a problem call has been reported and the relevant capture file has been identified, it will be a lot easier to manage if you filter out any non-SIP or RTP traffic. See the Capture & Display Filters section above for more details on configuring a display filter.

  1. Open the relevant capture file

  2. Enter sip or rtp in the display filter box. The input field’s background should turn green when your syntax is good

  3. Click "Apply" to only show SIP and RTP traffic

  4. Select FileExport Specified Packets

  5. In the "Export Specified Packets" window, make sure the "Displayed" radio button is selected

  6. Enter a filename for the filtered capture

  7. Check the "Compress with gzip" box

  8. Click "Save"

  9. (Optional) Forward the capture to Clearfly for evaluation