WO2010063104A1

WO2010063104A1 - Method and apparatus for measuring ip network performance characteristics

Info

Publication number: WO2010063104A1
Application number: PCT/CA2009/001735
Authority: WO
Inventors: Loki Michael Jorgenson; Christopher Robert Norris; Fredrick Klassen
Original assignee: Apparent Networks, Inc.
Priority date: 2008-12-02
Filing date: 2009-12-02
Publication date: 2010-06-10

Abstract

The present invention provides a method and apparatus for characterizing a path of a packet-based network in the presence of one or more forms of sub-optimal operation thereof. One or more sequences of packets are transmitted along the path such that each packet of the one or more sequences of packets traverses at least a portion of said path. These one or more sequences of packets are configured in order to elicit one or more responses from the packet-based network. Test data representative of responses of the packet-based network to transmission of the one or more sequences of packets is collected, wherein this test data is subsequently remediated at least in part based on the one or more forms of sub-optimal operation of the packet-based network and converted into remediated data and the remediated data is analysed to develop one or more characterizations of the path.

Description

METHOD AND APPARATUS FOR MEASURING IP NETWORK PERFORMANCE CHARACTERISTICS

FIELD OF THE INVENTION

[0001] The present invention pertains to the field of packet-based network evaluation and in particular to a method and apparatus for measuring IP network performance characteristics.

BACKGROUND

[0002] A typical packet-based data communication network comprises a number of packet handling devices interconnected by data links, a collection of which forms a path. The packet handling devices may comprise, for example, routers, switches, bridges, firewalls, gateways, hubs or similar devices. The data links may comprise physical media segments such as electrical cables of various types, fibre optic cables, and the like or transmission type media such as radio links, laser links, ultrasonic links, or similar links. Various communication protocols may be used to carry data across the data links, wherein data can be carried between two points in such a network by traversing a path, which includes one or more data links connecting the two points.

[0003] These communication networks, for example IP networks, routinely support applications which can be sensitive to network conditions such as network congestion levels. Typical such applications can include real-time services such as remote login applications, gaming applications, Voice-over-IP, IPTV and other forms of audio or video streaming, for example. When network conditions deteriorate due to the presence of other traffic or undesired network behavior, end-user Quality of Experience (QoE) for such applications can degrade, sometimes significantly due to factors such as delay, bandwidth limitations, jitter and packet loss, among others.

[0004] As such, measuring the performance of networks can be essential to their management. In addition, measuring performance, can also be important for planning and assessment of a particular network's relative value to a business. As such, measurement can provide the underpinning for monitoring and diagnostic systems and also be a key consideration in the management of applications that depend on the network.

[0005] As is known, Internet Protocol (IP) networks were not specifically designed to support measurement, however there are various techniques for making measurements that are largely based opportunistically on extant functionalities of the network. For example, statistics may be gathered at the interfaces of network devices (such as switches, routers or end-hosts) using Simple Network Management Protocol (SNMP) or Remote Monitoring (RMON). Alternately, File Transfer Protocol (FTP) may be used to transfer data across an end-to-end network path or Internet Control Message Protocol (ICMP) Echo may be used to send packets on a round-trip between devices. From these various sources of data, different analyses may extract measures such as available bandwidth, achievable capacity, latency, or jitter, among others. One technique in particular relies on the use of extant mechanisms like ICMP Echo to send tightly grouped bursts of packets on a round trip along a path between two end-hosts. These packet bursts are sometimes referred to as "packet trains". By analyzing how the packets of a burst are separated in time from each other when they arrive back at their transmission point, it is possible to estimate the upper limit regarding achievable capacity of a path of the network as defined by the smallest bottleneck along the end-to-end path. The available bandwidth, or available end-to-end capacity not consumed by cross-traffic, can also be measured on a dynamic basis as traffic varies. The subsequent analysis of this effect is referred to as "packet train dispersion" (PTD) analysis.

[0006] For example, packet dispersion techniques can be used to characterize the performance properties of an end-to-end IP-based network path. Two or more packets are transmitted from an origin device to a remote device and then, depending on the implementation, these packets are usually echoed back to the origin device. The packets are transmitted such that they are immediately contiguous, namely such that there is little or no gap in time between the end of a leading packet and the beginning of the subsequent packet. As is known, successful application of dispersion analysis requires a high degree of control of the packet timing registration in order to detect relatively small changes in separation between packets.

[0007] Assuming two packets traveling together, one after the other with a small gap ΔT separating them, they will maintain this separation indefinitely unless they encounter a buffer or queue on a network device in the mid-path or other impediment to their transmission. If a network device can transmit as quickly as it receives, the packets will leave the queue of that device as they arrive and the gap will be maintained. However, if a network device forwards the packets at a reduced rate, the second packet will end up waiting in the queue while the first packet is being transmitted. Once the first packet is transmitted, the second will be transmitted immediately after and the gap will be reduced towards substantially zero. Alternately, if other packets from cross-traffic enter the queue in between the two packets, this may delay transmission of the second packet and the subsequent gap between the first and second packet may be increased.

[0008] By analyzing the changes in packet separation, it is possible to determine characteristics of the network path including the bandwidth bottleneck (the point of least capacity) which defines the end-to-end bandwidth, and the average availability of the end-to-end bandwidth with respect to other traffic. Accurate analysis typically requires that packet timestamp registration be tightly controlled, that sufficient statistics be gathered, and that the specific circumstances of the network path under test are properly accounted for.

[0009] For example, packet pair dispersion (PPD) involves the use of only two packets, wherein this is an economical approach that limits the number of packets sent out onto the networks. However, this technique can suffer from lack of precision due to the sparseness of the timing statistics of a small number of packets. An improvement over packet pair in this regard is to send some number of packets N, typically N=K) but as many as several hundred. With additional packets, the statistical variation is reduced, resulting in more predictable measures, particularly on high capacity networks, for example 100 Mbps and higher.

[0010] For example, packet train dispersion (PTD) involves sending N packets, where N > 2. Again, there is little or no gap between the N packets and they are referred to as a "train" or a "burst". When packets in the train pass through a bandwidth bottleneck, they are stretched out such that the time to transmit each packet increases with decreasing transfer rate. The separation between each packet is substantially zero as each packet must wait in queue while the preceding packet is transmitted. If the packets then arrive at their destination, an estimation of the bandwidth can be derived by measuring the time from the first packet to last packet, and the number of bytes contained within that burst, and evaluating the ratio of bytes transferred over time.

[0011] However, if during packet transmission an increase in transfer rate is subsequently encountered, the a packet is forwarded more quickly and the following packet must finish arriving before it can be forwarded. In this instance, a time gap appears between each packet that is commensurate with the previous bottleneck bandwidth and the same ratio can be evaluated. This effect is sometimes referred to in the art as the Van Jacobson effect. An example of this effect is illustrated in Figure 1 , wherein data is being sent from the sending host 150 to the receiving host 155. The packets 100 are being transmitted and subsequently reach a bottleneck 120, which slows down the transfer of the packets, in this case resulting in the packets having no separation there between. Upon passing through this bottleneck, a separation 130 or time gap between the packets 140 can be created due at least in part to the differential speeds with which the packets are being transmitted in the two regions of the network.

[0012] Packet train dispersion can be effective in theory and within the laboratory, however it suffers from various limitations in practice. As is known, there are many behaviours on IP networks, particularly the Internet, that obscure or interfere with the collection of the requisite data for analysis using PTD. For example, the presence of traffic on the network can introduce noise into the data. Furthermore, symptoms of network dysfunction such as loss or reordering can also obstruct the subsequent data analysis using PTD. Finally, variation and inconsistency in the implementation of networks can make the measurement results and analysis thereof unreliable or inaccurate due to the inapplicability of the underlying network models assumed for the respective analysis.

[0013] Various improvements have been disclosed that overcome some of these limitations, for example, filtering burst samples for minimum round trip time (RTT), analyzing measures of the variance of the round trip times to determine the bandwidth or constructing a "virtual best burst" (composed of the minimum RTT for each of the packets in the burst from the set of all corresponding packets in all bursts). As is known, these various filtering and interpretation mechanisms can mitigate the effect of cross- traffic on the burst statistics and increase the accuracy of bandwidth estimates under various conditions. [0014] Other improvements regarding analysis, can include implementations that affect how the data relating to the transmission of the bursts of packets is gathered. For example, the need for highly precise clocks and packet timing registration or the specification of the number and size of packets that should be sent in packet trains. These techniques can reduce the effects inherent to the operating systems and hardware interfaces that are part of a multi-tasking system.

[0015] However, during the analysis of the collected information relating to packet transmission, data which is reflective of conditions of dysfunction or unusual network configurations are typically set aside as intractable and beyond the scope of their techniques. This may result in the requirement of further measurements in order that a sufficient amount of acceptable test data has been collected. However, as the occurrence of packet loss, packet re-ordering, unsymmetric bandwidths, half-duplex operation, and other such conditions are quite common on most networks, it is desired to have effective techniques that can be applied more broadly than the extant approaches.

[0016] As such, there is a need for a new method and apparatus that will aid in the characterization of a network path in the presence of sub-optimal operation thereof.

[0017] This background information is provided for the purpose of making known information believed by the applicant to be of possible relevance to the present invention. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present invention.

SUMMARY OF THE INVENTION

[0018] An object of the present invention is to provide a method and apparatus for measuring IP network performance characteristics. In accordance with one aspect of the present invention there is provided a method for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub-optimal operation, said method comprising the steps of: generating one or more sequences of packets and transmitting the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; collecting test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; generating remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and analysing the remediated data to develop one or more characterizations of the path.

[0019] In accordance with another aspect of the present invention there is provided an apparatus for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub-optimal operation, the apparatus comprising: a generation module configured to generate one or more sequences of packets and to transmit the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; a collection module configured to collect test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; a remediation module configured to generate remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and an analysis module configured to analyse the remediated data to develop one or more characterizations of the path.

[0020] In accordance with another aspect of the present invention there is provided a computer readable medium having recorded thereon statement and instructions which, when executed by a computer processor, cause the processor to execute a method for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub-optimal operation, said method comprising the steps of: generating one or more sequences of packets and transmitting the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; collecting test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; generating remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and analysing the remediated data to develop one or more characterizations of the path. BRIEF DESCRIPTION OF THE FIGURES

[0021] Figure 1 illustrates a Van Jacobson diagram which schematically illustrates packet transmission through a network bottleneck.

[0022] Figure 2 is a schematic view of an example path through a network from a first location to a second location according to one embodiment of the present invention.

[0023] Figure 3 is a schematic representation of a sequence of packets configured as a burst load of packets according to one embodiment of the present invention.

[0024] Figure 4 is a schematic representation of a sequence of packets configured as a stream of packets according to one embodiment of the present invention.

[0025] Figure 5 is a schematic representation of an analysis system in accordance with one embodiment of the present invention.

[0026] Figure 6 is a schematic representation of a received sequence of packets and the remediated data derived therefrom according to one embodiment of the present invention.

[0027] Figure 7 is a schematic representation of a transmitted sequence of packets, received sequences of packets having packet loss and remediated data derived therefrom according to one embodiment of the present invention.

[0028] Figure 8 is a schematic representation of datagram remediation of the timings associated with a sequence of packets for determination of capacity, according to one embodiment of the present invention.

[0029] Figure 9 is a schematic representation of timing remediation using a datagram configured as a primer packet associated with a sequence of packets, wherein the remediated data enables determination of capacity, according to one embodiment of the present invention.

[0030] Figure 10 is a schematic representation of datagram remediation of the timings associated with a sequence of packets for determination of capacity, according to one embodiment of the present invention. [0031] Figure 1 1 is a schematic representation of estimated capacity as a function of the number of packets in a sequence of packets, according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Definitions

[0032] The term "sequence of packets" is used to define datagrams, bursts of packets, streams of packets, or other configuration of packets. For example, datagrams can be single packets transmitted with relatively large inter-packet separations in time. For example, bursts can be groups of packets transmitted with relatively small inter-packet spacing and relatively large inter-burst separations. In addition, streams can be sequences of datagrams or bursts transmitted with a relatively fixed separation therebetween. Other configurations of sequences of packets would be readily understood by a worker skilled in the art.

[0033] The term "path" is used to define a packet transfer route within a network, typically defined between a source host and a destination host. The source host is the location of the source of the one or more sequences of packets and the destination host is the desired destination of the one or more sequences of packets. A path can be defined as a one way path, namely a one way route within the network which is defined between the source host and the destination host. A path can be also be defined as a closed loop type path, wherein the path defines an outgoing route from the source host to the destination host and further defines a return route from the destination host to the source host. In some embodiments, the outgoing route can be the same as the return route and in other embodiments, the outgoing route can be different from the return route.

[0034] The term "packet handling device" is used herein to describe devices on a packet- switched network which accept packets or sequences of packets and reply or forward the packets. Examples of packet handling devices include routers, switches, bridges, firewalls, gateways, hubs or similar devices. Packet handling devices typically function using one or more predetermined sets of protocols on at least OSI layers 3 and below, or other compatible layer in another packet based network model. As used herein, a packet handling device may also refer to an OSI layer 2 device, or similar device, such as an Ethernet switch for handling frames.

[0035] The term "sub-optimal operation" is used to define a characteristic of a network or path of a network, which is representative of a potential problem or condition associated with the network or path. A characteristic of a network or path that can be indicative of sub-optimal operation can be a hardware or software implementation of one or more of the packet handling devices or network links associated with the network or path or other physical or configuration aspect of the network or path. In addition, a characteristic of a network or path that can be indicative of sub-optimal operation can be a transient condition of network operation, and can be for example associated with traffic on the network. For example, instances of packet loss, jitter, latency, packet reordering, unsymmetric capacities and half-duplex operation are examples of sub-optimal operation of a network or path. Other characteristics of a network or path that can be associated with sub-optimal operation would be readily understood by a worker skilled in the art.

[0036] The term "capacity", "total capacity" or "data transfer capacity" are used herein interchangeable to define the maximum amount of data that can be transmitted along a network path in a given time period. For example, the capacity or total capacity of a network path can be synonymous with bandwidth or maximum achievable bandwidth or linespeed or throughput. Furthermore, available capacity is used to define the amount of the total capacity of a network path available for data transfer at a given time, with respect to existing traffic or other limiting factors. Furthermore, the ratio of [(total capacity minus available capacity) to total capacity] is defined as the utilization of the network path, which may be expressed as a percentage or other applicable format.

[0037] The term "duplex" is used to define a characteristic of the network path that corresponds with simultaneous transfer of data in both directions between a first node and a second node of the network. This typically applies to the measurable capacity of a given network interface or network path or segment that is the sum of both directions and potentially limited by one direction in the case where the capacities in each direction are not symmetric. Duplex measurement in general typically cannot distinguish contributions from each direction and typically treats the measure or characteristic as a conflated value. [0038] The term "simplex" is used to define a characteristic of the network path that corresponds with the transfer of data in only one direction between a first node and a second node of the network. Measurements may be taken that are specific to the flow of data in each direction. As such simplex is used to define a methodology which enables the separation or dissemination of one or more characteristics of the outgoing path and the returning path of the path between the first node and second node. Furthermore, simplex can be used to define a methodology which enables the determination of one or more characteristics of only the outgoing path or the returning path.

[0039] As used herein, the term "about" refers to a +/- 10% variation from the nominal value. It is to be understood that such a variation is always included in any given value provided herein, whether or not it is specifically referred to.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0041] The present invention provides a method and apparatus for characterizing a path of a packet-based network, and in particular for characterizing a path in the presence of one or more forms of sub-optimal operation of the path. Initially the method and apparatus generates one or more sequences of packets and transmits these one or more sequences of packets along the path of the packet-based network, such that each packet of the one or more sequences of packets traverses at least a portion of said path. These one or more sequences of packets are configured in order to elicit one or more responses from the packet-based network. In addition, the method and apparatus collects test data representative of responses of the packet-based network to the transmission of the one or more sequences of packets. Furthermore, this collected test data is subsequently remediated and converted into remediated data. The manner in which the remediation of the test data is conducted is at least in part based on the one or more forms of sub-optimal operation of the packet-based network. Finally, the remediated data is subsequently analysed in order to develop one or more characterizations of the path. For example, the analysis of the remediated data may be performed using packet train dispersion techniques or packet pair dispersion techniques or other analysis techniques as would be readily understood by a worker skilled in the art. [0042] According to embodiments of the present invention, the remediation of the test data is performed in order alleviate the impact of one or more sub-optimal operations of the packet based network or path on the subsequent analysis of the collected responses to the initial transmission of the one or more sequences of packets. This remediation of the test data, is essentially performed in order to reconfigure the initially collected test data, in order that this collected information can be suitably used with the respective analysis technique, which is based on particular underlying assumptions for the operation and functionality of a path of the network being assessed.

[0043] In one embodiment of the present invention, the sub-optimal operation of the path results in packet reordering. In this embodiment, the remediation of the test data is performed in order to alleviate the effect that packet reordering has on the subsequent analysis used for evaluation of characteristics of the path of the network. The test data is collected and subsequently converted into remediated data. The remediation process for generation of the remediated data, is configured and accordingly performed in a manner that the remediated data is substantially compatible with the analysis subsequently performed to determine one or more characteristics of a path of the network, thereby substantially alleviating the effect of packet reordering thereon.

[0044] In one embodiment of the present invention, the sub-optimal operation of the path results in packet loss. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to alleviate the effect that packet loss has on the subsequent analysis used for evaluation of characteristics of the path of the network. In this manner, the remediated data is substantially compatible with the analysis subsequently performed which enables the determination of one or more characteristics of a path of the network, thereby substantially alleviating the effect of packet loss thereon.

[0045] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the characteristics of the network path exhibited from a source host to a destination host are different from the characteristics of the network path exhibited from the destination host to the source host. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to separate information indicative of the outgoing path from the source host to the destination host and the return path from the destination host to the source host. In this manner, the remediated data is substantially compatible with the analysis subsequently performed, while enabling the determination of one or more characteristics of the outgoing path and/or one or more characteristics of the return path, thereby substantially alleviating the effect of unsymmetrical operation of the network.

[0046] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the collected test data reflective of the timings associated with one or more sequences of packets, which includes multiple packets in a burst type configuration, is remediated using collected test data reflective of the timings associated with one or more sequences of packets, which include a single packet or datagram. The conversion of the test data into remediated data enables the subsequent evaluation of the capacity of the path exhibiting unsymmetrical characteristics, and thus this capacity is indicative of the rate at which sustained two-way information traffic can flow along the path.

[0047] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the collected test data is reflective of the timings associated with one or more sequences of packets, wherein each of the one or more sequences of packets includes multiple packets in a burst type configuration, preceded by a packet or datagram by a predetermined amount of time. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to separately evaluate capacity indicative of the outgoing path from the source host to the destination host and capacity of the return path from the destination host to the source host.

[0048] In one embodiment of the present invention, the sub-optimal operation of the path results in the path exhibiting an undesired level of variability of its determined capacity characteristics. In this embodiment, the collected test data is reflective of the timings associated with one or more sequences of packets, wherein each of the one or more sequences of packets includes multiple packets in a burst type configuration, preceded by a packet or datagram. The conversion of the test data into remediated data enables the subsequent evaluation of the capacity of the path having a desired limited variability. Active Probing Techniques

[0049] For measuring and evaluation of a packet-based network, the present invention provides for active probing, which can comprise configuring and transmitting one or more sequences of packets into a network along a network path and observing a response, which may vary according to the mechanisms implemented along the network path. Network responses can be collected as test data by the source host, destination host, or other host. Furthermore, the conversion of the test data into the remediated data and optionally the analysis of the remediated data can be performed by a computing device located at or associated with the source host, destination host or other host.

[0050] In various embodiments of the present invention, active probing can be configured to elicit responses from one or more packet handling devices such as ICMP messages, ICMP Echo Reply packets, packet acknowledgements, and the like. ICMP messages may include Destination Unreachable messages such as Host or Port Unreachable, Time Exceeded messages such as TTL expired in transit, and the like. Packet acknowledgements may be generated by sending packets according to a specific protocol, such as TCP or UDP at Layer 3 or Ethernet or PPP at Layer 2, and observing the receipt of these packets either directly or via generated acknowledgements for protocols supporting reliable packet delivery.

[0051] In some embodiments of the present invention, at least part of the active probing of the network path is modified based at least in part on the sub-optimal operation of the path. For example, the protocol, size or other characterization of the one or more sequences of packets being generated for active probing may at least in part be based on the sub-optimal operation of the path. For example, based on the sub- optimal operation of the path, a particular remediation action is required and in order for this remediation action to be performed, the transmission and collection of test data indicative of specific formats of one or more sequences of packets may be required. In addition, the transmission of the one or more sequences of packets and/or the collection of the responses to the one or more sequences of packets as test data, may be modified based at least in part on the sub-optimal operation of the path. [0052] Aspects related to probing in several embodiments of the present invention are described herein. It is noted that each aspect can occur alone or in combination in various embodiments of the present invention.

Example of Active Probing

[0053] Figure 2 illustrates an example of a portion of a network 10, wherein the network comprises an arrangement of network devices 14 interconnected by data links 16. The network devices may comprise, for example, routers, switches, bridges, hubs, gateways and the like. The data links may comprise physical media segments such as electrical cables, fibre optic cables, or the like or transmission type media such as radio links, laser links, ultrasonic links, or the like. In addition an analysis system 17 is connected to the network 10.

[0054] Also connected to the network 10 are mechanisms for sending one or more sequences of packets along a path 34 and receiving the packets after they have traversed the path. In the illustrated embodiment, path 34 is a closed loop, wherein packets originate at a test packet sequencer 20, travel along the path 34 to a reflection point 18, and then propagate back to the test packet sequencer 20. The path, however, does not need to be a closed loop, for example, an open path may be configured such that the mechanism for dispatching the one or more sequences of packets may be separated from the mechanism that receives the packets after they have traversed the path.

[0055] The packets of the one or more sequences of packets being sent along a path during the active probing can be of varying sizes, wherein the largest size of packet is defined by the MTU supported by the path to the selected end host or destination host. For example, if a packet larger than the MTU is transmitted during an active probing session, a fragmentation response will typically be received by the test packet sequencer 20.

[0056] Furthermore, these one or more sequences of packets transmitted during the active probing can be datagrams, bursts or streams, for example. Datagrams can be single packets with large inter-packet separations in time, for example the separation between the single packets can be in the range of hundreds of milliseconds. In addition, tight datagrams can be datagrams transmitted with relatively small inter-packet spacings. Bursts can be groups of a number of packets that are separated by small or virtually zero inter-packet separations. The inter-burst separation in time can be relatively large and can be in the range of hundreds of milliseconds. Furthermore, a burstload can be an extended burst of many packets back-to-back, which can typically be more than an order of magnitude larger than a burst. For example, a burstload can comprise in the order of hundreds of packets. In addition, streams can be sequences of bursts of for example a fixed size and number wherein there is a fixed separation of time between bursts. Other configurations of sequences of packets would be readily understood by a worker skilled in the art. Each of these configurations of sequences of packets can be used alone or in combination to actively sample a path of a network in order to characterise its end-to-end performance, however the level of characterisation can be dependent on the form of the sequences of packets used during the active probing session.

[0057] In the example portion of a network as illustrated in Figure 2, the test packet sequencer 20 can record information about the times at which the one or more sequences of packets are dispatched and times at which returning packets are received. Alternately, wherein the path is a one way, or open path, a first mechanism, for example a test packet sequencer, can be positioned at the origin or source host in order to dispatch a sequence of packets and a second mechanism can be positioned at the destination host, and can be used to receive the sequence of packets. In this configuration, the first and second mechanisms would collect timings related to the departure and receipt of a sequence of packets.

[0058] In one embodiment, a sequence of packets is a burstload, an example configuration of which is illustrated in Figure 3. A burstload 310 comprises a plurality of packets for example N packets 360, wherein these packets have an interpacket spacing 300 of approximately a zero interpacket spacing. In this embodiment each packet has been identified as having a size S 350, however in alternate configurations of a burstload the size of the packets can vary within the burstload. Furthermore, the number of packets in a burstload is typically an order of magnitude greater than the number of packets within a burst. For example a burstload can comprise about 100, 200, 400 or other large number of packets.

[0059] Having specific regard to streams, this format of a sequence of packets represents a type of network response sampling that is unique to specific applications. For example, in one embodiment a stream is a parameterized sampling that substantially corresponds to a network load generated by real-time application traffic, for example VoIP traffic. As illustrated in Figure 4, this example of a stream comprises M bursts, 410, 420 and 430, each composed of N packets 460 having a size 5, 440, and approximately a zero inter-packet spacing within a burst. Each burst is separated by a fixed timing equal to t, 450. The parameterization of the values selected for M, N, S and t can be mapped to the choices of codec being used for an associated real-time application, for example VoIP and the number of simultaneous real-time connections. In one embodiment, appropriate values of the parameters for packet size, 5, and burst separation timing, t are defined by the codec being used, and the number of simultaneous real-time connections provides an appropriate value for the burst size, N. In addition, the number of bursts, M, can be specific to the resolution of the statistics that are required for a particular active sampling procedure. In one embodiment, the number of bursts used in a sampling procedure is approximately 50, however this can vary depending on the desired detail of path evaluation. Other configurations of streams would be readily understood by a worker skilled in the art.

[0060] With further reference to the example of a portion of a network path as illustrated in Figure 2, the test packet sequencer 20 that dispatches one or more sequences of test packets 30 each comprising one or more test packets 32 is connected to the network 14. The path 34 extends from the test packet sequencer 20 through routers 14A, 14B, and 14C to a computer 19 from where the packets are routed back through routers 14C, 14B, and 14A to return to test packet sequencer 20. There are various ways to cause packets 32 to traverse a path 34. For example, the packets 32 may comprise Internet Control Message Protocol (ICMP) ECHO packets directed to end host 19 which will automatically generate an ICMP ECHO REPLY packet in response to each ICMP ECHO packet or alternately an ICMP ECHO packet may result in an ICMP TTL Expiry packet being created in response at some mid-path device. For another example, packets 32 could be another type of packet protocol, such as packets formatted according to the Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) protocol wherein packets of these protocols are port specific and may generate a ICMP Port Unreachable packet from an end host or an ICMP TTL Expiry packet from a mid-path device, for example. Such packets could be sent to end host 19 and then returned to test packet sequencer 20 by software or hardware at end host 19, for example UDP echo daemon software, in the form of a UDP ECHO packet. Furthermore, when multiple packets form a sequence of packets, the sequence of packets can comprise packets of a single configuration or packets of varying configurations. For example, a sequence of packets can include some packets configured using ICMP, while other packets of the same sequence of packets are configured using UDP. Other configurations of a sequence of packets would be readily understood by a worker skilled in the art.

[0061] As packets 32 pass along path 34 through network devices 14 and data links 16, individual packets 32 may be delayed by different amounts and some packets 32 may be lost in transit. Various characteristics of the network devices 14 and data links 16 along path 34 can be determined by observing how the transmission characteristic data derived from different packets in the sequences varies, which can include measures relating to latency, capacity, delay variation, packet loss among others, for example.

[0062] Analysis system 17 receives the test data 33, wherein the analysis system may comprise a programmed computer. Analysis system 17 may be hosted in a common device or located at a common location with test packet sequencer 20 or may be separate therefrom. As long as analysis system 17 can receive test data 33, its precise location is a matter of convenience. Furthermore, the analysis system can be configured as a single mechanism for remediation of the test data and analysis thereof and equally may comprise multiple modules, wherein a particular module is configured to perform a specific task. In addition, the analysis system can be configured as a single computing device, and equally may comprise multiple computing devices communicatively linked in order to perform the required tasks.

[0063] An embodiment of the analysis system is illustrated in Figure 5. In this embodiment, the analysis system 558 includes a generation module 550, which is configured to generate and transmit one or more sequences of packets and a collection module 552, which is configured to collect the test data resulting from the transmission. In addition, the analysis system 558 comprises a remediation module 554, which is configured to remediate the test data into remediated data and an analysis module 556 which is configured to perform one or more analyses on the remediated data in order to determine one or more characteristics of the network path being evaluated. As is illustrated, each of these modules can be interconnected either to one another directly or indirectly through another module. [0064] Test data 33 comprises information regarding packets which have traversed path 34. This information may include information regarding one or more of lost packets, final inter-packet separation, and information such as hop number, hop address, measured and reported MTU, and error flags. Test data 33 may comprise information regarding one or more of the sequence of packets transmitted including variables such as packet size (the number of bytes in a packet), burst size (the number of packets in a burst), and initial inter-packet separation (the time between packets in a burst at transmission). For example, ICMP messages collected as test data, such as ICMP Echo Reply, TTL Expiry, or Port Unreachable, may contain in their payload the header information, including the packet that triggered the ICMP message, at the point that the ICMP message was triggered. Further, the test data 33 may include derivatives of one or more of these variables, for example packet sequence can be derived from inter-packet separation. Higher order variables may also be derived as admixtures of these variables, for example through the use of sequences of packets having a distribution of packet sizes within a distribution of inter-packet separations.

[0065] The analysis system 17 furthermore determines remediated data that can be determined directly or indirectly using the test data and in light of the one or more forms of sub-optimal operation of the path. In addition, the analysis system further determines one or more characteristics of the path from the remediated data.

Remediation of Test Data

[0066] According to embodiments of the present invention, the remediation of the test data is performed in order alleviate the impact of one or more sub-optimal operations of the packet based network or path on the subsequent analysis of the collected responses to the initial transmission of the one or more sequences of packets. This remediation of the test data, is essentially performed in order to reconfigure the initially collected test data, in order that this collected information can be suitably used with the respective analysis technique, which is based on particular underlying assumptions for the operation and functionality of a path of the network being assessed.

[0067] For example, in the presence of a sub-optimal operation of a path, the collected and unremediated test data may not be suitable for the respective analysis technique, due to the sub-optimal operation of the path violating one or more particular underlying assumptions for the operation and functionality of a path which is relied upon by the analysis technique. If the respective analysis technique is applied to the collected unremediated test data, the analysis technique may result in a null, false, incorrect or inaccurate determination of one or more characteristics of the path. As such, the remediation of the test data is performed in order alleviate the impact of one or more sub-optimal operations of the packet based network or path on the subsequent analysis of the collected responses to the initial transmission of the one or more sequences of packets.

[0068] According to embodiments of the present invention, remediation of the test data into remediated data can be performed in order to alleviate the impact of for example, packet reordering, packet loss, unsymmetrical path configurations and network capacity. For example, according to embodiments of the present invention, there is provided techniques for packet reordering remediation, packet loss remediation, simplex measurement remediation, datagram remediation, datagram remediation using a primer packet and remediation based on network capacity. In embodiments of the present invention, multiple remediation techniques may be applied to the test data, under appropriate conditions, in order to determine the remediated data, for example, packet reordering remediation may be applied as well as packet loss remediation, or simplex measurement remediation may be applied together with packet reordering remediation or datagram remediation using a primer packet or other combination of two or more remediation techniques, as would be readily understood by a worker skilled in the art.

Packet Reordering Remediation

[0069] In one embodiment of the present invention, the sub-optimal operation of the path results in packet reordering. In this embodiment, the remediation of the test data is performed in order to alleviate the effect that packet reordering has on the subsequent analysis used for evaluation of characteristics of the path of the network. The test data is collected and subsequently converted into remediated data. The remediation process for generation of the remediated data, is configured and accordingly performed in a manner that the remediated data is substantially compatible with the analysis subsequently performed to determine one or more characteristics of a path of the network, thereby substantially alleviating the effect of packet reordering thereon. [0070] Currently, capacity estimation methods using packet train dispersion (PTD) explicitly excludes trains that exhibit re-ordering of the packets, based on the fact that the presence of re-ordering violates the underlying assumptions of the network models that are used in the analysis. For example, reordering is considered dysfunctional within the standard network models because it should not be possible for one packet to "pass" another on a serial connection. Most network links are serial, processing packets on a first-in first-out (FIFO) basis. Therefore the basic assumptions of PTD require that each packet be constrained by the preceding and following packets. As such, if packets arrive out of order, the constraints of a serial path cannot be assumed. Furthermore, in constructing a "virtual best burst" as is currently performed, the correspondence between a particular packet received out of order and the members of a virtual best burst is ambiguous.

[0071] The most typical cause of reordering is the presence of a multi-channel link somewhere in the end-to-end path of the network, which can include cases such as: 1) certain router architectures; 2) certain operating system architectures; 3) multi-channel links between mid-path routers; 4) per-packet load-sharing without session or connection identification; and the like as would be readily understood by a worker skilled in the art.

[0072] According to embodiments of the present invention, in each of these cases, it can be presumed that the effective capacity of the multiple channels can be appropriately represented by the overall timing of a burst that passes through them. As such, regardless of packet reordering of the one or more sequences of packets relative to the original transmission time thereof, the effective bulk transfer rate of the path is equivalent to the time for all packets of the sequence of packets to arrive. As such, for the purposes of PTD analysis, the timings of an arriving burst containing reordering can be re- arranged, ranked in order of arrival time, and then re-numbered in a manner such that the first packet to arrive is labelled as number one, the second packet to arrive is labelled as number two, and so on, regardless of each of the packets original transmission time.

[0073] An example of this embodiment of the present invention is illustrated in Figure 6, wherein a sequence of packets includes a total of seven packets, numbered 1 to 7, which indicates the initial order in which in the packets were transmitted. This sequence of packets can be originally transmitted such that all seven of the packets are transmitted with substantially no inter-packet separation. As illustrated in Figure 6, upon reception of the sequence of packets, the order of the packets 500 has changed and there is also temporal separations 505 between these packets. The arrival time in relation to the original order to the transmission of the packets is also illustrated 510, and this can be considered to represent a portion of the test data. According to an embodiment of the present invention, the generation of the remediated data from the test data results in the sequence of packets being renumbered 520 in sequential order, regardless of their actual reception order, with their reception times being according rearranged 530.

[0074] According to the present invention, with reordered packet sequences rearranged in order of reception, and thereby converted into remediated data, they can be analyzed according to PTD analysis. The resulting capacity estimate for the path can subsequently be interpreted as: 1) the effective transfer rate of the multi-channel link, if it is the smallest bottleneck in the path, as experienced by an application and 2) representative of the smallest bottleneck, if it is other than the multi-channel link, since the packets entering the bottleneck would retain their order while in transit of a serial link.

[0075] In addition, in a case where a desired analysis is being performed using the "virtual best burst" technique, the remediated data representative of the de-ordered timings of the sequence of packets can be mapped directly to a virtual best burst according to the order of the packets of the sequence of packets upon reception.

Packet Loss Remediation

[0076] In one embodiment of the present invention, the sub-optimal operation of the path results in packet loss. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to alleviate the effect that packet loss has on the subsequent analysis used for evaluation of characteristics of the path of the network. In this manner, the remediated data is substantially compatible with the analysis subsequently performed which enables the determination of one or more characteristics of a path of the network, thereby substantially alleviating the effect of packet loss thereon.

[0077] Currently, bursts that include packet loss must be excluded from the PTD analysis. This reasoning is based on the assumption that if a packet is lost in transit, subsequent contiguous packets can at times have a gap separating them that is smaller than otherwise possible due to the presence of the otherwise missing packet. As such, the burst cannot be assumed to have passed the same number of bytes through a constraining bottleneck and so the timings are not reliable, for the subsequent analysis using PTD.

[0078] According to the present invention, under certain circumstances, packet loss associated with the transmission of one or more sequences of packets can be disregarded and a PTD analysis can still be applied to determine a network capacity. For example, within a sequence of packets that includes N packets, wherein N=K) can be considered typical, the requirement that there be no loss within the entire sequence of packets can be relaxed when: 1) loss within the sequence of packets results in a sub-set of the packets within the sequence of packets being received contiguously, namely there is no intervening packet loss; 2) the number of packets sent in a sequence of packets is sufficiently large and/or the packet loss is sufficiently low that the number of packets in a contiguous sub-set is sufficiently large, for example typically 5 or greater; 3) only a sequence of packets containing loss after the contiguous sub-set of packets received can be used, wherein the contiguous sub-set must start with the first packet transmitted; and 4) timings from any sub-set of packets within a sequence of packets containing loss can be used to construct one or more sub-set virtual best bursts of one or more of a variety of sizes up to the number of packets in the sub-set of packets.

[0079] According to this embodiment of the present invention, the remediated data can include multiple sub- set sequences of packets which include one or more sequences of packets for example, each comprising from about 5 packets up to the largest contiguous number of packets available in the sequence of packets, for example 10 packets or more. This remediated data can subsequently be used to construct one or more virtual best bursts having a minimum size of 5 packets up to the largest contiguous number of packets available in the sequence of packets, for example 10 packets or more.

[0080] In an embodiment of the present invention, Figure 7 illustrates an originally transmitted sequence of packets 600 configured as a burst of packets. This figures also illustrates possible configurations of the received packets 610 of the sequence of packets, wherein packet loss may have occurred. In addition, Figure 7, further illustrates sub-sets of sequences of packets 620, which may be determined based on the format of the packet loss associated with the received one or more sequences of packets. For example, arrows 625, illustrate the different sub-sets of packets 620, which may be associated with each of the configurations of the received sequences of packets 610 which may or may not include packet loss. Having further regard to the illustrated examples of possible received packets configurations 610, the third configuration of received packets is representative of multiple packets being lost. As illustrated, according to embodiments of the present invention, there are no sub-set of sequences of packets that may be derived from this received configuration, as the second packet and tenth packets are lost, and as such the remaining received packets do not comply with the above defined relaxation requirements relating to packets loss, namely a sub-set sequence of packets containing approximately 5 packets does not also contain the first packet transmitted.

[0081] For example, in the presence of packet loss, there can be a larger number of smaller sub-sets available and fewer larger ones. In addition, where there are sufficient statistics for the largest size of sub-set sequences of packets, an adequate value for capacity can be determined from a largest virtual best burst constructed from this remediated data. However, when there are insufficient statistics for the larger sizes, smaller sub-set virtual bursts may yield a more reliable measure. In addition, according to an embodiment of the present invention, the highest measurement for the capacity of all sub-set virtual best bursts is assumed to be the optimal value to be derived from a set of sequences of packets containing loss insofar as each sub-set represents a value from a distribution of values with a maximum.

[0082] In one embodiment of the present invention, a variation on this packet loss remediation can allow for sub-sets of sequence of packets to be used even after loss has appeared. This approach requires assumptions to be made that are not consistent with the virtual best burst approach. In particular, the filtering of the test data by minimum RTT cannot be used to select out the best values, as timings for a given packet in a sequence of packets vary according to any loss preceding it. However the above defined packet loss remediation approach may provide a means for estimating achievable capacity with some changes to the analytic approach. For example, virtual best bursts could be composed for sequences of packets, which have the loss at the same point in the sequence. As an example, all sequences of packets which have lost the second packet could be used to compose a special virtual best burst for the balance of the sequence of packets, or sub-set of packets. Subsequently, the analysis could assume that the special case of the second packet being lost and adjust the PTD analysis to model this case accordingly.

[0083] Alternatively, sub-sets of packets following loss could be analyzed without resorting to PTD. Rather, alternate methods for estimating capacity could be used such as local mode identification using histograms or kernel density estimation. These methods instead rely on correlations in the statistical distribution to identify trends in multiple samples.

Simplex Measurement Remediation

[0084] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the characteristics of the network path exhibited from a source host to a destination host are different from the characteristics of the network path exhibited from the destination host to the source host. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to separate information indicative of the outgoing path from the source host to the destination host and the return path from the destination host to the source host. In this manner, the remediated data is substantially compatible with the analysis subsequently performed, while enabling the determination of one or more characteristics of the outgoing path and/or one or more of the return path, thereby substantially alleviating the effect of unsymmetrical operation of the network.

[0085] It can be advantageous to measure the performance of an arbitrary path of a network from a single point by exploiting the general availability of ICMP on the majority of IP hosts. In this basic approach using a closed loop type path, the resulting measures collected assume that the path in each direction is symmetric or that the target use of the path is fully duplex. However, some paths are unsymmetric and are not adequately measured by this approach. In addition, an application using a path of a network is often unsymmetric in its desired functionality, due to for example a requirement that the application necessitates unsymmetric transmission volume, for example transmission and reception of packets by the application.

[0086] In these cases of unsymmetric outgoing and return paths, it can be desirable to measure the characteristics of a network path in each direction in order to identify differences between the outgoing path and the return path. Mechanisms for this form of analysis can include sending user datagram protocol (UDP) packets to an arbitrary port in order to generate an ICMP Port Unreachable. In this configuration, the returning packets are much smaller than those transmitted, which is different from the basic case where the packets are of the same size. This difference in packet sizes in the two directions along the path can make it possible to distinguish the bottlenecks in one direction along the path from those in the other direction.

[0087] According to an embodiment of the present invention, there is provided a means for measuring the performance of an end-to-end path when the destination host has been instrumented with software to echo back packets in a specific manner. In this embodiment, a sequence of packets can be sent from a source host to a destination host and echoed back to the source host. Timestamps can be taken at each end of the path and the effect of any dispersion accumulated along the outgoing path, namely from the source host to the destination host, is removed by the destination host before sending the sequence of packets back to the source host. The test data collected in this manner, can be converted into remediated data such that the timestamps of the packets transmitted and received are considered. In this manner, the subsequent analysis treats each set of timestamps as distinct, thereby enabling the generation of one or more performance measures for each direction of the path. For example, this general methodology may be considered similar to the extant 2-way or "duplex" approach in how it samples and analyzes, and therefore this embodiment of the present invention, can provide a simple extension of that 2-way methodology to a 1-way or "simplex" measurement.

[0088] However, with this type of sampling there is typically a need to synchronize the clocks at the two ends of the measured path, namely to synchronize the source host clock with the destination host clock. According to an embodiment of the present invention, there is provided a method for at least measuring the capacity using the above noted simplex methodology, wherein the clocks of the source host and destination host do not need to be synchronized.

[0089] According to one embodiment of the present invention, simplex sampling and measurement can be initialized such that one or more sequences of packets can be sent from the source host to the destination host in order to synchronize the clocks thereof. [0090] According to another embodiment of the present invention, simplex sampling and measurement can be performed following the subsequent steps. Initially, the source host can be implemented to perform duplex sampling wherein it sends a sequence of packets of maximum size with no gap between the packets, to the destination host, wherein the destination host echoes the sequence of packets back to the source host immediately as each packet arrives and the source host subsequently timestamps each of the returning packets as they arrive.

[0091] Subsequently the source host sends another sequence of packets of maximum size with no gap between the packets to the destination host, wherein this sequence of packets may be marked for simplex handling at the destination host. The destination host receives each packet of the sequence of packets and holds them and records the timestamp of the arrival of each of the packets of the sequence of packets, according to the local clock. Subsequent to the arrival of all of the packets of the sequence of packets, the destination host prepares a sequence of packets to echo back to the source host, wherein these packets are the same size and have the same contents as the arriving sequence of packets, but without any dispersion gap between the packets of the sequence of packets. Furthermore, prior to the destination host transmitting this sequence of packets, the destination host includes the previously recorded arrival timestamps of the packets within the payload of one or more of the packets being sent to the source host. In addition, the destination host holds all of the packets of the prepared sequence of packets for a holding period of time, which is also recorded in the payload(s) of one or more of the packets, before transmitting the sequence of packets back to the source host. The period of time may be determined based on: 1 ) at least being greater than the time taken to prepare the sequence of packets for transmission to the source host; 2) an amount of time to receive the incoming sequence of packets, write the arrival timestamps into the payload(s) of the sequence of packets for the reply, prepare the sequence of packets for reply, and plus a fixed interval; or 3) an amount of time to receive the incoming sequence of packets, write the arrival timestamps into the payload(s) of the sequence of packets for the reply, prepare the sequence of packets for reply, for example send reply as soon as possible. Finally, the source host records the arrival timestamps of the returning packets of the sequence of packets, and also decodes the arrival timestamps and holding period of time from the packet payload(s) that were placed there by the source host. This sequence of transmission between the source host and the destination host can be repeated a sufficient number of time in order to generate a set of timestamps from multiple sequences of packets such that an adequate analysis can be performed.

[0092] According to embodiments of the present invention, the set of timestamps that can be explicitly found or derived from the above noted test data for each sample, which is indicative of the remediated data, can include: 1 ) transmit timestamps for each packet from the originating host defined in local source host time; 2) arrival timestamps for each packet at the destination host, in non-local time, namely destination host time, and subsequently relative timing between arriving packets at the destination host; 3) the holding period of time of the sequence of packets before echoed back from the destination host, defined in absolute time; and 4) arrival timestamps for each packet of the sequence of packets at the source host, in local source host time.

[0093] According to embodiments of the present invention, from this data, it is possible to derive the round trip time (RTT) including the holding period of time in terms of the local time at the source host. From this value, a reduced RTT can be calculated that excludes the holding period of time and can be effectively equivalent to a normal round trip time. Depending on the implementation, this reduced RTT may be calculated for one or more packets, or for the entire sequence of packets. Subsequently, either an optimal sequence of packets with the shortest round trip time can be identified, or a virtual best burst can be constructed composed of the times for each packet with the least round trip time.

[0094] In some embodiments of the present invention, the sequence of packets with the shortest round trip time, or the related best virtual burst, can be assumed to have completed the end-to-end round trip with the minimum possible time as it may be considered that this sequence of packets did not encounter any cross-traffic or be otherwise delayed beyond the amount that will be common to every packet making that trip. As such, the packet dispersion recorded at each end of the path, namely at the destination host and the source host, can be assumed to be representative of the bottlenecks inherent to the path.

[0095] In embodiments of the present invention, PTD can then be applied to the relative time of the arriving bursts at the destination host to determine the outgoing path capacity and similarly, PTD can be applied to the arriving bursts at the source host to determine the return path capacity.

[0096] In one embodiment of the present invention, by selecting the appropriate set of timestamps, the available capacity or achievable capacity in each direction of the path can be measured. Selecting the preferred timestamps according to minimum RTT remains a desired approach, however, without synchronizing the clocks of the source host and the destination host, it becomes necessary to remediate the available test data to reveal the best choice. For example, it can be assumed that the minimum RTT travel time for both the sequence of packets travelling the outgoing path and the sequence of packets travelling the return path is an adequate filter, where the overall trip time for both is measured in the local time of the source host. However, as described herein, the overall trip time also includes holding period of time at the destination host that is recorded as a time difference. By subtracting the holding period of time from the overall trip time yields the RTT composed only of travel time. This RTT value can be minimized to select the desired outgoing packet and return packet pair. Subsequently, the intra-burst dispersion, namely the time the first and last packets in each sequence of packets, which is known for both the outgoing path and the return path can be analyzed by PTD to determine capacity in each direction.

[0097] According to the embodiments of the present invention, for this specific analysis, no synchronization of the clocks at the source host and the destination host is required. However, should the clocks be synchronized, other measures may be determined from the remediated data indicative of simplex sampling, wherein these other measures may include one or more of one-way propagation delay, utilization, jitter, and the like.

[0098] Without synchronization estimates of one-way propagation delay, utilization, jitter and the like can be made by making one or more assumptions such that contributions from each direction of packet travel are comparable. Further, by examining the relative variation in timings in each separate direction, relative levels of difference between the outgoing path and the return path can be estimated, to either confirm or reject the assumption of symmetry. It is also possible to roughly project the levels of difference and correct the symmetric estimates to account for detected indications that unsymmetric outgoing and return paths. [0099] In one embodiment of the present invention, the above remediated data indicative of simplex sampling, when taken in combination with the duplex samples that also may be sent between the source host and destination host, the measured capacity for duplex or the two way, return transmission of packets, may be compared with each of the simplex capacity measures in the outgoing and incoming directions, in order to assess the degree to which the path is unsymmetric, the presence of half-duplex, and the like along the path, or the like.

Datagram Remediation

[0100] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the collected test data reflective of the timings associated with one or more sequences of packets, which includes multiple packets in a burst type configuration, is remediated using collected test data reflective of the timings associated with one or more sequences of packets, which includes a single packet or datagram. The conversion of the test data into remediated data enables the subsequent evaluation of the capacity of the path exhibiting unsymmetrical characteristics, and thus this capacity is indicative of the rate at which sustained two-way information traffic can flow along the path.

[0101] According to one embodiment of the present invention, remediation of the test data into remediated data can be provided such that the PTD method can be applied and such that an automatic and accurate assessment of the capacity according to duplex can be determined, wherein the evaluated capacity is sensitive to the presence of half-duplex mode along the path.

[0102] In one embodiment of the present invention, the remediation of the test data can also account for other unsymmetric conditions that may be present along the path. For example, even in a full-duplex environment, there may be various unsymmetric conditions between the outgoing path and return path of the path being evaluated that are half-duplex-like. That is, the capacities in one direction may be limited compared to the other due to design, dysfunction, or other circumstance. For example, some network interface cards (NICs) are known to mitigate collisions and packet loss associated with unintentional duplex mismatch by delaying transmission when receiving packets. [0103] According to an embodiment of the present invention, the remediation of the test data is enabled by the collection of test data relating to the timing measurements of one or more sequences of packets configured as a duplex datagram together with timing measurements of one or more sequences of packets configured as bursts. This combination of the collected timings is used to remediate this test data prior to applying PTD analysis to derive capacity of a path, such that this evaluation of the capacity is sensitive to the presence of half-duplex mode along the path.

[0104] In one embodiment of the present invention, multiple iterations of sequences of packets configured as probative packets are sent to a destination host, such that they are echoed back to the source host. For example, in one embodiment, these probative packets can be configured as ICMP Echo packets, however other suitable protocols would be readily understood by a worker skilled in the art. In one embodiment, each iteration of of the probative packets comprises at least of one instance of a datagram of large size and a burst of packets of that same large size.

[0105] According to this embodiment, from the collected records of the timestamps of the datagrams and bursts, the best overall burst timestamps are selected. The criteria for "best" can be variable, however in one embodiment "best" may be assumed to be the shortest time for the round trip of all packets in the burst. Similarly, the best overall datagram timing is selected, and in one embodiment "best" may be assumed as the shortest round trip time. Other criteria for the selection of "best" burst and "best" datagram would be readily understood by a worker skilled in the art.

[0106] Subsequently, the round-trip time for the first packet of the "best" burst is compared to the round-trip time for the "best" datagram. The minimum value for the round-trip time is used to represent the round-trip time for the first packet in the burst. This is illustrated in Figure 8, wherein the round trip time for the datagram 710 is used to represent the round trip time associated with the first packet of the burst 700. The capacity of the path can subsequently be calculated as the ratio of the number of bytes in the burst, less the first packet, over the time between the first and last packet in the burst. For example, the payload of the first packet, regardless of the use of the datagram, does not contribute to the total number of bytes transmitted, as the time of arrival of each packet is measured from the time it is fully arrived including payload, not from when its header initially is received. [0107] For example, with further reference to Figure 8, the datagram 710 round-trip time is much less than that of the first packet in the burst 700. This type of difference in round trip time is typical when a half-duplex interface is present at the destination host. In this example, the datagram is immediately sent back by to the source from the destination host once it received. In contrast when operating in half-duplex mode, the interface at the destination host must hold the first packet of the burst while it continues receiving the subsequent packets. As such, the first packet is typically not echoed back until most or all packets of the burst have arrived, thereby increasing the time for the first packet's round-trip. However, the overall time that the burst is held up is commensurate with its size, and thus related to the data transfer rate of the interface at the destination host.

[0108] However, if the destination host's interface is set to full-duplex, the first packet of the burst will be returned immediately. In this instance the round-trip timing for the first packet of the burst will typically be comparable to the round trip timing of the datagram.

[0109] According to embodiments of the present invention, by using this combined set of timings to calculate the capacity of the path, the presence of half-duplex is automatically included in the analysis. For example, the capacity of a path is reduced by a factor of 2 for half-duplex, due to the influence of the datagram's shorter round trip time. This representation of the capacity substantially accurately reflects the rate at which sustained two-way traffic can flow along the path evaluated.

[0110] While the above describes the effect at the destination host of the path under test, similar effects can be observed when another interface along the end-to-end path is set to half-duplex configuration. The effect of this half-duplex configuration on the determined capacity depends on the capacity of the links, the latency, and the size of the burst. For example, if the entire end-to-end path is full-duplex except for one device in the mid-path, it is possible for all of the packets to be passed forward to the destination host before any packets are subsequently echoed back to the source host. The delay of the first packet in the burst when a mid-path interface is half-duplex will typically decrease as the latency between that interface and the destination host increases. As such this network configuration will effect the calculated value of the capacity according to an embodiment of the present invention, as it will gradually increase with increasing latency between the mid-path half-duplex interface and the full-duplex destination host.

[0111] According to embodiments of the present invention, any significant difference between the datagram timing and the first burst packet timing acts as a clear indication that half-duplex is present somewhere along the path between the source host and the destination host.

Datagram Remediation Using Primer Packet

[0112] In one embodiment of the present invention, the sub-optimal operation of the path results in a path exhibiting unsymmetrical characteristics. In this embodiment, the collected test data is reflective of the timings associated with one or more sequences of packets, wherein each of the one or more sequences of packets includes multiple packets in a burst type configuration, preceded by a packet or datagram by a predetermined amount of time. In this embodiment, the test data is collected and subsequently converted into remediated data, wherein the remediation of the test data is performed in order to separately evaluate capacity indicative of the outgoing path from the source host to the destination host and capacity of the return path from the destination host to the source host.

[0113] In two-way or duplex measurements, the combined datagram and burst timings may be applied to detect the presence of half-duplex and calculate the two-way capacity of a path from a source host to a destination host. However, this method for detecting duplex configuration cannot be suitably applied to simplex or one way measurements.

[0114] A sequence of packets can be configured as a probative burst using both a datagram and a burst at the same time. In this case, the datagram can be referred to as a "primer packet" and precedes the burst by a predetermined amount of time. The primer packet is essentially configured in this manner in order to optimize the path between the source host and the destination host by "priming" all of the interfaces just prior to the transmission of the probative burst. In this manner, delays in processing the burst packets due to interfaces being inactive or otherwise unprepared may be mitigated. Nominally the primer packet is transmitted some fixed time prior to the burst. For example, this fixed time can be selected such that the primer packet has substantially no interaction with the subsequently transmitted sequence of packets, while being sufficiently short enough such that interfaces being "primed" do not return to an inactive or otherwise unprepared state prior to transmission of the sequence of packets. For example, according to some embodiment of the present invention, the fixed time can be 8ms, or other suitable predetermined amount of time as would be readily understood.

[0115] In one embodiment of the present invention, the primer packet can also be used in simplex measurements of a path. In other embodiments, the use of a primer packet may be further extended to overcome some of the challenges of time synchronization between the source host and the destination host, as without time syncing, transit times cannot be properly measured between the source host and the destination host. As such, datagram transit times cannot be directly compared to burst packet transit times in either direction.

[0116] According to an embodiment of the present invention, the primer packet may be treated as if it is a datagram, wherein its transmission timing relative to the burst can be used to provide timing information between the datagram and first packet of the burst. Since the difference in transmission timing between the primer packet and the first packet of the burst is fixed, it can be monitored at the destination host for change.

[0117] As such, the combination of the burst and primer which is received with the minimum overall round-trip time can be selected for use with the simplex measurement remediation as defined above. This minimum round trip time of this particular combination of the burst and primer packet can be indicative of the optimal conditions along the path, for example, the least amount of or no cross-traffic, which may be encountered during transmission along the path. The timings for the sequence of packets including the primer packet and the burst which were recorded at the destination host for the selected burst and primer combination can be examined to determine the separation between the primer packet and the first burst packet. If the gap at reception exceeds the original fixed transmission gap, for example 8 ms, it can be presumed that the additional gap represents the extra time required for transmission of the burst packet. In this instance, the timing measure and associated with that of the first packet in the burst can be replaced by the timing of the primer timing however subtracting the fixed gap time period. Figure 9 illustrates this embodiment of the present invention, wherein the timing of the primer packet 810, minus the 8ms, replaces the timing of that recorded for the first packet of the burst 800. [0118] In addition, if the reception gap between the primer packet and the first packet of the burst is less than the original transmission gap, the timing of the first burst packet remains unchanged. The subsequently combined burst timing can be used to calculate the capacity. In the event that the primer timing decreases the first burst packet timing, the capacity will be reduced.

[0119] In embodiments of the present invention, while this approach can be used for detecting and measuring half-duplex capacity in duplex testing, it can also detect other conditions that either treat streaming traffic differently than datagram traffic or are an artificial effect imposed by the interfaces at either the source host or the destination host.

Capacity Remediation

[0120] In one embodiment of the present invention, the sub-optimal operation of the path results in the path exhibiting an undesired level of variability of its determined capacity characteristics. In this embodiment, the collected test data is reflective of the timings associated with one or more sequences of packets, wherein each of the one or more sequences of packets includes multiple packets in a burst type configuration, preceded by a packet or datagram. The conversion of the test data into remediated data enables the subsequent evaluation of the capacity of the path having a desired limited variability.

[0121] Under particular circumstances, for example high capacity networks e.g. Gigabit Ethernet and the like, network interfaces may incidentally introduce variations in packet timings. For example, high performance network interface cards (NICs) may perform packet handling by grouping transmitted or received packets and processing them together. For example, if a NIC is performing packet handling based on groups of packets, during the collection of test data, a smaller datagram may appear to arrive as quickly as a larger datagram, or a first packet of a sequence of packets may appear to arrive later than a single packet or datagram, or several packets of a sequence of packets may appear to arrive at the same time. Thus this operational functionality of the NIC may result in timings for a single packet, for example a datagram, being significantly different from that of the first packet in a sequence of packets. An example of this type of group processing of packets 905 of a sequence of packets by a NIC is illustrated in Figure 10, wherein packets of a sequence of packets are processed in groups of four. According to embodiments of the invention, using a datagram 915, for example a large datagram, and its associated timing, instead of the timing parameters for a first packet of a sequence of packets, can alleviate the variability of the capacity determination for the network path being evaluated.

[0122] When the number of packets in a sequence of packets is varied for the same network path, it can be observed that there is a dependence of the estimated capacity based on the size of the sequence of packets. For example and with further reference to Figure 10, the network path capacity which would be determined should the sequence of packets include four packets, would be substantially different from the network capacity determined should there have been five packets in the sequence of packets. This result can be readily identified by the discontinuity on the round trip time calculated for the fourth packet and the fifth packet as can be seen in Figure 10.

[0123] According to embodiments of the present invention, the capacity of a network path can be evaluated based on the determination of an asymptotic limit of a curve reflective of the estimated capacity versus the number of packets in a sequence of packets which was used for the estimated capacity determination. For example, Figure 1 1 illustrates an example of a curve 1005 having an asymptotic limit 1007 which may be derived for the sampling points 1002, wherein each sampling point is reflective of the estimated capacity of the network path based on the sequence of packets comprising a particular number of packets. For example, this limiting value or asymptotic limit, can represent an idealized estimate wherein the number of packets in a sequence of packets approaches infinity, which substantially reproduces the effect of the method of flooding for the determination of capacity. The estimated capacity curve resulting from this process of capacity estimation can be fitted with a suitable function using one or a plurality of curve fitting functions and/or statistical methods for example least squares or other method as would be readily understood by a worker skilled in the art.

[0124] According to embodiments of the present invention, a series of sequences of packets having different numbers of packets are sent along the network path under consideration, wherein an estimate of capacity can be determined for these varying the size of the sequences of packets. For example, and with further reference to Figure 1 1 , as there are a total of 17 sampling points, these estimates of capacity may have been determined by sending 17 sequences of packets which had packet numbers ranging from 2 to 18 packets.

[0125] In some embodiments, sequences of packets comprising the largest number (N) of packets desired can be sent along the path, and the capacity can be determined based on the responses of the first predetermined number (n) of packets in the sequence of packets, where n < N. For example, and with further regard to Figure 1 1 , there are a total of 17 sampling points for the estimated capacity. Each of these sampling points can be determined by sending sequences of packets having 18 packets, and subsequently estimating the capacity of the network path under consideration based on the first two packets, first three packets, first four packets, and so on until all 18 packets of the sequence of packets are used.

[0126] According to one embodiment of the present invention, the determination of an appropriate expression for the estimated capacity curve can depend upon the specific circumstances. For the specific example illustrated in Figure 1 1 , a suitable function for fitting the illustrated data can be defined as follows:

Capacity(n) = A (Bn - I )

(Bn + 1)

where A is the asymptotic limit value, B describes the rate at which the curve converges to A and n is the number of packets in the sequence of packets. As would be readily understood by a worker skilled in the art, there are a plurality of different functions that can be defined to fit a particular set of data, and as such, the above function is for illustrative purposes only and is not to be considered limiting to the scope of the present invention.

Variations

[0127] In embodiments of the present invention, probing and evaluation can be repeated continuously over long, or effectively indefinite, periods of time. That case is sometimes referred to as "continuous monitoring". Where evaluations over finite-periods may provide an instantaneous snapshot of the current condition, continuous monitoring may allow for on-going comparison of network conditions according to different sequence of packet configurations. This can be of interest as traffic conditions are often highly transient, with peak loads happening at certain times of the day, or due to changing network circumstances or application usage.

[0128] It will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. In particular, it is within the scope of the invention to provide a computer program product or program element, or a program storage or memory device such as a transmission medium, magnetic or optical wire, tape or disc, or the like, for storing signals readable by a machine, for controlling the operation of a computer according to the method of the invention and/or to structure its components in accordance with the system of the invention.

[0129] Further, each step of the method may be executed on any general computer, such as a personal computer, server or the like and pursuant to one or more, or a part of one or more, program elements, modules or objects generated from any programming language, such as C++, Java, PL/1 , or the like. In addition, each step, or a file or object or the like implementing each said step, may be executed by special purpose hardware or a circuit module designed for that purpose.

[0130] AU such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

WE CLAIM:

1. A method for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub-optimal operation, said method comprising the steps of: a) generating one or more sequences of packets and transmitting the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; b) collecting test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; c) generating remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and d) analysing the remediated data to develop one or more characterizations of the path.

2. The method according to claim 1 , wherein the sub-optimal operation of the path results in reordering of one or more packets within the one or more sequences of packets and generating the remediated data includes renumbering one or more of the packets of a specific sequence of packets based on order of arrival.

3. The method according to claim 1 , wherein the sub-optimal operation of the path results in loss of one or more packets within the one or more sequences of packets and generating the remediated data includes dividing a specific sequence of packets into one or more sub-sets of contiguous packets.

4. The method according to claim 1 , wherein the sub-optimal operation of the path results in the path exhibiting unsymmetrical characteristics between an outgoing path and a returning path and generating one or more sequences of packets includes: a) generating and transmitting one or more sequences of packets configured for duplex handling; and b) generating and transmitting one or more sequences of packets configured for simplex handling.

5. The method according to claim 4, wherein the path includes a destination host and a source host, wherein subsequent to arrival of all packets of a specific sequence of packets configured for simplex handling, the destination host prepares a returning sequence of packets substantially identical to the specific sequence of packets and contiguously transmits the returning sequence of packets to the source host.

6. The method according to claim 5, wherein the destination host performs actions including: a) receiving and time stamping each packet of the one or more sequences of packets configured for simplex handling; and b) recording the time stamping in a payload of one or more packets of the returning sequence of packets.

7. The method according to claim 6, wherein the destination host waits a predetermined amount of time prior to transmitting the returning sequence of packets, and records said predetermined amount of time in a payload of one or more of the returning sequence of packets.

8. The method according to claim 4, wherein the remediated data enables determination of one or more characteristics of the outgoing path and/or one or more characteristics of the returning path.

9. The method according to claim 4, wherein the remediated data enables determination of capacity of the path indicative of sustained two-way traffic flow along the path.

10. The method according to claim 4, wherein the remediated data enables determination of capacity of the outgoing path and/or capacity of the returning path.

1 1. The method according to claim 1 , wherein generating the one or more sequences of packets and/or transmitting the one or more sequences of packets is modified at least in part based on the sub-optimal operation of the path.

12. The method according to claim 1 , wherein the sub-optimal operation of the path results in unsymmetrical characteristics between an outgoing path and a returning path, wherein generating one or more sequences of packets includes generating a duplex datagram and a specific sequence of packets, wherein generating remediated data includes combining timings representative of the duplex datagram with timings representative of the specific sequence of packets.

13. The method according to claim 12, wherein the duplex datagram is transmitted a predetermined amount of time prior to the specific sequence of packets.

14. The method according to claim 1 , wherein the sub-optimal operation of the path results in variability of a determined capacity, wherein generating remediated data includes evaluating a curve representative of capacity verses number of packets in the sequence of packets used for capacity determination.

15. The method according to claim 14, wherein evaluating the curve includes determining an asymptotic limit of the curve.

16. The method according to claim 1 , wherein the path is a closed path.

17. An apparatus for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub-optimal operation, the apparatus comprising: a) a generation module configured to generate one or more sequences of packets and to transmit the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; b) a collection module configured to collect test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; c) a remediation module configured to generate remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and d) an analysis module configured to analyse the remediated data to develop one or more characterizations of the path.

18. The apparatus according to claim 17, wherein the sub-optimal operation of the path results in reordering of one or more packets within the one or more sequences of packets and the remediation module is configured to generate the remediated data by renumbering one or more of the packets of a specific sequence of packets based on order of arrival.

19. The apparatus according to claim 17, wherein the sub-optimal operation of the path results in loss of one or more packets within the one or more sequences of packets and the remediation module is configured to generate the remediated data by dividing a specific sequence of packets into one or more sub-sets of contiguous packets.

20. The apparatus according to claim 17, wherein the sub-optimal operation of the path results in the path exhibiting unsymmetrical characteristics between an outgoing path and a returning path and the generation module is configured to generate and transmit one or more sequences of packets configured for duplex handling and generate and transmit one or more sequences of packets configured for simplex handling.

21. The apparatus according to claim 20, wherein the path includes a destination host and a source host, wherein subsequent to arrival of all packets of a specific sequence of packets configured for simplex handling, the destination host is configured to prepare a returning sequence of packets substantially identical to the specific sequence of packets and contiguously transmit the returning sequence of packets to the source host.

22. The apparatus according to claim 21 , wherein the destination host is configured to receive and time stamp each packet of the one or more sequences of packets configured for simplex handling and record the time stamps in a payload of one or more packets of the returning sequence of packets.

23. The apparatus according to claim 22, wherein the destination host is configured to wait a predetermined amount of time prior to transmitting the returning sequence of packets and further configured to record said predetermined amount of time in the payload of one or more packets of the returning sequence of packets.

24. The apparatus according to claim 20, wherein the remediated data enables determination of one or more characteristics of the outgoing path and/or one or more characteristics of the returning path.

25. The apparatus according to claim 20, wherein the remediated data enables determination of capacity of the path indicative of sustained two-way traffic flow along the path.

26. The apparatus according to claim 20, wherein the remediated data enables determination of capacity of the outgoing path and/or capacity of the returning path.

27. The apparatus according to claim 17, wherein the generation module is configured to generate the one or more sequences of packets and/or transmit the one or more sequences of packets based at least in part based on the sub-optimal operation of the path.

28. The apparatus according to claim 17, wherein the sub-optimal operation of the path results in unsymmetrical characteristics between an outgoing path and a returning path, wherein the generation module is configured to generate one or more sequences of packets which include a duplex datagram and a specific sequence of packets and the remediation module is configured to combine timings representative of the duplex datagram with timings representative of the specific sequence of packets during remediation of the test data.

29. The apparatus according to claim 28, wherein the duplex datagram is transmitted a predetermined amount of time prior to the specific sequence of packets.

30. The apparatus according to claim 17, wherein the sub-optimal operation of the path results in variability of a determined capacity, wherein the remediation module is configured to evaluate a curve representative of capacity verses number of packets in the sequence of packets used for capacity determination.

31. The apparatus according to claim 30, wherein the remediation module is configured to determine an asymptotic limit of the curve.

32. The apparatus according to claim 17, wherein the path is a closed path.

33. A computer readable medium having recorded thereon statement and instructions which, when executed by a computer processor, cause the processor to execute a method for characterizing a path of a packet-based network, said path including two or more nodes, wherein the path is exhibiting one or more forms of sub- optimal operation, said method comprising the steps of: a) generating one or more sequences of packets and transmitting the one or more sequences of packets along the path of the packet-based network, each packet of the one or more sequences of packets traversing at least a portion of said path; b) collecting test data representative of responses of the packet-based network to transmission of the one or more sequences of packets; c) generating remediated data from the test data, the remediated data generated at least in part based on the one or more forms of sub-optimal operation; and d) analysing the remediated data to develop one or more characterizations of the path.