Common TCP Evaluation Suite
draft-irtf-iccrg-tcpeval-01

Abstract

This document presents an evaluation test suite for the initial assessment of proposed TCP modifications. The goal of the test suite is to allow researchers to quickly and easily evaluate their proposed TCP extensions in simulators and testbeds using a common set of well-defined, standard test cases, in order to compare and contrast proposals against standard TCP as well as other proposed modifications. This test suite is not intended to result in an exhaustive evaluation of a proposed TCP modification or new congestion control mechanism. Instead, the focus is on quickly and easily generating an initial evaluation report that allows the networking community to understand and discuss the behavioral aspects of a new proposal, in order to guide further experimentation that will be needed to fully investigate the specific aspects of such proposal.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on January 5, 2015.

Copyright Notice

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Traffic generation
• 2.1. Desirable model characteristics
• 2.2. Tmix
• 2.2.1. Base Tmix trace files for tests
• 2.3. Loads
• 2.3.1. Varying the Tmix traffic load
• 2.3.2. Dealing with non-stationarity
• 2.4. Packet size distribution
• 2.4.1. Potential revision
• 3. Achieving reliable results in minimum time
• 3.1. Background
• 3.2. Equilibrium or Steady State
• 3.2.1. Note on the offered load in NS2
• 3.3. Accelerated test start up time
• 4. Basic scenarios
• 4.1. Basic topology
• 4.2. Traffic
• 4.3. Flows under test
• 4.4. Scenarios
• 4.4.1. Data Center
• 4.4.2. Access Link
• 4.4.3. Trans-Oceanic Link
• 4.4.4. Geostationary Satellite
• 4.4.5. Wireless LAN
• 4.4.6. Dial-up Link
• 4.5. Metrics of interest
• 4.6. Potential Revisions
• 5. Latency specific experiments
• 5.1. Delay/throughput tradeoff as function of queue size
• 5.1.1. Topology
• 5.1.2. Flows under test
• 5.1.3. Metrics of interest
• 5.2. Ramp up time: completion time of one flow
• 5.2.1. Topology and background traffic
• 5.2.2. Flows under test
• 5.2.3. Metrics of interest
• 5.3. Transients: release of bandwidth, arrival of many flows
• 5.3.1. Topology and background traffic
• 5.3.2. Flows under test
• 5.3.3. Metrics of interest
• 6. Throughput- and fairness-related experiments
• 6.1. Impact on standard TCP traffic
• 6.1.1. Topology and background traffic
• 6.1.2. Flows under test
• 6.1.3. Metrics of interest
• 6.2. Intra-protocol and inter-RTT fairness
• 6.2.1. Topology and background traffic
• 6.2.2. Flows under test
• 6.2.3. Metrics of interest
• 6.3. Multiple bottlenecks
• 6.3.1. Topology and traffic
• 6.3.2. Metrics of interest
• 7. Implementations
• 8. Acknowledgements
• 9. Informative References
• Appendix A. Discussions on Traffic
• Authors' Addresses

1. Introduction

This document describes a common test suite for the initial assessment of new TCP extensions or modifications. It defines a small number of evaluation scenarios, including traffic and delay distributions, network topologies, and evaluation parameters and metrics. The motivation for such an evaluation suite is to help researchers in evaluating their proposed modifications to TCP. The evaluation suite will also enable independent duplication and verification of reported results by others, which is an important aspect of the scientific method that is not often put to use by the networking community. A specific target is that the evaluations should be able to be completed in a reasonable amount of time by simulation, or with a reasonable amount of effort in a testbed.

It is not possible to provide TCP researchers with a complete set of scenarios for an exhaustive evaluation of a new TCP extension; especially because the characteristics of a new extension will often require experiments with specific scenarios that highlight its behavior. On the other hand, an exhaustive evaluation of a TCP extension will need to include several standard scenarios, and it is the focus of the test suite described in this document to define this initial set of test cases.

These scenarios generalize current characteristics of the Internet such as round-trip times (RTT), propagation delays, and buffer sizes. It is envisaged that as the Internet evolves these will need to be adjusted. In particular, we expect buffer sizes will need to be adjusted as latency becomes increasingly important.

The scenarios specified here are intended to be as generic as possible, i.e., not tied to a particular simulation or emulation platform. However, when needed some details pertaining to implementation using a given tool are described.

This document has evolved from a “round-table” meeting on TCP evaluation, held at Caltech on November 8-9, 2007, reported in [TESTSUITE08]. This document is the first step in constructing the evaluation suite; the goal is for the evaluation suite to be adapted in response to feedback from the networking community. It revises draft-irtf-tmrg-tests-02 [I-D-TMRG-TESTS].

The traces used and a sample implementation (including patched ns-2) are available from: http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG

2. Traffic generation

Congestion control concerns the response of flows to bandwidth limitations or to the presence of other flows. Cross-traffic and reverse-path traffic are therefore important to the tests described in this suite. Such traffic can have the desirable effect of reducing the occurrence of pathological conditions, such as global synchronization among competing flows, that might otherwise be mis-interpreted as normal average behaviours of those protocols [FLOYD03][MASCOLO06]. This traffic must be reasonably realistic for the tests to predict the behaviour of congestion control protocols in real networks, and also well-defined so that statistical noise does not mask important effects.

2.1. Desirable model characteristics

Most scenarios use traffic produced by a traffic generator, with a range of start times for user sessions, flow sizes, and the like, mimicking the traffic patterns commonly observed in the Internet. It is important that the same “amount” of congestion or cross-traffic be used for the testing scenarios of different congestion control algorithms. This is complicated by the fact that packet arrivals and even flow arrivals are influenced by the behavior of the algorithms. For this reason, a pure open-loop, packet-level generation of traffic where generated traffic does not respond to the behaviour of other present flows is not suitable. Instead, emulating application or user behaviours at the end points using reactive protocols such as TCP in a closed-loop fashion results in a closer approximation of cross-traffic, where user behaviours are modeled by well-defined parameters for source inputs (e.g., request sizes for HTTP), destination inputs (e.g., response size), and think times between pairs of source and destination inputs. By setting appropriate parameters for the traffic generator, we can emulate non-greedy user-interactive traffic (e.g., HTTP 1.1, SMTP and remote login), greedy traffic (e.g., P2P and long file downloads), as well as long-lived but non-greedy, non-interactive flows (or thin streams).

This approach models protocol reactions to the congestion caused by other flows in the common paths, although it fails to model the reactions of users themselves to the presence of congestion. A model that includes end-users’ reaction to congestion is beyond the scope of this draft, but we invite researchers to explore how the user behavior, as reflected in the flow sizes, user wait times, and number of connections per session, might be affected by the level of congestion experienced within a session [ROSSI03].

2.2. Tmix

There are several traffic generators available that implement a similar approach to that discussed above. For now, we have chosen to use the Tmix [WEIGLE06] traffic generator. Tmix is available for the NS2 and NS3 simulators, and can generate traffic for testbeds (for example GENI [GENITMIX]).

Tmix represents each TCP connection by a connection vector (CV) consisting of a sequence of (request-size, response-size, think-time) triples, thus representing bi-directional traffic. Connection vectors used for traffic generation can be obtained from Internet traffic traces.

2.2.1. Base Tmix trace files for tests

The traces currently defined for use in the test suite are based on campus traffic at the University of North Carolina (see [TRACES] for a description of construction methods and basic statistics).

The traces have an additional “m” field added to each connection vector to provide each direction’s maximum segment size for the connection. This is used to provide the packet size distribution described in Section 2.4.

These traces contain a mixture of connections, from very short flows that do not exist for long enough to be “congestion controlled”, to long thin streams, to bulk file transfer like connections.

The traces are available at: http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG

Each of the nine bidirectional trace files are named with the following convention:

    rAsI.org

where the I is the number of the Tmix initiator node, and A is the number of the tmix acceptor node, when the traffic sources are set up in the dumbbell configuration shown in Figure 2.

2.3. Loads

While the protocols being tested may differ, it is important that we maintain the same “load” or level of congestion for the experimental scenarios. For many of the scenarios, such as the basic ones in Section 4, each scenario is run for a range of loads, where the load is varied by varying the rate of session arrivals.

2.3.1. Varying the Tmix traffic load

To adjust the traffic load for a given scenario, the connection start times for flows in a Tmix trace are scaled as follows. Connections are actually started at:

    experiment_cv_start_time = scale * cv_start_time              (1)

where cv_start_time denotes the connection vector start time in the Tmix traces and experiment_cv_start_time is the time the connection starts in the experiment. Therefore, the smaller the scale the higher (in general) the traffic load.

2.3.1.1. Notes

Changing the connection start times also changes the way the traffic connections interact, potentially changing the “clumping” of traffic bursts.

Very small changes in the scaling parameter can cause disproportionate changes in the offered load. This is due to possibility of the small change causing the exclusion or inclusion of a CV that will transfer a very large amount of data.

2.3.2. Dealing with non-stationarity

The Tmix traffic traces, as they are, offer a non-stationary load. This is exacerbated for tests that do not require use of the full trace files, but only a portion of them. While removing this non-stationarity does also remove some of the “realism” of the traffic, it is necessary for the test suite to produce reliable and consistent results.

A more stationary offered load is achieved by shuffling the start times of connection vectors in the Tmix trace file. The trace file is logically partitioned into n-second bins, which are then shuffled using a Fisher-Yates shuffle [SHUFFLEWIKI], and the required portions written to shuffled trace files for the particular experiment being conducted.

2.3.2.1. Bin size

The bin size is chosen so that there is enough shuffling with respect to the test length. The offered traffic per test second from the Tmix trace files depends on a scale factor (see Section 2.3.1), which is related to the capacity of the bottleneck link. The shuffling bin size (in seconds) is set at:

   b = 500e6 / C                                                  (2)

where C is the bottleneck link’s capacity in bits per second, and 500e6 is a scaling factor (in bits).

Thus for the access link scenario described in Section 4.4.2, the bin size for shuffling will be 5 seconds.

2.3.2.2. NS2 implementation specifics

The tcl scripts for this process are distributed with the NS2 example test suite implementation. Care must be taken when using this algorithm, so that the given random number generator and the same seed are employed, or else the resulting experimental traces will be different.

2.4. Packet size distribution

For flows generated by the traffic generator, 10% of them use 536-byte packets, and 90% 1500-byte packets. The base Tmix traces described in Section 2.2.1 have been processed at the connection level to have this characteristic. As a result, packets in a given test will be roughly, but not be exactly, in this proportion. However, the proportion of offered traffic will be consistent for each experiment.

2.4.1. Potential revision

As Tmix can now read and use a connection’s Maximum Segment Size (MSS) from the trace file, it will be possible to produce Tmix connection vector trace files where the packet sizes reflect actual measurements.

3. Achieving reliable results in minimum time

This section describes the techniques used to achieve reliable results in the minimum test time.

3.1. Background

Over a long time, because the session arrival times are to a large extent independent of the transfer times, load could be defined as:

    A = E[f]/E[t],

where E[f] is the mean session (flow) size in bits transferred, E[t] is the mean session inter-arrival time in seconds, and A is the load in bps.

It is important to test congestion control protocols in “overloaded” conditions. However, if A > C, where C is the capacity of the bottleneck link, then the system has no equilibrium. In long-running experiments with A > C, the expected number of flows would keep on increasing with time (because as time passes, flows would tend to last for longer and longer, thus “piling up” with newly-arriving ones). This means that, in an overload scenario, some measures will be very sensitive to the duration of the tests.

3.2. Equilibrium or Steady State

Ideally, experiments should be run until some sort of equilibrium results can be obtained. Since every test algorithm can potentially change how long this may take, the following approach is adopted:

1. Traces are shuffled to remove non-stationarity (see Section 2.3.2.)
2. The experiment run time is determined from the traffic traces. The shuffled traces are compiled such that the estimate of traffic offered in the second third of the test is equal to the estimate of traffic offered in the final third of the test, to within a 5% tolerance. The length of the trace files becomes the total experiment run time (including the warm up time).
3. The warmup time until measurements start, as shown in Section 4, is calculated as the time at which the NS2 simulation of standard TCP achieves “steady state”. In this case, warmup time is determined as the time required so the measurements have statistically similar first and second half results. The metrics used as reference are: the bottleneck raw throughput, and the average bottleneck queue size. The latter is stable when A » C and A « C, but not when A ~= C. In this case the queue is not a stable measure, and just the raw bottleneck throughput is used.

3.2.1. Note on the offered load in NS2

The offered load in an NS2 simulation using one-way TCP will be higher than the estimated load. One-way TCP uses fixed TCP segment sizes, so all transmissions that would normally use a segment size less than the maximum segment size (in this case 496B or 1460B), such as at the end of a block of data, or for short queries or responses, will still be sent as a maximum segment size packet.

3.3. Accelerated test start up time

Tmix traffic generation does not provide an instant constant load. It can take quite a long time for the number of simultaneous TCP connections, and thus the offered load, to build up. To accelerate the system start up, the system is “prefilled” to a state close to “steady state”. This is done by starting initial sessions over a shorter interval than they would normally start, and biasing the sessions started to longer sessions. Details of how this is achieved follow.

Connections that start before t=prefill_t in the Tmix traces, are selected with a bias toward longer sessions (connections which are estimated to continue past the long_flow_bias time (see Figure 1)). These selected connections are then started at an accelerated rate by starting them over the time interval prefill_si.

The prefill_t (in seconds) calculation is based on the following heuristic:

    prefill_t = 1.5 * targetload * maxRTT                         (3)

where maxRTT is the median maximum RTT in the particular topology, and targetload is given as a percentage. This generally works quite well, but requires some adjustment for very high BDP scenarios. Experiment tables specify the prefill_t value to be used in each experiment.

The long_flow_bias threshold is set at

    long_flow_bias = prefill_t / 2 .                              (4)

These values are not optimal, but have been experimentally determined to give reasonable results.

The start up time interval, prefill_si, is calculated as follows:

    prefill_si = total_pfcb / (C * TL / 100.0)                    (5)

where total_pfcb is the total number of bits estimated to be sent by the prefill connections, C is the capacity of the bottleneck link, and TL is the target offered load as a percentage.

This procedure has the effect of quickly bringing the system to a loaded state. From this point the system runs until t = warmup (as calculated in Section 3.2), after which moment statistics are computed.

                                       |<----- test_duration ----->|
                                       |                           |
              prefill_si               |                           |
                |<-->|                 |                           |
|--------|------|----|-----------------|---------------------------|
t=0      |      |    |<---- warmup --->|
         |      |    |                 |
         |      |    t = prefill_t     t = warmup + prefill_t 
         |      |
         |      t = prefill_t - prefill_si
         |
         t = long_flow_bias 

Node 1                                                      Node 4
       \_                                                _/
         \_                                            _/
           \_ __________     Central      __________ _/
             |          |     link       |          |
Node 2 ------| Router 1 |----------------| Router 2 |------ Node 5
            _|__________|                |__________|_
          _/                                          \_
        _/                                              \_
Node 3 /                                                  \ Node 6

Node_1----Wireless_1..                                      Node_4 
                     :.                                    / 
                      :...   Base   central link          / 
Node_2----Wireless_2 ....:..Station-------------- Router_2 --- Node_5
                      ...: (Router 1)                     \ 
                     .:                                    \ 
Node_3----Wireless_3.:                                      Node_6 

           logical link
       X--------------------X
       |                    | 
       v                    |
   n1--+-- .                |  _n4 
            :               V /
   n2--+-- .:.C0-------------C1---n5
            :                 \_
   n3--+-- .                    n6

              T_n2                        T_n4
               |                           |
               |                           |
         T_n1  |                           |  T_n3
            \  |                           | /
             \ |                           |/
      B_n1--- R1--------------------------R2--- B_n4
             / |                           |\
            /  |                           | \
        B_n2   |                           |  B_n5
               |                           |
              B_n3                        B_n6

           T                                  T
            \                                /
             \                              /
              R1--------------------------R2
             /                              \
            /                                \
           U                                  U

              A_1                                  A_4
              B_1                                  B_4 
                 \                                / 
                  \        central link          / 
          A_2 --- Router_1 -------------- Router_2 --- A_5 
          B_2     /                              \     B_5
                 /                                \ 
              A_3                                  A_6
              B_3                                  B_6

 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >
  __________ 0ms _________________ 0ms __________________ 30ms ____
 |   ................  |   ................  |   ................  |
 |   :              :  |   :              :  |   :              :  |
 |   :              :  |   :              :  |   :              :  |
0ms  :              : 30ms :              : 0ms  :              : 0ms
 |   ^              V  |   ^              V  |   ^              V  |

   > - - - - - - - - - - - - - - - - - - - - - - - - - - -  - - - >
    __________ 10ms _______________ 10ms ________________ 10ms ___ 
   |   ...............  |   ...............  |   ...............  |
   |   :             :  |   :             :  |   :             :  |
   |   :             :  |   :             :  |   :             :  |
  10ms :             : 10ms :             : 10ms :             : 10ms
   |   ^             V  |   ^             V  |   ^             V  |

           Node_1          Node_3     Node_5         Node_7 
               \              |          |             / 
                \             |10ms      |10ms        /10ms
              0ms\            |          |           / 
                  \      A    |    B     |   C      /
                Router1 ---Router2---Router3--- Router4  
                  /    10ms   |   10ms   |  10ms    \ 
                 /            |          |           \ 
            10ms/             |10ms      |10ms        \ 0ms
               /              |          |             \ 
          Node_2           Node_4     Node_6         Node_8 
    
 Flow 1: Node_1 <--> Node_8    
 Flow 2: Node_2 <--> Node_3
 Flow 3: Node_4 <--> Node_5
 Flow 4: Node_6 <--> Node_7

  A = E[f]/E[t],