+---+                                                           +---+
|S1 |====== \                 Forward -->              / =======|R1 |
+---+       \\                                        //        +---+
             \\                                      //
+---+       +-----+                               +-----+       +---+
|S2 |=======|  A  |------------------------------>|  B  |=======|R2 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
(...)         //                                     \\         (...)
             //          <-- Backward                 \\
+---+       //                                         \\       +---+
|Sn |====== /                                           \ ======|Rn |
+---+                                                           +---+

Figure 1: Example of A Testbed Topology

• Define the test case:
  • General description: describes the motivation and the goals of the test case.
  • Expected behavior: describes the desired rate adaptation behavior.
  • Define a list of metrics to evaluate the desired behavior: this indicates the minimum set of metrics (e.g., link utilization, media sending rate) that a proposed algorithm needs to measure to validate the expected rate adaptation behavior. It should also indicate the time granularity (e.g., averaged over 10ms, 100ms, or 1s) for measuring certain metrics. Typical measurement interval is 200ms.
• Define testbed topology: every test case needs to define an evaluation testbed topology. Figure 1 shows such an evaluation topology. In this evaluation topology, S1..Sn are traffic sources. These sources generate media traffic and use either congestion control algorithm under investigation. R1..Rn are the corresponding receivers. A test case can have one or more such traffic sources (S) and their corresponding receivers (R). The path from the source to destination is denoted as "forward" and the path from a destination to a source is denoted as "backward". The following basic structure of the test case has been described from the perspective of media generating endpoints attached on the left-hand side of Figure 1. In this setup, the media flows are transported in forward direction and corresponding feedback/control messages are transported in the backward direction. However, it is also possible to set up the test with media in both forward and backward directions. In that case, unless otherwise specified by the test case, it is expected that the backward path does not introduce any congestion related impairments and has enough capacity to accommodate both media and feedback/control messages. It should be noted that depending on the test cases it is possible to have different path characteristics in either of the directions.
• Define testbed attributes:
  • Duration: defines the duration of the test in seconds.
  • Path characteristics: defines the end-to-end transport level path characteristics of the testbed for a particular test case. Two sets of attributes describe the path characteristics, one for the forward path and the other for the backward path. The path characteristics for a particular path direction is applicable to all the Sources "S" sending traffic on that path. If only one attribute is specified, it is used for both path directions, however, unless specified the reverse path has no capacity restrictions and no path loss.
    • Path direction: forward or backward.
    • Bottleneck-link capacity: defines minimum capacity of the end-to-end path
    • Reference bottleneck capacity: defines a reference value for the bottleneck capacity for test cases with time-varying bottleneck capacities. All bottleneck capacities will be specified as a ratio with respect to the reference capacity value.
    • One-way propagation delay: describes the end-to-end latency along the path when network queues are empty, i.e., the time it takes for a packet to go from the sender to the receiver without encountering any queuing delay.
    • Maximum end-to-end jitter: defines the maximum jitter that can be observed along the path.
    • Bottleneck queue type: for example, Droptail, FQ-CoDel, or PIE.
    • Bottleneck queue size: defines the size of queue in terms of queuing time when the queue is full (in milliseconds).
    • Path loss ratio: characterizes the non-congested, additive, losses to be generated on the end-to-end path. MUST describe the loss pattern or loss model used to generate the losses.
  • Application-related: defines the traffic source behavior for implementing the test case
    • Media traffic Source: defines the characteristics of the media sources. When using more than one media source, the different attributes are enumerated separately for each different media source.
      • Media type: Video/Voice
      • Media flow direction: forward, backward or both.
      • Number of media sources: defines the total number of media sources
      • Media codec: Constant Bit Rate (CBR) or Variable Bit Rate (VBR)
      • Media source behavior: describes the media encoder behavior. It defines the main parameters that affect the adaptation behavior. This may include but is not limited to:
        • Adaptability: describes the adaptation options. For example, in the case of video it defines the following ranges of adaptation: bit rate, frame rate, video resolution. Similarly, in the case of voice, it defines the range of bit rate adaptation, the sampling rate variation, and the variation in packetization interval.
        • Output variation : for a VBR encoder it defines the encoder output variation from the average target rate over a particular measurement interval. For example, on average the encoder output may vary between 5% to 15% above or below the average target bit rate when measured over a 100 ms time window. The time interval over which the variation is specified must be provided.
        • Responsiveness to a new bit rate request: the lag in time between a new bit rate request from the congestion control algorithm and actual rate changes in encoder output. Depending on the encoder, this value may be specified in absolute time (e.g. 10ms to 1000ms) or other appropriate metric (e.g. next frame interval time).

        More detailed discussions on expected media source behavior, including those from synthetic video traffic sources, is at

        [I-D.ietf-rmcat-video-traffic-model].
      • Media content: describes the chosen media sequences; For example, test sequences are available at: [xiph-seq] and [HEVC-seq].
      • Media timeline: describes the point when the media source is introduced and removed from the testbed. For example, the media source may start transmitting immediately when the test case begins, or after a few seconds.
      • Startup behavior: the media starts at a defined bit rate, which may be the minimum, maximum bit rate, or a value in between (in Kbps).
    • Competing traffic source: describes the characteristics of the competing traffic source, the different types of competing flows are enumerated in [I-D.ietf-rmcat-eval-criteria].
      • Traffic direction: forward, backward or both.
      • Type of sources: defines the types of competing traffic sources. Types of competing traffic flows are listed in [I-D.ietf-rmcat-eval-criteria]. For example, the number of TCP flows connected to a web browser, the mean size and distribution of the content downloaded.
      • Number of sources: defines the total number of competing sources of each media type per traffic direction.
      • Congestion control: enumerates the congestion control used by each type of competing traffic.
      • Traffic timeline: describes when the competing traffic starts and ends in the test case.
  • Additional attributes: describes attributes essential for implementing a test case which are not included in the above structure. These attributes MUST be well defined, so that the other implementers of that particular test case are able to implement it easily.

Any attribute can have a set of values (enclosed within "[]"). Each member value of such a set MUST be treated as different value for the same attribute. It is desired to run separate tests for each such attribute value.

The test cases described in this document follow the above structure.

4. Recommended Evaluation Settings

This section describes recommended test case settings and could be overwritten by the respective test cases.

4.1. Evaluation metrics

To evaluate the performance of the candidate algorithms the implementers MUST log enough information to visualize the following metrics at a fine enough time granularity:

1. Flow level:
   1. End-to-end delay for the congestion controlled media flow.
   2. Variation in sending bit rate and goodput. Mainly observing the frequency and magnitude of oscillations.
   3. Packet losses observed at the receiving endpoint.
   4. Feedback message overhead.
   5. Convergence time - time to reach steady state for the congestion controlled media flow(s).
2. Transport level:
   1. Bandwidth utilization.
   2. Queue length (milliseconds at specified path capacity):
      • average over the length of the session.
      • 5 and 95 percentile.
      • median, maximum, minimum.

4.2. Path characteristics

Each path between a sender and receiver as described in Figure 1 have the following characteristics unless otherwise specified in the test case.

• Path direction: forward and backward.
• Reference bottleneck capacity: 1Mbps.
• One-Way propagation delay: 50ms. Implementers are encouraged to run the experiment with additional propagation delays mentioned in [I-D.ietf-rmcat-eval-criteria]
• Maximum end-to-end jitter: 30ms. Jitter models are described in [I-D.ietf-rmcat-eval-criteria]
• Bottleneck queue type: Drop tail. Implementers are encouraged to run the experiment with other AQM schemes, such as FQ-CoDel and PIE.
• Bottleneck queue size: 300ms.
• Path loss ratio: 0%.

Examples of additional network parameters are discussed in [I-D.ietf-rmcat-eval-criteria].

For test cases involving time-varying bottleneck capacity, all capacity values are specified as a ratio with respect to a reference capacity value, so as to allow flexible scaling of capacity values along with media source rate range. There exist two different mechanisms for inducing path capacity variation: a) by explicitly modifying the value of physical link capacity; or b) by introducing background non-adaptive UDP traffic with time-varying traffic rate. Implementers are encouraged to run the experiments with both mechanisms for test cases specified in Section 5.1, Section 5.2, and Section 5.3.

4.3. Media source

Unless otherwise specified, each test case will include one or more media sources as described below.

• Media type: Video
  • Media codec: VBR
  • Media source behavior:
    • Adaptability:
      • Bit rate range: 150 Kbps - 1.5 Mbps. In real-life applications the bit rate range can vary a lot depending on the provided service, for example, the maximum bit rate can be up to 4Mbps. However, for running tests to evaluate the congestion control algorithms it is more important to have a look at how they are reacting to certain amount of bandwidth change. Also it is possible that the media traffic generator used in a particular simulator or testbed is not capable of generating higher bit rate. Hence we have selected a suitable bit rate range typical of consumer-grade video conferencing applications in designing the test case. If a different bit rate range is used in the test cases, then the end-to-end path capacity values will also need to be scaled accordingly.
      • Frame resolution: 144p - 720p (or 1080p). This resolution range is selected based on the bit rate range. If a different bit rate range is used in the test cases then the frame resolution range also need to be selected suitably.
      • Frame rate: 10fps - 30fps. This frame rate range is selected based on the bit rate range. If a different bit rate range is used in the test cases then the frame rate range also need to be adjusted suitably.
    • Variation from target bit rate: +/-5%. Unless otherwise specified in the test case(s), bit rate variation SHOULD be calculated over one (1) second period of time.
    • Responsiveness to new bit rate request: 100ms
  • Media content: The media content should represent a typical video conversational scenario with head and shoulder movement. We recommend to use Foreman video sequence.
  • Media startup behavior: 150Kbps. It should be noted that applications can use smart ways to select an optimal startup bit rate value for a certain network condition. In such cases the candidate proposals MAY show the effectiveness of such smart approach as an additional information for the evaluation process.

• Media type: Audio
  • Media codec: CBR
  • Media bit rate: 20Kbps

5. Basic Test Cases

5.1. Variable Available Capacity with a Single Flow

In this test case the bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [I-D.ietf-rmcat-cc-requirements], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:

• an intermediate bottleneck
• change in available capacity (e.g., due to interface change, routing change, abrupt arrival/departure of background non-adaptive traffic).
• maximum media bit rate is greater than link capacity. In this case, the application will attempt to ramp up to its maximum bit rate, since the link capacity is limited to a value lower, the congestion control scheme is expected to stabilize the sending bit rate close to the available bottleneck capacity.

It should be noted that the exact variation in available capacity due to any of the above depends on the underlying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviors in a certain network environment.

Expected behavior: the candidate algorithm is expected to detect the path capacity constraint, converges to the bottleneck link's capacity and adapt the flow to avoid unwanted oscillation when the sending bit rate is approaching the bottleneck link's capacity. The oscillations occur when the media flow(s) attempts to reach its maximum bit rate but overshoots the usage of the available bottleneck capacity then to rectify, it reduces the bit rate and starts to ramp up again.

Evaluation metrics : as described in Section 4.1.

                             Forward -->
+---+       +-----+                               +-----+       +---+
|S1 |=======|  A  |------------------------------>|  B  |=======|R1 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
                          <-- Backward

+---+                                                         +---+
|S1 |===== \                                         / =======|R1 |
+---+      \\             Forward -->               //        +---+
            \\                                     //
            +-----+                               +-----+
            |  A  |------------------------------>|  B  |
            |     |<------------------------------|     |
            +-----+                               +-----+
              //                                    \\
             //          <-- Backward                \\
+---+       //                                        \\       +---+
|S2 |====== /                                          \ ======|R2 |
+---+                                                          +---+

	 +---+                                                         +---+
	 |S1 |===== \                Forward -->              / =======|R1 |
	 +---+      \\                                       //        +---+
	             \\                                     //
       		  +-----+                               +-----+
	          |  A  |------------------------------>|  B  |
	          |     |<------------------------------|     |
	          +-----+                               +-----+
	             //                                     \\
	            //            <-- Backward               \\
	+---+      //                                         \\       +---+
	|R2 |===== /                                           \ ======|S2 |
	+---+                                                          +---+
	

+---+                                                         +---+
|S1 |===== \                Forward -->              / =======|R1 |
+---+      \\                                       //        +---+
            \\                                     //
+---+       +-----+                               +-----+       +---+
|S2 |=======|  A  |------------------------------>|  B  |=======|R2 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
              //                                     \\
             //          <-- Backward                 \\
+---+       //                                         \\       +---+
|S3 |====== /                                           \ ======|R3 |
+---+                                                           +---+

	 +--+                                                       +--+
	 |S1|===== \              Forward -->              / =======|R1|
	 +--+      \\                                     //        +--+
	            \\                                   //
            +-----+                             +-----+
            |  A  |---------------------------->|  B  |
            |     |<----------------------------|     |
            +-----+                             +-----+
	             //                                   \\
	            //        <-- Backward                 \\
	+-----+    //                                       \\    +-----+
	|S_tcp|=== /                                         \ ===|R_tcp|
	+-----+                                                   +-----+
	

                             Forward ---->


             +---+          +---+        +---+      +---+
             |S2 |          |R2 |        |S3 |      |R3 |
             +---+          +---+        +---+      +---+
               |              |            |          |
	       |              |            |          |
+---+       +-----+       +-----+      +-----+     +-----+       +---+
|S1 |=======|  A  |------>|  B  |----->|  C  |---->|  D  |=======|R1 |
+---+       |     |<------|     |<-----|     |<----|     |       +---+
            +-----+       +-----+      +-----+     +-----+

                     1st                       2nd
              Bottleneck (A->B)          Bottleneck (C->D)

                           <------ Backward