+--------------------------------+
			|Case| Number of traffic         |
			|    +-----+----+----+-----------+
			|    |Video|Webs| CBR| FTP (IW10)|
			+----|-----|----|----|-----------|
			| A  |  0  |  1 |  1 |     0     |
			|----|-----|----|----|-----------|
			| B  |  0  |  1 |  1 |     1     |
			|----|-----|----|----|-----------|
			| C  |  1  |  1 |  1 |     0     |
			+----|-----|----|----|-----------|
			| D  |  1  |  1 |  1 |     0     |
			+----|-----|----|----|-----------|
			| E  |  1  |  1 |  1 |     1     |
			+----+-----+----+----+-----------+
		

Figure 3: Bursty traffic scenarios

For each of these scenarios, the graphs described in Section 2.3 MUST be generated. In addition, other metrics such as end-to-end latency, jitter, flow completion time, QoE MUST be generated.

3.2.3. Inter-RTT and intra-protocol fairness

TCP dynamics are a driving force for AQM design. It is therefore important to evaluate against a set of RTT (e.g., from 5 ms to 200 ms). Also, asymmetry in terms of RTT between various paths SHOULD be considered so that the fairness between the flows can be discussed as one may react faster to congestion than another. The impact of the scheduling and the AQM introduced on this lack of fairness SHOULD be evaluated.

Moreover, introducing an AQM and/or scheduling schemes may result in the absence of fairness between the flows, even when the RTTs are identical. This potential lack of fairness SHOULD be evaluated.

The topology that SHOUD be exploited is the one of Figure 1:

• to evaluate the inter-RTT fairness, for each run, two flows (Flow1.1 and Flow1.2) SHOULD be introduced and the set of RTT SHOULD be: RTTA1.1 in [5ms;100ms] and RTTA1.2 in [100ms;200ms].
• to evaluate the impact of the RTT value on the AQM performance and the intra-protocol fairness, for each run, two flows (Flow1.1 and Flow1.2) SHOULD be introduced and the set of RTT SHOULD be: RTTA1.1 in [5ms;200ms] and RTTA1.2 in [5ms;200ms], with (RTTA1.1)=(RTTA1.2).

These flows MUST have the same congestion control algorithm.

The output that MUST be measured is the ratio between the average goodput values of the two flows (Section 2.2.3) and the packet drop rate for each flow (Section 2.2.2).

3.2.4. Fluctuating network conditions

Network devices experience varying operating conditions depending on factors such as time of day, deployment scenario etc. For example:

• Traffic and congestion levels are higher during peak hours than off-peak hours.
• A queue's draining rate could vary depending on other queues. A low load on high priority queue implies higher draining rate for lower priority queues.

If the target context is a stable environment, the tester MUST illustrate their stability over time.

In context where the network conditions can vary over time, an AQM scheme MUST be robust enough to control network latencies under fluctuating network conditions, without the need for operator tuning of AQM parameters. This document will evaluate AQM proposals under varying congestion levels and varying draining rates.

3.2.4.1. Topology Description

The topology is presented in Figure 1. For this scenario, the capacities of the links MUST be set to 10Mbps and the RTTs to 100ms.

3.2.4.2. Mild Congestion

This scenario helps evaluate how an AQM scheme adapts to a light load of incoming traffic resulting in mild congestion -- packet drop rates less than 1%. The scenario consists of 4-5 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second. Each TCP flow transfers a large file for a period of 50s.

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.2.4.3. Medium Congestion

This scenario helps evaluate how an AQM scheme adapts to incoming traffic resulting in medium congestion -- packet drop rates between 1%-3%. The scenario consists of 10-20 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second. Each TCP flow transfers a large file for a period of 50s.

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.2.4.4. Heavy Congestion

This scenario helps evaluate how an AQM scheme adapts to incoming traffic resulting in heavy congestion -- packet drop rates between 5%-10%. The scenario consists of 30-40 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second. Each TCP flow transfers a large file for a period of 50s.

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.2.4.5. Varying Available Bandwidth

This scenario helps evaluate how an AQM scheme adapts to varying available bandwidth on the outgoing link. To simulate varying draining rates, the bottleneck bandwidth between nodes 'Central L' and Central R' vary over the course of the experiment as follows -- 100Mbps during 0-50s, 10Mbps during 50-100s, 100Mbps during 100-150s. The scenario consists of 50 TCP New Reno flows between sender A and receiver B. All TCP flows start at random times during the initial second. Each TCP flow transfers a large file for a period of 150s. In order to better assess the impact of draining rates on the AQM behavior, the tester MUST compare its performance with those of tail-drop.

For this scenario, the graphs described in Section 2.3 MUST be generated. Moreover, one graph MUST be generated for each of the three phases previously detailed.

3.3. Diverse Network Environments

This section presents scenarios with are related to specific network environments. These classical network environments which COULD be considered to evaluate the performance of the AQM under tests.

Each subsection presents a generic scenario and a use case. The scenarios are classified according to a relation between the delay (high, medium or low) and the capacity (high, medium, low). One scenario details as well that the impact of the sizes of the buffers should be evaluated. The guidelines selected those which are of interest for the evaluation of the performance of AQM proposals. On top of these abstracted scenario, these guidelines present use cases for each selected scenario, by proposing a carefully dimensioned topology.

3.3.1. Medium bandwidth, medium delay: Wi-Fi

This scenario is introduced to carefully evaluate AQM proposals in a generic context, where the link between the delay and the bandwidth is not specific and assess how AQM proposals can control latency in this context.

We refer to Figure 1 to detail the topology:

• Sender A to Central L: capacity=100Mbps, RTT=10ms;
• Central L to Central R: capacity=20Mbps, RTT=10ms;
• Central R to Receiver B: capacity=100Mbps, RTT=10ms;
• The tester MAY include a packet loss rate of 1 to 3% on the Wi-Fi link (between Central L and Central R).

The traffic that MUST generated between the sender A and the receiver B is:

• Five repeating TCP transfers: repeating 5MB file transmission;
• One continuous TCP transfer: continuous file transmission;
• Four HTTP web traffic (repeated download of 700kB);

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.3.2. Low bandwidth, high delay: Rural broadband networks and satellite links

In the context of low bandwith and high delay, the burst absorption capacity of an AQM is seriously challenged. Indeed, due to the important bandwith-delay product, the sending buffer should be large, resulting in potentially large congestion windows and large bursts arrivals to the gateways. The tolerance to large incoming bursts is a key feature of an AQM introduced in this context: this is the reason why this challenging context is detailed in these guidelines.

We refer to Figure 1 to detail the topology:

• Sender A to Central L: capacity=10Mbps, RTT=10ms;
• Central L to Central R: capacity=1Mbps, RTT=200ms;
• Central R to Receiver B: capacity=10Mbps, RTT=10ms;

The traffic that MUST generated between the sender A and the receiver B is:

• Five repeating TCP transfers: repeating 5MB file transmission;
• One continuous TCP transfer: continuous file transmission;
• Four HTTP web traffic (repeated download of 700kB);

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.3.3. High bandwidth, low delay: data centers

In the context of high bandwith and low delay, the specific characteristics require updated thresholds, which determine the behavior of an AQM. As a result, the auto-tuning of an AQM is seriously challenged. This is the reason why this challenging context is detailed in these guidelines.

We refer to Figure 1 to detail the topology:

• Sender A to Central L: capacity=1Gbps, RTT=0.1ms;
• Central L to Central R: capacity=1Gbps, RTT=0.1ms;
• Central R to Receiver B: capacity=1Gbps, RTT=0.1ms;

The traffic that MUST generated between the sender A and the receiver B is:

• Four repeating TCP transfers: repeating 5MB file transmission;

For this scenario, the graphs described in Section 2.3 MUST be generated.

3.3.4. Low and high buffers

The size of the buffers impacts on AQMs performance, whether its algorithm is based on the queue length or the queueing delay. The tester MAY consider cases where the buffer is low (i.e. 1/10 BDP) and when the buffer is large (i.e. 10 BDP).

We refer to Figure 1 to detail the topology:

• Sender A to Central L: capacity=100Mbps, RTT=10ms;
• Central L to Central R: capacity=20Mbps, RTT=10ms;
• Central R to Receiver B: capacity=100Mbps, RTT=10ms;

The traffic that MUST generated between the sender A and the receiver B is:

• Five repeating TCP transfers: repeating 5MB file transmission;
• One continuous TCP transfer: continuous file transmission;
• Four HTTP web traffic (repeated download of 700kB);

For this scenario, the graphs described in Section 2.3 MUST be generated. Moreover, these guidelines advise to plot the characteristics of the queue (such as queue length or queuing delay) as explained in Section 2.1.2.

4. Deployment

This section details deployment issues that MUST be discussed, such as stability, implementation cost, implementation feasibility, control knobs, etc.

4.1. Operator control knobs and auto-tuning

One of the biggest hurdles for RED deployment was/is its parameter sensitivity to operating conditions -- how difficult it is to tune important RED parameters for a deployment in order to get maximum benefit from the RED implementation. Fluctuating congestion levels and network conditions add to the complexity. Incorrect parameter values lead to poor performance. Naturally, various network operators felt it unwise to turn on the AQM (RED) implementation.

Any AQM scheme is likely to have parameters whose values affect the AQM's control law and behavior. Exposing all these parameters as control knobs to a network operator (or user) can easily result in an unsafe AQM deployment. Unexpected AQM behavior ensues when parameter values are not set properly. A minimal number of control knobs minimizes the number of ways a, possible naive, user can break the AQM system. Fewer control knobs make the AQM scheme more user-friendly and easier to deploy and debug.

We recommend that an AQM scheme SHOULD minimize the control knobs exposed for operator tuning. An AQM scheme SHOULD expose only those knobs that control the "larger" AQM behavior such as queue delay threshold, queue length threshold, etc.

Additionally, an AQM scheme's safety is directly related to its stability under varying operating conditions such as varying traffic profiles and fluctuating network conditions, as described in Section 3.2.4 and in Section 3.2.1. Operating conditions vary often and hence it is necessary that the AQM MUST remain stable under these conditions without the need for additional external tuning. If AQM parameters require tuning under these conditions, then the AQM MUST self-adapt necessary parameter values by employing auto-tuning techniques.

4.2. Parameter sensitivity and stability analysis

An AQM scheme's control law is the primary means by which the AQM controls queuing delay. Hence understanding the AQM control law is critical to understanding AQM behavior. The AQM's control law may include several input parameters whose values affect the AQM output behavior and stability. Additionally, AQM schemes may auto-tune parameter values in-order to maintain stability under different network conditions (such as different congestion levels, draining rates or network environments). The stability of these auto-tuning techniques is also important to understand.

AQM proposals SHOULD provide background material showing control theoretic analysis of the AQM control law and the input parameter space within which the control law operates as expected. For parameters that are auto-tuned, the material SHOULD include stability analysis of the auto-tuning mechanism(s) as well. Such analysis helps the WG understand AQM control law better and the network conditions/deployments under which the AQM is stable.

The impact of every externally tuned parameter MUST be discussed. As an example, if an AQM proposal needs various external tuning to work on different network environments presented in Section 3, these external modifications MUST be clear for deployment issues. Also, the frequency at which some parameters are re-configured MUST be evaluated, as it may impact the capacity of the AQM to absorb incoming bursts of packets.

4.3. Implementation cost

An AQM's successful deployment is directly related to its ease of implementation. Network platforms may need hardware or software implementations of the AQM. Depending on a platform's capabilities and limitations, the platform may or may not be able to implement some or all parts of the AQM logic.

AQM proposals SHOULD provide pseudo-code for the complete AQM scheme, highlighting generic implementation-specific aspects of the scheme such as "drop-tail" vs. "drop-head", inputs from platform (current queueing delay, queue length), computations involved, need for timers etc. This helps identify costs associated with implementing the AQM on a particular hardware or software platform. Also, it helps WG understand what kind of platforms can easily support the AQM and which cannot.

AQM proposals SHOULD highlight parts of AQM logic that are platform dependent and discuss if and how AQM behavior could be impacted by the platform. For example, a queue-delay based AQM scheme requires current queuing delay as input from the platform. If the platform already maintains this value, then it is trivial to implement the AQM logic on the platform. On the other hand, if the platform provides indirect means to estimate queuing delay (ex: timestamps, deque rate etc.), then the AQM behavior is sensitive to how good the queuing delay estimate turns out on that platform. Highlighting the AQM's sensitivity to queuing delay estimate helps implementers identify optimal means of implementing the AQM on a platform.

4.4. Interaction with packet scheduling

On top of the introduction of an AQM scheme, a router may schedule the transmission of packets in a specific manner by introducing a scheduling scheme. This algorithm may create sub-queues and integrate an AQM scheme on each of these sub-queues. Another scheduling policy may modify the way packets are sequenced, modifying the timestamp of each packet.

Both schedulers and AQMs can be introduced when packet are either enqued or dequed. If both schedulers and AQM are implemented when packet are enqued, their interaction should not be a major issue. However, if one is introduced when packets are enqued and the others when they are dequed, there may be destructive interactions.

The scheduling and the AQM schemes conjointly impact on the end-to-end performance. During the evaluation process of an AQM proposal, the tester MUST discuss the feasibility to add scheduling on top of its algorithm. This discussion MAY detail if AQM is placed while packets are enqued and dequed.

4.5. ECN behavior

Apart from packet drops, Explicit Congestion Notification (ECN) is an alternative means to signal data senders about network congestion. A network device explicitly marks specific bit(s) in packet headers to convey congestion information to data senders. A data sender implementing ECN treats the marked packet as if it were a packet drop and reacts the same way as it would to a packet drop. Note that ECN minimizes performance penalties, since packets do not have to be retransmitted.

An AQM scheme SHOULD support ECN and SHOULD leverage ECN as an initial means to control queuing delay before resorting to packet drops. An AQM scheme SHOULD self-adapt and remain stable even with faulty and/or unresponsive ECN implementations en-route.

4.6. Packet sizes and congestion notification

An AQM scheme may be considering packet sizes while generating congestion signals. [I-D.ietf-tsvwg-byte-pkt-congest] discusses the motivations behind the same. For example, control packets such as DNS requests/responses, TCP SYNs/ACKs are small, and their loss can severely impact application performance. An AQM scheme may therefore be biased towards small packets by dropping them with smaller probability compared to larger packets. However, such an AQM scheme is unfair to data senders generating larger packets. Data senders, malicious or otherwise, are motivated to take advantage of the AQM scheme by transmitting smaller packets, and could result in unsafe deployments and unhealthy transport and/or application designs.

An AQM scheme SHOULD adhere to recommendations outlined in [I-D.ietf-tsvwg-byte-pkt-congest], and SHOULD NOT provide undue advantage to flows with smaller packets.

5. Comparing AQMs

This memo recognizes that the guidelines mentioned above may be used for comparing AQMs. This memo recommends that AQM schemes MUST be compared against both performance (Section 3) and deployment (Section 4) categories. In addition, this section details how best to achieve a fair comparison of AQM schemes by avoiding certain pitfalls.

5.1. Performance comparison

AQM schemes MUST be compared against all the generic scenarios discussed in Section 3.2. AQM schemes MAY be compared for specific network environments such as data center, home networks etc. For a particular network environment, AQM schemes MUST be compared against all the scenarios listed for that network environment Section 3.3. For each evaluation scenario, the schemes MUST be compared against the metrics discussed under that scenario. Moreover, if an AQM scheme's parameter(s) were externally tuned for optimization or other purposes, these values MUST be disclosed.

Note that AQM schemes belong to different varieties such as queue-length based scheme (ex: RED) or queue-delay based scheme (ex: CoDel, PIE). Also, AQM schemes expose different control knobs associated with different semantics. For example, while both PIE and CoDel are queue-delay based schemes and each expose a knob to control the queueing delay -- PIE's "queueing delay reference" vs. CoDel's "queueing delay target", the two schemes' knobs have different semantics resulting in different control points. Such differences in AQM schemes can be easily overlooked while making comparisons.

This document recommends the following procedures for a fair performance comparison of two AQM schemes:

1. Comparable control parameters and comparable input values: Carefully identify the set of parameters that control similar behavior between the two AQM schemes and ensure these parameters have comparable input values. For example, while comparing how well a queue-length based AQM X controls queueing delay vs. queue-delay based AQM Y, identify the two schemes' parameters that control queue delay and ensure that their input values are comparable. Similarly, to compare two AQMs on how well they accommodate bursts, identify burst-related control parameters and ensure they are configured with similar values.
2. Compare over a range of input configurations: There could be situations when the set of control parameters that affect a specific behavior have different semantics between the two AQM schemes. As mentioned above, PIE's knob to control queue delay has different semantics from CoDel's. In such situations, the schemes MUST be compared over a range of input configurations. For example, compare PIE vs. CoDel over the range of delay input configurations -- 5ms, 10ms, 15ms etc.

5.2. Deployment comparison

AQMs MUST be compared against the deployment criteria discussed in Section 4.

6. Acknowledgements

This work has been partially supported by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700).

7. Contributors

Many thanks to Gorry Fairhurst, Amadou B. Bagayoko, Chamil Kulatunga Michael Welzl, Fred Baker, Rong Pan and David Collier-Brown for detailed and wise feedback on this document.

8. IANA Considerations

This memo includes no request to IANA.

9. Security Considerations

All drafts are required to have a security considerations section.

10. References

10.1. Normative References

[I-D.ietf-aqm-recommendation]	Baker, F. and G. Fairhurst, "IETF Recommendations Regarding Active Queue Management", Internet-Draft draft-ietf-aqm-recommendation-01, January 2014.
[I-D.ietf-tsvwg-byte-pkt-congest]	Briscoe, B. and J. Manner, "Byte and Packet Congestion Notification", Internet-Draft draft-ietf-tsvwg-byte-pkt-congest-12, November 2013.
[RFC2119]	Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, 1997.
[RFC5136]	Chimento, P. and J. Ishac, "Defining Network Capacity", RFC 5136, 2008.

10.2. Informative References

[RFC5348]	Floyd, S., Handley, M., Padhye, J. and J. Widmer, "TCP Friendly Rate Control (TFRC): Protocol Specification", RFC 5348, September 2008.
[TCPEVAL2008]	Andrew, L., Marcondes, C., Floyd, S., Dunn, L., Guillier, R., Gang, W., Eggert, L., Ha, S. and I. Rhee, "Towards a common TCP evaluation suite", PFLDnet 6th, 2008.
[BB2012]	CACM Staff, , "BufferBloat: what's wrong with the internet?", Commun. ACM vol. 55, 2008.
[DOCSIS2013]	White, G. and D. Rice, "Active Queue Management Algorithms for DOCSIS 3.0", Technical repport - Cable Television Laboratories , 2013.

Appendix A. Additional Stuff

This becomes an Appendix.

Authors' Addresses