Internet DRAFT - draft-briscoe-conex-policing
draft-briscoe-conex-policing
ConEx B. Briscoe
Internet-Draft BT
Intended status: Informational February 14, 2014
Expires: August 18, 2014
Network Performance Isolation using Congestion Policing
draft-briscoe-conex-policing-01
Abstract
This document describes why policing using congestion information can
isolate users from network performance degradation due to each
other's usage, but without losing the multiplexing benefits of a LAN-
style network where anyone can use any amount of any resource.
Extensive numerical examples and diagrams are given. The document is
agnostic to how the congestion information reaches the policer. The
congestion exposure (ConEX) protocol is recommended, but other tunnel
feedback mechanisms have been proposed.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
Briscoe Expires August 18, 2014 [Page 1]
Internet-Draft Congestion Policing February 2014
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Example Bulk Congestion Policer . . . . . . . . . . . . . . . 4
3. Network Performance Isolation: Intuition . . . . . . . . . . 8
3.1. The Problem . . . . . . . . . . . . . . . . . . . . . . . 8
3.2. Approach . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3. Terminology . . . . . . . . . . . . . . . . . . . . . . . 11
3.4. Simple Boundary Model of Congestion Control . . . . . . . 11
3.5. Long-Running Flows . . . . . . . . . . . . . . . . . . . 12
3.6. On-Off Flows . . . . . . . . . . . . . . . . . . . . . . 14
3.6.1. Numerical Examples Without Policing . . . . . . . . . 16
3.6.2. Congestion Policing of On-Off Flows . . . . . . . . . 19
3.7. Weighted Congestion Controls . . . . . . . . . . . . . . 20
3.8. A Network of Links . . . . . . . . . . . . . . . . . . . 22
3.8.1. Numerical Example: Isolation from Focused Load . . . 23
3.8.2. Isolation in the Short-Term . . . . . . . . . . . . . 24
3.8.3. Encouraging Load Balancing . . . . . . . . . . . . . 24
3.9. Tenants of Different Sizes . . . . . . . . . . . . . . . 25
3.10. Links of Different Sizes . . . . . . . . . . . . . . . . 25
3.11. Diverse Congestion Control Algorithms . . . . . . . . . . 26
4. Parameter Setting . . . . . . . . . . . . . . . . . . . . . . 29
5. Security Considerations . . . . . . . . . . . . . . . . . . . 30
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30
7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 30
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 30
9. Informative References . . . . . . . . . . . . . . . . . . . 30
Appendix A. Summary of Changes between Drafts . . . . . . . . . 33
1. Introduction
This document is informative, not normative. It describes why
congestion policing isolates the network performance of different
parties who are using a network, and why it is more useful than other
forms of performance isolation such as peak and committed information
rate policing, weighted round-robin or weighted fair-queueing.
The main point is that congestion policing isolates performance even
when the load applied by everyone is highly variable, where variation
may be over time, or by spreading traffic across different paths (the
hose model). Unlike other isolation techniques, congestion policing
is an edge mechanism that works over a pool of links across whole
network, not just at one link. If the load from everyone happens to
be constant, and there is a single bottleneck, congestion policing
Briscoe Expires August 18, 2014 [Page 2]
Internet-Draft Congestion Policing February 2014
gives a nearly identical outcome to weighted round-robin or weighted
fair-queuing at that bottleneck. But otherwise, congestion policing
is a far more general solution.
The document complements others that describe various network
arrangements that could use congestion policing, such as in a
broadband access network [I-D.briscoe-conex-initial-deploy], in a
mobile access network [I-D.ietf-conex-mobile] or in a data centre
network [I-D.briscoe-conex-data-centre]. Nonetheless, all the
numerical examples use the data centre scenario.
The key to the solution is the use of congestion-bit-rate rather than
bit-rate as the policing metric. _How _this works is very simple and
quick to describe Section 2.
However, it is much more difficult to understand _why_ this approach
provides performance isolation. In particular, why it provides
performance isolation across a network of links, even though there is
apparently no isolation mechanism in each link. Section 3 builds up
an intuition for why the approach works, and why other approaches
fall down in different ways. The bullets below provide a summary of
that explanation, which builds from the simple case of long-running
flows through a single link up to a full meshed network with on-off
flows of different sizes and different behaviours:
o Starting with the simple case of long-running flows focused on a
single bottleneck link, tenants get weighted shares of the link,
much like weighted round robin, but with no mechanism in any of
the links. This is because losses (or ECN marks) are random, so
if one tenant sends twice as much bit-rate it will suffer twice as
many lost bits (or ECN-marked bits). So, at least for constant
long-running flows, regulating congestion-bits gives the same
outcome as regulating bits;
o In the more realistic case where flows are not all long-running
but a mix of short to very long, it is explained that bit-rate is
not a sufficient metric for isolating performance; how _often_ a
tenant is sending (or not sending) is the significant factor for
performance isolation, not whether bit-rate is shared equally
whenever a source happens to be sending;
o Although it might seem that data volume would be a good measure of
how often a tenant is sending, we then show why it is not. For
instance, a tenant can send a large volume of data but hardly
affect the performance of others -- by being more responsive to
congestion. Using congestion-volume (congestion-bit-rate over
time) in a policer encourages large data senders to be more
responsive (to yield), giving other tenants much higher
Briscoe Expires August 18, 2014 [Page 3]
Internet-Draft Congestion Policing February 2014
performance while hardly affecting their own performance.
Whereas, using straight volume as an allocation metric provides no
distinction between high volume sources that yield and high volume
sources that do not yield (the widespread behaviour today);
o We then show that a policer based on the congestion-bit-rate
metric works across a network of links treating it as a pool of
capacity, whereas other approaches treat each link independently,
which is why the proposed approach requires none of the
configuration complexity on switches that is involved in other
approaches.
o We also show that a congestion policer can be arranged to limit
bursts of congestion from sources that focus traffic onto a single
link, even where one source may consist of a large aggregate of
sources.
o We show that a congestion policer rewards traffic that shifts to
less congested paths (e.g. multipath TCP or virtual machine
motion).
o We show that congestion policing works on the pool of links,
irrespective of whether individual links have significantly
different capacities.
o We show that a congestion policer allows a wide variety of
responses to congestion (e.g. New Reno TCP [RFC5681], Cubic TCP,
Compound TCP, Data Centre TCP [DCTCP] and even unresponsive UDP
traffic), while still encouraging and enforcing a sufficient
response to congestion from all sources taken together, so that
the performance of each application is sufficiently isolated from
others.
2. Example Bulk Congestion Policer
In order to explain why a congestion policer works, we first describe
a particular design of congestion policer, called a bulk congestion
policer. The aim is to give a concrete example to motivate the
protocol changes necessary to implement such a policer. This is
merely an informative document, so there is no intention to
standardise this or any specific congestion policing algorithm. The
aim is for different implementers to be free to develop their own
improvements to this design.
A number of other documents describe deployment arrangements that
include a congestion policer, for instance in a broadband access
network [I-D.briscoe-conex-initial-deploy], in a mobile access
network [I-D.ietf-conex-mobile] and in a data centre network
Briscoe Expires August 18, 2014 [Page 4]
Internet-Draft Congestion Policing February 2014
[I-D.briscoe-conex-data-centre]. The congestion policer described in
the present document could be deployed in any of these arrangements.
However, to be concrete, the example link capacities, topology and
terminology used in the present document assume a multi-tenant data
centre scenario [I-D.briscoe-conex-data-centre]. It is believed that
similar examples could be drawn up for the other scenarios, just
using different numbers for link capacities, etc and replacing some
terminology, e.g. substituting 'end-customer' for 'tenant'
throughout.
Figure 1 illustrates a bulk congestion policer. It is very similar
to a regular token bucket for policing bit-rate except the tokens do
not represent bits, rather they represent 'congestion-bits'. Using
Congestion Exposure (ConEx) [I-D.ietf-conex-abstract-mech] is
probably the easiest way to understand what congestion-bits are.
However, there are other valid ways to signal congestion information
to a congestion policer without ConEx (e.g. tunnelled feedback
described in [I-D.briscoe-conex-data-centre].
Using ConEx as an example, a ConEx-based congestion token bucket
ignores any packets unless they are ConEx-marked. The ConEx protocol
includes audit capabilities that force the sender to reveal
information it receives about losses anywhere on the downstream path
(e.g. at the points marked X in Figure 1). The sender signals this
information with a ConEx-marking in the IP header of the packets it
sends into the network, using an arrangement called re-inserted
feedback or re-feedback. So, every time the sender detects a loss of
a 1500B packet, it has to apply a ConEx-marking to a subsequent 1500B
packet (or it has to ConEx-mark at least as many bytes made up of
smaller packets). In Figure 1 ConEx-marked packets are shown tagged
with a '*'.
A bulk congestion policer is deployed at every point where traffic
enters an operator's network (similar to the Diffserv architecture).
At any one point of entry, the operator assigns a congestion token
bucket to each tenant that is sending traffic into the network. The
operator assigns a contracted token-fill-rate w1 to tenant 1, w2 to
tenant 2, and in general wi to tenant with index i, where i = 1,2,...
This contracted token-fill-rate, wi, can be thought of like a weight,
where a higher weight allows a tenant to contribute more traffic
through a downstream congested link or to contribute traffic into
more congested links, as we shall see.
Tenants wanting to send more traffic can be assigned a higher weight
(probably paying more for it). Typically, tenants might sign-up to a
traffic contract in two parts:
Briscoe Expires August 18, 2014 [Page 5]
Internet-Draft Congestion Policing February 2014
1. a peak rate, which represents the capacity of their access to the
network
2. a congestion token-fill rate, which limits how much traffic can
be sent that limits what others are sending.
In Figure 1, as each packet arrives, the classifier assigns it to the
congestion token bucket of the relevant tenant. The token bucket
ignores all non-ConEx-marked packets - they do not drain any tokens
from the bucket. Otherwise, if a packet is ConEx-marked and ,say,
1500B in size it will drain 1500B of tokens from the congestion token
bucket. If the tenant is contributing to more congestion than its
contracted congestion-fill-rate, the bucket level will decrease, and
if it persists, the bucket level will become low enough to cause the
policer to start discarding a proportion of all that tenant's traffic
(whether ConEx-marked or not).
______________________
| | | | Legend |
|w1 |w2 |wi | |
| | | | [_] [_]packet stream |
V V V | |
congestion . . . | [*] marked packet |
token bucket| . | | . | __|___| | ___ |
__|___| | . | | |:::| | \ / policer |
| |:::| __|___| | |:::| | /_\ |
| +---+ | +---+ | +---+ | |
bucket depth | : | : | : | /\ marking meter |
controls the | . | : | . | \/ |
policer _V_ . | : | . |______________________|
____\ /__/\___________________________ downstream
/[*] /_\ \/ [_] | : [_] | : [_] \ /->network
class-/ | . | . \ / /--->
ifier/ _V_ . | . \ / /
__,--.__________________\ /__/\___________________\______/____/ loss
`--' [_] [*] [_] /_\ \/ [_] | . [*] / \ \-X--->
\ | . / \-->
\ _V_ : / \ loss
\__________________________\ /__/\_____/ \-X--->
[_] [*] [_] [*] [_] /_\ \/ [_] \
\--->
Figure 1: Bulk Congestion Policer Schematic
Note that this is called a bulk congestion policer because it polices
all the flows of a tenant even if they spread out in a hose pattern
across numerous downstream networks and even if only a few of these
flows are primarily to blame for contributing heavily to downstream
Briscoe Expires August 18, 2014 [Page 6]
Internet-Draft Congestion Policing February 2014
congestion, while others find plenty of capacity downstream. This
approach is motivated by simplicity rather than precision (if
precision were required other policer designs could be used).
Simplicity might be more important than precision if capacity were
generally well-provisioned and the policer was intended to intervene
only rarely to limit unusually extreme behaviour. Also, a network
might take such a simple but somewhat heavy-handed approach in order
to motivate tenants to implement more complex flow-specific controls
themselves [CongPol].
Various more sophisticated congestion policer designs have been
evaluated [CPolTrilogyExp]. In these experiments, it was found that
it is better if the policer gradually increases discards as the
bucket becomes empty. Also isolation between tenants is better if
each tenant is policed based on the combination of two buckets, not
one (Figure 2):
1. A deep bucket (that would take minutes or even hours to fill at
the contracted fill-rate) that constrains the tenant's long-term
average rate of congestion (wi)
2. a very shallow bucket (e.g. only two or three MTU) that is filled
considerably faster than the deep bucket (c * wi), where c = ~10,
which prevents a tenant storing up a large backlog of tokens then
causing congestion in one large burst.
In this arrangement each marked packet drains tokens from both
buckets, and the probability of policer discard is taken as the worse
of the two buckets.
Briscoe Expires August 18, 2014 [Page 7]
Internet-Draft Congestion Policing February 2014
| |
Legend: |c*wi |wi
See previous figure V V
. .
. | . | deep bucket
_ _ _ _ _ _ _ _ _ _ _ _ |___|
| . |:::|
|_ _ _ _ _ _ _ |___| |:::|
| shallow +---+ +---+
worse of the| bucket
two buckets| \____ ____/
triggers| \ / both buckets
policing V : drained by
___ . marked packets
___________\ /___________________/ \__________________
[_] [_] /_\ [_] [*] [_] \ / [_] [_] [_]
Figure 2: Dual Congestion Token Bucket (in place of each single
bucket in the previous figure)
This design of congestion policer will be sufficient to understand
the rest of the document. However, it should be remembered that this
is only an example, not an ideal design. Also it should be
remembered that other mechanisms such as tunnel feedback can be used
instead of ConEx.
3. Network Performance Isolation: Intuition
3.1. The Problem
Network performance isolation traditionally meant that each user
could be sure of a minimum guaranteed bit-rate. Such assurances are
useful if traffic from each tenant follows relatively predictable
paths and is fairly constant. If traffic demand is more dynamic and
unpredictable (both over time and across paths), minimum bit-rate
assurances can still be given, but they have to be very small
relative to the available capacity, because a large number of users
might all want to simulataneously share any one link, even though
they rarely all use it at the same time.
This either means the shared capacity has to be greatly overprovided
so that the assured level is large enough, or the assured level has
to be small. The former is unnecessarily expensive; the latter
doesn't really give a sufficiently useful assurance.
Round robin or fair queuing are other forms of isolation that
guarantee that each user will get 1/N of the capacity of each link,
Briscoe Expires August 18, 2014 [Page 8]
Internet-Draft Congestion Policing February 2014
where N is the number of active users at each link. This is fine if
the number of active users (N) sharing a link is fairly predictable.
However, if large numbers of tenants do not typically share any one
link but at any time they all could (as in a data centre), a 1/N
assurance is fairly worthless. Again, given N is typically small but
could be very large, either the shared capacity has to be expensively
overprovided, or the assured bit-rate has to be worthlessly small.
The argument is no different for the weighted forms of these
algorithms: WRR & WFQ).
Both these traditional forms of isolation try to give one tenant
assured instantaneous bit-rate by constraining the instantaneous bit-
rate of everyone else. This approach is flawed except in the special
case when the load from every tenant on every link is continuous and
fairly constant. The reality is usually very different: sources are
on-off and the route taken varies, so that on any one link a source
is more often off than on.
In these more realistic (non-constant) scenarios, the capacity
available for any one tenant depends much more on _how often_
everyone else uses a link, not just _how much_ bit-rate everyone
else would be entitled to if they did use it.
For instance, if 100 tenants are using a 1Gb/s link for 1% of the
time, there is a good chance each will get the full 1Gb/s link
capacity. But if just six of those tenants suddenly start using the
link 50% of the time, whenever the other 94 tenants need the link,
they will typically find 3 of these heavier tenants using it already.
If a 1/N approach like round-robin were used, then the light tenants
would suddently get 1/4 * 1Gb/s = 250Mb/s on average. Round-robin
cannot claim to isolate tenants from each other if they usually get
1Gb/s but sometimes they get 250Mb/s (and only 10Mb/s guaranteed in
the worst case when all 100 tenants are active).
In contrast, congestion policing is the key to network performance
isolation because it focuses policing only on those tenants that go
fast over congested path(s) excessively and persistently over time.
This keeps congestion below a design threshold everywhere so that
everyone else can go fast. In this way, congestion policing takes
account of highly variable loads (varying in time and varying across
routes). And, if everyone's load happens to be constant, congestion
policing converges on the same outcome as the traditional forms of
isolation.
The other flaw in the traditional approaches to isolation, like WRR &
WFQ, is that they actually prevent long-running flows from yielding
to brief bursts from lighter tenants. A long-running flow can yield
to brief flows and still complete nearly as soon as it would have
Briscoe Expires August 18, 2014 [Page 9]
Internet-Draft Congestion Policing February 2014
otherwise (the brief flows complete sooner, freeing up the capacity
for the longer flow sooner--see Section 3.7). However, WRR & WFQ
prevent flows from even seeing the congestion signals that would
allow them to co-ordinate between themselves, because they isolate
each tenant completely into separate queues.
In summary, superficially, traditional approaches with separate
queues sound good for isolation, but:
1. not when everyone's load is variable, so each tenant has no
assurance about how many other queues there will be;
2. and not when each tenant can no longer even see the congestion
signals from other tenants, so no-one's control algorithms can
determine whether they would benefit most by pushing harder or
yielding.
3.2. Approach
It has not been easy to find a way to give the intuition on why
congesiton policing isolates performance, particularly across a
networks of links not just on a single link. The approach used in
this section, is to describe the system as if everyone is using the
congestion response they would be forced to use if congestion
policing had to intervene. We therefore call this the boundary model
of congestion control. It is a very simple congestion response, so
it is much easier to understand than if we introduced all the square
root terms and other complexity of New Reno TCP's response. And it
means we don't have to try to decribe a mix of responses.
The purpose of congestion policing is not to intervene in everyone's
rate control all the time. Rather it is to encourage each tenant to
_avoid_ being policed -- to keep the aggregate of all their flows'
rates within an overall envelope that is sufficiently responsive to
congestion. Nonetheless, congestion policing can and will enforce a
congestion response if a particular tenant sends traffic that is
completely unresponsive to congestion. This upper bound enforced by
everyone else's congestion policers is what ensures that each
tenant's minimum performance is isolated from the combined effect of
everyone else.
We cannot emphasise enough that the intention is not to make
individual flows conform to this boundary response to congestion.
Indeed the intention is to allow a diverse evolving mix of congestion
responses, but constrained in total within a simple overall envelope.
After describing and further justifying using the simple boundary
model of congestion control, we start by considering long-running
Briscoe Expires August 18, 2014 [Page 10]
Internet-Draft Congestion Policing February 2014
flows sharing one link. Then we will consider on-off traffic, before
widening the scope from one link to a network of links and to links
of different sizes. Then we will depart from the initial simplified
model of congestion control and consider diverse congestion control
algorithms, including completely unresponsive end-systems.
Formal analysis to back-up the intuition provided by this section
will be made available in a more extensive companion technical report
[conex-dc_tr].
3.3. Terminology
The term congestion probability will be used as a generisation of
either loss probability or ECN marking probability.
It is necessary to carefully distinguish:
o congestion bit-rate (or just congestion rate), which is an
absolute measure of the rate of congested bits
o congestion probability, which is a relative measure of the
proportion of congested bits to all bits
Sometimes people loosely talk of loss rate when they mean loss
probability (measured in %). In this document, we will keep to
strict terminology, so loss rate would be measured in lost packets
per second, or lost bits per second.
3.4. Simple Boundary Model of Congestion Control
The boundary model of congestion control ensures a flow's bit-rate is
inversely proportional to the congestion level that it detects. For
instance, if congestion probability doubles, the flow's bit-rate
halves. This is called a scalable congestion control because it
maintains the same rate of congestion signals (marked or dropped
packets) no matter how fast it goes. Examples from the research
community are Relentless TCP [Relentless] and Scalable TCP [STCP].
In production systems, TCP algorithms based on New Reno [RFC5681]
have been widely replaced by alternatives closer to this scalable
ideal (e.g. Cubic TCP in Linux and Compound TCP in Windows), because
at high rates New Reno generated congestion signals too infrequently
to track available capacity fast enough [RFC3649]. More recent TCP
updates (e.g. Data Centre TCP [DCTCP] in Windows 8 and available in
Linux) are becoming closer still to the scalable ideal.
For instance, consider a scenario where a flow with scalable
congestion control is alone in a 1Gb/s link, then another similar
Briscoe Expires August 18, 2014 [Page 11]
Internet-Draft Congestion Policing February 2014
flow from another tenant joins it. Both will push up the congestion
probability, which will push down their rates until they together fit
into the link. Because the flow's rate has to halve to accomodate
the new flow, congestion probability will double (lets say from
0.002% to 0.004%), by our initial assumption of a two similar
scalable congestion controls. When it is alone on the link, the
congestion-bit-rate of the flow is 20kb/s (=1Gb/s * 0.002%), and when
it shares the link it is still 20kb/s (= 500Mb/s * 0.004%).
In summary, a congestion control can be considered scalable if the
bit-rate of packets carrying congestion signals (the congestion-bit-
rate) always stays the same no matter how much capacity it finds
available. This ensures there will always be enough signals in a
round trip time to keep the dynamics under control.
Reminder: Making individual flows conform to this boundary (scalable)
response to congestion is a non-goal. Although we start this
explanation with this specific simple end-system congestion response,
this is just to aid intuition.
3.5. Long-Running Flows
Table 1 shows various scenarios where each of five tenants has
contracted for 40kb/s of congestion-bit-rate in order to share a 1Gb/
s link. In order to help intuition, we start with the (unlikely)
scenario where all their flows are long-running. A number of long-
running flows will try to use all the link capacity, so for
simplicity we take utilisation as a round figure of 100%.
In the case we have just described (scenario A) neither tenant's
policer is intervening at all, because both their congestion
allowances are 40kb/s and each sends only one flow that contributes
20kb/s of congestion -- half the allowance.
Briscoe Expires August 18, 2014 [Page 12]
Internet-Draft Congestion Policing February 2014
+--------+-------------+----------+----------+----------+-----------+
| Tenant | contracted | scenario | scenario | scenario | scenario |
| | congestion- | A | B | C | D |
| | bit-rate | # : Mb/s | # : Mb/s | # : Mb/s | # : Mb/s |
| | kb/s | | | | |
+--------+-------------+----------+----------+----------+-----------+
| | | | | | |
| (a) | 40 | 1 : 500 | 5 : 250 | 5 : 200 | 5 : 250 |
| (b) | 40 | 1 : 500 | 3 : 250 | 3 : 200 | 2 : 250 |
| (c) | 40 | - : --- | 3 : 250 | 3 : 200 | 2 : 250 |
| (d) | 40 | - : --- | 2 : 250 | 2 : 200 | 1 : 125 |
| (e) | 40 | - : --- | - : --- | 2 : 200 | 1 : 125 |
| | Congestion | 0.004% | 0.016% | 0.02% | 0.016% |
| | probability | | | | |
+--------+-------------+----------+----------+----------+-----------+
Table 1: Bit-rates that a congestion policer allocates to five
tenants sharing a 1Gb/s link with various numbers (#) of long-running
flows all using 'scalable congestion control' of weight 20kb/s
Scenario B shows a case where four of the tenants all send 2 or more
long-running flows. Recall, from the assumption of scalable
controls, that each flow always contributes 20kb/s no matter how fast
it goes. Therefore the policers of tenants (a-c) limit them to two
flows-worth of congestion (2 x 20kb/s = 40kb/s). Tenant (d) is only
asking for 2 flows, so it gets them without being policed, and all
four get the same quarter share of the link.
Scenario C is similar, except the fifth tenant (e) joins in, so they
all get equal 1/5 shares of the link.
In Scenario D, only tenant (a) sends more than two flows, so (a)'s
policer limits it to two flows-worth of congestion, and everyone else
gets the number of flows-worth that they ask for. This means that
tenants (d) & (e) get less than everyone else, which is fine because
they asked for less than they would have been allowed. (Similarly,
in Scenarios A & B, some of the tenants are inactive, so they get
zero, which is also less than they could have had if they had
wanted.)
With lots of long-running flows, as in scenarios B & C, congestion
policing seems to emulate per-tenant round robin scheduling,
equalising the bit-rate of each tenant, no matter how many flows they
run. By configuring different contracted allowances for each tenant,
it can easily be seen that congestion policing could emulate weighted
round robin (WRR), with the relative sizes of the allowances acting
as the weights.
Briscoe Expires August 18, 2014 [Page 13]
Internet-Draft Congestion Policing February 2014
Scenario D departs from round-robin. This is deliberate, the idea
being that tenants are free to take less than their share in the
short term, which allows them to take more at other times, as we
shall see in Section 3.7. In Scenario D, policing focuses only on
the tenant (a) that is continually exceeding its contract. This
policer focuses discard solely on tenant a's traffic so that it
cannot cause any more congestion at the shared link (shown as 0.016%
in the last row).
To summarise so far, ingress congestion policers control congestion-
bit-rate in order to indirectly assure a minimum bit-rate per tenant.
With lots of long-running flows, the outcome is somewhat similar to
WRR, but without the need for any mechanism in each queue.
However, aiming to behave like WRR is only useful when all flows are
infinitely long. When load is variable because transfers are of
finite size, congestion policing properly isolates one tenant's
performance from the behaviour of others, unlike WRR would, as will
now be explained.
3.6. On-Off Flows
Figure 3 compares two example scenarios where tenant 'b' regularly
sends small files in the top chart and the same size files but more
often in the bottom chart (a higher 'on-off ratio'). This is the
typical behaviour of a Web server as more clients request more files
at peak time. Meanwhile, in this example, tenant c's behaviour
doesn't change between the two scenarios -- it sends a couple of
large files, each starting at the same time in both cases.
The capacity of the link that 'b' and 'c' share is shown as the full
height of the plot. The files sent by 'b' are shown as little
rectangles. 'b' can go at the full bit-rate of the link when 'c' is
not sending, which is represented by the tall thin rectangles part-
way through the trace labelled 'b'. We assume for simplicity that
'b' and 'c' divide up the bit-rate equally. So, when both 'b' and
'c' are sending, the 'b' rectangles are half the height (bit-rate)
and twice the width (duration) relative to when 'b' sends alone. The
area of a file to be transferred stays the same, whether tall and
thin or short and fat, because the area represents the size of the
file (bit-rate x duration = file size). The files from 'c' look like
inverted castellations, because 'c' uses half the link rate while
each file from 'b' completes, then 'c' can fill the link until 'b'
starts the next file. The cross-hatched areas represent idle times
when no-one is sending.
For this simple scenario we ignore start-up dynamics and just focus
on the rate and duration of flows that are long enough to stabilise,
Briscoe Expires August 18, 2014 [Page 14]
Internet-Draft Congestion Policing February 2014
which is why they can be represented as simple rectangles. We will
introduce the effect of flow startups later.
In the bottom case, where 'b' sends more often, the gaps between b's
transfers are smaller, so 'c' has less opportunity to use the whole
line rate. This squeezes out the time it takes for 'c' to complete
its file transfers (recall a file will always have the same area
which represents its size). Although 'c' finishes later, it still
starts the next flow at the same time. In turn, this means 'c' is
sending during a greater proprotion of b's transfers, which extends
b's average completion time too.
^ bit-rate
|
|---------------------------------------,--.---------,--.----,-------
| | |\/\/\/\/\| |/\/\|
| c | b|/\/\/\/\/| b|\/\/| c
|------. ,-----. ,-----. | |\/\/\/\/\| |/\/\| ,--
| b | | b | | b | | |/\/\/\/\/| |\/\/| | b
| | | | | | | |\/\/\/\/\| |/\/\| |
+------'------'-----'------'-----'------'--'---------'--'----'----'-->
time
^ bit-rate
|
|---------------------------------------------------.--,--.--,-------
| |/\| |\/|
| c |\/| b|/\| c
|------. ,-----. ,-----. ,-----. ,-----. ,-----./\| |\/| ,----
| b | | b | | b | | b | | b | | b |\/| |/\| | b
| | | | | | | | | | | |/\| |\/| |
+------'--'-----'--'-----'--'-----'--'-----'--'-----'--'--'--'--'---->
time
Figure 3: In the lower case, the on-off ratio of 'b' has increased,
which extends all the completion times of 'c' and 'b'
Round-robin would do little if anything to isolate 'c' from the
effect of 'b' sending files more often. Round-robin is designed to
force 'b' and 'c' to share the capacity equally when they are both
active. But in both scenarios they already share capacity equally
when they are both active. The difference is in how often they are
active. Round-robin and other traditional fair queuing techniques
don't have any memory to sense that 'b' has been active more of the
time.
In contrast, a congestion policer can tell when one tenant is sending
files more frequently, by measuring the rate at which the tenant is
contributing to congestion. Our aim is to show that policers will be
Briscoe Expires August 18, 2014 [Page 15]
Internet-Draft Congestion Policing February 2014
able to isolate performance properly by using the right metric
(congestion bit-rate), rather than using the wrong metric (bit-rate),
which doesn't sense whether the load over time is large or small.
3.6.1. Numerical Examples Without Policing
The usefulness of the congestion bit-rate metric will now be
illustrated with the numerical examples in Table 2. The scenarios
illustrate what the congestion bit-rate would be without any policing
or scheduling action in the network. Then this metric can be
monitored and limited by a policer, to prevent one tenant from
harming the performance of others.
The 2nd & 3rd columns (file-size and inter-arrival time) fully
represent the behaviour of each tenant in each scenario. All the
other columns merely characterise the outcome in various ways. The
inter-arrival time (T) is the average time between starting one file
and the next. For instance, in scenario E tenant 'b' sends a 16Mb
file every 200ms on average. The formula in the heading of some
columns shows how the column was derived from other columns.
Scenario E is contrived so that the three tenants all offer the same
load to the network, even though they send files of very different
size (S). The files sent by tenant 'a' are 100 times smaller than
those of tenant 'b', but 'a' sends them 100 times more often. In
turn, b's files are 100 times smaller than c's, but 'b' in turn sends
them 100 times more often. Graphically, the scenario would look
similar to Figure 3, except with three sizes of file, not just two.
Scenarios E-G are designed to roughly represent various distributions
of file sizes found in data centres, but still to be simple enough to
facilitate intuition, even though each tenant would not normally send
just one size file.
The average completion time (t) and the maximum were calculated from
a fairly simple analytical model (documented in a campanion technical
report [conex-dc_tr]). Using one data point as an example, it can be
seen that a 1600Mb (200MB) file from tenant 'c' completes in 1905ms
(about 1.9s). The files that are 100 times smaller complete 100
times more quickly on average. In fact, in this scenario with equal
loads, each tenant perceives that their files are being transferred
at the same rate of 840Mb/s on average (file-size divided by
completion time, as shown in the 'apparent bit-rate' column). Thus
on average all three tenants perceive they are getting 84% of the 1Gb
/s link on average (due to the benefit of multiplexing and
utilisation being low at 240Mb/s / 1Gb/s = 24% in this case).
The completion times of the smaller files vary significantly,
depending on whether a larger file transfer is proceeding at the same
Briscoe Expires August 18, 2014 [Page 16]
Internet-Draft Congestion Policing February 2014
time. We have already seen this effect in Figure 3, where, when
tenant b's files share with 'c', they take twice as long to complete
as when they don't. This is why the maximum completion time is
greater than the average for the small files, whereas there is
imperceptible variance for the largest files.
The final column shows how congestion bit-rate will be a useful
metric to enforce performance isolation (as we said, the table
illustrates the situation before any enforcement mechanism is added).
In the case of equal loads (scenario E), average congestion bit-rates
are all equal. In scenarios F and G average congestion bit-rates are
higher, because all tenants are placing much more load on the network
over time, even though each still sends at equal rate to others when
they are active together. Figure 3 illustrated a similar effect in
the difference between the top and bottom scenarios.
The maximum instantaneous congestion bit-rate is nearly always 20kb/
s. That is because, by definition, all the tenants are using scalable
congestion controls with a constant congestion rate of 20kb/s, and
each tenant's flows rarely overlap because the per-tenant load in
these scenarios is fairly low. As we saw in Section 3.4, the
congestion rate of a particular scalable congestion control is always
the same, no matter how many other flows it competes with.
Once it is understood that the congestion bit-rate of one scalable
flow is always 'w' and doesn't change whenever a flow is active, it
becomes clear what the congestion bit-rate will be when averaged over
time; it will simply be 'w' multiplied by the proportion of time that
the tenant's file transfers are active. That is, w*t/T. For
instance, in scenario E, on average tenant b's flows start 200ms
apart, but they complete in 19ms. So they are active for 19/200 =
10% of the time (rounded). A tenant that causes a congestion bit-
rate of 20kb/s for 10% of the time will have an average congestion-
bit-rate of 2kb/s, as shown.
To summarise so far, no matter how many more files other tenants
transfer at the same time, each scalable flow still contributes to
congestion at the same rate, but it contributes for more of the time,
because it squeezes out into the gap before the next flow from that
tenant starts.
Briscoe Expires August 18, 2014 [Page 17]
Internet-Draft Congestion Policing February 2014
+-----+------+--------+------+-------------+-----------+------------+
| Ten | File | Ave. | Ave. | Completion | Apparent | Congest- |
| ant | size | inter- | load | time | bit-rate | ion bit- |
| | S | arr- | S/T | ave : max | ave : min | rate |
| | Mb | ival | Mb/s | t | S/t | ave : max |
| | | T | | ms | Mb/s | w*t/T |
| | | ms | | | | kb/s |
+-----+------+--------+------+-------------+-----------+------------+
| | | | | Scenario E | | |
| a | 0.16 | 2 | 80 | 0.19 : 0.48 | 840 : 333 | 2 : 20 |
| b | 16 | 200 | 80 | 19 : 35 | 840 : 460 | 2 : 20 |
| c | 1600 | 20000 | 80 | 1905 : 1905 | 840 : 840 | 2 : 20 |
| | | | ____ | | | |
| | | | 240 | | | |
| | | | | Scenario F | | |
| a | 0.16 | 0.67 | 240 | 0.31 : 0.48 | 516 : 333 | 9 : 20 |
| b | 16 | 50 | 320 | 29 : 42 | 557 : 380 | 11 : 20 |
| c | 1600 | 10000 | 160 | 3636 : 3636 | 440 : 440 | 7 : 20 |
| | | | ____ | | | |
| | | | 720 | | | |
| | | | | Scenario G | | |
| a | 0.16 | 0.67 | 240 | 0.33 : 0.64 | 481 : 250 | 10 : 20 |
| b | 16 | 40 | 400 | 32 : 46 | 505 : 345 | 16 : 40 |
| c | 1600 | 10000 | 160 | 4543 : 4543 | 352 : 352 | 9 : 20 |
| | | | ____ | | | |
| | | | 800 | | | |
+-----+------+--------+------+-------------+-----------+------------+
Single link of capacity 1Gb/s. Each tenant uses a scalable congestion
control which contributes a congestion-bit-rate for each flow of w =
20kb/s.
Table 2: How the effect on others of various file-transfer behaviours
can be measured by the resulting congestion-bit-rate
In scenario F, clients have increased the rate they request files
from tenants a, b and c respectively by 3x, 4x and 2x relative to
scenario E. The tenants send the same size files but 3x, 4x and 2x
more often. For instance tenant 'b' is sending 16Mb files four times
as often as before, and they now take longer as well -- nearly 29ms
rather than 19ms -- because the other tenants are active more often
too, so completion gets squeezed to later. Consequently, tenant 'b'
is now sending 57% of the time, so its congestion-bit-rate is 20kb/s
* 57% = 11kb/s. This is nearly 6x higher than in scenario E,
reflecting both b's own increase by 4x and that this increase
coincides with everyone else increasing their load.
Briscoe Expires August 18, 2014 [Page 18]
Internet-Draft Congestion Policing February 2014
In scenario G, tenant 'b' increases even more, to 5x the load it
offered in scenario E. This results in average utilisation of 800Mb/s
/ 1Gb/s = 80%, compared to 72% in scenario F and only 24% in scenario
E. 'b' sends the same files but 5x more often, so its load rises 5x.
Completion times rise for everyone due to the overall rise in load,
but the congestion rates of 'a' and 'c' don't rise anything like as
much as that of 'b', because they still leave large gaps between
files. For instance, tenant 'c' completes each large file transfer
in 4.5s (compared to 1.9s in scenario E), but it still only sends
files every 10s. So 'c' only sends 45% of the time, which is
reflected in its congestion bit-rate of 20kb/s * 45% = 9kb/s.
In contrast, on average tenant 'b' can only complete each medium-
sized file transfer in 32ms (compared to 19ms in scenario E), but on
average it starts sending another file after 40ms. So 'b' sends 79%
of the time, which is reflected in its congestion bit-rate of 20kb/s
* 79% = 16kb/s (rounded).
However, during the 45% of the time that 'c' sends a large file, b's
completion time is higher than average (as shown in Figure 3). In
fact, as shown in the maximum completion time column, 'b' completes
in 46ms, but it starts sending a new file after 40ms, which is before
the previous one has completed. Therefore, during each of c's large
files, 'b' sends 46/40 = 116% of the time on average.
This actually means 'b' is overlapping two files for 16% of the time
on average and sending one file for the remaining 84%. Whenever two
file transfers overlap, 'b' will be causing 2 x 20kb/s = 40kb/s of
congestion, which explains why tenant b in scenario G is the only
case with a maximum congestion rate of 40kb/s rather than 20kb/s as
in every other case. Over the duration of c's large files, 'b' would
therefore cause congestion at an average rate of 20kb/s * 84% + 40kb/
s * 16% = 23kb/s (or more simply 20kb/s * 116% = 23kb/s). Of course,
when 'c' is not sending a large file, 'b' will contribute less to
congestion, which is why its average congestion rate is 16kb/s
overall, as discussed earlier.
3.6.2. Congestion Policing of On-Off Flows
Still referring to the numerical examples in Table 2, we will now
discuss the effect of limiting each tenant with a congestion policer.
The network operator might have deployed congestion policers to cap
each tenant's average congestion rate to 16kb/s. None of the tenants
are exceeding this limit in any of the scenarios, but tenant 'b' is
just shy of it in scenario G. Therefore all the tenants would be free
to behave in all sorts of ways, such as those of scenarios E-G.
Briscoe Expires August 18, 2014 [Page 19]
Internet-Draft Congestion Policing February 2014
However, they would be prevented from degrading the performance of
the other tenants beyond the point reached by tenant 'b' in scenario
G. If tenant 'b' added more load, the policer would prevent the extra
load entering the network by focusing drop solely on tenant 'b',
preventing the other tenants from experiencing any more congestion
due to tenant 'b'. Then tenants 'a' and 'c' would be assured the
(average) apparent bit-rates shown, whatever the behaviour of 'b'.
If 'a' added more load, 'c' would not suffer. Instead 'b' would go
over limit and its rate would be trimmed during congestion peaks,
sacrificing some of its lead to 'a'. Similarly, if 'c' added more
load, 'b' would be made to sacrifice some of its performance, so that
'a' would not suffer. Further, if more tenants arrived to share the
same link, the policer would force 'b' to sacrifice performance in
favour of the additional tenants.
There is nothing special about a policer limit of 16kb/s. The example
when discussing infinite flows used a limit of 40kb/s per tenant.
And some tenants can be given higher limits than others (e.g. at an
additional charge). If the operator gives out congestion limits that
together add up to a higher amount but it doesn't increase the link
capacity, it merely allows the tenants to apply more load (e.g. more
files of the same size in the same time), but each with lower bit-
rate.
3.7. Weighted Congestion Controls
At high speed, congestion controls such as Cubic TCP, Data Centre
TCP, Compound TCP etc all contribute to congestion at widely
differing rates, which is called their 'aggressiveness' or 'weight'.
So far, we have made the simplifying assumption of a scalable
congestion control algorithm that contributes to congestion at a
constant rate of w = 20kb/s. We now assume tenant 'c' uses a similar
congestion control to before, but with different parameters in the
algorithm so that its weight is still constant, but much lower, at w
= 2.2kb/s.
Tenant 'b' is sending smaler files than 'c', so it still uses w =
20kb/s. Then, when the two compete for the 1Gb/s link, they will
share it in proportion to their weights, 20:2.2 (or 90%:10%). That
is, when competing, 'b' and 'c' will respectively get (20/22.2)*1Gb/s
= 900Mb/s and (2.2/22.2)*1Gb/s = 100Mb/s of the 1Gb/s link. Figure 4
shows the situation before (upper) and after (lower) this change.
When the two compete, 'b' transfers each file 9/5 faster than before
(900Mb/s rather than 500Mb/s), so it completes them in 5/9 of the
time. 'b' still contributes congestion at the same rate of 20kb/s,
but for 5/9 less time than before. Therefore, relative to before,
Briscoe Expires August 18, 2014 [Page 20]
Internet-Draft Congestion Policing February 2014
'b' uses up its allowance only just over half as quickly, and it
completes each transfer nearly twice as fast.
Even though 'b' completes each small file much faster, 'c' should
complete its file transfer hardly any later than before. Although
tenant 'b' goes faster, as each 'b' file finishes, it gets out of the
way sooner. Irrespective of its lower weight, 'c' can still use the
full link when 'b' is not present, because weights only determine
shares between active flows. Because 'c' has the whole link sooner
it can catch up to the same point it would have reached in its
download if 'b' had been slower but longer. Tenant 'c' will probably
lose some completion time because it has to accelerate and decelerate
more. But, whenever it is sending a file, 'c' gains (20kb/s -2kb/s)
= 18kb of allowance every second, which it can use for other
transfers.
^ bit-rate
|
|---------------------------------------------------.--,--.--,-------
| |/\| |\/|
| c |\/| b|/\| c
|------. ,-----. ,-----. ,-----. ,-----. ,-----./\| |\/| ,----
| b | | b | | b | | b | | b | | b |\/| |/\| | b
| | | | | | | | | | | |/\| |\/| |
+------'--'-----'--'-----'--'-----'--'-----'--'-----'--'--'--'--'---->
time
^ bit-rate
|
|---------------------------------------------------.--,--.--,-------
|---. ,---. ,---. ,---. ,---. ,---. |/\| |\/| ,---.
| | | | | | c | | | | | | |\/| b|/\| c| |
| | | | | | | | | | | | |/\| |\/| | |
| b | | b | | b | | b | | b | | b | |\/| |/\| | b |
| | | | | | | | | | | | |/\| |\/| | |
+---'-----'---'----'---'----'---'----'---'----'---'-'--'--'--'--'---'>
time
Figure 4: Weighted congestion controls with equal weights (upper) and
unequal (lower)
It seems too good to be true that both tenants gain so much and lose
so little by 'c' reducing its aggressiveness. Effectively 'c' allows
everyone else's shorter flows to proceed as if the long flows were
hardly there, and 'c' hardly loses out at all itself. The gains are
unlikely to be as perfect as this simple model predicts, but we
believe they will be nearly as substantial.
Briscoe Expires August 18, 2014 [Page 21]
Internet-Draft Congestion Policing February 2014
It might seem that everyone can keep gaining by everyone agreeing to
reduce their weights, ad infinitum. However, the lower the weight,
the less signals the congestion control gets, so it starts to lose
its control during dynamics. Nonetheless, congestion policing should
encourage congestion control designs to keep reducing their weights,
but they will have to stop when they reach the minimum necessary
congestion in order to maintain sufficient control signals.
3.8. A Network of Links
So far we have only considered a single link. Congestion policing at
the network edge is designed to work across a network of links,
treating them all as a pool of resources, as we shall now explain.
We will use the dual-homed data centre topology shown in Figure 5
(stretching the bounds of ASCII art) as a simple example of a pool of
resources.
In this case where there are 48 servers (H1, H2, ... Hn where n=48)
on the left, with on average 8 virtual machines (VMs) running on each
(e.g. server n is running Vn1, Vn2, ... to Vnm where m = 8). Each
server is connected by two 1Gb/s links, one to each top-of-rack
switch S1 & S2. To the right of the switches, there are 6 links of
10Gb/s each, connecting onwards to customer networks or to the rest
of the data centre. There is a total of 48 *2 *1Gb/s = 96Gb/s
capacity between the 48 servers and the 2 switches, but there is only
6 *10Gb/s = 60Gb/s to the right of the switches. Nonetheless, data
centres are often designed with some level of contention like this,
because at the ToR switches a proportion of the traffic from certain
hosts turns round locally towards other hosts in the same rack.
virtual hosts switches
machines
V11 V12 V1m __/
* * ... * H1 ,-.__________+--+__/
\___\__ __\____/`-' __-|S1|____,--
`. _ ,' ,'| |_______
. H2 ,-._,`. ,' +--+
. . `-'._ `.
. . `,' `.
. ,' `-. `.+--+_______
Vn1 Vn2 Vnm / `-_|S2|____
* * ... * Hn ,-.__________| |__ `--
\___\__ __\____/`-' +--+ \__
\
Figure 5: Dual-Homed Topology -- a Simple Resource Pool
Briscoe Expires August 18, 2014 [Page 22]
Internet-Draft Congestion Policing February 2014
The congestion policer proposed in this document is based on the
'hose' model, where a tenant's congestion allowance can be used for
sending data over any path, including many paths at once. Therefore,
any one of the virtual machines on the left can use its allowance to
contribute to congestion on any or all of the 6 links on the right
(or any other link in the diagram actually, including those from the
server to the switches and those turning back to other hosts).
Nonetheless, if congestion policers are to enforce performance
isolation, they should stop one tenant squeezing the capacity
available to another tenant who needs to use a particular bottleneck
link or links (e.g. to reach a particular receiver). Policing should
work whether the offending tenant is acting deliberately or merely
carelessly.
3.8.1. Numerical Example: Isolation from Focused Load
Consider the following scenario: two tenants have equal congestion
allowances of 40kb/s, and let's imagine they each run servers that
send about 1 flow per second. All their flows use the same scalable
congestion control and all use the same aggressiveness or weight of
20kb/s.
Tenant A has clients all over the network, so it spreads its traffic
over numerous links, and its flows usually complete within 1 second.
Therefore, on average, A has one flow running at any one instant, so
it consumes 20kb/s from its 40kb/s congestion allowance.
Then tenant B decides (carelessly or deliberately) to concentrate all
its flows into one of the links that A sometimes uses for one of its
flows. All the flows through this new bottleneck (A's, B's and those
from other tenants) will still each consume 20kb/s of allowance while
they are active (Section 3.6), but they will all take longer to
complete. Let's say they take 15 times longer to complete on average
(i.e. 15s not 1s). And let's assume 5% of tenant A's flows pass
through this link. Then each tenant will consume congestion
allowance at:
o Tenant A: (95% * 1 * 20kb/s) + (5% * 15 * 20kb/s) = 34kb/s
o Tenant B: (100% * 15 * 20kb/s) = 300kb/s
Tenant B's congestion token bucket will therefore drain 7.5 times
faster than it is filling (300 / 40 = 7.5) so it will rapidly empty
and start dropping some of tenant B's traffic before it enters the
network. In fact, once tenant B's bucket is empty, it will be
limited to no more than 40kb/s of congestion at the bottleneck link,
rather than 300kb/s otherwise. In effect, the policer shifts packet
Briscoe Expires August 18, 2014 [Page 23]
Internet-Draft Congestion Policing February 2014
drops in excess of the 40kb/s allowance out of the shared link an
into itself, to focus excess discards only on tenant B.
Once tenant B's bucket empties, tenant A's completion time for the
flows through this bottleneck would not return to 1s, but it would be
limited to about 2s (not 15s), and it would consume its congestion
allowance at about 21kb/s, well within its limt of 40kb/s.
3.8.2. Isolation in the Short-Term
It seems problematic that tenant B's congestion policer takes time to
empty before it kicks in, during which time other tenants suffer
reduced performance. This is deliberate--to allow some give and take
over time. Nonetheless, it is possible to limit this problem as
well, using the dual token bucket already described in Figure 5. The
second shallow bucket sets an immediate limit on how much tenant B
can harm the performance of others. The fill-rate of the shallow
bucket is c times that of the deeper bucket (let's say c = 8), so
tenant B is immediately limited to 8 * 40kb/s = 320kb/s of
congestion. In the scenario described above, tenant B only reaches
300kb/s but at least this backstop prevents congestion from tenant B
ever exceeding 320kb/s.
3.8.3. Encouraging Load Balancing
A policer may not even have to directly intervene for tenants to be
protected; it might encourage load balancing to remove the problem
first. Load balancing might either be provided by the network
(usually just random), or some of the tenants might themselves
actively shift traffic off the increasingly congested bottleneck and
onto other paths. Some of them might be using the multipath TCP
protocol (MPTCP -- see experimental [RFC6356]) that would achieve
this automatically, or ultimately if the congestion signals
persisted, automatic VM placement algorithms might shift their
virtual machine to a different endpoint to circumvent the congestion
hot-spot completely. Even if some tenants were not using MPTCP or
could not shift easily, others shifting away would achieve the same
outcome. Essentially, the deterrent effect of congestion policers
encourages everyone to even out congestion across the network,
shifting load away from hot spots. Then performance isolation
becomes an emergent property of everyone's behaviour, due to the
deterrent effect of policers, rather than always through explicit
policer intervention.
In such an environment, a policer is needed that shares out the whole
pool of network resources, not one that just controls shares of each
link individually.
Briscoe Expires August 18, 2014 [Page 24]
Internet-Draft Congestion Policing February 2014
In contrast, enforcement mechanisms based on scheduling algorithms
like WRR or WFQ have to be deployed at each link, and each one works
in ignorance of how much of other links the tenant is using. These
algorithms were designed for networks with a single known bottleneck
per customer (e.g. many access networks). However, in data centres
there are many potential bottlenecks and each tenant generally only
uses a share of a small number of them. A mechanism like WRR would
not isolate anyone's performance if it gave every tenant the right to
use the same share of all the links in the network, without regard to
how many they were using.
3.9. Tenants of Different Sizes
The scenario in Section 3.8 assumed tenants A & B had equal
congestion allowances. If one tenant bought a huge allowance (e.g.
500kb/s) while another tenant bought a small allowance (e.g. 10kb/s),
it would seem that the former could focus all its traffic on one link
to swamp the latter.
At this point, it needs to be pointed out that congestion allowances
do not preclude the need for decent capacity planning. If we further
assume that tenant A has bought the same access capacity as tenant B,
50 times more congestion allowance implies tenant A expects to be
provided with 50 times the contention ratio of tenant B, i.e. any
single links they share will be big enough for B to still have enough
if it gets 1 share for every 50 used by A.
However, there is still a problem in that A can store up congestion
allowance and cause much more than 500kb/s of congestion in one
burst. The dual token bucket arrangement (Figure 5) limits tenant A
to c * 500kb/s of congestion, but if c=8 as before, 8 * 500kb/s = 4Mb
/s of congestion in a burst would seriously swamp tenant B if all
focused on one link where tenant B was quietly consuming its 10kb/s
allowance.
The answer to this problem is that the burst limit c can be smaller
for larger tenants, tending to 1 for the largest tenants (i.e. a
single shallow bucket would suffice for huge tenants). The reasoning
is that the c factor allows a tenant some give-and-take over time,
but the aggregate of traffic from a larger tenant consists of enough
flows that it has sufficient give-and-take within its own aggregate
without needing give-and-take from others.
3.10. Links of Different Sizes
Congestion policing treats a Mb/s of capacity in one link as
identical to a Mb/s of capacity in another link, even if the size of
each link is different. For instance, consider the case where one of
Briscoe Expires August 18, 2014 [Page 25]
Internet-Draft Congestion Policing February 2014
the three links to the right of each switch in Figure 5 were upgraded
to 40Gb/s while the other two remained at 10Gb/s (perhaps to
accommodate the extra traffic from a couple of the dual homed 1Gb/s
servers being upgraded to dual-homed 10Gb/s).
A congestion control algorithm running at the same rate will cause
the same level of congestion probability, whatever size link it is
sharing. For example:
o If 50 equal flows share a 10Gb/s link (10Gb/s / 50 = 200Mb/s each)
they will cause 0.01% congestion probability;
o If 200 of the same flows share a 40Gb/s link (40Gb/s / 200 = 200Mb
/s each) they will still cause 0.01% congestion probability;
This is because the congestion probability is determined by the
congestion control algorithms, not by the link.
Therefore, if an average of 300 flows were spread across the above
links (1x 40Gb/s and 2 x 10Gb/s), the numbers on each link would tend
towards respectively 200:50:50, so that each flow would get 200Mb/s
and each link would have 0.01% congestion on it.
Sometimes, there might be more flows on one link, resulting in less
than 200Mb/s per flow and congestion higher than 0.01%. However,
whenever the congestion level was greater on one link than another,
congestion policing would encourage flows to balance out the
congestion level across the links (as long as some flows could use
congestion balancing mechanisms like MPTCP).
In summary, all the outcomes of congestion policing described so far
(emulating WRR, etc) apply across a pool of diverse link sizes just
as much as they apply to single links, without any need for the links
to signal to each other nor to a central controller.
3.11. Diverse Congestion Control Algorithms
Throughout this explanation we have assumed a scalable congestion
control algorithm, which we justified (Section 3.4) as the 'boundary'
case if congestion policing were to intervene, which is all that is
relevant when considering whether the policer can enforce performance
isolation.
The defining difference between the scalable congestion we have
assumed and the congestion controls in widespread production
operating systems (New Reno, Compound, Cubic, Data Centre TCP etc) is
the way congestion probability decreases as flow-rate increases (for
a long-running flow). With a scalable congestion control, if flow-
Briscoe Expires August 18, 2014 [Page 26]
Internet-Draft Congestion Policing February 2014
rate doubles, congestion probability halves. Whereas, with most
production congestion controls, if flow-rate doubles, congestion
probability reduces to less than half. For instance, New Reno TCP
reduces congestion to a quarter. The responses of Cubic and Compound
are closer to the ideal scalable control than to New Reno, but they
do not depart too far from New Reno to ensure they can co-exist
happily with it.
Congestion policing still works, whether or not the congestion
controls in daily use by tenants fit the scalable model. A bulk
congestion policer constrains the sum of all the congestion controls
being used by a tenant so that they collectively remain below an
aggregate envelope that is itself shaped like the sum of many
scalable algorithms. Bulk congestion policers will constrain the
overall congestion effect (the sum) of any mix of algorithms within
it, including flows that are completely unresponsive to congestion.
This will now be explained using Figure 6 (if the ASCII art is
incomprehensible, see a similar plot at Figure 3 of [CongPol]).
bit-rate[b/s]
/ \
| * / / / / / / / / / / / / / / / / /
| +New Reno * Policed / / / / / / / / / / / / /
| + TCP * / / / / / / / / / / / / / / / /
| + * / / / / / / / / / / / / / / /
| - - - - - -+- - - - - - - - -*- - - - - - - - - - - - - - -
| + * / / / / / unresponsive/ /
| + * / / / / / / / / / / /
| + * / / / / / / / / / /
| +* / / / / / / /
| * / / /+/ /
| * / /
|____________________________________________________________\
0 ' 0.2%' ' 0.4%' 'congestion/
Figure 6: Schematic Plot of Bulk Policing of Traffic with Different
Congestion Responses
The congestion policer prevents the aggregate of a tenant's traffic
from operating in the hatched region shown in Figure 6 (there is
deliberately no vertical scale, because the plot is schematic only).
The tenant can have either high congestion or high rate, but not both
at the same time. It can be seen that the single New Reno TCP flow
responds to congestion in a similar way, but with a flatter slope.
It can also be seen that, at 0.25% prevailing congestion, this
policer would allow roughly two simultaneous New Reno TCP flows
without intervening. At about 0.45% congestion it would allow only
Briscoe Expires August 18, 2014 [Page 27]
Internet-Draft Congestion Policing February 2014
one, and above 0.45% it would start to police even one flow. But at
much lower congestion levels typical in a high-capacity data centre
(e.g. 0.01%) the policer would allow numerous New Reno TCP flows
(extrapolating the curve upwards).
The horizontal plot represents an unresponsive (UDP) flow. The
congestion policer would allow one of these flows to run unimpeded up
to roughly 0.3% congestion, when the policer would effectively take
control and impose its own congestion response on this flow, dropping
more UDP packets for higher levels of prevailing congestion. If a
tenant ran two of these UDP flows (or one at twice the rate), the
policer would leave them alone unless prevailing congestion was above
about 0.22% when it would intervene (by extrapolation of the visible
plots).
The New Reno TCP curve follows the characteristic k/sqrt(p) rule,
while the 'Policed' curve follows a simple w/p rule, where k is the
New Reno constant and w is the constant contracted fill-rate of a
particular policer. A scalable TCP (not shown) would follow a v/p
rule where v is the weight of the individual flow. If we imagine a
scenario where w = 8v, the policer would allow 8 of these scalable
TCP flows (averaged over time), and this would be true at any
congestion level, because the curvature of 8 scalable TCP's would
exacatly match the 'Policed' curve.
So far, the discussion has focused on the congestion avoidance phase
of each congestion response. As each TCP flow starts, it typically
increases exponentially (TCP slow-start) until it overshoots
available capacity causing a spike of loss due to the feedback delay
over the following round-trip. The ConEx audit mechanism requires
the source to send credit markings in advance to cover the risk of
these spikes of loss. Credit marked ConEx packets drain tokens from
the congestion policer similarly to regular ConEx marked packets,
which reduces the available congestion allowance for other flows. If
the tenant starts new flows fast enough, these credit markings alone
will empty the token bucket so that the policer starts to discard
some packets from the start-up phase before they can enter the
network. (The tunnel feedback method for signalling congestion back
to the policer lacks such a credit mechanism, which is an important
advantage of the ConEx protocol, given Internet traffic generally
consists of a high proportion of short flows.)
A tenant could run a mix of multiple different types of TCP and UDP
flows, as long as, when all stacked up, the sum of them all still
fitted under the 'Policed' curve at the prevailing level of
congestion.
Briscoe Expires August 18, 2014 [Page 28]
Internet-Draft Congestion Policing February 2014
4. Parameter Setting
The primary parameter to set for each tenant is the contracted fill-
rate of their congestion token bucket. For a server sending
predominantly long-running flows (e.g.a video server or an indexing
robot filling search databases), this fill-rate can be determined
from the required number of simultaneous TCP flows, and the typical
congestion-bit-rate of one TCP flow.
For a scalable TCP flow, the congestion bit-rate is constant. But
currently (2013) TCP Cubic is predominant, and the congestion bit-
rate of one Cubic flow is not constant, but depends somewhat on bit-
rate and RTT. For example, a Cubic flow with 100Mb/s throughput and
50ms RTT, has a congestion-bit-rate of ~2kbs, whereas at 200Mb/s and
5ms RTT, its congestion-bit-rate is ~6kb/s. Therefore, a reasonable
rule of thumb for Cubic's congestion-bit-rate at rates and RTTs of
about this order is 5kb/s.
Therefore, if a tenant wanted to serve no more on average than 12
simultaneous TCP Cubic flows, their contracted congestion-rate should
be roughly 12 * 5kb/s = 60kb/s.
The contracted fill rate of a congestion policer should not need to
change as flow-rates and link capacities scale, assuming the tenant
still utilises the higher capacity to the same extent.
A congestion policer based on the dual token bucket in Figure 2 also
needs to specify the c factor that determined the maximum congestion-
rate allowed, rather than the average. A theoretical approach to
setting this factor is being worked on but, in the mean time, a
figure of about 10 seems reasonable for a tenant sending only a few
simultaneous flows (see Section 3.8.2), while Section 3.9 explains
that the c factor should tend to reduce down to 1 for very large
tenants.
The dual token bucket makes isolation fairly insensitive to the depth
of the deeper bucket, because the shallower bucket constrains the
congestion burst rate, which makes the congestion burst size less
important. It is believed that the deeper bucket depth could be
minutes or even hours of token fill. The depth of the shallower
bucket should be just a few MTU--more experimentation is needed to
determine how few.
Indeed, all the above recommendations on parameter setting are best
considered as initial values to be used in experiments to determine
whether other values would be better.
Briscoe Expires August 18, 2014 [Page 29]
Internet-Draft Congestion Policing February 2014
No recommendations can yet be given for parameter settings for a
tenant running a large proportion of short flows. This requires
further experimentation.
5. Security Considerations
This document is about performance isolation, therefore the whole
document concerns security. Where ConEx is used, the specification
of the ConEx Abstract Mechanism [I-D.ietf-conex-abstract-mech] should
be consulted for security matters, particularly the design of audit
to assure the integrity of ConEx signals.
6. IANA Considerations
{Section to be removed by the RFC Editor}. This document does not
require actions by IANA.
7. Conclusions
{ToDo}
8. Acknowledgments
Bob Briscoe is part-funded by the European Community under its
Seventh Framework Programme through the Trilogy 2 project (ICT-
317756). The views expressed here are solely those of the author.
9. Informative References
[CPolTrilogyExp]
Raiciu, C., Ed., "Progress on resource control", Trilogy
EU 7th Framework Project ICT-216372 Deliverable 9,
December 2009, <http://trilogy-project.org/publications/
deliverables.html>.
[CongPol] Jacquet, A., Briscoe, B., and T. Moncaster, "Policing
Freedom to Use the Internet Resource Pool", Proc ACM
Workshop on Re-Architecting the Internet (ReArch'08) ,
December 2008,
<http://bobbriscoe.net/projects/refb/#polfree>.
[DCTCP] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel,
P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data
Center TCP (DCTCP)", ACM SIGCOMM CCR 40(4)63--74, October
2010, <http://portal.acm.org/citation.cfm?id=1851192>.
Briscoe Expires August 18, 2014 [Page 30]
Internet-Draft Congestion Policing February 2014
[I-D.briscoe-conex-data-centre]
Briscoe, B. and M. Sridharan, "Network Performance
Isolation in Data Centres using Congestion Policing",
draft-briscoe-conex-data-centre-01 (work in progress),
February 2013.
[I-D.briscoe-conex-initial-deploy]
Briscoe, B., "Initial Congestion Exposure (ConEx)
Deployment Examples", draft-briscoe-conex-initial-
deploy-03 (work in progress), July 2012.
[I-D.ietf-conex-abstract-mech]
Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx)
Concepts and Abstract Mechanism", draft-ietf-conex-
abstract-mech-08 (work in progress), October 2013.
[I-D.ietf-conex-mobile]
Kutscher, D., Mir, F., Winter, R., Krishnan, S., Zhang,
Y., and C. Bernardos, "Mobile Communication Congestion
Exposure Scenario", draft-ietf-conex-mobile-03 (work in
progress), February 2014.
[RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows",
RFC 3649, December 2003.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, September 2009.
[RFC6356] Raiciu, C., Handley, M., and D. Wischik, "Coupled
Congestion Control for Multipath Transport Protocols", RFC
6356, October 2011.
[Relentless]
Mathis, M., "Relentless Congestion Control", Proc. Int'l
Wkshp on Protocols for Future, Large-scale & Diverse
Network Transports (PFLDNeT'09) , May 2009,
<http://www.hpcc.jp/pfldnet2009/Program.html>.
[STCP] Kelly, T., "Scalable TCP: Improving Performance in
Highspeed Wide Area Networks", ACM SIGCOMM Computer
Communication Review 32(2), April 2003,
<http://www-lce.eng.cam.ac.uk/~ctk21/scalable/>.
[conex-dc_tr]
Briscoe, , "Network Performance Isolation in Data Centres
by Congestion Exposure to Edge Policers", BT Technical
Report TR-DES8-2011-004, November 2011.
Briscoe Expires August 18, 2014 [Page 31]
Internet-Draft Congestion Policing February 2014
To appear
Briscoe Expires August 18, 2014 [Page 32]
Internet-Draft Congestion Policing February 2014
Appendix A. Summary of Changes between Drafts
From briscoe-00 to briscoe-01: No technical differences; merely
clarified throughout & updated references.
From briscoe-conex-data-centre-00 to briscoe-conex-policing-00: This
draft was separated out from the -00 version of
[I-D.briscoe-conex-data-centre], taken mostly from what was
section 4. Changes from that draft are listed below:
From section 4 of draft-briscoe-conex-data-centre-00:
+ New Introduction
+ Added section 2. "Example Bulk Congestion Policer"
+ Rearranged and clarified the previous introductory text that
preceded what is now Section 3.4 into three subsections:
3.1 "The Problem"
3.2 "Approach"
3.3 "Terminology"
+ Corrected numerical examples:
- Section 3.4 "Long-RunningFlows": 0.04% to 0.004% and
400kb/s to 40kb/s
- Section 3.6.1 "Numerical Examples Without Policing": 10kb
/s to 20kb/s
- section 3.7 "Weighted Congestion Controls": 22.2 to 20
+ Clarified where necessary, especially the descriptions of
the scenarios in 3.7 "Weighted Congestion Controls" and 3.8
"A Network of Links".
+ Section 3.8 "A Network of Links": added a numerical example
including congestion burst limiting that was not covered at
all previously
+ Added Section 3.9 "Tenants of Different Sizes"
+ Filled in absent text in:
- Section 3.11 "Diverse Congestion Controls"
Briscoe Expires August 18, 2014 [Page 33]
Internet-Draft Congestion Policing February 2014
- Section 4. "Parameters Settings"
- Section 5. "Security Considerations"
+ Minor corrections throughout and updated references.
Author's Address
Bob Briscoe
BT
B54/77, Adastral Park
Martlesham Heath
Ipswich IP5 3RE
UK
Phone: +44 1473 645196
EMail: bob.briscoe@bt.com
URI: http://bobbriscoe.net/
Briscoe Expires August 18, 2014 [Page 34]
