Internet DRAFT - draft-ietf-avt-rtcptest
draft-ietf-avt-rtcptest
Internet Engineering Task Force Audio Video Transport WG
Internet Draft J.Rosenberg,H.Schulzrinne
draft-ietf-avt-rtcptest-01.txt Bell Laboratories,Columbia U.
June 25, 1999
Expires: December 1999
Conformance Tests for RTP Scalability Algorithms
STATUS OF THIS MEMO
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as work in progress.
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This document defines a set of tests that can be run on an RTP imple-
mentation to determine the level of conformance with the various sca-
lability algorithms defined in the draft version of RTP. These tests
do not require access to internal implementations, and use black box
testing techniques to determine compliance.
1 Introduction
The draft standard version of RTP [1] contains a number of algorithms
and behaviors related to transmission and reception of RTCP packets.
These algorithms include periodic transmission of RTCP packets, SSRC
collision detection, and allocation of differing RTCP rates to
senders and receivers. Many of these algorithms (forward reconsidera-
tion, reverse reconsideration, and BYE reconsideration) relate to
J.Rosenberg,H.Schulzrinne [Page 1]
Internet Draft RTCP Test June 25, 1999
scalability features for large groups [2,3], and have been added in
the transition from proposed to draft standard. Other mechanisms have
been added to enhance robustness for high bandwidth sessions, or to
allow greater flexibility in choosing RTCP bandwidth usage.
It is essential for the evolution of RTP to evaluate the interopera-
bility of these features. However, many of these algorithms are local
computations which affect timers and other session variables. These
do not get reflected in the content of messages that are sent by RTP
participants. As a result, correctness and interoperability of this
features is difficult to assess. Other features of RTP (such as SSRC
collision) occur in unusual circumstances, and will not likely show
up in a typical interop test.
To get around these problems, this document proposes a suite of con-
formance tests that can be run on an implementation to test its com-
pliance to various components of the draft standard RTP. These tests
are "black box" tests, and do not require access to the internal
workings of the implementation being tested.
2 Testing Architecture
The basic architecture for performing the tests is shown in Figure 1.
--------------
| |
| test |
| instrument |
| |
--------------
|
-------------------------------- LAN
|
----------------
| |
| RTP |
| implementation |
| |
----------------
Figure 1: Testing Architecture
The test instrument is connected on the same LAN as the RTP
J.Rosenberg,H.Schulzrinne [Page 2]
Internet Draft RTCP Test June 25, 1999
implementation being tested. It is assumed that the test instrument
is preconfigured with the addresses and ports used by the RTP imple-
mentation, and is also aware of the RTCP bandwidth and
sender/receiver fractions. The tests can be conducted using either
multicast or unicast.
The test instrument must be capable of sending arbitrarily crafted
RTP and RTCP packets to the RTP implementation. The instrument should
also be capable of receiving packets sent by the RTP implementation,
parsing them, and computing metrics based on those packets.
It is furthermore assumed that a number of basic controls over the
RTP implementation exist. These controls are:
o the ability to force the implementation to send or not send
RTP packets at any desired point in time;
o the ability to force the application to terminate its involve-
ment in the RTP session, and for this termination to be known
immediately to the test instrument;
o the ability to set the session bandwidth and RTCP sender and
receiver fractions;
The second of these is required only for the test of BYE reconsidera-
tion, and is the only aspect of these tests not easily implementable
by pure automation. It will generally require manual intervention to
terminate the session from the RTP implementation and to convey this
to the test instrument through some non-RTP means.
3 Basic Behavior
The first test is to verify basic correctness of the implementation
of the RTCP transmission rules. This basic behavior consists of:
o periodic transmission of RTCP packets;
o randomization of the interval for RTCP packet transmission;
o correct implementation of the randomization interval computa-
tions, with unconditional reconsideration.
These features are tested in the following manner. The RTP implemen-
tation acts as a session receiver, and never sends any RTP packets.
The implementation is configured with a large session bandwidth, say
1 Mb/s. This will cause the implementation to use the minimal inter-
val of 5s rather than the small interval based on the session
bandwidth and membership size. The implementation will generate RTCP
J.Rosenberg,H.Schulzrinne [Page 3]
Internet Draft RTCP Test June 25, 1999
packets at this minimal interval, on average. The test instrument
generates no packets, but receives the RTCP packets generated by the
implementation. When an RTCP packet is received, the time is noted by
the test instrument. The difference in time between each pair of sub-
sequent packets (called the interval) is computed. These intervals
are stored, so that statistics based on these intervals can be com-
puted. It is recommended that this observation process operate for at
least 20 minutes.
An implementation passes this test if the intervals have the follow-
ing properties:
o the minimum interval is never less than 2 seconds or more than
2.5 seconds;
o the maximum interval is never more than 7 seconds or less than
5.5 seconds;
o the average interval is between 4.5 and 5.5 seconds;
o the number of intervals between x and x+500ms is less than the
number of intervals between x+500ms and x+1s, for any x.
In particular, an implementation fails if the packets are sent with a
constant interval.
4 Step join backoff
The main purpose of the reconsideration algorithm is to avoid a flood
of packets that might occur when a large number of users simultane-
ously join an RTP session. Reconsideration therefore exhibits a back-
off behavior in sending of RTCP packets when group sizes increase.
This aspect of the algorithm can be tested in the following manner.
The implementation begins operation. The test instrument waits for
the arrival of the first RTCP packet. When it arrives, the test
instrument notes the time and then immediately sends 100 RTCP RR
packets to the implementation, each with a different SSRC and SDES
CNAME. The test instrument should ensure that each RTCP packet is of
the same length. The instrument should then wait until the next RTCP
packet is received from the implementation, and the time of such
reception is noted.
Without reconsideration, the next RTCP packet will arrive within a
short period of time. With reconsideration, transmission of this
packet will be delayed. The earliest it can arrive depends on the
RTCP session bandwidth, receiver fraction, and average RTCP packet
size. The RTP implementation should be using the exponential
J.Rosenberg,H.Schulzrinne [Page 4]
Internet Draft RTCP Test June 25, 1999
averaging algorithm defined in the specification to compute the aver-
age RTCP packet size. Since this is dominated by the received packets
(the implementation has only sent one itself), the average will be
roughly equal to the length of the RTCP packets sent by the test
instrument. Therefore, the minimum amount of time between the first
and second RTCP packets from the implementation is:
T > 101 * S / ( B * Fr * (e-1.5) * 2 )
Where S is the size of the RTCP packets sent by the test instrument,
B is the RTCP bandwidth (normally five percent of the session
bandwidth), Fr is the fraction of RTCP bandwidth allocated to
receivers (normally 75 percent), and e is the natural exponent.
Without reconsideration, this minimum interval Te would be much
smaller:
Te > MAX( [ S / ( B * Fr * (e-1.5) * 2 ) ] , [ 2.5 / (e-1.5) ] )
B should be chosen sufficiently small so that T is around 60 seconds.
Reasonable choices for these parameters are B = 950 bits per second,
and S = 1024 bits. An implementation passes this test if the interval
between packets is not less than T above, and not more than 3 times
T.
The test may be repeated for the case when the RTP implementation is
a sender. This is accomplished by having the implementation send RTP
packets at least once a second. In this case, the interval between
the first and second RTCP packets should be no less than:
T > S / ( B * Fs * (e-1.5) * 2 )
Where Fs is the fraction of RTCP bandwidth allocated to senders, usu-
ally 25%. Note that this value of T is significantly smaller than the
interval for receivers.
5 Steady State Behavior
J.Rosenberg,H.Schulzrinne [Page 5]
Internet Draft RTCP Test June 25, 1999
In addition to the basic behavior in section 3, an implementation
should correctly implement a number of other, slightly more advanced
features:
o scale the RTCP interval with the group size;
o correctly divide bandwidth between senders and receivers;
o correctly compute the RTCP interval when the user is a sender
The following tests are meant to test these features.
5.1 Scaling the Interval
The implementation begins operation as a receiver. The test instru-
ment waits for the first RTCP packet from the implementation. When it
arrives, the test instrument notes the time, and immediately sends 50
RTCP RR packets and 50 RTCP SR packets to the implementation, each
with a different SSRC and SDES CNAME. The test instrument then sends
50 RTP packets, using the 50 SSRC from the RTCP SR packets. The test
instrument should ensure that each RTCP packet is of the same length.
The instrument should then wait until the next RTCP packet is
received from the implementation, and the time of such reception is
noted. The difference between the reception of the RTCP packet and
the reception of the previous is computed and stored. In addition,
after every RTCP packet reception, the 100 RTCP and 50 RTP packets
are retransmitted by the test instrument. This ensures that the
sender and member status of the 100 users does not time out. The test
instrument should collect the interval measurements figures for at
least 100 RTCP packets.
With 50 senders, the implementation should not try to divide the RTCP
bandwidth between senders and receivers, but rather group all users
together and divide the RTCP bandwidth equally. The test is deemed
successful if the average RTCP interval is within 5% of:
T = 101* S/B
Where S is the size of the RTCP packets sent by the test instrument,
and B is the RTCP bandwidth. B should be chosen sufficiently small so
that the value of T is on the order of tens of seconds or more. Rea-
sonable values are S=1024 bits and B=3.4 kb/s.
5.2 Compensating for Senders
J.Rosenberg,H.Schulzrinne [Page 6]
Internet Draft RTCP Test June 25, 1999
The test of section 5.1 is repeated. However, the test instrument
sends 10 RTP packets instead of 50, and 10 RTCP SR and 90 RTCP RR
instead of 50 of each. In addition, the implementation is made to
send at least one RTP packet between transmission of every one of its
own RTCP packets.
In this case, the average RTCP interval should be within 5% of:
T = 11 * S / (B * Fs)
Where S is the size of the RTCP packets sent by the test instrument,
B is the RTCP bandwidth, and Fs is the fraction of RTCP banwidth
allocated for senders (normally 25%). The values for B and S should
be chosen small enough so that T is on the order of tens of seconds.
Reasonable choices are S=1024 bits and B=1.5 kb/s.
6 Reverse Reconsideration
The reverse reconsideration algorithm is effectively the opposite of
the normal reconsideration algorithm. It causes the RTCP interval to
be reduced more rapidly in response to decreases in the group member-
ship. This is advantageous in that it keeps the RTCP information as
fresh as possible, and helps avoids some premature timeout problems.
These tests verify the basic correctness of reverse reconsideration.
6.1 Test I
In this test, the implementation joins the session as a receiver. As
soon as it sends its first RTCP packet, the test instrument sends 100
RTCP RR packets, each of the same length S, and a different SDES
CNAME and SSRC in each. It then waits for the implementation to send
another RTCP packet. Once it does, the test instrument sends 100 BYE
packets, each one containing a different SSRC, but matching an SSRC
from one of the initial RTCP packets. Each BYE should also be the
same size as the RTCP packets sent by the test instrument. This is
easily accomplished by using a BYE reason to pad out the length. The
time of the next RTCP packet from the implementation is then noted.
The delay T between this (the third RTCP packet) and the previous
should be no more than:
T < 3 * S / (B * Fr * (e-1.5) * 2)
J.Rosenberg,H.Schulzrinne [Page 7]
Internet Draft RTCP Test June 25, 1999
Where S is the size of the RTCP and BYE packets sent by the test
instrument, B is the RTCP bandwidth, Fr is the fraction of the RTCP
bandwidth allocated to receivers, and e is the natural exponent. B
should be chosen such that T is on the order of tens of seconds. A
reasonable choice is S=1024 bits and B=168 bits per second.
This test demonstrates basic correctness of implementation. An imple-
mentation without reverse reconsideration will not send its next RTCP
packet for nearly 100 times as long as the above amount.
6.2 Test II
In this test, the implementation joins the session as a receiver. As
soon as it sends its first RTCP packet, the test instrument sends 100
RTCP RR packets, each of the same length S, followed by 100 BYE pack-
ets, also of length S. Each RTCP packet carries a different SDES
CNAME and SSRC, and is matched with precisely one BYE packet with the
same SSRC. This will cause the implementation to see a rapid increase
and then rapid drop in group membership.
The test is deemed succesful if the next RTCP packet shows up T
seconds after the first, and T is within:
2.5 / (e-1.5) < T < 7.5 / (e-1.5)
This tests correctness of the maintenance of the pmembers variable.
An incorrect implementation might try to execute reverse reconsidera-
tion every time a BYE is received, as opposed to only when the group
membership drops below pmembers. If an implementation did this, it
would end up sending an RTCP packet immediately after receiving the
stream of BYE's. For this test to work, B must be chosen to be a
large value, around 1Mb/s.
7 BYE Reconsideration
The BYE reconsideration algorithm works in much the same fashion as
regular reconsideration, except applied to BYE packets. When a user
leaves the group, instead of sending a BYE immediately, it may delay
transmission of its BYE packet if others are sending BYE's.
The test for correctness of this algorithm is as follows. The RTP
implementation joins the group as a receiver. The test instrument
waits for the first RTCP packet. When the test instrument receives
this packet, the test instrument immediately sends 100 RTCP RR pack-
ets, each of the same length S, and each containing a different SSRC
J.Rosenberg,H.Schulzrinne [Page 8]
Internet Draft RTCP Test June 25, 1999
and SDES CNAME. Once the test instrument receives the next RTCP
packet from the implementation, the RTP implementation is made to
leave the RTP session, and this information is conveyed to the test
instrument through some non-RTP means. The test instrument then sends
100 BYE packets, each with a different SSRC, and each matching an
SSRC from a previously transmitted RTCP packet. Each of these BYE
packets is also of size S. Immediately following the BYE packets, the
test instrument sends 100 RTCP RR packets, using the same SSRC/CNAMEs
as the original 100 RTCP packets.
The test is deemed successful if the implementation either never
sends a BYE, or if it does, the BYE is not received by the test
instrument earlier than T seconds after the implementation left the
session, where T is:
T = 100 * S / ( 2 * (e-1.5) * B * Fr)
S is the size of the RTCP and BYE packets, e is the natural exponent,
B is the RTCP bandwidth, and Fr is the RTCP bandwidth fraction for
receivers. S and B should be chosen so that T is on the order of 50
seconds. A reasonable choice is S=1024 bits and B=1.1 kb/s.
The transmission of the RTCP packets is meant to verify that the
implementation is ignoring non-BYE RTCP packets once it decides to
leave the group.
8 Timing out members
Active RTP participants are supposed to send periodic RTCP packets.
When a participant leaves the session, they may send a BYE, however
this is not required. Furthermore, BYE reconsideration may cause a
BYE to never be sent. As a result, participants must time out other
participants who have not sent an RTCP packet in a long time. Accord-
ing to the specification, participants who have not sent an RTCP
packet in the last 5 intervals are timed out. This test verifies that
these timeouts are being performed correctly.
The RTP implementation joins a session as a receiver. The test
instrument waits for the first RTCP packet from the implementation.
Once it arrives, the test instrument sends 100 RTCP RR packets, each
with a different SDES and SSRC, and notes the time. This will cause
the implementation to believe that there are now 101 group partici-
pants, causing it to increase its RTCP interval. The test instrument
continues to monitor the RTCP packets from the implementation. As
each RTCP packet is received, the time of its reception is noted, and
J.Rosenberg,H.Schulzrinne [Page 9]
Internet Draft RTCP Test June 25, 1999
the interval between RTCP packets is stored. The 100 participants
spoofed by the test instrument should eventually time out at the RTP
implementation. This should cause the RTCP interval to be reduced to
its minimum.
The test is deemed successful if the interval between RTCP packets
after the first is no less than:
Ti > 101 * S / ( 2 * (e-1.5) * B * Fr)
and this minimum interval is sustained no later than Td seconds after
the transmission of the 100 RR's, where Td is:
Td = 7 * 101 * S / ( B * Fr )
and the interval between RTCP packets after this point is no less
than:
Tf > 2.5 / (e-1.5)
For this test to work, B and S must be chosen so Ti is on the order
of minutes. Recommended values are S = 1024 bits and B = 1.9 kbps.
9 SSRC Randomization
According to the RTP specification, SSRC's are supposed to be chosen
randomly and uniformly over a 32 bit space. This randomization is
beneficial for several reasons:
o It reduces the probability of collisions in large groups.
o It simplifies the process of group sampling [4], which depends
on the uniform distribution of SSRC's across the 32 bit space.
Unfortunately, verifying that a random number has 32 bits of uniform
randomness requires a large number of samples. The procedure below
gives only a rough validation to the randomness used for generating
the SSRC.
J.Rosenberg,H.Schulzrinne [Page 10]
Internet Draft RTCP Test June 25, 1999
The test runs as follows. The RTP implementation joins the group as a
receiver. The test instrument waits for the first RTCP packet. It
notes the SSRC in this RTCP packet. The test is repeated 2500 times,
resulting in a collection of 2500 SSRC.
The are then placed into 25 bins. An SSRC with value X is placed into
bin FLOOR(X/(2**32 / 25)). The idea is to break the 32 bit space into
25 regions, and compute the number of SSRC in each region. Ideally,
there should be 40 SSRC in each bin. Of course, the actual number in
each bin is a random variable whose expectation is 40. With 2500
SSRC, the coefficient of variation of the number of SSRC in a bin is
.1, which means the number should be between 36 and 44. The test is
thus deemed successful if each bin has no less than 30 and no more
than 50 SSRC.
Running this test may require substantial amounts of time, particu-
larly if there is no automated way to have the implementation join
the session. In such a case, the test can be run fewer times. With 26
tests, half of the SSRC should be less than 2**31, and the other half
higher. The cofficient of variation in this case is .2, so the test
is successful if there are more than 8 SSRC less than 2**31, and less
than 26.
In general, if the SSRC is collected N times, and there are B bins,
the coefficient of variation of the number of SSRC in each bin is
given by:
coeff = SQRT( (B-1)/N )
10 SSRC Collisions
RTP has a provision for SSRC collisions. These collisions occur when
two different session participants choose the same SSRC. In this
case, both participants are supposed to send a BYE, leave the ses-
sion, and rejoin with a different SSRC, but the same CNAME. The pur-
pose of this test is to verify that this function is present in the
implementation.
The test is straightforward. The RTP implementation is made to join
the multicast group as a receiver. The test instrument waits for the
first RTCP packet. Once it arrives, the test instrument notes the
CNAME and SSRC from the RTCP packet. The test instrument then gen-
erates an RTCP receiver report. This receiver report contains an SDES
J.Rosenberg,H.Schulzrinne [Page 11]
Internet Draft RTCP Test June 25, 1999
chunk with an SSRC matching that of the RTP implementation, but with
a different CNAME. At this point, the implementation should send a
BYE RTCP packet (containing an SDES chunk with the old SSRC and
CNAME), and then rejoin, causing it to send a receiver report con-
taining an SDES chunk, but with a new SSRC and the same CNAME.
The test is deemed succesful if the RTP implementation sends the RTCP
BYE and RTCP RR as described above within one minute of receiving the
colliding RR from the test instrument.
11 Rapid SR's
In the draft version of RTP, the minimum interval for RTCP packets
can be reduced for large session bandwdiths. The reduction applies to
senders only. The recommended algorithm for computing this minimum
interval is 360 divided by the RTP session bandwidth, in kbps. For
bandwidths larger than 72 kbps, this interval is less than 5 seconds.
This test verifies the ability of an implementation to use a lower
RTCP minimum interval when it is a sender in a high bandwidth ses-
sion. The test can only be run on implementations that support this
reduction, since it is optional.
The RTP implementation is configured to join the session as a sender.
The session is configured to use 360 kbps. If the recommended algo-
rithm for computing the reduced minimum interval is used, the result
is a 1 second interval. If the RTP implementation uses a different
algorithm, the session bandwidth should be set in such a way to cause
the reduced minimum interval to be 1 second.
Once joining the session, the RTP implementation should begin to send
both RTP and RTCP packets. The interval between RTCP packets is meas-
ured and stored until 100 intervals have been collected.
The test is deemed successful if the smallest interval is no less
than 1/2 a second, and the largest interval is no more than 1.5
seconds. The average should be close to 1 second.
12 Changes since -00
The following changes have been made since version -00 of this
specification:
o The introduction was rewritten to reflect the new broader
scope of the tests. Previously, the tests verified only the
features new in draft standard. Now, the tests include some
features present in the proposed standard.
J.Rosenberg,H.Schulzrinne [Page 12]
Internet Draft RTCP Test June 25, 1999
o A test for verifying uniform distribution of SSRC was added.
o A test for verifying reduction of the minimum interval for
high bandwidth sessions was added.
o A test for verifying SSRC collision detection was added.
o A test for verifying timeouts was added.
13 Author's Addresses
Jonathan Rosenberg
Lucent Technologies, Bell Laboratories
101 Crawfords Corner Rd.
Holmdel, NJ 07733
Rm. 4C-526
email: jdrosen@bell-labs.com
Henning Schulzrinne
Columbia University
M/S 0401
1214 Amsterdam Ave.
New York, NY 10027-7003
email: schulzrinne@cs.columbia.edu
14 Bibliography
[1] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: a
transport protocol for real-time applications," Internet Draft,
Internet Engineering Task Force, Mar. 1999. Work in progress.
[2] J. Rosenberg and H. Schulzrinne, "Timer reconsideration for
enhanced RTP scalability," in IEEE Infocom , (San Francisco, Califor-
nia), March/April 1998.
[3] J. Rosenberg and H. Schulzrinne, "New results in RTP scalabil-
ity," Internet Draft, Internet Engineering Task Force, Dec. 1997.
Work in progress.
[4] J. Rosenberg and H. Schulzrinne, "Sampling of the group member-
ship in RTP," Internet Draft, Internet Engineering Task Force, Apr.
1999. Work in progress.
J.Rosenberg,H.Schulzrinne [Page 13]
Internet Draft RTCP Test June 25, 1999
Full Copyright Statement
Copyright (c) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this docu-
ment itself may not be modified in any way, such as by removing the
copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of develop-
ing Internet standards in which case the procedures for copyrights
defined in the Internet Standards process must be followed, or as
required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MER-
CHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
J.Rosenberg,H.Schulzrinne [Page 14]
