Internet DRAFT - draft-welch-streaming-test
draft-welch-streaming-test
Switch Testing for Streaming Media Applications July 2006
Network Working Group J. Welch
Internet Draft IneoQuest Technologies
Intended Category: Informational Muhammad Waris Sagheer
Cisco Systems
Sameer Satyam
Cisco Systems
Javed Asghar
Cisco Systems
Syed Nawaz
Cisco Systems
Andre Dufour
Agilent Technologies
July, 2006
Switch Testing for Streaming Media Applications
draft-welch-streaming-test-00.txt
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than a "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/lid-abstracts.html
Welch,Sagheer,Salyam, Expires January 2007 [Page 1]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This memo provides information for the Internet community. It does
not specify an Internet standard. Distribution of this memo is
unlimited.
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Abstract
This memo defines a set of supplementary tests for networking
interconnection devices (switches) that can be used to evaluate and
compare their performance when used with streaming media. The Media
Delivery Index (MDI) [i3] measurement is employed as a convenient
stream quality indicator of input and output streams to indicate
cumulative stream jitter and packet loss under test load conditions.
Typical operating profiles are defined to focus test efforts on
common switch applications for unidirectional streaming media to
reduce the amount of testing required and to encourage device and
system evaluation in advance of deployment through use of a set of
reference tests.
The supplementary tests defined in this memo are intended for
Information only.
1.
Introduction
There has been considerable progress over the last several years in
the development of methods to provide for Quality of Service (QoS)
over packet switched networks to improve the delivery of streaming
media and other time and packet loss sensitive applications such as
[i1], [i2]. QoS mechanisms are required for many practical converged
networks involving streaming media applications such as video
transport to assure the availability of network bandwidth by
Welch,Sagheer,Salyam, Expires January 2007 [Page 2]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
providing upper limits on the number of flows admitted to a network
as well as to bound the packet jitter introduced by the network.
These bounds are required to dimension a receiver`s buffer to
properly display the video in real time without buffer overflow or
underflow. Just as a receiver’s buffer has limited depth and is
subject to overflow considerations, so too are intermediate switch
node buffers limited in depth and subject to flow variation stresses.
QoS implementations and queue management strategies in switch devices
vary widely and it has been difficult to compare resultant switch
performance in the presence of streaming media. Owing to their real
time behavior and persistent nature, streaming media applications are
particularly sensitive to packet arrival time jitter and packet loss.
Cumulative jitter and loss performance are especially critical in
gauging end user perception of compressed video stream quality as any
loss is frequently visually perceptible and significant cumulative
jitter can cause loss in terminal equipment due to dejitter buffer
overflow or underflow. Subjective voice stream perception quality
has been shown to be more tolerant of loss for acceptable quality.
Due to the wide variety of possible network application uses,
traditional data application oriented tests often do not reflect
perceived switch performance when used in streaming media
applications.
Traditional network interconnect device evaluation criteria such as
outlined in RFC 2544 specify procedures for evaluating device
performance expressed as maximum frame forwarding rates for given
frame sizes, frame formats, broadcast/multicast frames, management
frames, and during routing updates and with forwarding filters
enabled. Bursty traffic is considered as well by specifying test
conditions of various length bursts. Device throughput is
characterized by the fastest rate that can be sustained without frame
loss. Latency measurements are considered by measurement of a frame
propagation time while the device is under load. Frame burst
handling, system recovery times from oversubscription, and reset
recovery times are also detailed. These measurements provide
standardized test methods and consistent reporting guides so that
device performance can be compared with different vendors and the
devices’ suitability for handling specific mixed data applications
can be assessed.
RFC 2889 extends the methodology of RFC 2544 for MAC layer switches
to include “…forwarding performance, congestion control, latency,
address handling and filtering.”[RFC 2889] It also includes test
Welch,Sagheer,Salyam, Expires January 2007 [Page 3]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
conditions for fully meshed traffic, consideration for address
learning, frame formats and sizes, duplexity, burst size, number of
addresses per port, flooding performance, illegal frame handling, and
broadcast frame performance. As in RFC 2544, standardized test
methods and consistent reporting guides are detailed.
Methods exist to control the stream jitter introduced by devices
forwarding streaming media [i4] through ingress flow allocation and
policing, forwarding management, and link bandwidth allocation. The
design and configuration tasks necessary to achieve optimal
performance for these functions are often complex. Hence, there is a
need for a verification and evaluation procedure for streaming media.
This note addresses the need.
Traditional methods of evaluating VoIP streams have proven adequate
to date. Video streaming media device performance has not been
adequately addressed and is the focus of this note. For converged
profile tests, it is suggested that existing, conventional VoIP tests
be used for verifying the performance of VoIP such as ITU-T P.862
[i7], Y.1541 [i8], and G.107 [i9] along with equipment intended to
provide such results in addition to the tests described here. If the
defined test profile includes VoIP traffic, such tests should ideally
be performed simultaneously with the tests described here.
This note proposes and describes the following approach for device
evaluation:
-Identification of a set of profiles representing common traffic load
types, levels, and flow paths to be used during the evaluation of a
switch intended for streaming media. These profiles are based on
common configurations found in many Cable and Telco streaming media
transport networks. Some suggested profiles are included and others
may be added to expand the coverage of the tests to include more
traffic classes.
-Generation of realistic, repeatable traffic load test conditions to
characterize switches with the use of the media delivery index (MDI)
as the indicator of flow quality for streaming media. MDI, which
measures cumulative packet jitter and packet loss, is becoming
increasingly common as a metric of streaming media quality.
-Use of the MDI Characteristic Curve as the means for showing the
results of switch testing for streaming media.
Welch,Sagheer,Salyam, Expires January 2007 [Page 4]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
Determining switch contribution to jitter and loss under known
traffic profiles provides users confidence that a particular switch
will adequately function in a particular target environment. For
example, a video on demand (VoD) server head end environment with
known server-delivered traffic, MDI characteristics, flow paths, and
known downstream decoder. Specifications can require that a switch
and the remaining transport system contribute no more than a maximum
MDI footprint (additional jitter and loss) for successful unimpaired
stream delivery where success is ultimately measured by a network-
unimpaired decode of the stream. Switches characterized in these
terms simplify system design and assure streaming media delivery
quality for a given range of load conditions.
For other than streaming media performance, the existing methods
described above can and should be used to represent switch
performance.
Thus, this note is intended to describe a realistic and practical set
of supplementary tests for switches intended to carry streaming media
whose results can readily be used to:
1. Compare switches of various design, configuration, setup, and
manufacturer,
2. Determine whether so characterized devices will deliver the
expected performance for specific system requirements, and
3. Verify that a specific system is configured to deliver expected
operational streaming media delivery performance.
4. In addition, these tests can assist in planning and configuring
buffer sizes properly over single and multiple hop networks in the
presence of bursty streaming traffic.
The Media Delivery Index (MDI) measurement is employed as the media
quality indicator since it conveniently captures cumulative stream
jitter and packet loss performance through its delay factor (DF) and
media loss rate (MLR) components.
2.
Testing Criteria
Welch,Sagheer,Salyam, Expires January 2007 [Page 5]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
The measure of network switch performance is the magnitude of MDI
modification, or MDI footprint, it imposes: this indicates how it
has degraded or improved a stream’s MDI. The MDI footprint of a
device or network segment is the maximum difference between the MDI
delay factor at the input of the device to that measured at the
output for the same stream. Taking the maximum delay factor recorded
for each stream at the input and comparing it to the corresponding
stream’s maximum DF recorded over the test period at the output and,
from these maxima, selecting the largest one demonstrates the
device’s worst case footprint. Ideally, the DF change will be low
indicating that the amount of stream jitter present at the input has
not been substantially increased and no loss was introduced. It is
important to make the measurements on all streams during the test as
some streams can be significantly affected while other streams may
not be affected at all based on their relative arrival times at a
queue. It is also important that the switch’s load profile including
the input active port count (including exactly which ports on a
multiport device are active), per port stream count, bit rates,
forwarding paths, and delay factors accurately represent the worst
case combinations for the intended application, since these factors
may change the effects that a switch has on streams.
A switch that unduly delays forwarding packets from input queue to
output queue or bursts data to its output queue, and subsequently to
the port output, exhibits poorer stream performance than one that
avoids introducing packet bursts. This poorer bursty performance can
accumulate in successive downstream devices and may ultimately result
in a downstream queue overflow (or an underflow at the destination
decoder) and lost packets. Devices with ports that have per stream
rate shaped outputs forward better stream characteristics to
downstream nodes reducing the chances of a downstream queue overflow.
The MDI Characteristic Curves that result from the tests described in
this note can be utilized by a service provider in an iterative
fashion to adequately dimension queue depths and behavior, traffic
shaping, and other switching infrastructure resources during both lab
certification and system commissioning.
3.
Profiles
Switches should be tested using input traffic profiles representative
of the intended applications of the devices. Test results should be
plotted as MDI Characteristic Curves as described below. MDI
Welch,Sagheer,Salyam, Expires January 2007 [Page 6]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
Characteristic Curves can be used to determine operating limits for
the switch for a given number of input streams which have given
levels of MDI.
Switch implementation options such as how ASICs and backplanes are
partitioned, how architecture tradeoffs are made, how available
buffer memory is allocated, etc. can greatly affect the performance
being discussed. Unless otherwise specified, manufacturers should
quote performance based on worst case configurations with notes on
how streaming performance would be affected for other configurations.
For example, if selecting output ports on a switch results in worse
MDI Characteristic Curve performance if the active output ports are
on different blades (modules), then the manufacturer should note the
differences and tradeoffs. Alternatively, the device specification
can be made such that any configuration of port selection will
satisfy the (probably lower performance) specification. Testing time
can be shortened and the process of determining MDI Characteristic
Curves expedited if only the worst case numbers are specified. These
advantages must be weighed against under representing the device
performance in more ideal configurations. In any case, the
operational hardware and software configurations must be noted along
with the MDI Characteristic curve test results.
Testing and plotting MDI Characteristic Curves for the following
profiles is recommended based on the profiles’ prevalence in existing
deployments. These profiles can provide a minimum baseline for
device comparison. A relatively small number of test conditions are
included to minimize testing time and costs in order to encourage
testing and publication. Additional profiles may be used by vendors
and/or users to represent other applications as demand warrants.
Vendors may wish to expand this list in order to fully characterize a
device that is capable of a particularly demanding system
configuration. Users may wish to test a configuration that more
exactly represents an expected, intended, or anticipated network
load. Additional profiles may be included in testing results that
are adjusted for port count, stream count, and stream mix by
manufacturers, for example, to better represent a device’s
capabilities and/or by users to better represent a target user
configuration for testing and characterization. The listed profiles
in this section should be included in a device characterization so
that reference performance levels can be compared from manufacturer
to manufacturer. If the number of available switch ports or port
speeds or other switch implementation constraints prevent testing a
Welch,Sagheer,Salyam, Expires January 2007 [Page 7]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
particular profile, it should be noted in the testing results.
Testing configurations beyond the included profiles is encouraged.
Some other typical profiles likely to be of interest include
combinations of: SD/MPTS, HD/MPTS, SD/HD/MPTS, various ratios of
multicast to unicast as would be encountered in a service provider’s
head end consisting of both broadcast and on-demand flows, and flow
paths for device ports within a blade and between blades. Profiles
may also include redundant architectures based on mesh networks,
redundant rings, with and without QoS configurations, VLANs, etc. A
supplemental set of profiles would also likely include the reduced
bit rates for SD and HD streams when used with MPEG4 part 10
compression.
Note that the profiles are intended to describe an input traffic mix
representative of a service provider’s or user’s operating
environment. A switch’s performance might best be represented by a
family of MDI Characteristic Curves for typical or recommended switch
configurations. For example, a full featured metro class switch
might have sets of MDI Characteristic Curves for VLAN configurations,
converged network Diffserv priorities, etc. A lower cost, smaller
edge switch may only need MDI Characteristic Curves for the Broadcast
head end Profile.
The profiles in this section assume 1 Gb/s links which are in common
use today. Stream counts used may be scaled with bandwidth if other
link speeds are of interest. The stream bit rates specified in this
section are stream payload rates and may vary somewhat depending on
the specific encapsulation protocols selected.
3.1 All Standard Definition (SD) Streams:
This profile includes only SD streams at 3.75 Mb/s destined for
uncongested output port(s). 3.75 Mb/s is the typical rate of
transmission at which MPEG2 compressed standard definition streams
are transported today in cable and telco environments. This profile
assumes that all input streams can be destined for any single output
port and that the output port will never be oversubscribed.
Characteristic Curves for input streams in the following cases are
required. Each row below varies the number of input streams per
input port. For each of these combinations, the Characteristics are
plotted when the streams are transmitted out 1, 2, 4, 8, or 16 ports.
This profile is representative of head end applications where all
Welch,Sagheer,Salyam, Expires January 2007 [Page 8]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
streams are encoded as SD and the switch is aggregating potentially
hundreds of streams from off-air receivers and encoders or other
feeds. The aggregated streams are then forwarded to possibly a metro
or regional distribution system using a variety of possible ring,
mesh, and point-to-point architectures. In this application the
streams use multicast destination addresses. The switch may or may
not be specifically configured for multicast operation. The number
of streams per output port should be equal and should sum to the
total number of input streams.
No. of switch No. of Streams per No. of switch
input ports Input port output ports
1 250 1, 2, 4, 8, 16
2 125 1, 2, 4, 8, 16
4 62 1, 2, 4, 8, 16
8 31 1, 2, 4, 8, 16
16 15 1, 2, 4, 8, 16
32 7 1, 2, 4, 8, 16
3.2 All High Definition (HD) Streams
This profile includes only HD streams at 15.0 Mb/s destined for
uncongested output port(s). 15 Mb/s is the typical rate of
transmission at which MPEG2 compressed high definition streams are
transported today in cable and telco environments. This profile
assumes that all input streams can be destined for any single output
port and that the output port will never be oversubscribed. MDI
Characteristic Curves generated for the following cases are required.
Each row below varies the number of input streams per input port.
For each of these combinations, the MDI Characteristic Curves are
plotted when the streams are transmitted out 1, 2, 4, 8, or 16 ports.
This profile is representative of head end applications where all
streams are encoded as HD and the switch is aggregating potentially
hundreds of streams from off-air receivers and encoders or other
feeds. The aggregated streams are then forwarded to possibly a metro
or regional distribution system using a variety of possible ring,
mesh, and point-to-point architectures. In this application the
Welch,Sagheer,Salyam, Expires January 2007 [Page 9]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
streams use multicast destination addresses. The switch may or may
not be specifically configured for multicast operation. The number
of streams per output port should be equal and should sum to the
total number of input streams.
No. of switch No. of Streams per No. of switch
input Ports input port output ports
1 60 1, 2, 4, 8, 16
2 30 1, 2, 4, 8, 16
4 15 1, 2, 4, 8, 16
8 7 1, 2, 4, 8, 16
Welch,Sagheer,Salyam, Expires January 2007 [Page 10]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
3.3 All Multi Program Transport Streams (MPTS)
This profile includes only MPTS streams at 38 Mb/s destined for
uncongested output port(s). 38 Mb/s is the typical rate of
transmission at which multiple programs multiplexed together at a
head end into a Multi Program Transport Stream (MPTS) are transported
today in cable and telco environments. This profile assumes that all
input streams can be destined for any single output port and that the
output port will never be oversubscribed. MDI Characteristic Curves
for the following cases are required. Each row below varies the
number of input streams per input port. For each of these
combinations, the MDI Characteristic Curves are plotted when the
streams are transmitted out 1, 2, 4, 8, or 16 ports. This profile is
representative of head end applications where all streams are
multiplexed into MPTSs and the switch is aggregating these streams
from the multiplexers. The aggregated streams are then forwarded to
possibly a metro or regional distribution system using a variety of
possible ring, mesh, and point-to-point architectures. In this
application the streams use unicast destination addresses to effect
transport to a downstream demultiplexer before transport to an end
user. The number of streams per output port should be equal and
should sum to the total number of input streams.
No. of switch No. of Streams per No. of switch
input ports input port output ports
1 25 1, 2, 4, 8
2 12 1, 2, 4, 8
4 6 1, 2, 4, 8
8 2 1, 2, 4, 8
3.4 SD/HD Stream Mix
This profile includes an SD/HD stream mix of 120 streams of SD 3.75
Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested
output port(s). A mix of SD and HD streams are typical in many
current head ends where HD streams are still in limited availability
or must be limited due to downstream distribution plant limitations.
This profile assumes that all input streams can be destined for any
Welch,Sagheer,Salyam, Expires January 2007 [Page 11]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
single output port and that the output port will never be
oversubscribed. MDI Characteristic Curves for the following cases
are required. Each row below varies the number of input streams per
input port. For each of these combinations, the MDI Characteristic
Curves are plotted when the streams are transmitted out 1, 2, 4, or 8
ports. This profile is representative of head end applications where
there are a mix of streams encoded as SD and HD and the switch is
aggregating potentially hundreds of streams from off-air receivers
and encoders or other feeds. The aggregated streams are then
forwarded to possibly a metro or regional distribution system using a
variety of possible ring, mesh, and point-to-point architectures. In
this application the streams use multicast destination addresses.
The switch may or may not be specifically configured for multicast
operation. The number of streams per output port should be equal and
should sum to the total number of input streams.
No. of switch No. of Streams per No. of switch
Input ports input port (SD/HD) output ports
1 120/30 1, 2, 4, 8
2 64/16 1, 2, 4, 8
4 32/8 1, 2, 4, 8
8 16/4 1, 2, 4, 8
3.5 VoD Headend Switch
This profile includes an SD/HD stream mix of 120 streams of SD 3.75
Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested
output port(s). In this application the streams use unicast
destination addresses. This profile requires a minimum of 8
simultaneously active input ports representing a small to moderate
size VoD server array serving a mix of SD and HD streams. This
profile assumes that all input streams from an input port can be
destined for any output port and that the output ports will never be
oversubscribed. MDI Characteristic Curves for the following cases
are required. Each row below varies the number of active input
ports. For each row of combinations, the MDI Characteristic Curves
are plotted when the streams are transmitted out 8, 16, 24, … up to
the maximum number of blade and/or switch output ports. The maximum
number of blade and/or switch output ports is determined either by
the device manufacturer based on device physical or performance
Welch,Sagheer,Salyam, Expires January 2007 [Page 12]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
limits, or by the user based on a maximum projected configuration.
The number and type of streams per output port should be equal and
should sum to the number of input streams per input port.
Direct mapping of input port traffic to one other output port is
used.
No. of switch No. of Streams per No. of switch
Input ports input port (SD/HD) output ports
8 120/30 8
16 120/30 16
24 120/30 24
3.6 Backbone Ring hub site Switch
This profile includes two 10 Gigabit ports representing connections
to a backbone with a traffic mix of SD/HD streams, VoIP streams, and
various levels of non-streaming data with a mix of broadcast and
unicast stream addressing. This profile is intended to represent
converged backbone traffic, say on a metropolitan ring, with a hub
site accessing and forwarding the backbone traffic for local
distribution thus representing a midstream location in a provider
network. This profile assumes that 50% of the backbone traffic is
video, 20% is VoIP, 20% is Data and that most traffic arriving on one
backbone port is forwarded to another backbone port while a
percentage of the traffic is forwarded to local Gigabit ports and not
forwarded to the second backbone port.
Priority marking and handling of classes of traffic such as video,
voice, and data may be optionally included in this profile.
This profile includes an SD/HD stream mix of 1000 streams of SD 3.75
Mb/s, 300 streams of HD, 10000 VoIP streams at 64 Kb/s, and 2 Gb/s
data traffic on the inbound backbone port with all streams employing
unicast addressing. The data traffic should consist of 25%
utilization each of 64, 512, and 1024, and 1500 byte packets. This
profile requires a minimum of 2 active Gigabit output ports
representing the traffic being forwarded to the hub site. The hub
site traffic consists of 100 SD streams, 20 HD streams, 1000 VoIP
streams, and 2% IP data utilization. The remainder of the traffic is
forwarded to the downstream backbone port. Thus, direct mapping of
Welch,Sagheer,Salyam, Expires January 2007 [Page 13]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
input port traffic to three other output ports, the backbone output
port and two local hub output ports, is defined.
This profile assumes that the streams from an input port destined for
local Gigabit port(s) will never oversubscribe those ports.
Likewise, the profile assumes that the streams from an input port
destined for a backbone output port will never oversubscribe that
port. MDI Characteristic Curves for the output port cases described
below are required.
10 Gb/s backbone ports load:
No. of Data No. of
backbone No. of Streams per Util backbone output
Input ports Input port (SD/HD/VoIP) % ports
1 800/200/10000 20 1
1 Gb/s Hub ports load:
No. of Streams per Data No. of
output port (SD/HD/VoIP) Util Gigabit output
% ports
50/10/500 1% 2
This represents a backbone load of approximately 3Gb/s (SD video) +
3Gb/s (HD video) + 1Gb/s (VoIP) + 2Gb/s (data) = 9 Gb/s and a hub
Gigabit drop port with 10% of the backbone load.
3.7 SD/HD Stream Multicast Edge Switch
This profile includes an SD/HD stream mix of 120 streams of SD 3.75
Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested
output port(s). A mix of SD and HD streams are typical in many
current head ends where HD streams are still in limited availability
or must be limited due to downstream plant limitations. This profile
assumes that all input streams can be destined for any single output
port and that the output port will never be oversubscribed. MDI
Characteristic Curves for the following cases are required. Each row
Welch,Sagheer,Salyam, Expires January 2007 [Page 14]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
below varies the number of input streams per input port. For each of
these combinations, the MDI Characteristic Curves are plotted when
the streams are transmitted out 1, 2, 4, 8 ports and the maximum
number of available output ports for the device under test. This
profile is representative of edge multicast applications where there
are a mix of streams encoded as SD and HD and the switch is
distributing potentially hundreds of streams from up stream
distribution networks. The distributed streams are then forwarded to
STBs directly or via DSLAMs. In this application the streams use
multicast distribution mechanisms and the switch must replicate
inbound streams to all requesting ports. This profile should include
the configuration in which all output ports are requesting all
possible input streams simultaneously.
No. of switch input No. of Streams per No. out ports
Ports input port (SD/HD)
1 120/30 1, 2, 4, 8, max
2 60/15 1, 2, 4, 8, max
4 30/7 1, 2, 4, 8, max
8 15/3 1, 2, 4, 8, max
4.
Test Stream Source
The stream sources used for testing should be capable of generating
the required number of test streams at the appropriate bit rates and
with appropriate addressing and QoS tagging flexibility as described
in the test Profiles. They should be able to generate the streams
with industry common, streaming media protocol encapsulations such as
ISO 13818 Transport Streams over UDP/IP and/or RTP as identified as
part of the profile description used for testing. The encapsulations
and resultant packet sizes should represent realistic traffic
characteristics as described in the profile.
The stream sources must also have the capability to increase the MDI
Delay Factor (cumulative jitter) of the test streams to be able to
create the output MDI Characteristic Curves. A given Delay Factor
can be created with a variety of specific inter-packet gap times and
burst sizes. For the tests in this document, the stream sources
should use the minimum inter-packet gap times compliant with the link
layer protocol employed to create elevated Delay Factors. This
Welch,Sagheer,Salyam, Expires January 2007 [Page 15]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
causes the maximum utilization of switch queues when the switches are
subject to multiple coincident input streams.
The stream sources must also have the capability to synchronize the
test streams such that they arrive at the switch being tested
simultaneously. This causes the maximum utilization of switch queues
and represents worst case test conditions. Such transient conditions
can be encountered on realistic deployed networks.
5.
Test Stream Analyzer
The output stream analyzer used for testing should be capable of
identifying analyzing the MDI for all streams simultaneously that the
device or system under test produces.
6.
Test Procedure
Configure a test profile with appropriate stimulus streams and stream
monitors and begin transmission.
Run each profile test for two minutes to assure no packet loss with
this load and then record results. Note that since an MDI
Characteristic curve point must have no packet loss, a test can be
terminated as soon as packet loss is detected (that is, as soon as
the media loss rate (MLR) is greater than zero.) If no loss is
detected, the test should run for at least two minutes to determine
the maximum stream DF. After at least two minutes of operation,
determine the test stream which has experienced the largest increase
in DF by comparing each test stream’s maximum output DF measured
during the test and subtracting that test stream’s input DF. Record
that output stream’s maximum DF on the MDI Characteristic Curve
described below. For profiles that include multiple types of input
streams such as SD, HD, etc., an MDI Characteristic Curve for each
type of stream is required; therefore, the comparison of input to
output DF must be executed for each stream of the same type.
In the event of loss, decrease the number of input test streams and
perform the measurement described above to acquire other DF points on
the MDI Characteristic Curve.
Welch,Sagheer,Salyam, Expires January 2007 [Page 16]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
Begin with the input streams’ nominal DF and increase the input
streams’ Delay Factor in steps of 2ms and repeat the Profile’s test
to obtain other members of the family of Characteristic Curves for
the Profile, as shown below.
7.
MDI Characteristic Curve
The MDI Characteristic Curve is a graph of the output streams’ delay
factor vs. the number of lossless output streams that can be achieved
for a given test profile, while holding the input Delay Factor
constant as shown below.
Output
DF (ms)
9| ^^^^^^(input DF=5)
8| ^^
7| ^
6| ^
5|^^ ********(input DF=3)
4| ***
3|*****
2|
1|
+----+---+---+---+---+---Lossless stream count
10 20 30 40 50
Example MDI Characteristic Curve
For the above example, note that for ideal input streams which have
low cumulative jitter (DF = 3 curve), 40 is the maximum number of
output streams without loss that the tested switch delivers since
this is where the input DF = 3 curve terminates. When the streams’
input DF is increased to 5 ms, the tested switch delivers only up to
30 streams without loss. Note also that as the number of input
streams increase, the output DF also typically changes. Referring to
the input DF=5 curve, while the maximum number of lossless streams is
30, at 30 output streams the output DF has reached 9 ms vs 5 ms when
the switch is less loaded and delivering less streams.
Welch,Sagheer,Salyam, Expires January 2007 [Page 17]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
For profiles that include multiple types of input streams such as SD,
HD, etc., an MDI Characteristic Curve for each type of stream is
required. Optionally, separate MDI Characteristic Curves may be
generated based on other stream characteristics such as priority,
output port number, destination address type, etc.
8.
Application of Test Data
MDI Characteristic curves can be used in conjunction with stream
source characteristics and stream consuming device characteristics to
determine whether the transport system is adequately configured to
deliver acceptable stream quality.
For example, by measuring the worst case DF of a network device’s
input stream and applying that DF to the network device’s MDI
Characteristic Curve, the output MDI curve for that device indicates
the worst case output performance to be expected for that stream.
That value can be compared to the requirements of the downstream
stream consuming device to determine if it has an adequate dejitter
buffer to deliver lossless performance.
9.
Summary
A supplementary procedure for characterizing the quality of streaming
media flows by network switches and systems was described. The
results of the testing are documented in a series of Characteristic
Curves representing the MDI footprint of the device under test.
These curves describe a realistic, practical, and convenient method
of describing the cumulative jitter performance of the device. They
can be used to compare the performance of switches of various design,
configuration, setup, and manufacturer and to infer whether so
characterized devices will deliver the needed streaming media
quality.
For applications other than streaming media performance, previously
existing methods described above can and should be used to represent
switch performance.
The Media Delivery Index metric is employed as the media quality
indicator. It conveniently captures cumulative stream jitter and
Welch,Sagheer,Salyam, Expires January 2007 [Page 18]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
packet loss performance through its delay factor and media loss rate
components.
Comments are solicited and should be addressed to the authors via
Jim.Welch@IneoQuest.com.
10.
Security Considerations
The measurements identified in this document do not directly affect
the security of a network or user. Actions taken in response to
these measurements which may affect the available bandwidth of the
network or availability of a service are beyond the scope of this
document.
While most applications for the testing described is envisioned to be
performed in a laboratory or pre-deployment setting, performing the
measurements described in this document only requires examination of
payload header information, such as MPEG transport stream headers
and/or RTP headers to determine nominal stream bit rate and sequence
number information. Content may be encrypted without affecting these
measurements. Therefore, content privacy is not expected to be a
concern even on publicly accessible networks.
11.
Informative References
i1. R. Braden et al., `Resource Reservation Protocol ` Version 1
Functional Specification`, RFC 2205, 1997.
i2. V. Raisanen, `Implementing Service Quality in IP Networks`, John
Wiley & Sons Ltd., 2003.
i3. Welch, Clark, ‘A Proposed Media Delivery Index’, RFC 4445, 2006.
i4. Van Jacobson, Kathleen Nichols, Kedar Poduri, Internet Draft
draft-ietf-diffserv-pdb-vw-00.txt, July 2000, “The ‘Virtual Wire’
Per-Domain Behavior”
i5. S. Bradner, `Benchmarking Methodology for Network Interconnect
Devices`, RFC2544, 1999.
i6. R. Mandeville, `Benchmarking Methodology for LAN Switching
Devices`, RFC2889, 2000.
i7. `Perceptual evaluation of speech quality (PESQ): An objective
method for end-to-end speech quality assessment of narrow-band
telephone networks and speech codecs`, ITU-T P.862, February 2001
i8. `Network performance objectives for IP-based services`, ITU-T
Y.1541
Welch,Sagheer,Salyam, Expires January 2007 [Page 19]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
i9. `The E-model, a computational model for use in transmission
planning`, ITU-T G.107, February 2003
12.
Acknowledgments
The authors gratefully acknowledge the contributions of Marc Todd and
Jesse Beeson of IneoQuest Technologies, Inc.
13.
Authors' Address
James Welch
IneoQuest Technologies, Inc
170 Forbes Blvd
Mansfield, Massachusetts 02048
508 618 0312
Jim.Welch@ineoquest.com
Muhammad Waris Sagheer
Cisco Systems, Inc
170 West Tasman Drive
San Jose, California 95134-1706
408 853 6682
waris@cisco.com
Sameer Satyam
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
408 525 4697
sameers@cisco.com
Javed Asghar
Cisco Systems
170 West Tasman Drive
San Jose, California 95134-1706
408 853 4078
jasghar@cisco.com
Syed Nawaz
Cisco Systems
170 West Tasman Drive
Welch,Sagheer,Salyam, Expires January 2007 [Page 20]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
San Jose, California 95134-1706
408 853 5453
snawaz@cisco.com
Andre Dufour
Agilent Technologies
2500-4710 Kingsway
Burnaby, BC V5H 4M2
Canada
604 454 3405
adufour@agilent.com
14.
Copyright Notice
Copyright (C) The Internet Society (2006). This document is subject to
the rights, licenses and restrictions contained in BCP 78, and except
as set forth therein, the authors retain all their rights.
15.
Disclaimer
This document and the information contained herein are provided on an
'AS IS' basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.'
16.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the ISOC's procedures with respect to rights in ISOC Documents can
be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
Welch,Sagheer,Salyam, Expires January 2007 [Page 21]
Asghar,Nawaz, Dufour
Switch Testing for Streaming Media Applications July 2006
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Welch,Sagheer,Salyam, Expires January 2007 [Page 22]
Asghar,Nawaz, Dufour
