| Network Working Group | A. Morton |
| Internet-Draft | AT&T Labs |
| Intended status: Standards Track | M. Bagnulo |
| Expires: October 05, 2014 | UC3M |
| P. Eardley | |
| BT | |
| April 03, 2014 |
Active Performance Metric Sub-Registry
draft-ietf-ippm-registry-active-00
This memo defines the Active Performance Metrics sub-registry of the Performance Metric Registry. This sub-registry will contain Active Performance Metrics, especially those defined in RFCs prepared in the IP Performance Metrics (IPPM) Working Group of the IETF, and possibly applicable to other IETF metrics. Three aspects make IPPM metric registration difficult: (1) Use of the Type-P notion to allow users to specify their own packet types. (2) Use of flexible input variables, called Parameters in IPPM definitions, some of which determine the quantity measured and others of which should not be specified until execution of the measurement. (3) Allowing flexibility in choice of statistics to summarize the results on a stream of measurement packets.
This memo proposes a way to organize registry entries into columns that are well-defined, permitting consistent development of entries over time (a column may marked NA if it is not applicable for that metric). The design is intended to foster development of registry entries based on existing reference RFCs, whilst each column serves as a check-list item to avoid omissions during the registration process. Every entry in the registry, before IANA action, requires Expert review as defined by concurrent IETF work in progress "Registry for Performance Metrics" (draft-manyfolks-ippm-metric-registry).
The document contains two examples: a registry entry for an active Performance Metric entry based on RFC3393 and RFC5481, and a registry entry for an end-point Performance Metric based on RFC 7003. The examples are for Informational purposes and do not create any entry in the IANA registry.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on October 05, 2014.
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
[ISSUES
1. REAL-TIME OR INPUT PARAMETER [CONSISTENT WITH REGISTRY I-D] closed - just Parameter
2. CHANGED STREAM PARAMETER TO STREAM INPUT PARAMETER I didn't find any instances of this change - closed
3. I PREFER KEEPING THE CATEGORY-COLUMN HIERARCHY - ok we keep it
4. RATHER THAN BLANK COLUMNS, SHOULD WE HAVE 'NOT APPLICABLE' [MAYBE EVEN IANA REGISTERED??] sounds good to Al, used NA.
5. THE EXAMPLES ARE INFORMATIONAL NOT STANDARDS TRACK yes of course - -Closed.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Note: Efforts to synchronize terminology with [I-D.manyfolks-ippm-metric-registry] will likely be incomplete until both drafts are stable.
This memo defines the Active Performance Metrics sub-registry of the Performance Metric Registry. This sub-registry will contain Active Performance Metrics, especially those defined in RFCs prepared in the IP Performance Metrics (IPPM) Working Group of the IETF, according to their framework [RFC2330]. Three aspects make IPPM metric registration difficult: (1) Use of the Type-P notion to allow users to specify their own packet types. (2) Use of Flexible input variables, called Parameters in IPPM definitions, some which determine the quantity measured and others which should not be specified until execution of the measurement. (3) Allowing flexibility in choice of statistics to summarize the results on a stream of measurement packets. This memo uses terms and definitions from the IPPM literature, primarily [RFC2330], and the reader is assumed familiar with them or may refer questions there as necessary.
This sub-registry is part of the Performance Metric Registry [I-D.manyfolks-ippm-metric-registry] which specifies that all sub-registries must contain at least the following fields: the identifier, the name, the status, the requester, the revision, the revision date, the description for each entry, and the reference specifications used as the foundation for the Registered Performance Metric (see [I-D.manyfolks-ippm-metric-registry]).
Although there are several standard templates for organizing specifications of performance metrics (see [RFC2679] for an example of the traditional IPPM template, based to large extent on the Benchmarking Methodology Working Group's traditional template in [RFC1242], and see [RFC6390] for a similar template), none of these templates was intended to become the basis for the columns of an IETF-wide registry of metrics. As we examined the aspects of metric specifications which need to be registered, it was clear that none of the existing metric templates fully satisfies the particular needs of a registry.
One clear motivation for having such a registry is to allow a controller to request a measurement agent to execute a measurement using a specific metric (see [I-D.ietf-lmap-framework]). Such a request can be performed using any control protocol that refers to the value assigned to the specific metric in the registry. Similarly, the measurement agent can report the results of the measurement and by referring to the metric value it can unequivocally identify the metric that the results correspond to.
There was a previous attempt to define a metric registry RFC 4148 [RFC4148]. However, it was obsoleted by RFC 6248 [RFC6248] because it was "found to be insufficiently detailed to uniquely identify IPPM metrics... [there was too much] variability possible when characterizing a metric exactly" which led to the RFC4148 registry having "very few users, if any".
Our approach learns from this by tightly defining each entry in the registry with only a few parameters open, if any. The idea is that entries in the registry represent different measurement methods. Each may require run-time parameters to set factors like source and destination addresses, which do not change the fundamental nature of the measurement and can be set just before measurement execution. The downside of this approach is that it could result in a large number of entries in the registry. We believe that less is more in this context - it is better to have a reduced set of useful metrics rather than a large set of metrics with questionable usefulness. Therefore it is required for all registries within the Performance Metric Registry (see [I-D.manyfolks-ippm-metric-registry]) that the registry only includes commonly used metrics that are well defined; hence we require expert review policies for the approval and assignment of entries in this sub-registry.
There are several side benefits of having a registry with well-chosen entries. First, the registry could serve as an inventory of useful and used metrics that are normally supported by different implementations of measurement agents. Second, the results of the metrics would be comparable even if they are performed by different implementations and in different networks, as the metric and method is unambiguously defined.
The registry constitutes a key component of a 'Characterization Plan'. It describes various factors that need to be set by the party controlling the measurements, for example: specific values for the parameters associated with the selected registry entry (for instance, source and destination addresses); and how often the measurement is made. The Characterization Plan determines the individual Measurement Tasks which Measurement Agents will be instructed to do and which they then execute autonomously.
Measurement Instructions might look something like: "Dear measurement agent: Please start test DNS(example.com) and RTT(server.com,150) every day at 2000 GMT. Run the DNS test 5 times and the RTT test 50 times. Do that when the network is idle. Generate both raw results and 99th percentile mean. Send measurement results to collector.com in IPFIX format". The Characterization Plan depends on the requirements of the controlling party. For instance the broadband consumer might want a one-off measurement made immediately to one specific server; a regulator might want the same measurement made once a day until further notice to the 'top 10' servers; whilst an operator might want a varying series of tests (some of which will be beyond those defined in an IETF registry) as determined from time to time by their operational support system. While the registries defined in this document help to define the Characterization Plan, its full specification falls outside the scope of this document, and other IETF work as currently chartered.
[I-D.manyfolks-ippm-metric-registry] defines the overall structure for a Performance Metric Registry and provides guidance for defining a sub registry.
This document defines the Active Performance Metrics Sub-registry; active metrics are those where the packets measured have been specially generated for the purpose.
A row in the registry corresponds to one Registered Performance Metric, with entries in the various columns specifying the metric. Section 3 defines the columns for a Registered Active Performance Metric.
As discussed in [I-D.manyfolks-ippm-metric-registry], each entry (row) must be tightly defined; the definition must leave open only a few parameters that do not change the fundamental nature of the measurement (such as source and destination addresses), and so promotes comparable results across independent implementations. Also, each registered entry must be based on existing reference RFCs (or other standards) for performance metrics, and must be operationally useful and have significant industry interest. This is ensured by expert review for every entry before IANA action.
This section defines the categories and columns of the registry. Below, categories are described at the 3.x heading level, and columns are at the 3.x.y heading level. The Figure below illustrates this organization. An entry (row) therefore gives a complete description of a Registered Metric.
Registry Categories and Columns, shown as
Category
------------------
Column | Column |
Common Registry Indexes and Information
---------------------------------------------------------------------
ID | Name | Status |Request | Rev| Rev.Date | Description | Ref Spec|
Metric Definition
-----------------------------------------
Reference Definition | Fixed Parameters |
Method of Measurement
----------------------------------------------------------------
Reference Method | Stream Type | Output | Output | Run-time |
| and Parameters | Type | Units | Param
Comments and Remarks
--------------------
Each column serves as a check-list item and helps to avoid omissions during registration and expert review. In some cases an entry (row) may have some columns without specific entries, marked Not Applicable (NA).
This category has multiple indexes to each registry entry. It is defined in [I-D.manyfolks-ippm-metric-registry]:
Defined in [I-D.manyfolks-ippm-metric-registry]. In order to have the document self contained, we could copy the definition from [I-D.manyfolks-ippm-metric-registry] here, but i guess we should do that once the definition in [I-D.manyfolks-ippm-metric-registry] is stable.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment than above.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment than above.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment than above.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment than above.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment than above.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment as the previous.
Defined in [I-D.manyfolks-ippm-metric-registry], same comment as the previous.
This category includes columns to prompt all necessary details related to the metric definition, including the RFC reference and values of input factors, called fixed parameters, which are left open in the RFC but have a particular value defined by the performance metric.
This entry provides references to relevant sections of the RFC(s) defining the metric, as well as any supplemental information needed to ensure an unambiguous definition for implementations.
Fixed Parameters are input factors whose value must be specified in the Registry. The measurement system uses these values.
Where referenced metrics supply a list of Parameters as part of their descriptive template, a sub-set of the Parameters will be designated as Fixed Parameters. For example, Fixed Parameters determine most or all of the IPPM Framework convention "packets of Type-P" as described in [RFC2330], such as transport protocol, payload length, TTL, etc.
A Parameter which is Fixed for one Registry entry may be designated as a Run-time Parameter for another Registry entry.
This category includes columns for references to relevant sections of the RFC(s) and any supplemental information needed to ensure an unambiguous method for implementations.
This entry provides references to relevant sections of the RFC(s) describing the method of measurement, as well as any supplemental information needed to ensure unambiguous interpretation for implementations referring to the RFC text.
Principally, two different streams are used in IPPM metrics, Poisson distributed as described in [RFC2330] and Periodic as described in [RFC3432]. Both Poisson and Periodic have their own unique parameters, and the relevant set of values is specified in this column.
Each entry for this column contains the following information:
The simplest example of stream specification is Singleton scheduling, where a single atomic measurement is conducted. Each atomic measurement could consist of sending a single packet (such as a DNS request) or sending several packets (for example, to request a webpage). Other streams support a series of atomic measurements in a "sample", with a schedule defining the timing between each transmitted packet and subsequent measurement.
For entries which involve a stream and many singleton measurements, a statistic may be specified in this column to summarize the results to a single value. If the complete set of measured singletons is output, this will be specified here.
Some metrics embed one specific statistic in the reference metric definition, while others allow several output types or statistics.
Each entry in the output type column contains the following information:
The output type defines the type of result that the metric produces. It can be the raw results or it can be some form of statistic. The specification of the output type must define the format of the output. In some systems, format specifications will simplify both measurement implementation and collection/storage tasks. Note that if two different statistics are required from a single measurement (for example, both "Xth percentile mean" and "Raw"), then a new output type must be defined ("Xth percentile mean AND Raw").
The measured results must be expressed using some standard dimension or units of measure. This column provides the units.
When a sample of singletons (see [RFC2330] for definitions of these terms) is collected, this entry will specify the units for each measured value.
Run-Time Parameters are input factors that must be determined, configured into the measurement system, and reported with the results for the context to be complete. However, the values of these parameters is not specified in the Registry, rather these parameters are listed as an aid to the measurement system implementor or user (they must be left as variables, and supplied on execution).
Where metrics supply a list of Parameters as part of their descriptive template, a sub-set of the Parameters will be designated as Run-Time Parameters.
The Data Format of each Run-time Parameter SHALL be specified in this column, to simplify the control and implementation of measurement devices.
Examples of Run-time Parameters include IP addresses, measurement point designations, start times and end times for measurement, and other information essential to the method of measurement.
Besides providing additional details which do not appear in other categories, this open Category (single column) allows for unforeseen issues to be addressed by simply updating this Informational entry.
This section is Informational.
This section gives an example registry entry for the active metric described in [RFC3393], on Packet Delay Variation.
This category includes multiple indexes to the registry entries, the element ID and metric name.
An integer having enough digits to uniquely identify each entry in the Registry.
A metric naming convention is TBD.
One possibility based on IPPM's framework is:
Act_IP-UDP-One-way-pdv-95th-percentile-Poisson
An assessment of packet delay variation with respect to the minimum delay observed on the stream.
This category includes columns to prompt the entry of all necessary details related to the metric definition, including the RFC reference and values of input factors, called fixed parameters.
See sections 2.4 and 3.4 of [RFC3393]. Singleton delay differences measured are referred to by the variable name "ddT".
Since the metric's reference supplies a list of Parameters as part of its descriptive template, a sub-set of the Parameters have been designated as designated as Fixed Parameters for this entry.
This category includes columns for references to relevant sections of the RFC(s) and any supplemental information needed to ensure an unambiguous methods for implementations.
See section 2.6 and 3.6 of [RFC3393] for singleton elements.
Poisson distributed as described in [RFC2330], with the following Parameters.
See section 4.3 of [RFC3393] for details on the percentile statistic.
The percentile = 95.
Data format is a 32-bit unsigned floating point value.
Individual results (singletons) should be represented by the following triple
[RFC5905] (32 bits) and is as follows: the first 16 bits represent the *signed* integer number of seconds; the next 16 bits represent the fractional part of a second.
if needed. The result format for ddT is *similar to* the short format in
See section 3.3 of [RFC3393] for singleton elements.
[RFC2330] recommends that when a time is given, it will be expressed in UTC.
The timestamp format (for T, Tf, etc.) is the same as in [RFC5905] (64 bits) and is as follows: the first 32 bits represent the unsigned integer number of seconds elapsed since 0h on 1 January 1900; the next 32 bits represent the fractional part of a second that has elapsed since then.
Since the metric's reference supplies a list of Parameters as part of its descriptive template, a sub-set of the Parameters have been designated as Run-Time Parameters for this entry. In related registry entries, some of the parameters below may be designated as Fixed Parameters instead.
Lost packets represent a challenge for delay variation metrics. See section 4.1 of [RFC3393] and the delay variation applicability statement[RFC5481] for extensive analysis and comparison of PDV and an alternate metric, IPDV.
This section is Informational.
This section gives an example registry entry for the end-point metric described in RFC 7003 [RFC7003], for RTCP-XR Burst/Gap Discard Metric reporting.
This category includes multiple indexes to the registry entries, the element ID and metric name.
An integer having enough digits to uniquely identify each entry in the Registry.
A metric naming convention is TBD.
TBD.
This category includes columns to prompt the entry of all necessary details related to the metric definition, including the RFC reference and values of input factors, called fixed parameters. Section 3.2 of [RFC7003] provides the reference information for this category.
Packets Discarded in Bursts:
The total number of packets discarded during discard bursts. The measured value is unsigned value. If the measured value exceeds 0xFFFFFD, the value 0xFFFFFE MUST be reported to indicate an over-range measurement. If the measurement is unavailable, the value 0xFFFFFF MUST be reported.
Fixed Parameters are input factors that must be determined and embedded in the measurement system for use when needed. The values of these parameters is specified in the Registry.
Threshold: 8 bits, set to value = 3 packets.
The Threshold is equivalent to Gmin in [RFC3611], i.e., the number of successive packets that must not be discarded prior to and following a discard packet in order for this discarded packet to be regarded as part of a gap. Note that the Threshold is set in accordance with the Gmin calculation defined in Section 4.7.2 of [RFC3611].
Interval Metric flag: 2 bits, set to value 11=Cumulative Duration
This field is used to indicate whether the burst/gap discard metrics are Sampled, Interval, or Cumulative metrics [RFC6792]:
I=10: Interval Duration - the reported value applies to the most recent measurement interval duration between successive metrics reports.
I=11: Cumulative Duration - the reported value applies to the accumulation period characteristic of cumulative measurements.
Senders MUST NOT use the values I=00 or I=01.
This category includes columns for references to relevant sections of the RFC(s) and any supplemental information needed to ensure an unambiguous methods for implementations. For the Burst/Gap Discard Metric, it appears that the only guidance on methods of measurement is in Section 3.0 of [RFC7003] and its supporting references. Relevant information is repeated below, although there appears to be no section titled "Method of Measurement" in [RFC7003].
Metrics in this block report on burst/gap discard in the stream arriving at the RTP system. Measurements of these metrics are made at the receiving end of the RTP stream. Instances of this metrics block use the synchronization source (SSRC) to refer to the separate auxiliary Measurement Information Block [RFC6776], which describes measurement periods in use (see [RFC6776], Section 4.2).
This metrics block relies on the measurement period in the Measurement Information Block indicating the span of the report. Senders MUST send this block in the same compound RTCP packet as the Measurement Information Block. Receivers MUST verify that the measurement period is received in the same compound RTCP packet as this metrics block. If not, this metrics block MUST be discarded.
Since RTCP-XR Measurements are conducted on live RTP traffic, the complete description of the stream is contained in SDP messages that proceed the establishment of a compatible stream between two or more communicating hosts. See Run-time Parameters, below.
The output type defines the type of result that the metric produces.
The measured results are apparently expressed in packets, although there is no section of [RFC7003] titled "Metric Units".
Run-Time Parameters are input factors that must be determined, configured into the measurement system, and reported with the results for the context to be complete. However, the values of these parameters is not specified in the Registry, rather these parameters are listed as an aid to the measurement system implementor or user (they must be left as variables, and supplied on execution).
The Data Format of each Run-time Parameter SHALL be specified in this column, to simplify the control and implementation of measurement devices.
SSRC of Source: 32 bits As defined in Section 4.1 of [RFC3611].
SDP Parameters: As defined in [RFC4566]
Session description v= (protocol version number, currently only 0)
o= (originator and session identifier : username, id, version number, network address)
s= (session name : mandatory with at least one UTF-8-encoded character)
i=* (session title or short information) u=* (URI of description)
e=* (zero or more email address with optional name of contacts)
p=* (zero or more phone number with optional name of contacts)
c=* (connection information—not required if included in all media)
b=* (zero or more bandwidth information lines) One or more Time descriptions ("t=" and "r=" lines; see below)
z=* (time zone adjustments)
k=* (encryption key)
a=* (zero or more session attribute lines)
Zero or more Media descriptions (each one starting by an "m=" line; see below)
m= (media name and transport address)
i=* (media title or information field)
c=* (connection information — optional if included at session level)
b=* (zero or more bandwidth information lines)
k=* (encryption key)
a=* (zero or more media attribute lines — overriding the Session attribute lines)
An example Run-time SDP description follows:
v=0
o=jdoe 2890844526 2890842807 IN IP4 192.0.2.5
s=SDP Seminar i=A Seminar on the session description protocol
u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe)
c=IN IP4 233.252.0.12/127
t=2873397496 2873404696
a=recvonly
m=audio 49170 RTP/AVP 0
m=video 51372 RTP/AVP 99
a=rtpmap:99 h263-1998/90000
TBD.
This section is Informational. (?)
This section gives an example registry entry for the <type of metric and specification reference> .
This category includes multiple indexes to the registry entries, the element ID and metric name.
An integer having enough digits to uniquely identify each entry in the Registry.
A metric naming convention is TBD.
A metric Description is TBD.
This category includes columns to prompt the entry of all necessary details related to the metric definition, including the RFC reference and values of input factors, called fixed parameters.
<possible section reference>.
Fixed Parameters are input factors that must be determined and embedded in the measurement system for use when needed. The values of these parameters is specified in the Registry.
<list fixed parameters>
This category includes columns for references to relevant sections of the RFC(s) and any supplemental information needed to ensure an unambiguous methods for implementations.
For <metric>.
<section reference>
<list of stream parameters>.
<references>
The output type defines the type of result that the metric produces.
The measured results are expressed in <units>,
<section reference>.
Run-time Parameters are input factors that must be determined, configured into the measurement system, and reported with the results for the context to be complete.
<list of run-time parameters>
<reference(s)>.
Additional (Informational) details for this entry
This registry has no known implications on Internet Security.
IANA is requested to create The Active Performance Metric Sub-registry within the Performance Metric Registry defined in [I-D.manyfolks-ippm-metric-registry]. The Sub-registry will contain the following categories and (bullet) columns, (as defined in section 3 above):
Common Registry Indexes and Info
Metric Definition
Method of Measurement
Comments and Remarks
The authors thank Brian Trammell for suggesting the term "Run-time Parameters", which led to the distinction between run-time and fixed parameters implemented in this memo, and the IPFIX metric with Flow Key as an example.