| Network Working Group | S. Wenger |
| Internet-Draft | J. Lennox |
| Updates: 5104 (if approved) | Vidyo, Inc. |
| Intended status: Standards Track | B. Burman |
| Expires: July 14, 2017 | M. Westerlund |
| Ericsson | |
| January 10, 2017 |
Using Codec Control Messages in the RTP Audio-Visual Profile with Feedback with Layered Codecs
draft-ietf-avtext-avpf-ccm-layered-04
This document updates RFC5104 by fixing a shortcoming in the specification language of the Codec Control Message Full Intra Request (FIR) as defined in RFC5104 when using it with layered codecs. In particular, a Decoder Refresh Point needs to be sent by a media sender when a FIR is received on any layer of the layered bitstream, regardless on whether those layers are being sent in a single or in multiple RTP flows. The other payload-specific feedback messages defined in RFC 5104 and RFC 4585 as updated by RFC 5506 have also been analyzed, and no corresponding shortcomings have been found.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 14, 2017.
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
The Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF) [RFC4585] and Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF) [RFC5104] specify a number of payload-specific feedback messages which a media receiver can use to inform a media sender of certain conditions, or make certain requests. The feedback messages are being sent as RTCP receiver reports, and RFC 4585 specifies timing rules that make the use of those messages practical for time-sensitive codec control.
Since the time those RFCs were developed, layered codecs have gained in popularity and deployment. Layered codecs use multiple sub-bitstreams called layers to represent the content in different fidelities. Depending on the media codec and its RTP payload format in use, a number of options exist how to transport those layers in RTP. With reference to A Taxonomy of Semantics and Mechanisms for Real-Time Transport Protocol (RTP) Sources [RFC7656]): Section 5.
The dependency relationship between layers in a truly layered, pyramid-shaped bitstream forms a directed graph, with the base layer at the root. Enhancement layers depend on the base layer and potentially on other enhancement layers, and the target layer and all layers it depends on have to be decoded jointly in order to re-create the uncompressed media signal at the fidelity of the target layer. Such a layering structure is assumed henceforth; for more exotic layering structures please see
Implementation experience has shown that the Full Intra Request (FIR) command as defined in [RFC5104] is underspecified when used with layered codecs and when more than one RTP stream is used to transport the layers of a layered bitstream at a given fidelity. In particular, from the [RFC5104] specification language it is not clear whether an FIR received for only a single RTP stream of multiple RTP streams covering the same layered bitstream necessarily triggers the sending of a Decoder Refresh Point (as defined in [RFC5104] section 2.2) for all layers, or only for the layer which is transported in the RTP stream that the FIR request is associated with.
This document fixes this shortcoming by:
While, clearly, the reaction to FIR for layered codecs in [RFC5104] and companion documents is underspecified, it appears that this is not the case for any of the other payload-specific codec control messages defined in any of [RFC4585], [RFC5104]. A brief summary of the analysis that led to this conclusion is also included in this document.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
The remainder of this section replaces the definition of Decoder Refresh Point in section 2.2 of [RFC5104] in its entirety.
Decoder Refresh Point: A bit string, packetized in one or more RTP packets, that completely resets the decoder to a known state.
Examples for "hard" single layer decoder refresh points are Intra pictures in H.261 [H.261], H.263 [H.263], MPEG-1 [MPEG-1], MPEG-2 [MPEG-2], and MPEG-4 [MPEG-4]; Instantaneous Decoder Refresh (IDR) pictures in H.264 [H.264], and H.265 [H.265]; and Keyframes in VP8 [RFC6386] and VP9 [I-D.grange-vp9-bitstream]. "Gradual" decoder refresh points may also be used; see for example H.264 [H.264]. While both "hard" and "gradual" decoder refresh points are acceptable in the scope of this specification, in most cases the user experience will benefit from using a "hard" decoder refresh point.
A decoder refresh point also contains all header information above the syntactical level of the picture layer that is conveyed in-band. In [H.264], for example, a decoder refresh point contains those parameter set Network Adaptation Layer (NAL) units that generate parameter sets necessary for the decoding of the following slice/data partition NAL units. (That is assuming the parameter sets have not been conveyed out of band.)
When a layered codec is in use, the above definition--in particular, the requirement to completely reset the decoder to a known state--implies that the decoder refresh point includes hard or gradual single layer decoder refresh points for all layers.
A media receiver or middlebox may decide to send a FIR command based on the guidance provided in Section 4.3.1 of [RFC5104]. When sending the FIR command, it MUST target the RTP stream that carries the base layer of the layered bitstream, and this is done by setting the Feedback Control Information (FCI, and in particular the SSRC field therein) to refer to the SSRC of the forward RTP stream that carries the base layer.
When a Full Intra Request Command is received by the designated media sender in the RTCP stream associated with any of the RTP streams in which any layer of a layered bitstream are sent, the designated media sender MUST send a Decoder Refresh Point [DRP] as defined above at its earliest opportunity. The requirements related to congestion control on the forward RTP streams as specified in sections 3.5.1. and 5. of [RFC5104] apply for the RTP streams both in isolation and combined.
Note: the requirement to react to FIR commands associated with enhancement layers is included for robustness and backward compatibility reasons.
The above modifications to RFC 5104 unambiguously define how to deal with FIR when layered bitstreams are in use. However, it is surprisingly difficult to identify the use of a layered bitstream. In general, it is expected that implementers know when layered bitstreams (in its commonly understood sense: with inter-layer prediction between pyramided-arranged layers) are in use and when not, and can therefore implement the above updates to RFC 5104 correctly. However, there are scenarios in which layered codecs are employed creating non-pyramid shaped bitstreams. Those scenarios may be viewed as somewhat exotic today but clearly are supported by certain video coding syntaxes, such as H.264/SVC. When blindly applying the above rules to those non-pyramid-arranged layering structures, suboptimal system behavior would result. Nothing would break, and there would not be an interoperability failure, but the user experience may suffer through the sending or receiving of Decoder Refresh Points at times or on parts of the bitstream that are unnecessary from a user experience viewpoint. Therefore, this informative section is included that provides the current understanding of when a layered bitstream is in use and when not.
The key observation made here is that the RTP payload format negotiated for the RTP streams, in isolation, is not necessarily an indicator for the use of a layered bitstream. Some layered codecs (including H.264/SVC) can form decodable bitstreams including only (one or more) enhancement layers, without the base layer, effectively creating simulcastable sub-bitstreams within a single scalable bitstream (as defined in the video coding standard), but without inter-layer prediction. In such a scenario, it is potentially, though not necessarily, counter-productive to send a decoder refresh point on all RTP streams using that payload format and SSRC. It is beyond the scope of this document to discuss optimized reactions to FIRs received on RTP streams carrying such exotic bitstreams.
One good indication of the likely use of pyramid-shaped layering with interlayer prediction is when the various RTP streams are "bound" together on the signaling level. In an SDP environment, this would be the case if they are marked as being dependent on each other using The Session Description Protocol (SDP) Grouping Framework [RFC5888] and the layer dependency RFC 5583 [RFC5583].
Between them, AVPF [RFC4585] and Codec Control Messages [RFC5104] define a total of seven Payload-specific Feedback messages. For the FIR command message, guidance has been provided above. In this section, some information is provided with respect to the remaining six codec control messages.
PLI is defined in section 6.3.1 of [RFC4585]. The prudent response to a PLI message received for an enhancement layer is to "repair" that enhancement layer and all dependent enhancement layers through appropriate source-coding specific means. However, the reference layer(s) used by the enhancement layer for which the PLI was received does not require repair. The encoder can figure out by itself what constitutes a dependent enhancement layer and does not need help from the system stack in doing so. Thus, there is nothing that needs to be specified herein.
SLI is defined in section 6.3.2 of [RFC4585]. The current understanding is that the prudent response to a SLI message received for an enhancement layer is to "repair" the affected spatial area of that enhancement layer and all dependent enhancement layers through appropriate source-coding specific means. As in PLI, the reference layers used by the enhancement layer for which the SLI was received do not need to be repaired. Again, as in PLI, the encoder can determine by itself what constitutes a dependent enhancement layer and does not need help from the system stack in doing so. Thus, there is nothing that needs to be specified herein. SLI has seen very little implementation and, as far as it is known, none in conjunction with layered systems.
RPSI is defined in section 6.3.3 of [RFC4585]. While a technical equivalent of RPSI has been in use with non-layered systems for many years, no implementations are known in conjunction of layered codecs. The current understanding is that the reception of an RPSI message on any layer indicating a missing reference picture forces the encoder to appropriately handle that missing reference picture in the layer indicated, and all dependent layers. Thus, RPSI should work without further need for specification language.
TSTN/TSTR are defined in section 4.3.2 and 4.3.3 of [RFC5104], respectively. The TSTR request communicates guidance of the preferred trade-off between spatial quality and frame rate. A technical equivalent of TSTN/TSTR has seen deployment for many years in non-scalable systems.
The Temporal-Spatial Trade-off request and notification messages include an SSRC target, which, similarly to FIR, may refer to an RTP stream carrying a base layer, an enhancement layer, or multiple layers. Therefore, the current understanding is that the semantics of the message applies to the layers present in the targeted RTP stream.
It is noted that per-layer TSTR/TSTN is a mechanism that is, in some ways, counterproductive in a system using layered codecs. Given a sufficiently complex layered bitstream layout, a sending system has flexibility in adjusting the spatio/temporal quality balance by adding and removing temporal, spatial, or quality enhancement layers. At present it is unclear whether an allowed (or even recommended) option to the reception of a TSTR is to adjust the bit allocation within the layer(s) present in the addressed RTP stream, or to adjust the layering structure accordingly--which can involve more than just the addressed RTP stream.
Until there is a sufficient critical mass of implementation practice, it is probably prudent for an implementer not to assume either of the two options or any middleground that may exist between the two. Instead, it is suggested that an implementation be liberal in accepting TSTR messages, and upon receipt responding in TSTN indicating "no change". Further, it is suggested that new implementations do not send TSTR messages except when operating in SRST mode as defined in [RFC7656]. Finally implementers are encouraged to contribute to the IETF documentation of any implementation requirements that make per-layer TSTR/TSTN useful.
VBCM is defined in section 4.3.4 of [RFC5104]. What was said above for RPSI [RPSI] applies here as well.
The authors want to thank Mo Zanaty for useful discussions.
This memo includes no request to IANA.
The security considerations of AVPF [RFC4585] (as updated by Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences [RFC5506]) and Codec Control Messages [RFC5104] apply. The clarified response to FIR does not introduce additional security considerations.
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997. |
| [RFC4585] | Ott, J., Wenger, S., Sato, N., Burmeister, C. and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 10.17487/RFC4585, July 2006. |
| [RFC5104] | Wenger, S., Chandra, U., Westerlund, M. and B. Burman, "Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, February 2008. |
| [RFC5506] | Johansson, I. and M. Westerlund, "Support for Reduced-Size Real-Time Transport Control Protocol (RTCP): Opportunities and Consequences", RFC 5506, DOI 10.17487/RFC5506, April 2009. |
NOTE TO RFC EDITOR: Please remove this section prior to publication.
draft-wenger-avtext-avpf-ccm-layered-00-00: initial version
draft-ietf-avtext-avpf-ccm-layered-00: resubmit as avtext WG draft per IETF95 and list confirmation by Rachel 4/25/2016
draft-ietf-avtext-avpf-ccm-layered-00: In section "Identifying the use of Layered Codecs (Informative)", removed last sentence that could be misread that the explicit signaling of simulcasting in conjunction with payload formats supporting layered coding implies no layering.
draft-ietf-avtext-avpf-ccm-layered-01: clarifications in section 5.
draft-ietf-avtext-avpf-ccm-layered-02: addressing WGLC comments, mostly editorial; see reflector discussions 09/2016
draft-ietf-avtext-avpf-ccm-layered-03: addressing AD writeup comments, editorial