| Payload Working Group | J. Uberti |
| Internet-Draft | S. Holmer |
| Intended status: Standards Track | M. Flodman |
| Expires: September 10, 2015 | |
| J. Lennox | |
| D. Hong | |
| Vidyo | |
| March 9, 2015 |
RTP Payload Format for VP9 Video
draft-uberti-payload-vp9-01
This memo describes an RTP payload format for the VP9 video codec. The payload format has wide applicability, as it supports applications from low bit-rate peer-to-peer usage, to high bit-rate video conferences. It includes provisions for temporal and spatial scalability.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 10, 2015.
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
This memo describes an RTP payload specification applicable to the transmission of video streams encoded using the VP9 video codec [I-D.grange-vp9-bitstream]. The format described in this document can be used both in peer-to-peer and video conferencing applications.
TODO: VP9 description. Please see [I-D.grange-vp9-bitstream].
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
The VP9 codec can maintain up to eight reference frames, of which up to three can be referenced or updated by any new frame.
VP9 also allows a reference frame to be resampled and used as a reference for another frame of a different resolution. This allows internal resolution changes without requiring the use of key frames.
These features together enable an encoder to implement various forms of coarse-grained scalability, including temporal, spatial and quality scalability modes, as well as combinations of these, without the need for explicit scalable coding tools.
Temporal layers define different frame rates of video; spatial and quality layers define different and possibly dependent representations of a single input frame. Spatial layers allow a frame to be encoded at different resolutions, whereas quality layers allow a frame to be encoded at the same resolution but at different qualities (and thus with different amounts of coding error). VP9 supports quality layers as spatial layers without any resolution changes; hereinafter, the term "spatial layer" is used to represent both spatial and quality layers.
This payload format specification defines how such temporal and spatial scalability layers can be described and communicated.
Layers are designed (and MUST be encoded) such that if any layer, and all higher layers, are removed from the bitstream along any of the two dimensions, the remaining bitstream is still correctly decodable.
For terminology, this document uses the term "layer frame" to refer to a single encoded VP9 frame for a particular resolution/quality, and "super frame" to refer to all the representations (layer frames) at a single instant in time. A super frame thus consists of one or more layer frames, encoding different spatial layers.
Within a super frame, a layer frame with spatial layer ID equal to S, where S > 0, can depend on a frame with a lower spatial layer ID. This "inter-layer" dependency results in additional coding gain to the traditional "inter-picture" dependency, where a frame depends on previously coded frame in time. For simplicity, this payload format assumes that, within a super frame if inter-layer dependency is used, a spatial layer S frame can only depend on spatial layer S-1 frame when S > 0. Additionally, if inter-picture dependency is used, spatial layer S frame is assumed to only depend on prevously coded spatial layer S frame.
TODO: Describe how simulcast can be supported?
Given above simplifications for inter-layer and inter-picture dependencies, a flag (the D bit described below) is used to indicate whether a spatial layer S frame depends on spatial layer S-1 frame. Then a receiver only needs to know the inter-picture dependency structure for a given spatial layer frame in order to determine its decodability. Two modes of describing the inter-picture dependency structure are possible: "flexible mode" and "non-flexible mode". An encoder can only switch between the two on the very first packet of a key frame with temporal layer ID equal to 0.
In flexible mode, each packet can contain up to 3 reference indices, which identifies all frames referenced by the frame transmitted in the current packet for inter-picture prediction. This (along with the D bit) enables a receiver to identify if a frame is decodable or not and helps it understand the temporal layer structure so that it can drop packets as it sees fit. Since this is signaled in each packet it makes it possible to have very flexible temporal layer hierarchies and patterns which are changing dynamically.
In non-flexible mode, the inter-picture dependency (the reference indices) of a group of frames (GOF) MUST be pre-specified as part of the scalability structure (SS) data. In this mode, each packet will have an index to refer to one of the described frames, from which the frames referenced by the frame transmitted in the current packet for inter-picture prediction can be identified.
The SS data can also be used to specify the resolution of each spatial layer present in the VP9 stream.
This section describes how the encoded VP9 bitstream is encapsulated in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is RECOMMENDED. All integer fields in the specifications are encoded as unsigned integers in network octet order.
The general RTP payload format for VP9 is depicted below.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| VP9 payload descriptor (integer #bytes) |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : VP9 pyld hdr | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
+ |
: Bytes 2..N of VP9 payload :
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The VP9 payload descriptor and VP9 payload header will be described in the next section. OPTIONAL RTP padding MUST NOT be included unless the P bit is set.
Figure 1
The remaining RTP header fields are used as specified in [RFC3550].
In flexible mode (with the F bit below set to 1), The first octets after the RTP header are the VP9 payload descriptor, with the following structure.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|I|P|L|F|B|E|V|-| (REQUIRED)
+-+-+-+-+-+-+-+-+
I: |M| PICTURE ID | (RECOMMENDED)
+-+-+-+-+-+-+-+-+
M: | EXTENDED PID | (RECOMMENDED)
+-+-+-+-+-+-+-+-+
L: | T |U| S |D| (CONDITIONALLY RECOMMENDED)
+-+-+-+-+-+-+-+-+ -\
P,F: | P_DIFF |X|N| (CONDITIONALLY RECOMMENDED) .
+-+-+-+-+-+-+-+-+ . - up to 3 times
X: |EXTENDED P_DIFF| (OPTIONAL) .
+-+-+-+-+-+-+-+-+ -/
V: | SS |
| .. |
+-+-+-+-+-+-+-+-+
Figure 2
In non-flexible mode (with the F bit below set to 0), The first octets after the RTP header are the VP9 payload descriptor, with the following structure.
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|I|P|L|F|B|E|V|-| (REQUIRED)
+-+-+-+-+-+-+-+-+
I: |M| PICTURE ID | (RECOMMENDED)
+-+-+-+-+-+-+-+-+
M: | EXTENDED PID | (RECOMMENDED)
+-+-+-+-+-+-+-+-+
L: |GOF_IDX| S |D| (CONDITIONALLY RECOMMENDED)
+-+-+-+-+-+-+-+-+
| TL0PICIDX | (CONDITIONALLY REQUIRED)
+-+-+-+-+-+-+-+-+
V: | SS |
| .. |
+-+-+-+-+-+-+-+-+
Figure 3
The mandatory first octet is followed by the extension data fields that are enabled:
The scalability structure (SS) data describes the resolution of each layer frame within a super frame as well as the inter-picture dependencies for a group of frames (GOF). If the VP9 payload descriptor's "V" bit is set, the SS data is present in the position indicated in Figure 2 and Figure 3.
+-+-+-+-+-+-+-+-+
V: | N_S |Y| N_G |
+-+-+-+-+-+-+-+-+ -\
Y: | WIDTH | (OPTIONAL) .
+ + .
| | (OPTIONAL) .
+-+-+-+-+-+-+-+-+ . - N_S + 1 times
| HEIGHT | (OPTIONAL) .
+ + .
| | (OPTIONAL) .
+-+-+-+-+-+-+-+-+ -/ -\
N_G: | T |U| R |-|-| (OPTIONAL) .
+-+-+-+-+-+-+-+-+ -\ . - N_G + 1 times
| P_DIFF | (OPTIONAL) . - R times .
+-+-+-+-+-+-+-+-+ -/ -/
Figure 4
In a scalable stream sent with a fixed pattern, the SS data SHOULD be included in the first packet of every key frame. This is a packet with P bit equal to zero, S or D bit equal to zero, B bit equal to 1, and temporal layer ID (TID) equal to 0. The SS data SHOULD also be included in the first packet of the first frame in which the SS changes. If the SS data is included in a frame with TID not equal to 0, it MUST also be repeated in the first packet of the first frame with a lower TID, until TID equals to 0.
TODO: need to describe VP9 payload header.
VP9 frames are fragmented into packets, in RTP sequence number order, beginning with a packet with the B bit set, and ending with a packet with the RTP marker bit set. There is no mechanism for finer-grained access to parts of a VP9 frame.
TODO
The VP9 payload descriptor defined in Section 4.2 above contains an optional PictureID parameter. One use of this parameter is included to enable use of reference picture selection index (RPSI) and slice loss indication (SLI), both defined in [RFC4585].
TODO: Update to indicate which frame within the picture.
The reference picture selection index is a payload-specific feedback message defined within the RTCP-based feedback format. The RPSI message is generated by a receiver and can be used in two ways. Either it can signal a preferred reference picture when a loss has been detected by the decoder -- preferably then a reference that the decoder knows is perfect -- or, it can be used as positive feedback information to acknowledge correct decoding of certain reference pictures. The positive feedback method is useful for VP9 used as unicast. The use of RPSI for VP9 is preferably combined with a special update pattern of the codec's two special reference frames -- the golden frame and the altref frame -- in which they are updated in an alternating leapfrog fashion. When a receiver has received and correctly decoded a golden or altref frame, and that frame had a PictureID in the payload descriptor, the receiver can acknowledge this simply by sending an RPSI message back to the sender. The message body (i.e., the "native RPSI bit string" in [RFC4585]) is simply the PictureID of the received frame.
TODO: Update to indicate which frame within the picture.
The slice loss indication is another payload-specific feedback message defined within the RTCP-based feedback format. The SLI message is generated by the receiver when a loss or corruption is detected in a frame. The format of the SLI message is as follows [RFC4585]:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| First | Number | PictureID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5
Here, First is the macroblock address (in scan order) of the first lost block and Number is the number of lost blocks. PictureID is the six least significant bits of the codec-specific picture identifier in which the loss or corruption has occurred. For VP9, this codec-specific identifier is naturally the PictureID of the current frame, as read from the payload descriptor. If the payload descriptor of the current frame does not have a PictureID, the receiver MAY send the last received PictureID+1 in the SLI message. The receiver MAY set the First parameter to 0, and the Number parameter to the total number of macroblocks per frame, even though only parts of the frame is corrupted. When the sender receives an SLI message, it can make use of the knowledge from the latest received RPSI message. Knowing that the last golden or altref frame was successfully received, it can encode the next frame with reference to that established reference.
TODO: this example is copied from the VP8 payload format specification, and has not been updated for VP9. It may be incorrect.
The use of RPSI and SLI is best illustrated in an example. In this example, the encoder may not update the altref frame until the last sent golden frame has been acknowledged with an RPSI message. If an update is not received within some time, a new golden frame update is sent instead. Once the new golden frame is established and acknowledged, the same rule applies when updating the altref frame.
| Event | Sender | Receiver | Established reference |
|---|---|---|---|
| 1000 | Send golden frame PictureID = 0 | ||
| Receive and decode golden frame | |||
| 1001 | Send RPSI(0) | ||
| 1002 | Receive RPSI(0) | golden | |
| ... | (sending regular frames) | ||
| 1100 | Send altref frame PictureID = 100 | ||
| Altref corrupted or lost | golden | ||
| 1101 | Send SLI(100) | golden | |
| 1102 | Receive SLI(100) | ||
| 1103 | Send frame with reference to golden | ||
| Receive and decode frame (decoder state restored) | golden | ||
| ... | (sending regular frames) | ||
| 1200 | Send altref frame PictureID = 200 | ||
| Receive and decode altref frame | golden | ||
| 1201 | Send RPSI(200) | ||
| 1202 | Receive RPSI(200) | altref | |
| ... | (sending regular frames) | ||
| 1300 | Send golden frame PictureID = 300 | ||
| Receive and decode golden frame | altref | ||
| 1301 | Send RPSI(300) | altref | |
| 1302 | RPSI lost | ||
| 1400 | Send golden frame PictureID = 400 | ||
| Receive and decode golden frame | altref | ||
| 1401 | Send RPSI(400) | ||
| 1402 | Receive RPSI(400) | golden |
Note that the scheme is robust to loss of the feedback messages. If the RPSI is lost, the sender will try to update the golden (or altref) again after a while, without releasing the established reference. Also, if an SLI is lost, the receiver can keep sending SLI messages at any interval allowed by the RTCP sending timing restrictions as specified in [RFC4585], as long as the picture is corrupted.
This payload format has two required parameters.
This registration is done using the template defined in [RFC6838] and following [RFC4855].
The receiver MUST ignore any fmtp parameter unspecified in this memo.
The media type video/VP9 string is mapped to fields in the Session Description Protocol (SDP) [RFC4566] as follows:
An example of media representation in SDP is as follows:
m=video 49170 RTP/AVPF 98
a=rtpmap:98 VP9/90000
a=fmtp:98 max-fr=30; max-fs=3600;
TODO: Update this for VP9
RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550], and in any applicable RTP profile. The main security considerations for the RTP packet carrying the RTP payload format defined within this memo are confidentiality, integrity and source authenticity. Confidentiality is achieved by encryption of the RTP payload. Integrity of the RTP packets through suitable cryptographic integrity protection mechanism. Cryptographic system may also allow the authentication of the source of the payload. A suitable security mechanism for this RTP payload format should provide confidentiality, integrity protection and at least source authentication capable of determining if an RTP packet is from a member of the RTP session or not. Note that the appropriate mechanism to provide security to RTP and payloads following this memo may vary. It is dependent on the application, the transport, and the signaling protocol employed. Therefore a single mechanism is not sufficient, although if suitable the usage of SRTP [RFC3711] is recommended. This RTP payload format and its media decoder do not exhibit any significant non-uniformity in the receiver-side computational complexity for packet processing, and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological data. Nor does the RTP payload format contain any active content.
Congestion control for RTP SHALL be used in accordance with RFC 3550 [RFC3550], and with any applicable RTP profile; e.g., RFC 3551 [RFC3551]. The congestion control mechanism can, in a real-time encoding scenario, adapt the transmission rate by instructing the encoder to encode at a certain target rate. Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2 to identify non-reference frames and discard them in order to reduce network congestion. Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).
The IANA is requested to register the following values:
- Media type registration as described in Section 6.1.