Internet-Draft | RTP Payload Format for JPEG XS | June 2021 |
Lugan, et al. | Expires 19 December 2021 | [Page] |
This document specifies a Real-Time Transport Protocol (RTP) payload format to be used for transporting JPEG XS (ISO/IEC 21122) encoded video. JPEG XS is a low-latency, lightweight image coding system. Compared to an uncompressed video use case, it allows higher resolutions and video frame rates, while offering visually lossless quality, reduced power consumption, and end-to-end latency confined to a fraction of a video frame.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 19 December 2021.¶
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
This document specifies a payload format for packetization of JPEG XS [ISO21122-1] encoded video signals into the Real-time Transport Protocol (RTP) [RFC3550].¶
The JPEG XS coding system offers compression and recompression of image sequences with very moderate computational resources while remaining robust under multiple compression and decompression cycles and mixing of content sources, e.g. embedding of subtitles, overlays or logos. Typical target compression ratios ensuring visually lossless quality are in the range of 2:1 to 10:1, depending on the nature of the source material. The end-to-end latency can be confined to a fraction of a video frame, typically between a small number of lines down to below a single line.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
JPEG XS is a low-latency lightweight image coding system for coding continuous-tone grayscale or continuous-tone color digital images.¶
This coding system provides an efficient representation of image signals through the mathematical tool of wavelet analysis. The wavelet filter process separates each component into multiple bands, where each band consists of multiple coefficients describing the image signal of a given component within a frequency domain specific to the wavelet filter type, i.e. the particular filter corresponding to the band.¶
Wavelet coefficients are grouped into precincts, where each precinct includes all coefficients over all bands that contribute to a spatial region of the image.¶
One or multiple precincts are furthermore combined into slices consisting of an integer number of precincts. Precincts do not cross slice boundaries, and wavelet coefficients in precincts that are part of different slices can be decoded independently of each other. Note, however, that the wavelet transformation runs across slice boundaries. A slice always extends over the full width of the image, but may only cover parts of its height.¶
A JPEG XS codestream is formed by (in the given order):¶
The JPEG XS codestream format, including the definition of all markers, is further defined in [ISO21122-1]. It represents sample values of a single image, without any interpretation relative to a color space.¶
While the information defined in the codestream is sufficient to reconstruct the sample values of one image, the interpretation of the samples remains undefined by the codestream itself. This interpretation is given by the video support box and the color specification box which contain significant information to correctly play the JPEG XS stream. The layout and syntax of these boxes, together with their content, are defined in [ISO21122-3].¶
The video support box provides information on the maximum bitrate, the frame rate, the interlaced mode (progressive or interlaced), the subsampling image format, the informative timecode of the current JPEG XS frame, the profile, level/sublevel used, and optionally on the buffer model and the mastering display metadata.¶
Note that the profile and level/sublevel, specified by respectively the Ppih and Plev fields, specify limits on the capabilities needed to decode the codestream and handle the output. Profiles represent a limit on the required algorithmic features and parameter ranges used in the codestream. The combination of level and sublevel defines a lower bound on the required throughput for a decoder in respectively the image (or decoded) domain and the codestream (or coded) domain. The actual defined profiles and levels/sublevels, along with the associated values for the Ppih and Plev fields, are defined in [ISO21122-2].¶
The color specification box indicates the color primaries, transfer characteristics, matrix coefficients and video full range flag needed to specify the color space of the video stream.¶
The concatenation of a video support box, a color specification box, and a JPEG XS codestream forms a JPEG XS picture segment.¶
In the case of a progressive video stream, each JPEG XS frame consists of one single JPEG XS picture segment.¶
In the case of an interlaced video stream, each JPEG XS frame is made of two concatenated JPEG XS picture segments. The codestream of each picture segment corresponds exclusively to one of the two fields of the interlaced frame. Both picture segments SHALL contain identical boxes (i.e. concatenation of the video support box and the color specification box is byte exact the same for both picture segments of the frame).¶
Note that the interlaced mode, as signaled by the frat field [ISO21122-3] in the video support box, indicates either progressive, interlaced top-field first, or interlaced bottom-field first mode. Thus, in the case of interlaced content, its value SHALL also be identical in both picture segments.¶
This section specifies the payload format for JPEG XS streams over the Real-time Transport Protocol (RTP) [RFC3550].¶
In order to be transported over RTP, each JPEG XS stream is transported in a distinct RTP stream, identified by a distinct Synchronization source (SSRC) [RFC3550].¶
A JPEG XS stream is divided into Application Data Units (ADUs), each ADU corresponding to a single JPEG XS frame.¶
An ADU is made of several packetization units. If a packetization unit is bigger than the maximum size of an RTP packet payload, the unit is split into multiple RTP packet payloads, as illustrated in Figure 1. As seen there, each packet SHALL contain (part of) one and only one packetization unit. A packetization unit may extend over multiple packets. The payload of every packet SHALL have the same size (based e.g. on the Maximum Transfer Unit of the network), except (possibly) the last packet of a packetization unit. The boundaries of a packetization unit SHALL coincide with the boundaries of the payload of a packet (excluding the payload header), i.e. the first (resp. last) byte of the packetization unit SHALL be the first (resp. last) byte of the payload (excluding its header).¶
There are two different packetization modes defined for this RTP payload format.¶
Due to the constant bit-rate of JPEG XS, the codestream packetization mode guarantees that a JPEG XS RTP stream will produce a constant number of bytes per video frame, and a constant number of RTP packets per video frame. If an implementation wishes to provide the same guarantee with the slice packetization mode, it will need to use an additional mechanism. This can involve a constraint at the rate allocation stage in the JPEG XS encoder to impose a constant bit-rate at the slice level, the usage of padding data, or the insertion of empty RTP packets (i.e. an RTP packet whose payload data is empty).¶
The format of the RTP header is specified in [RFC3550] and reprinted in Figure 4 for convenience. This RTP payload format uses the fields of the header in a manner consistent with that specification.¶
The RTP payload (and the settings for some RTP header bits) for packetization units are specified in Section 4.3.¶
The version (V), padding (P), extension (X), CSRC count (CC), sequence number, synchronization source (SSRC) and contributing source (CSRC) fields follow their respective definitions in [RFC3550].¶
The remaining RTP header information to be set according to this RTP payload format is set as follows:¶
If progressive scan video is being transmitted, the marker bit denotes the end of a video frame. If interlaced video is being transmitted, it denotes the end of the field. The marker bit SHALL be set to 1 for the last packet of the video frame/field. It SHALL be set to 0 for all other packets.¶
A dynamically allocated payload type field that designates the payload as JPEG XS video.¶
The RTP timestamp is set to the sampling timestamp of the content. A 90 kHz clock rate SHALL be used.¶
As specified in [RFC3550] and [RFC4175], the RTP timestamp designates the sampling instant of the first octet of the video frame to which the RTP packet belongs. Packets SHALL NOT include data from multiple video frames, and all packets belonging to the same video frame SHALL have the same timestamp. Several successive RTP packets will consequently have equal timestamps if they belong to the same video frame (that is until the marker bit is set to 1, marking the last packet of the video frame), and the timestamp is only increased when a new video frame begins.¶
If the sampling instant does not correspond to an integer value of the clock, the value SHALL be truncated to the next lowest integer, with no ambiguity.¶
The first four bytes of the payload of an RTP packet in this RTP payload format are referred to as the payload header. Figure 5 illustrates the structure of this payload header.¶
The payload header consists of the following fields:¶
The T bit is set to indicate that packets are sent sequentially by the transmitter. This information allows a receiver to dimension its input buffer(s) accordingly. If T=0, nothing can be assumed about the transmission order and packets may be sent out-of-order by the transmitter. If T=1, packets SHALL be sent sequentially by the transmitter. The T bit value SHALL be identical for all packets of the RTP stream.¶
The K bit is set to indicate which packetization mode is used. K=0 indicates codestream packetization mode, while K=1 indicates slice packetization mode. In the case that the Transmission mode (T) is set to 0 (out-of-order), the slice packetization mode SHALL be used and K SHALL be set to 1. This is required, because only the slice packetization mode supports out-of-order packet transmission. The K bit value SHALL be identical for all packets of the RTP stream.¶
The L bit is set to indicate the last packet of a packetization unit. As the end of the video frame also ends the packet containing the last unit of the video frame, the L bit is set whenever the M bit is set. If codestream packetization mode is used, L bit and M bit are equivalent.¶
These two I bits are used to indicate how the JPEG XS frame is scanned (progressive or interlaced). In case of an interlaced frame, they also indicate which JPEG XS picture segment the payload is part of (first or second).¶
The frame (F) counter identifies the video frame number modulo 32 to which a packet belongs. Frame numbers are incremented by 1 for each video frame transmitted. The frame number, in addition to the timestamp, may help the decoder manage its input buffer and bring packets back into their natural order.¶
The Slice and Extended Packet (SEP) counter is used differently depending on the packetization mode.¶
The packet (P) counter identifies the packet number modulo 2048 within the current packetization unit. It is set to 0 at the start of the packetization unit and incremented by 1 for every subsequent packet (if any) belonging to the same unit. Practically, if codestream packetization mode is enabled, this field counts the packets within a JPEG XS picture segment and is extended by the SEP counter when it overruns. If slice packetization mode is enabled, this field counts the packets within a slice or within the JPEG XS header segment.¶
The payload data of a JPEG XS RTP stream consists of a concatenation of multiple JPEG XS frames. Within the RTP stream, all of the video support boxes and all of the color specification boxes SHALL retain their respective layouts for each JPEG XS frame. Thus, each video support box in the RTP stream SHALL define the same sub boxes. The effective values in the boxes are allowed to change under the condition that their relative byte offsets SHALL NOT change.¶
Each JPEG XS frame is the concatenation of one or more packetization unit(s), as explained in Section 4.1. Figure 6 depicts this layout for a progressive video frame in the codestream packetization mode, Figure 7 depicts this layout for an interlaced video frame in the codestream packetization mode, Figure 8 depicts this layout for a progressive video frame in the slice packetization mode and Figure 9 depicts this layout for an interlaced video frame in the slice packetization mode. The Frame counter value is not indicated because the value is constant for all packetization units of a given video frame.¶
In order to facilitate proper synchronization between senders and receivers it is RECOMMENDED to implement traffic shaping and delivery timing in accordance with the Network Compatibility Model compliance definitions specified in [SMPTE-ST2110-21]. In such case, the session description SHALL signal the compliance with the media type parameter TP. The actual applied traffic shaping and timing delivery mechanism is outside the scope of this memo and does not influence the payload packetization.¶
Congestion control for RTP SHALL be used in accordance with [RFC3550], and with any applicable RTP profile: e.g., [RFC3551]. An additional requirement if best-effort service is being used is users of this payload format SHALL monitor packet loss to ensure that the packet loss rate is within acceptable parameters. Circuit Breakers [RFC8083] is an update to RTP [RFC3550] that defines criteria for when one is required to stop sending RTP Packet Streams and applications implementing this standard SHALL comply with it. [RFC8085] provides additional information on the best practices for applying congestion control to UDP streams.¶
This section specifies the required and optional parameters of the payload format and/or the RTP stream. All parameters are declarative, meaning that the information signaled by the parameters is also present in the payload data, namely in the payload header (see Section 4.3) or in the JPEG XS header segment [ISO21122-1] [ISO21122-3]. When provided, their respective values SHALL be consistent with the payload.¶
Signals the color difference signal sub-sampling structure.¶
Signals utilizing the non-constant luminance Y'C'B C'R signal format of Recommendation ITU-R BT.601-7, Recommendation ITU-R BT.709-6, Recommendation ITU-R BT.2020-2, or Recommendation ITU-R BT.2100 SHALL use the appropriate one of the following values for the Media Type Parameter "sampling":¶
YCbCr-4:4:4 (4:4:4 sampling) YCbCr-4:2:2 (4:2:2 sampling) YCbCr-4:2:0 (4:2:0 sampling)¶
Signals utilizing the Constant Luminance Y'C C'BC C'RC signal format of Recommendation ITU-R BT.2020-2 SHALL use the appropriate one of the following values for the Media Type Parameter "sampling":¶
CLYCbCr-4:4:4 (4:4:4 sampling) CLYCbCr-4:2:2 (4:2:2 sampling) CLYCbCr-4:2:0 (4:2:0 sampling)¶
Signals utilizing the constant intensity I CT CP signal format of Recommendation ITU-R BT.2100 SHALL use the appropriate one of the following values for the Media Type Parameter "sampling":¶
ICtCp-4:4:4 (4:4:4 sampling) ICtCp-4:2:2 (4:2:2 sampling) ICtCp-4:2:0 (4:2:0 sampling)¶
Signals utilizing the 4:4:4 R' G' B' or RGB signal format (such as that of Recommendation ITU-R BT.601, Recommendation ITU-R BT.709, Recommendation ITU-R BT.2020, Recommendation ITU-R BT.2100, SMPTE ST 2065-1 or ST 2065-3) SHALL use the following value for the Media Type Parameter sampling.¶
RGB (RGB or R' G' B' samples)¶
Signals utilizing the 4:4:4 X' Y' Z' signal format (such as defined in SMPTE ST 428-1) SHALL use the following value for the Media Type Parameter sampling.¶
XYZ (X' Y' Z' samples)¶
Key signals as defined in SMPTE RP 157 SHALL use the value key for the Media Type Parameter sampling. The Key signal is represented as a single component.¶
KEY (Samples of the key signal)¶
Signals utilizing a color sub-sampling other than what is defined here SHALL use the following value for the Media Type Parameter sampling.¶
UNSPECIFIED (Sampling signaled by the payload.)¶
Specifies the system colorimetry used by the image samples. Valid values and their specification are:¶
BT601-5 ITU-R Recommendation BT.601-5. BT709-2 ITU-R Recommendation BT.709-2. SMPTE240M SMPTE ST 240M. BT601 ITU-R Recommendation BT.601-7. BT709 ITU-R Recommendation BT.709-6. BT2020 ITU-R Recommendation BT.2020-2. BT2100 ITU-R Recommendation BT.2100 Table 2 titled "System colorimetry". ST2065-1 SMPTE ST 2065-1 Academy Color Encoding Specification (ACES). ST2065-3 SMPTE ST 2065-3 Academy Density Exchange Encoding (ADX). XYZ ISO/IEC 11664-1, section titled "1931 Observer". UNSPECIFIED Colorimetry is signaled in the payload by the color specification box of [ISO21122-3], or it must be manually coordinated between sender and receiver.¶
Signals utilizing the Recommendation ITU-R BT.2100 colorimetry SHOULD also signal the representational range using the optional parameter RANGE defined below. Signals utilizing the UNSPECIFIED colorimetry might require manual coordination between the sender and the receiver.¶
Transfer Characteristic System. This parameter specifies the transfer characteristic system of the image samples. Valid values and their specification are:¶
SDR Standard Dynamic Range video streams that utilize the OETF of ITU-R Recommendation BT.709 or ITU-R Recommendation BT.2020. Such streams SHALL be assumed to target the EOTF specified in ITU-R Recommendation BT.1886. PQ High dynamic range video streams that utilize the Perceptual Quantization system of ITU-R Recommendation BT.2100. HLG High dynamic range video streams that utilize the Hybrid Log-Gamma system of ITU-R Recommendation BT.2100. UNSPECIFIED Video streams whose transfer characteristics are signaled by the payload as specified in [ISO21122-3], or must be manually coordinated between sender and receiver.¶
A mapping of the parameters into the Session Description Protocol (SDP) [RFC8866] is provided for applications that use SDP.¶
The media type video/jxsv string is mapped to fields in the Session Description Protocol (SDP) [RFC8866] as follows:¶
All parameters of the media format SHALL correspond to the parameters of the payload. In case of discrepancies between payload parameter values and SDP fields, the values from the payload data SHALL prevail.¶
The receiver SHALL ignore any parameter that is not defined in Section 7.1.¶
An example SDP mapping for JPEG XS video is as follows:¶
m=video 30000 RTP/AVP 112 a=rtpmap:112 jxsv/90000 a=fmtp:112 packetmode=0;sampling=YCbCr-4:2:2; width=1920;height=1080;depth=10; colorimetry=BT709;TCS=SDR;RANGE=FULL;TP=2110TPNL¶
In this example, a JPEG XS RTP stream is to be sent to UDP destination port 30000, with an RTP dynamic payload type of 112 and a media clock rate of 90000 Hz. Note that the "a=fmtp:" line has been wrapped to fit this page, and will be a single long line in the SDP file. This example includes the TP parameter (as specified in Section 5).¶
When JPEG XS is offered over RTP using SDP in an offer/answer model [RFC3264] for negotiation for unicast usage, the following limitations and rules apply:¶
The IANA is requested to register the media type registration "video/jxsv" as specified in Section 7.1. The media type is also requested to be added to the IANA registry for "RTP Payload Format MIME types" <https://www.iana.org/assignments/rtp-parameters>.¶
RTP packets using the payload format defined in this memo are subject to the security considerations discussed in [RFC3550] and in any applicable RTP profile such as RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124]. This implies that confidentiality of the media streams is achieved by encryption.¶
However, as "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals like confidentiality, integrity, and source authenticity for RTP in general. This responsibility lies on anyone using RTP in an application. They can find guidance on available security mechanisms and important considerations in "Options for Securing RTP Sessions" [RFC7201]. Applications SHOULD use one or more appropriate strong security mechanisms.¶
Implementations of this RTP payload format need to take appropriate security considerations into account. It is important for the decoder to be robust against malicious or malformed payloads and ensure that they do not cause the decoder to overrun its allocated memory or otherwise misbehave. An overrun in allocated memory could lead to arbitrary code execution by an attacker. The same applies to the encoder, even though problems in encoders are typically rarer.¶
This payload format and the JPEG XS encoding do not exhibit any substantial non-uniformity, either in output or in complexity to perform the decoding operation and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological datagrams.¶
This payload format and the JPEG XS encoding do not contain code that is executable.¶
It is important to note that HD or UHDTV JPEG XS-encoded video can have significant bandwidth requirements (typically more than 1 Gbps for ultra high-definition video, especially if using high framerate). This is sufficient to cause potential for denial-of-service if transmitted onto most currently available Internet paths.¶
Accordingly, if best-effort service is being used, users of this payload format SHALL monitor packet loss to ensure that the packet loss rate is within acceptable parameters. Packet loss is considered acceptable if a TCP flow across the same network path, and experiencing the same network conditions, would achieve an average throughput, measured on a reasonable timescale, that is not less than the RTP flow is achieving. This condition can be satisfied by implementing congestion control mechanisms to adapt the transmission rate (or the number of layers subscribed for a layered multicast session), or by arranging for a receiver to leave the session if the loss rate is unacceptably high.¶
This payload format may also be used in networks that provide quality-of-service guarantees. If enhanced service is being used, receivers SHOULD monitor packet loss to ensure that the service that was requested is actually being delivered. If it is not, then they SHOULD assume that they are receiving best-effort service and behave accordingly.¶
The authors would like to thank the following people for their valuable contributions to this memo: Arnaud Germain, Alexandre Willeme, Gael Rouvroy, Siegfried Foessel, and Jean-Baptise Lorent.¶
Note to RFC Editor: This section may be removed after carrying out all the instructions of this section.¶
RFC XXXX is to be replaced by the RFC number this specification receives when published.¶