RTP Payload Format for Avatar Representation Format (ARF) Animation Stream

Introduction Avatars are digital representations of users in the metaverse, a set of virtual worlds where people can interact with each other in real-time. Users can customize different aspects of their avatars, such as clothing, accessories, and even physical attributes. Avatars allow users to express themselves and create a unique digital identity within the metaverse. The integration, animation, and representation of avatars in real-time communication services is essential to enable immersive experiences. specifies the Avatar Representation Format (ARF) to offer an interoperable exchange format for the storage, carriage and animation of avatars. It defines the "Avatar Animation Unit"(AAU) as a unit of packetization suitable for Avatar animation streams, and similar in essence to the NAL unit defined in some video specifications. This document describes how AAUs can be transmitted using the RTP protocol. This document follows recommendations in and for RTP payload format writers.

Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Definition, and abbreviations

General This document uses the definitions of the Avatar Representation Format . Some of these terms are provided here for convenience.

Definitions Animation Streams: timed data used to animate the base avatar.

Abbreviation ARF Avatar Representation Format AAU Avatar Animation Unit LoD Level of Detail

Avatar Representation Format(informative)

Overview of Avatar Representation Format The Avatar Representation Format (ARF) defines two key components of an avatar animation system: the Base Avatar Format, which describes static assets, and the Animation Stream Format, which describes the dynamic part of the avatar animation, and is the core subject of this document. The Base Avatar Format defines a structure for avatar models, among other things allowing them to be stored in digital asset repositories. This ensures that core avatar assets can be accessed and animated by receiving systems. On the other hand, the Animation Stream Format specifies how animation data is organized and transmitted between sender and receiver. It defines the encoding of facial and body animation, enabling data captured from input devices such as head-mounted displays (HMDs) and sensors to be consistently interpreted across different systems for animating associated avatars. describe an Avatar reference architecture.

Avatar Animation Streams Animation streams are timed data used to animate an avatar. In , this data includes skeletal, blend shape set, and other animation-related information. Animation stream format defines how animation data is structured and carried between senders and receivers. This format defines how facial and body animation information is encoded, allowing data captured from input devices like Head-Mounted Displays (HMDs) and sensors to be consistently interpreted across different systems for the animation of associated avatars. The animation streams may be read from a file, or generated on-the-fly as cameras and/or sensors capture a person's motion and generate corresponding commands to mimic this movement for an avatar that represents the user. Avatar animation samples are structured into a bitstream comprising a sequence of Avatar Animation Units (AAUs), defined in , and whose general structure is provided in . An avatar animation is associated with a Base Avatar, using an avatar ID. Each AAU is associated with an Avatar ID that indicates the target avatar to which the animation data applies. In addition, it is also associated with a Level of Detail (LoD), which indicates the level of detail of the asset to which the animation data is associated. The animation data within an AAU can for example be generated by a tracking and animation framework (e.g., OpenXR or ARKit) . defines thisidentified using a URN. The receiver is aware of the avatar IDs and/or levels of detail that are transmitted in a stream, and needs the appropriate assets to render the avatar animation. The method for accessing the assets is not described in this document. The receiver can for example use the avatar ID and level of detail associated with an AAU to transmit the AAU to an animation player instance that has the proper assets.

The structure of AAU Header(a) and Payload(b)

Payload format for ARF Animation Streams

General This section specifies details related to the RTP payload format definitions for the ARF Animation Streams defined in . Aspects related to RTP header, RTP payload header and general payload structure are defined.

RTP Header Usage The RTP header is defined in and represented in . Some of the header field values are interpreted as follows.

RTP header for Avatar Animation Unit Marker bit (M): 1 bit. The marker bit SHOULD be set to one in the first RTP packet after an idle period. This is aligned with the use of the marker bit in audio codecs. This can for example be used for jitter buffer adaptation. The marker bit in all other packets MUST be set to zero. Payload type (PT): 7 bits The assignment of a payload type MUST be performed either through the profile used or in a dynamic way. Sequence Number (SN): 16 bits Set and used in accordance with Timestamp: 32 bits A timestamp representing the sampling time of the earliest AAU (Avatar Animation Unit) in the payload. The AAU defines aau_timestamp in its payload . The timestamp in seconds can be calculated as: timestamp / timescale. Synchronization source (SSRC): 32 bits Used to identify the source of the RTP packets. By definition a single SSRC is used for all parts of a single bitstream. The remaining RTP header fields are used as specified in .

RTP Payload Header for Avatar Animation Unit The RTP Payload Header follows the RTP header. describes RTP Payload Header.

RTP Payload header for Avatar Animation D (Dependency, 1 bit): this field indicates whether an AAU included in the avatar animation packet payload is an independent AAU (D=0) or dependent (D=1). If D=1, the AAU is dependent on other AAUs for decoding. If D=0, the AAU can be decoded independently. Editor's Note: in the current version of all AAUs are independent AAUs. UT (Unit Type, 4 bits): this field indicates the type of the payload, which can be the type of the AAU for single unit payload, or the type of the payload otherwise, as shown in . L (Level of Detail, 3 bits): this field indicates the level of detail to which the AAU(s) within the RTP packet applies. If the RTP packet includes multiple AAUs, L MUST indicate the lowest LoD. AvID (Avatar ID, 8 bits): this field identifies the avatar to which the animation data in the payload of the packet applies. The avatar corresponds to the digital assets to be animated.

Payload structures

General Three different types of RTP packet payload structures are specified. A single unit packet contains a single AAU in the payload. A fragmentation unit contains a subset of an AAU. An aggregation packet contains multiple Avatar animation units in the payload. The unit type (UT) field of the RTP payload header, as shown in , identifies both the payload structure and, in the case of a single-unit structure, also identifies the type of AAU present in the payload. The Unit Types 1-5 in are defined in .

Payload structure type for Avatar The payload structures are represented in . The single unit payload structure is specified in . The fragmented unit payload structure is specified in . The aggregation unit payload structure is specified in .

RTP Transmission mode

Single Unit Payload Structure In a single unit payload structure, as described in , the RTP packet contains the RTP header, followed by the Payload Header and one single AAU. The Payload Header follows the structure described in . The payload contains an AAU as defined in .

Single AAU payload structure

Fragmented Unit Payload Structure In a fragmented unit payload structure, as described in , the RTP packet contains the RTP header, followed by the Payload Header, a Fragmented Unit (FU) header, and an AAU fragment. The Payload Header follows the structure described in . The value of the UT field of the Payload Header is 15. The FU header follows the structure described in .

Fragmentation unit header FU headers are used to enable fragmenting a single AAU into multiple RTP packets. Fragments of the same AAU MUST be sent in consecutive order with ascending RTP sequence numbers (with no other RTP packets within the same RTP stream being sent between the first and last fragment). FUs MUST NOT be nested, i.e., an FU MUST NOT contain a subset of another FU. describes a FU header, including the following fields:

Fragmentation unit header FUS (Fragmented Unit Start, 1 bit): this field MUST be set to 1 for the first fragment, and 0 for the other fragments. FUE (Fragmented Unit End, 1 bit): this field MUST be set to 1 for the last fragment, and 0 for the other fragments. RSV (Reserved, 3 bits): these bits MUST be set to 0 by the sender and ignored by the receiver. UT (Unit Type, 4 bits): this field indicates the type of the AAU this fragment belongs to, using values defined in .

Aggregation Packet Payload Structure In an aggregation packet, as described in , the RTP packet contains an RTP header, followed by a Payload Header, and, for each aggregated AAU, an AAU size followed by the AAU. The Payload Header follows the structure described in .

Single-Time Aggregation Packet shows a Single-Time Aggregation Packet (STAP), which can be used to transmit multiple avatar animation units that correspond to the same timestamp. For example, if two different AAUs are used for different animations for different parts of the avatar, they can be transmitted together in a single STAP. The default sizes of the avatar animation unit length field is 16 bits. The value of the UT field of the Payload Header is 13.

Multiple-time aggregation packet shows a multi-time aggregation packet. It is used to transmit multiple Avatar animation units with different timestamps, in one RTP packet. Multi-time aggregation can help reduce the number of packets, in environments where some delay is acceptable. The default sizes of the TS offset and the AAU length fields are 16 bits each. The value of the UT field of the Payload Header is 14. In case of MTAP, the timestamp offset field MUST be set to the value of (AAU-time of the animation unit - RTP timestamp of the packet). The timestamp offset of the earliest aggregation unit MUST always be zero. Therefore, the RTP timestamp of the MTAP is identical to the earliest AAU-time.

AAU Transmission Considerations The following considerations apply for the streaming of avatar animation units over RTP: In some multimedia conference scenarios using an RTP video mixer (e.g., when adding or selecting a new source), it is recommended to use Full Intra Request (FIR) feedback messages with avatar animation. The purpose of the FIR message is to cause an encoder to send a decoder refresh point at the earliest opportunity. In the context of avatar animation, an appropriate decoder refresh point is a configuration AAU. The configuration AAU point enables a decoder to be reset to a known state and be able to decode all AAUs following it.

Payload Format Parameters This section describes payload format optional parameters. A mapping of the parameters into the Session Description Protocol (SDP) is also provided for applications that use SDP. Equivalent parameters could be defined elsewhere for use with control protocols that do not use SDP.

Media Type Registration Update The receiver MUST ignore any parameter unspecified in this memo. Type name: application Subtype name: ampg Required parameters: N/A Optional parameters: Optional parameters are defined in the following section. Encoding considerations: This type is only defined for transfer via RTP . Security considerations: Please see section 11. Interoperability considerations: N/A Published specification: Please refer to Applications that use this media type: Any application that relies on Avatar media services over RTP Fragment identifier considerations: N/A Additional information: N/A Person & email address to contact for further information: Intended usage: COMMON Restrictions on usage: N/A Author: See Authors' Address section of this memo. Change controller: IETF avtcore@ietf.org (mailto:avtcore@ietf.org) Provisional registration? (standards tree only): No

Optional Parameters Definition version: It provides the year of the edition and amendment of the specifications followed by this RTP payload type. This parameters is defined in Table 3 of . framework: It provides a comma-separated list of the tracking framework names (URNs) used to generate the encoded stream. The URNs in this parameters corresponds to the URNs in Table 5, 6 of . avatar-ids: It provides an associations between avatar IDs for which animation data is carried in the animation stream, and their corresponding ARF containers. This parameter is provided as a comma-separated list of "key/value" pairs, where the key is the avatar id (an integer between 0 and 255 inclusive) and the value is a base64 encoded string. The semantic of the value is application dependent and can for example be a URL to the ARF container. The parameter avatar_id is defined in section 7 of . avatar-lods: It indicates which levels of detail are used in the avatar animation stream. This parameter is a comma-separated list of integers. Each item in this list corresponds to a level of detail as defined in section 7 of

Congestion Control Consideration General congestion control considerations for RTP transmission, as described in , also apply to avatar streaming over RTP. By adjusting the SDP 'avatar-lod' parameter, it is possible to reduce processing load and optimize bandwidth usage, thereby partially mitigating congestion issues. The ability to adapt to the level of detail dynamically allows senders or receivers to manage computational complexity and network resource consumption based on system constraints or user context. Moreover, in use cases such as video conferencing, different levels of detail may be applied to different parts of the avatar and transmitted via separate streams.

SDP Considerations The mapping of above defined payload format media type to the corresponding fields in the Session Description Protocol (SDP) is done according to . The media name in the "m=" line of SDP MUST be application. The encoding name in the "a=rtpmap" line of SDP MUST be ampg The clock rate in the "a=rtpmap" line may be any sampling rate and SHOULD match the acu timescale value of the AAU CONFIG unit . The OPTIONAL parameters (defined in ), when present, MUST be included in the "a=fmtp" line of SDP. This is expressed as a media type string, in the form of a semicolon-separated list of parameter=value pairs. An example of media representation corresponding to the avatar animation RTP payload in SDP is as follows:

SDP Offer/Answer Considerations When using the offer/answer procedure described in to negotiate the use of avatar animations, the following considerations apply: The SDP parameter version identifies the version of the avatar animation specification. It MUST be used symmetrically in SDP offer and answer, and it MUST NOT be changed in subsequent offers or answers within the same session. If it is not specified, the initial version of the specification SHOULD be assumed. Any receiver compliant with must accept any stream with a compatible version. The properties expressed using SDP parameters other than 'version' are provided as recommendations for efficient data transmission and are not binding, meaning that a sender is encouraged but not required to conform to the parameters specified by the receiver. These properties may be set to different values in offers and answers. These properties may be updated in subsequent offers or answers. These properties can be sent by a sender to reflect the characteristics of bitstreams and can be set by a receiver to reflect the capabilities and configurations of the local player device, or a preferred set of bitstream properties. The parameter frameworks indicates that the AAUs of the stream carry animation data that conforms to the one or more framework names (URNs) signalled with this parameter. The sender uses this parameter to indicate the formats of data transported within the AAUs of the stream. The receiver, to be able to render the animations, needs to support the formats associated with signalled frameworks. The receiver uses this parameter to indicate the desired framework names. The parameter avatar-ids indicates that a stream corresponds to the one or more avatar IDs signalled with this parameter. The sender uses this parameter to indicate that the AAUs of the stream carry data corresponding to the signalled avatar IDs. The receiver uses this parameter to indicate the avatar IDs it wishes to receive data for. The parameter avatar-lods indicates that the AAUs of the stream correspond to one or more levels of detail signalled with this parameter. The sender uses this parameter to indicate available LoDs, and the receiver uses it to select the desired LoD. To render the animations, the receiver MUST have loaded the corresponding assets associated with the selected level(s) of detail. A receiver may ignore any part of a received stream, e.g., that it does not have support for rendering.

Declarative SDP Considerations When avatar animation over RTP is offered with SDP in a declarative style, the parameters capable of indicating both bitstream properties as well as receiver capabilities are used to indicate only bitstream properties. For example, in this case, the parameters frameworks, avatar-ids, and avatar-lods declare the values used by the bitstream, not the capabilities and configurations for receiving bitstreams. A receiver of the SDP is required to support all parameters and values of the parameters provided; otherwise, the receiver MUST reject or not participate in the session. It falls on the creator of the session to use values that are expected to be supported by the receiving application.

IANA Considerations

Avatar Animation Media Registration New media types will be registered with IANA; see .

Security Considerations RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification , and in any applicable RTP profile such as RTP/AVP , RTP/AVPF , RTP/SAVP , or RTP/SAVPF . For example, an avatar may contain sensitive information derived from a user's personal data, and thus requires protection against leakage or tampering during transmission. When avatar data is delivered over a network or downloaded from a server, it is critical to ensure its integrity and confidentiality to prevent unauthorized access, modification, or confidentiality. However, as "Securing the RTP Protocol Framework: Why RTP Does Not Mandate a Single Media Security Solution" discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals like confidentiality, integrity, and source authenticity for RTP in general. This responsibility lays on anyone using RTP in an application. They can find guidance on available security mechanisms and important considerations in "Options for Securing RTP Sessions" . Applications SHOULD use one or more appropriate strong security mechanisms. The rest of this Security Considerations section discusses the security impacting properties of the payload format itself.