A Real-Time Transport Protocol (RTP) Header Extension for Mixer-to-Client Audio Level Indication
draft-ietf-avtext-mixer-to-client-audio-level-02

Abstract

This document describes a mechanism for RTP-level mixers in audio conferences to deliver information about the audio level of individual participants. Such audio level indicators are transported in the same RTP packets as the audio data they pertain to.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on November 10, 2011.

Copyright Notice

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Terminology
• 3. Protocol Operation
• 4. Header Format
• 5. Audio level encoding
• 6. Signaling Information
• 7. Security Considerations
• 8. IANA Considerations
• 9. Acknowledgments
• 10. Changes From Earlier Versions
• 10.1. Changes From Draft -01
• 10.2. Changes From Draft -00
• 11. References
• 11.1. Normative References
• 11.2. Informative References
• Appendix A. Reference Implementation
• Appendix A.1. AudioLevelCalculator.java
• Authors' Addresses

1. Introduction

The Framework for Conferencing with the Session Initiation Protocol (SIP) defined in RFC 4353 [RFC4353] presents an overall architecture for multi-party conferencing. Among others, the framework borrows from RTP [RFC3550] and extends the concept of a mixer entity "responsible for combining the media streams that make up a conference, and generating one or more output streams that are delivered to recipients". Every participant would hence receive, in a flat single stream, media originating from all the others.

Using such centralized mixer-based architectures simplifies support for conference calls on the client side since they would hardly differ from one-to-one conversations. However, the method also introduces a few limitations. The flat nature of the streams that a mixer would output and send to participants makes it difficult for users to identify the original source of what they are hearing.

Mechanisms that allow the mixer to send to participants cues on current speakers (e.g. the CSRC fields in RTP [RFC3550]) only work for speaking/silent binary indications. There are, however, a number of use cases where one would require more detailed information. Possible examples include the presence of background chat/noise/music/typing, someone breathing noisily in their microphone, or other cases where identifying the source of the disturbance would make it easy to remove it (e.g. by sending a private IM to the concerned party asking them to mute their microphone). A more advanced scenario could involve an intense discussion between multiple participants that the user does not personally know. Audio level information would help better recognize the speakers by associating with them complex (but still human readable) characteristics like loudness and speed for example.

One way of presenting such information in a user friendly manner would be for a conferencing client to attach audio level indicators to the corresponding participant related components in the user interface as displayed in Figure 1.

                      ________________________
                     |                        |
                     |  00:42 |  Weekly Call  |
                     |________________________|
                     |                        |
                     |                        |
                     | Alice |======    | (S) |
                     |                        |
                     | Bob   |=         |     |
                     |                        |
                     | Carol |          | (M) |
                     |                        |
                     | Dave  |===       |     |
                     |                        |
                     |________________________|

        

Implementing a user interface like the above requires analysis of the media sent from other participants. In a conventional audio conference this is only possible for the mixer since all other conference participants are generally receiving a single, flat audio stream and have therefore no immediate way of determining individual audio levels.

This document specifies an RTP extension header that allows such mixers to deliver audio level information to conference participants by including it directly in the RTP packets transporting the corresponding audio data.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

3. Protocol Operation

According to RFC 3550 [RFC3550] a mixer is expected to include in outgoing RTP packets a list of identifiers (CSRC IDs) indicating the sources that contributed to the resulting stream. The presence of such CSRC IDs allows RTP clients to determine, in a binary way, the active speaker(s) in any given moment. RTCP also provides a basic mechanism to map the CSRC IDs to user identities through the CNAME field. More advanced mechanisms, may exist depending on the signaling protocol used to establish and control a conference. In the case of the Session Initiation Protocol [RFC3261] for example, the Event Package for Conference State [RFC4575] defines a <src-id> tag which binds CSRC IDs to media streams and SIP URIs.

This document describes an RTP header extension that allows mixers to indicate the audio-level of every conference participant (CSRC) in addition to simply indicating their on/off status. This new header extension uses "General Mechanism for RTP Header Extensions" described in [RFC5285].

Each instance of this header contains a list of one-octet audio levels expressed in -dBov, with values from 0 to 127 representing 0 to -127 dBov(see Section 4 and Section 5). Appendix Appendix A provides a reference implementation indicating one way of obtaining such values from raw audio samples.

Every audio level value pertains to the CSRC identifier located at the corresponding position in the CSRC list. In other words, the first value would indicate the audio level of the conference participant represented by the first CSRC identifier in that packet and so forth. The number and order of these values MUST therefore match the number and order of the CSRC IDs present in the same packet.

When encoding audio level information, a mixer SHOULD include in a packet information that corresponds to the audio data being transported in that same packet. It is important that these values follow the actual stream as closely as possible. Therefore a mixer SHOULD also calculate the values after the original contributing stream has undergone possible processing such as level normalization, and noise reduction for example.

Note that in some cases a mixer may be sending an RTP audio stream that only contains audio level information and no actual audio. Updating a (web) interface conference module may be one reason for this to happen.

It may sometimes happen that a conference involves more than a single mixer. In such cases each of the mixers MAY choose to relay the CSRC list and audio-level information they receive from peer mixers (as long as the total CSRC count remains below 16). Given that the maximum audio level is not precisely defined by this specification, it is likely that in such situations average audio levels would be perceptibly different for the participants located behind the different mixers.

4. Header Format

The audio level indicators are delivered to the receivers in-band using the "General Mechanism for RTP Header Extensions" [RFC5285]. The payload of this extension is an ordered sequence of 8-bit audio level indicators encoded as per Section 5.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  ID   |  len  |0|  level 1    |0|  level 2    |0|  level 3   ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
          

Note that use of the two-byte header defined in RFC 5285 [RFC5285] follows the same rules the only change being the length of the ID and len fields.

5. Audio level encoding

The audio level header extension only carries the level of the audio in the RTP payload of the packet it is associated with. This information is carried in an RTP header extension element as defined by [RFC5285].

The audio level is defined in the same manner as is audio noise level in the RTP Payload Comfort Noise specification [RFC3389]. The overall magnitude of the noise level is encoded into the first byte of the payload, with spectral information about the noise in subsequent bytes. This specification's audio level parameter is defined so as to be identical to the comfort noise payload's noise-level byte.

                        0 1 2 3 4 5 6 7
                       +-+-+-+-+-+-+-+-+
                       |0|   level     |
                       +-+-+-+-+-+-+-+-+

        

The magnitude of the audio level is packed into the seven least significant bits of the single byte of the header extension, shown in Figure 3. The least significant bit of the audio level magnitude is packed into the least significant bit of the byte. The most significant bit of the byte is unused and always set to 0 as shown below in Figure 3.

The two-byte header defined in RFC 5285 [RFC5285] may also be used.

The audio level is expressed in -dBov, with values from 0 to 127 representing 0 to -127 dBov. dBov is the level, in decibels, relative to the overload point of the system, i.e. the maximum-amplitude signal that can be handled by the system without clipping.(Note: Representation relative to the overload point of a system is particularly useful for digital implementations, since one does not need to know the relative calibration of the analog circuitry.) For example, in the case of u-law (audio/pcmu) audio [ITU.G.711], the 0 dBov reference would be a square wave with values +/- 8031. (This translates to 6.18 dBm0, relative to u-law's dBm0 definition in Table 6 of G.711.)

The audio level for digital silence, for example for a muted audio source, MAY be represented as 127 (-127 dBov), regardless of the dynamic range of the encoded audio format.

Implementations MAY choose to measure audio levels prior to encoding them in the payload carried in the RTP payload, e.g. on raw linear PCM input.

The audio level header extension only carries the level of the audio in the RTP payload of the packet it is associated with, with no long-term averaging or smoothing applied.

To simplify implementation of the encoding procedures described here, this specification provides a sample Java implementation [ri] of an audio level calculator that helps obtain such values from raw linear PCM audio samples.

6. Signaling Information

The URI for declaring the audio level header extension in an SDP extmap attribute and mapping it to a local extension header identifier is "urn:ietf:params:rtp-hdrext:csrc-audio-level". There is no additional setup information needed for this extension (i.e. no extensionattributes).

An example attribute line in the SDP, for a conference might be:

        a=extmap:7 urn:ietf:params:rtp-hdrext:csrc-audio-level
        

The above mapping will most often be provided per media stream (in the media-level section(s) of SDP, i.e., after an "m=" line) or globally if there is more than one stream containing audio level indicators in a session.

Presence of the above attribute in the SDP description of a media stream indicates that some or all RTP packets in that stream would contain the audio level information RTP extension header.

Conferencing clients that support audio level indicators and have no mixing capabilities would not be able to content for this audio level extension and would hence have to always include the direction parameter in the "extmap" attribute with a value of "recvonly". Conference focus entities with mixing capabilities can omit the direction or set it to "sendrecv" in SDP offers. Such entities would need to set it to "sendonly" in SDP answers to offers with a "recvonly" parameter and to "sendrecv" when answering other "sendrecv" offers.

This speicification does not define use of the audio level extensions in video streams. Therefore, the extension defined in this document SHOULD NOT be advertised in anything but audio streams.

The following Figure 5 and Figure 6 show two example offer/answer exchanges between a conferencing client and a focus, and between two conference focus entities.

  v=0
  o=alice 2890844526 2890844526 IN IP6 host.example.com
  c=IN IP6 host.example.com
  t=0 0
  m=audio 49170 RTP/AVP 0 4
  a=rtpmap:0 PCMU/8000
  a=rtpmap:4 G723/8000
  a=extmap:1/recvonly urn:ietf:params:rtp-hdrext:csrc-audio-level

  v=0
  i=A Seminar on the session description protocol
  o=conf-focus 2890844730 2890844730 IN IP6 focus.example.net
  c=IN IP6 focus.example.net
  t=0 0
  m=audio 52543 RTP/AVP 0
  a=rtpmap:0 PCMU/8000
  a=extmap:1/sendonly urn:ietf:params:rtp-hdrext:csrc-audio-level
        

A client-initiated example SDP offer/answer exchange negotiating an audio stream with one-way flow of of audio level information.

  v=0
  i=Un seminaire sur le protocole de description des sessions
  o=fr-focus 2890844730 2890844730 IN IP6 focus.fr.example.net
  c=IN IP6 focus.fr.example.net
  t=0 0
  m=audio 49170 RTP/AVP 0
  a=rtpmap:0 PCMU/8000
  a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-level

  v=0
  i=A Seminar on the session description protocol
  o=us-focus 2890844526 2890844526 IN IP6 focus.us.example.net
  c=IN IP6 focus.us.example.net
  t=0 0
  m=audio 52543 RTP/AVP 0
  a=rtpmap:0 PCMU/8000
  a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:csrc-audio-level
        

An example SDP offer/answer exchange between two conference focus entities with mixing capabilities negotiating an audio stream with bidirectional flow of audio level information.

7. Security Considerations

1. This document defines a means of attributing audio level to a particular participant in a conference. An attacker may try to modify the content of RTP packets in a way that would make audio activity from one participant appear as coming from another.
2. Furthermore, the fact that audio level values would not be protected even in an SRTP session might be of concern in some cases where the activity of a particular participant in a conference is confidential.
3. Both of the above are concerns that stem from the design of the RTP protocol itself and they would probably also apply when using CSRC identifiers the way they were specified in RFC 3550 [RFC3550]. It is therefore important that according to the needs of a particular scenario, implementors and deployers consider use of header extension encryption [I-D.lennox-avtcore-srtp-encrypted-header-ext] or a lower level security and authentication mechanism.

8. IANA Considerations

This document defines a new extension URI that, if approved, would need to be added to the RTP Compact Header Extensions sub-registry of the Real-Time Transport Protocol (RTP) Parameters registry, according to the following data:

        Extension URI: urn:ietf:params:rtp-hdrext:csrc-audio-level
        Description:   Mixer-to-client audio level indicators
        Contact:       emcho@jitsi.org
        Reference:     RFC XXXX
        

Note to the RFC-Editor: please replace "RFC XXXX" by the number of this RFC.

9. Acknowledgments

Lyubomir Marinov contributed level measurement and rendering code.

Roni Even, Keith Drage, Ingemar Johansson, Michael Ramalho and several others provided helpful feedback over the dispatch mailing list.

Jitsi's participation in this specification is funded by the NLnet Foundation.

10. Changes From Earlier Versions

Note to the RFC-Editor: please remove this section prior to publication as an RFC.

10.1. Changes From Draft -01

• Removed code related the AudioLevelRenderer from "APPENDIX A. Reference Implementation" as it was considered an implementation matter by the working group.
• Modified the AudioLevelCalculator in "APPENDIX A. Reference Implementation" to take overload as a parameter.
• Clarified non-use of audio levels in video streams
• Closed the P.56 open issue. It was agreed on IETF 80 that P.56 is mostly about speech levels and the levels transported by the extension defined here should also be able to serve as an indication for noise.
• The Open Issues section has been removed as all issues that were in there are now resolved or clarified.
• Editorial changes for consistency with [I-D.ietf-avtext-client-to-mixer-audio-level].

10.2. Changes From Draft -00

• Added code for sound pressure calculation and measurement in "APPENDIX A. Reference Implementation".
• Changed affiliation for Emil Ivov.
• Removed "Appendix: Design choices".

11. References

11.1. Normative References

11.2. Informative References

Appendix A. Reference Implementation

This appendix contains Java code for a reference implementation of the level calculation and rendering methods.The code is not normative and by no means the only possible implementation. Its purpose is to help implementors add audio level support to mixers and clients.

The Java code contains an AudioLevelCalculator class that calculates the sound pressure level of a signal with specific samples. It can be used in mixers to generate values suitable for the level extension headers.

The implementation is provided in Java but does not rely on any of the language specific and can be easily ported to another.

Appendix A.1. AudioLevelCalculator.java

/**
 * Calculates the audio level of specific samples of a singal based on
 * sound pressure level.
 */
public class AudioLevelCalculator
{

    /**
     * Calculates the sound pressure level of a signal with specific
     * <tt>samples</tt>.
     *
     * @param samples the samples of the signal to calculate the sound
     * pressure level of. The samples are specified as an <tt>int</tt>
     * array starting at <tt>offset</tt>, extending <tt>length</tt>
     * number of elements and each <tt>int</tt> element in the specified
     * range representing a sample of the signal to calculate the sound
     * pressure level of. Though a sample is provided in the form of an
     * <tt>int</tt> value, the sample size in bits is determined by the
     * caller via <tt>overload</tt>.
     *
     * @param offset the offset in <tt>samples</tt> at which the samples
     * start
     *
     * @param length the length of the signal specified in
     * <tt>samples<tt> starting at <tt>offset</tt>
     *
     * @param overload the overload (point) of <tt>signal</tt>.
     * For example, <tt>overload</tt> may be {@link Byte#MAX_VALUE}
     * for 8-bit signed samples or {@link Short#MAX_VALUE} for
     * 16-bit signed samples.
     *
     * @return the sound pressure level of the specified signal
     */
    public static int calculateSoundPressureLevel(
        int[] samples, int offset, int length,
        int overload)
    {
        /*
         * Calcuate the root mean square of the signal i.e. the
         * effective sound pressure.
         */
        double rms = 0;

        for (; offset < length; offset++)
        {
            double sample = samples[offset];

            sample /= overload;
            rms += sample * sample;
        }
        rms = (length == 0) ? 0 : Math.sqrt(rms / length);

        /*
         * The sound pressure level is a logarithmic measure of the
         * effectivesound pressure of a sound relative to a reference
         * value and is measured in decibels.
         */
        double db;

        /*
         * The minimum sound pressure level which matches the maximum
         * of the sound meter.
         */
        final double MIN_SOUND_PRESSURE_LEVEL = 0;
        /*
         * The maximum sound pressure level which matches the maximum
         * of the sound meter.
         */
        final double MAX_SOUND_PRESSURE_LEVEL
            = 127 /* HUMAN TINNITUS (RINGING IN THE EARS) BEGINS */;

        if (rms > 0)
        {
            /*
             * The commonly used "zero" reference sound pressure in air
             * is 20 uPa RMS, which is usually considered the threshold
             * of human hearing.
             */
            final double REF_SOUND_PRESSURE = 0.00002;

            db = 20 * Math.log10(rms / REF_SOUND_PRESSURE);

            /*
             * Ensure that the calculated level is within the minimum
             * and maximum sound pressure level.
             */
            if (db < MIN_SOUND_PRESSURE_LEVEL)
                db = MIN_SOUND_PRESSURE_LEVEL;
            else if (db > MAX_SOUND_PRESSURE_LEVEL)
                db = MAX_SOUND_PRESSURE_LEVEL;
        }
        else
        {
            db = MIN_SOUND_PRESSURE_LEVEL;
        }

        return (int) db;
    }
}

          

AudioLevelCalculator.java

Authors' Addresses