Network Working Group | H. T. Alvestrand |
Internet-Draft | |
Intended status: Standards Track | February 25, 2013 |
Expires: August 29, 2013 |
Resolution Constraints in Web Real Time Communications
draft-alvestrand-constraints-resolution-02
This document specifies the constraints necessary for a Javascript application to successfully indicate to a browser that supports WebRTC what resolutions it desires on a video stream.
It also discusses the possible use of SDP to carry that information between browsers.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 29, 2013.
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
There are a number of scenarios where it's useful for a WebRTC application to indicate to the WebRTC implementation in the supported browser what the desired characteristics of a video stream are. These include, but are not limited to:
Similar considerations apply for framerate.
This draft is written in order to get something specific out to refer to during spec-writing and implementation. The text may eventually get merged into the JSEP specification, [I-D.ietf-rtcweb-jsep].
Consider the following (simplified) model of a video stream through a WebRTC application:
|<-------------- Browser A -------------------->| Camera ---> MediaStream A ---> Peerconnection A ------+ |<------- Application A ---------->| | v ^ v Signalling channel Internet (media) v ^ | |<------- Application B ---------->| | <video> tag <-- MediaStream B <--- Peerconnection B --+ |<-------------Browser B ----------------------->|
Both applications are running in browsers, with Application A connected to a camera that is able to deliver video streams up to HD quality (1280x720).
At one particular moment in time, the <video> tag in Application B is rendered as a thumbnail, and video is flowing to it in a 160x100 resolution; there is no need to send any more data, since no more pixels are available for its display anyway.
Then the user of Application B hits the "full-screen" button. There are now 1600x1200 pixels available for display.
Initially, Application B will splay the 160x100 image across the larger surface, because there is no other choice, but it will desire to have as many pixels as possible available to provide a high quality image.
At one particular moment in time, the camera is generating 1280x720, resulting in a 2 Mbits/second data flow from A to B. Congestion control signals that this data rate is no longer available; rather than letting the browser reduce the bandwidth of some flow of its choice, Application A decides that the high definition video is the feature that is least valuable. It can then apply a new constraint to Mediastream A, specifying that resolution should be at most 640x360; browser A is then responsible for making sure this decision is communicated to browser B (if it needs to be).
If application B is running on a slow machine (2000-class PC or 2010-class mobile phone), the maximum capacity of the video decoder may be 320x200 - Application B may then wish to indicate that application A should limit the stream sent across the network to that resolution - sending more bits isn't useful, because the receiver doesn't have enough capacity to decode and downscale the video stream.
As specified in the "v6 Settings model" being developed in the Media Capture Task Force (snapshot: http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/proposals/SettingsAPI_proposal_v6.html), the consumer of a video track in a MediaStream will have a "native resolution", which indicates what size video it's useful to push to it. The application can also set (and change) constraints on the video MediaStreamTrack, indicating which range of properties it sees useful for the purposes of the application.
In SDP, the "a=imageattr" attribute is available to provide information on the resolution of video streams described by an SDP m-line; a proposal [I-D.lennox-mmusic-sdp-source-selection] has been floated to provide similar information on a per-SSRC basis.
If both mechanisms are available, the choices available to the writer of application B in the "increase screen area" above are:
The advantage of the first method is that it does not require any SDP parsing or generation.
The advantage of the second method is that it will work when appliation A and application B are different applications; there is no need for them to have any private agreement on how to set bitrate. It does require both the implementation of constraints and that browser B has the ability to generate the proper constraints in the SDP.
The third method requires SDP parsing in browser A, but not SDP generation in browser B. It does require SDP manipulation in Javascript at application B.
In order to have either of the approaches to work, the specific constraints to use have to be defined. This section defines the constraints for resolution and framerate needed.
These constraints are usable in several places:
All of the constraints may be meaningful in both "mandatory" and "optional" forms.
See Section 6 for the actual definition of the constraints used here.
A constraint saying that we absolutely must have a minimum resolution of 1024x768:
getUserMedia({ video: { mandatory: { minWidth: 1024, minHeight: 768 } } }, successCallback, errorCallback);
A constraint saying that we'd prefer 60 frames per second, if available, and if we can get that, we'd like to limit the max resolution, but in all cases, the screen must be clamped to a 4:3 aspect ratio - 16:9 or odd aspect ratios are not acceptable to this application:
getUserMedia({ video: { mandatory: { minAspectRatio: 1.333, maxAspectRatio: 1.334 }, optional [ { minFrameRate: 60 }, { maxWidth: 640 }, { maxHeigth: 480 } ] } }, successCallback, errorCallback);
This document does not specify the exact mapping of constraints into imageattr values; this will have to be done before this mechanism can be depended on.
The examples below are thought exercises, based on [I-D.lennox-mmusic-sdp-source-selection] and [I-D.alvestrand-rtcweb-resolution].
An optional constraint has been applied to an incoming stream where both upper and lower are constrained to 320x200. The stream has been assigned to a hardware video decoder that can decode most resolutions up to 1024x768, in any aspect ratio, but only if all divisions are divisible by 4. The incoming stream has SSRC 1234.
Escaped line breaks are added for readability.
m=video a=remote-ssrc:1234 imageattr:* [x=320,y=200,q=1.0] \ [x=[120:4:1024],y=[100:4:768],q=0.2]
This document requests IANA to register constraints in the "RTCWeb Media Constraints" registry created by [I-D.burnett-rtcweb-constraints-registry]. NOTE: The registrations assume that this document is updated to no longer have "video" as part of the name, but have "video" as a field-of-use in the registration.
The definitions of width, height and aspect ratio are taken from [RFC6236].
The contact person is Harald Alvestrand <hta@google.com>.
Change control for the registration is with the IETF, as designated by the IESG.
Note that minFramerate defines a lower bound for the a=framerate attribute, which is itself defined as an upper limit; this means that even if a high framerate is negotiated, the actual framerate used may be lower due to temporary considerations (for instance CPU or bandwidth, or simply lack of movement in the picture).
No security considerations particular to these specific constraints have so far been identified.
Special thanks are given to Dan Burnett, Cullen Jennings, the IETF RTCWEB WG and the W3C WEBRTC WG for strongly influencing this memo, and to Per Kjellander for being the first to implement the constraints in getUserMedia.
[RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
[RFC4566] | Handley, M., Jacobson, V. and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006. |
[RFC6236] | Johansson, I. and K. Jung, "Negotiation of Generic Image Attributes in the Session Description Protocol (SDP)", RFC 6236, May 2011. |
[I-D.burnett-rtcweb-constraints-registry] | Burnett, D., "IANA Registry for RTCWeb Media Constraints", Internet-Draft draft-burnett-rtcweb-constraints-registry-02, October 2012. |
[I-D.lennox-mmusic-sdp-source-selection] | Lennox, J. and H. Schulzrinne, "Mechanisms for Media Source Selection in the Session Description Protocol (SDP)", Internet-Draft draft-lennox-mmusic-sdp-source-selection-05, October 2012. |
[I-D.alvestrand-rtcweb-resolution] | Alvestrand, H., "RTCWEB Resolution Negotiation", Internet-Draft draft-alvestrand-rtcweb-resolution-00, April 2012. |
[I-D.ietf-rtcweb-jsep] | Uberti, J. and C. Jennings, "Javascript Session Establishment Protocol", Internet-Draft draft-ietf-rtcweb-jsep-02, October 2012. |
[W3C.WD-mediacapture-streams-20120628] | Burnett, D. and A. Narayanan, "Media Capture and Streams", World Wide Web Consortium WD WD-mediacapture-streams-20120628, June 2012. |
[W3C.WD-webrtc-20120821] | Bergkvist, A., Burnett, D., Narayanan, A. and C. Jennings, "WebRTC 1.0: Real-time Communication Between Browsers", World Wide Web Consortium WD WD-webrtc-20120821, August 2012. |
Added the "Usage Scenarios" chapter.
Repointed the eventual target to be incorporation in the JSEP draft.
Made sure the constraints are consistently spelled in camelCase, with a small initial letter.
Moved a bit of the text around between sections, and referred to the "settings API" proposal from the Media Capture task force.