Internet DRAFT - draft-muthusamy-dispatch-haptics
draft-muthusamy-dispatch-haptics
DISPATCH Y. K. Muthusamy
Internet-Draft C. Ullrich
Intended status: Standards Track Immersion Corporation
Expires: 15 November 2021 14 May 2021
The 'haptics' Top-level Media Type
draft-muthusamy-dispatch-haptics-02
Abstract
This memo serves to register and document the 'haptics' top-level
media type, under which subtypes for representation formats for
haptics may be registered. This document also serves as a
registration application for a set of intended subtypes, which are
representative of some existing subtypes already in use.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 15 November 2021.
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Simplified BSD License text
as described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Simplified BSD License.
Muthusamy & Ullrich Expires 15 November 2021 [Page 1]
RFC XXXX The 'haptics' Top-level Media Type May 2021
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
2. Background and Justification . . . . . . . . . . . . . . . . 3
2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 4
2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4
2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4
2.5. Haptic Data Formats in Use . . . . . . . . . . . . . . . 5
2.6. Haptic Subtypes (envisioned standards) . . . . . . . . . 6
2.7. 'application' top-level type not suitable . . . . . . . . 6
3. Security Considerations . . . . . . . . . . . . . . . . . . . 7
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
4.1. Definition and Encoding . . . . . . . . . . . . . . . . . 8
4.2. Registration Procedure . . . . . . . . . . . . . . . . . 8
4.3. Subtype Registrations . . . . . . . . . . . . . . . . . . 8
4.3.1. IVS Haptics Type . . . . . . . . . . . . . . . . . . 8
4.3.2. HAPT Haptics Type . . . . . . . . . . . . . . . . . . 9
5. Normative References . . . . . . . . . . . . . . . . . . . . 10
6. Informative References . . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
1. Introduction
The term 'haptics' refers to the generation of touch-related
sensations in a device or interface. Haptics is widely used in
consumer devices in order to provide touch-based feedback to users.
The most common use of haptics is in mobile devices, where it is used
to provide feedback to users interacting with the touchscreen, e.g.,
typing on a virtual keyboard. Haptic technologies are unlike audio
and visual enabling technologies in the sense that they require some
form of actuation in order to create a tactile sensation. For mobile
phones and game controllers, these actuators are typically small
vibrating motors. For large touchscreens in vehicles, these
actuators can be specialized piezoelectric materials. Haptic
capabilities are found in nearly every modern smartphone and game and
virtual reality controller, making these devices an ideal target for
enhanced media experiences.
Internet Media Types [RFC6838] are used to label content carried over
Internet protocols. This document defines a new top-level type
'haptics' according to Section 4.2.7 of [RFC6838]. This top-level
type indicates that the content specifies haptic data. Under this
top-level type, different representation formats of haptics may be
registered.
Muthusamy & Ullrich Expires 15 November 2021 [Page 2]
RFC XXXX The 'haptics' Top-level Media Type May 2021
1.1. Terminology
The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT,
SHOULD,SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear
in this document, are to be interpreted as described in [RFC2119].
2. Background and Justification
Haptic signals provide an additional layer of entertainment and
sensory immersion for the user. Haptic tracks, in separate files,
can be combined with audio/video files and played back in sync to
provide an overall immersive media experience (audio, visual,
tactile) for the user. More recently, haptic tracks embedded in
standard file formats such as ISOBMFF (ISO Base Media File Format),
enable playback of the haptic signals over one or more actuators,
simultaneously with audio and video playback.
2.1. MPEG ISOBMFF
Historically, there has not been a registration of formats for
haptics. However, haptics has been proposed as a first-order media
type (at the same level as audio and video) in ISOBMFF. This
proposal was made to the MPEG Systems File Format sub-group in April
2020. The proposal was accepted and has progressed to DIS (Draft
International Standard) in October 2020 [ISOBMFF-AMD1][AMD1-Status].
Once it completes its progression through the MPEG standardization
stages (expected in October 2021), haptics will become part of the
ISO/IEC 14496-12 (ISOBMFF) standard. Given this development, a
strong case can be made for haptics to be added to the list of top-
level media type recognized by the IETF.
We envision the following designations for haptics in mp4 files, once
the top-level type 'haptics' is registered:
1. 'haptics/mp4' - mp4 files with just haptic tracks in them (e.g.,
streaming games, haptics files for haptic vests, belts, gloves,
etc.)
2. 'video/mp4' - mp4 files with video, audio, and haptics (to ensure
consistency with existing mp4 files with video content)
3. 'audio/mp4' - mp4 files with audio and haptics (to ensure
consistency with existing mp4 files with audio content without
any video)
Muthusamy & Ullrich Expires 15 November 2021 [Page 3]
RFC XXXX The 'haptics' Top-level Media Type May 2021
2.2. Haptic Sub-modalities
There are multiple sub-modalities of haptics:
* Vibrotactile (touch, vibration)
* Kinesthetic (force feedback)
* Surface (surface friction)
* Spatial, non-contact (ultrasound)
* Thermal (temperature)
Therefore, designating 'haptics' as a top-level media type would
enable the definition of data formats pertaining to these sub-
modalities in a more streamlined manner. This would not be possible
if 'haptics' were to be placed under other top-level types like
'audio', 'video', or 'application'.
2.3. Another Human Sense
The top-level media type 'audio' pertains to the human sense of
hearing, the top-level media type 'video' pertains to the human sense
of seeing, so it only makes sense for the (equally important) human
sense of touch to be represented by another top-level media type
'haptics'. Placing 'haptics' under 'audio' or 'video' is not
reflective of the kinds of files or use cases that would need haptics
but have nothing whatsoever to do with audio or video.
2.4. Commercial Uptake
Haptics is rapidly becoming a standard feature of consumer electronic
devices. For example:
* iPhone (191+ million units sold in 2019): native support for
haptic encoded data
* Android (1.18+ billion units sold in 2019): API support of haptic
buffers
* W3C (HTML vibration API): Optionally supported in mobile web
browsers
* Game consoles (39+ million units sold in 2019): MS Xbox, Sony
PlayStation, Nintendo Switch, etc.
* XR devices (9+ million units sold in 2019): OpenXR haptic API
Muthusamy & Ullrich Expires 15 November 2021 [Page 4]
RFC XXXX The 'haptics' Top-level Media Type May 2021
Haptic media is expected to be commonly exchanged between these
devices. Since they represent the majority of CE devices, a strong
case can be made for 'haptics' as a top-level media type.
2.5. Haptic Data Formats in Use
There are multiple instances of existing haptic data formats that
would live as sub-types under the proposed 'haptics' top-level media
type. While these subtypes have *not* been registered with IANA or
standardized (yet), the prevalence of these haptic data formats in a
large number of devices around the world, pre-dating the
standardization of haptic tracks in ISOBMFF, provides a compelling
argument for 'haptics' to be designated as a top-level media type:
* 'ahap': The AHAP haptic data format [AHAP] is currently the
standard encoding on all iOS devices + iOS connected game
peripherals. The format has seen usage and adoption beyond Apple
devices as well, with decoders available for Android and other XR
systems.
* 'ogg': Google has introduced a proprietary extension to the OGG
format in the latest version of Android 11. This encoding enables
haptic media to be stored in OGG files.
* 'ivs': The IVS haptic data format is currently a vendor-specific
format that is in use:
- In mobile phones from LG Electronics (specifically, the models
V30, V40, and the newest V50) that are sold worldwide
- In gaming phones from ASUS (specifically, models ROG, ROG Phone
II, ROG Phone 3) that are sold worldwide
* 'hapt': The HAPT haptic data format is currently a vendor-specific
format that is in use:
- In mobile haptic advertising (for W3C devices)
- The following Japanese game developers use the HAPT format as
part of Immersion's TouchSense SDK:
o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/)
o Craft&Meister: http://www.crafts-meister.co.jp/pc/
company_en.html (http://www.crafts-meister.co.jp/pc/
company_en.html)
Muthusamy & Ullrich Expires 15 November 2021 [Page 5]
RFC XXXX The 'haptics' Top-level Media Type May 2021
- Tencent is using the TouchSense SDK for their popular social
media application QQ and live streaming application NOW:
Immersion-Announces-Tencent-Licenses-TouchSense-Technology-
Deliver
(https://www.businesswire.com/news/home/20171026006443/en/
Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE-
Technology-Deliver)
Given the widespread use of these subtypes, it makes sense for
'haptics' to be a top-level media type.
2.6. Haptic Subtypes (envisioned standards)
The MPEG ISOBMFF proposal included an informative annex of known
haptic coding formats with proposed FourCC codes for them. These
codes are not registered yet, but the plan is indeed to standardize
these haptic coding formats in the near future. Once standardized,
they will also live as subtypes under the proposed 'haptics' top-
level media type:
* 'hmpg': the selected coding format from the MPEG Call for
Proposals on the Coded Representation of Haptics
[MPEG-Haptics-CfP]
* 'hiee': IEEE P1918.1.1 vibrotactile coding standard being
developed under the IEEE Tactile Internet initiative as part of
the 5G URLL profile.
* 'henm': enumerated effects haptic coding format (based on MIDI)
* 'havc': audio-to-vibe haptic coding format (automatic audio to
vibration conversion algorithms)
2.7. 'application' top-level type not suitable
From the above arguments, it is clear that haptics does not really
belong under any other media type. To reiterate, there are three
main reasons why the 'haptics' media type does not fit under the
'application' top-level type:
* haptics connects to a sensory system, touch/motion, directly, and
is more specific than the abstract 'application' type, and
* 'application' has historically been used for applications, i.e.,
code, which means it is viewed and treated with great care for
security. 'haptics' is not code, just as 'audio' and 'video' are
not code either.
Muthusamy & Ullrich Expires 15 November 2021 [Page 6]
RFC XXXX The 'haptics' Top-level Media Type May 2021
* haptics is a property of a media stream, it is not an application
under any normal definition. As such, it should be its own type.
3. Security Considerations
Haptics are interpreted data structures that represent collections of
different media rendering instructions intended to be decoded and
rendered on target device hardware. Haptic data can be represented
as collections of signal data and/or descriptive text in XML/JSON or
similar format. Signal data is typically not executed by endpoint
processors and represents minimal security risk. Descriptive text is
typically parsed and represented in memory using standard XML data
structures. This data is utilized to construct one or more signals
that are sent to the endpoint device hardware.
Because of the media/rendering nature of the data path for haptic
coded data the security profile of haptic data is expected to be
largely consistent with the security profile of visual and audio
media data.
As with any synthesized media data (audio, video, and haptics), there
is a security risk associated with execution of commands based on the
descriptive encoding either through its inherent extensibility or
through the insertion of arbitrary executable data in the descriptive
format itself. Indeed, media rendering systems are normally
implemented with a mix of user and kernel space execution since these
media must ultimately make their way to a hardware system. In
theory, malicious instructions present in descriptive haptic media
have the potential to execute arbitrary code in kernel space,
effectively bypassing system permissions structures and/or execution
sandboxes.
Haptics, audio, and video media have widespread use and careful
attention should be paid by operating system and device driver
implementors to ensure that synthesis and rendering signal paths do
not provide attack surfaces for malicious payloads. Ultimately, any
coded representation of haptic media is insufficient to implicitly
provide sufficient security and this protection should be enforced by
the operating system implementor.
4. IANA Considerations
This specification registers a new top-level type, 'haptics', in the
standards tree, adds it as an alternative value of "Type Name" in the
media types registration form [Media-Type-Registration], and
registers several subtypes for it.
Muthusamy & Ullrich Expires 15 November 2021 [Page 7]
RFC XXXX The 'haptics' Top-level Media Type May 2021
4.1. Definition and Encoding
'haptics' as the primary media content type indicates that the
content identified by it requires a certain haptics subsystem such as
low-level haptics APIs, which in turn will require hardware
capabilities such as one or more actuators to render the haptics
media. The 'haptics' media type does not provide any specific
information about the underlying data format and how the haptics
information should be interpreted -- the subtypes defined within a
'haptics' tree name the specific haptic formats. Unrecognized
subtypes of 'haptics' should be treated as 'application/octet-
stream'. Implementations may still pass unrecognized subtypes to the
haptics subsystem and associated rendering hardware.
4.2. Registration Procedure
New haptics formats should be registered using the online form
[Media-Type-Registration]. [RFC6838] should be consulted on
registration procedures. In particular, the haptics specification
should preferably be freely available.
Note that new parameter sub-values may be defined in the future. If
an implementation does not recognize a sub-value in the comma-
separated list, it should ignore the sub-value and continue
processing the other sub-values in the list.
4.3. Subtype Registrations
In this section, the initial entries under the top-level 'haptics'
media type are specified. They also serve as examples for future
registrations.
4.3.1. IVS Haptics Type
Type name: haptics
Subtype name: ivs
Required parameters: None
Optional parameters: None
Encoding considerations: Text/binary
Interoperability considerations: The IVS format is a device-
independent haptic effect coding. It is designed to enable
interoperability between distinct physical endpoints. Not all
devices may be able to render all effects present in an IVS file.
Muthusamy & Ullrich Expires 15 November 2021 [Page 8]
RFC XXXX The 'haptics' Top-level Media Type May 2021
Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 72 "Encoder Input
Format for Haptics" being developed by ISO/IEC JTC1/SC29 WG 2
[MPEG-Haptics-Encoder-Format].
Applications that use this media type: All applications that are able
to create, edit, or display haptic media content.
Additional information:
* File extension(s): Haptic file extensions used for IVS files: .ivs
(xml) and .ivt (binary)
* Macintosh file type code(s): (no code specified)
* Macintosh Universal Type Identifier code: None
* Fragment Identifier: None
* Deprecated Alias: None
Person & email address to contact for further information: Yeshwant
Muthusamy(ymuthusamy@immersion.com)
Change controller: Immersion Corporation
4.3.2. HAPT Haptics Type
Type name: haptics
Subtype name: hapt
Required parameters: None
Optional parameters: None
Encoding considerations: Text/binary
Interoperability considerations: The HAPT format is a device-
dependent haptic effect coding based on the RIFF coding standard. It
is designed to enable efficient coding of a device-specific haptic
effect.
Published specification: HAPT is a logical extension of the RIFF
standard [RIFF]
Applications that use this media type: All applications that are able
to create, edit, or display haptic media content.
Muthusamy & Ullrich Expires 15 November 2021 [Page 9]
RFC XXXX The 'haptics' Top-level Media Type May 2021
Additional information:
* File extension(s): Haptic file extensions used for HAPT files:
.hapt
* Macintosh file type code(s): (no code specified)
* Macintosh Universal Type Identifier code: None
* Fragment Identifier: None
* Deprecated Alias: None
Person & email address to contact for further information: Yeshwant
Muthusamy(ymuthusamy@immersion.com)
Change controller: Immersion Corporation
5. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type
Specifications and Registration Procedures", BCP 13,
RFC 6838, DOI 10.17487/RFC6838, January 2013,
<https://www.rfc-editor.org/info/rfc6838>.
6. Informative References
[ISOBMFF-AMD1]
"ISO/IEC 14496-12/CD AMD 1 Information technology - Coding
of audio-visual objects - Part 12: ISO base media file
format - Amendment 1: Support for new media types
(haptics, volumetric visual) and other improvements",
<https://www.iso.org/standard/81604.html>.
[AMD1-Status]
"Timeline of ISO/IEC 14496-12/CD Amd 1, Edition 6 -
Project ID: 81604, ISO/IEC JTC 1/SC 29/WG 3",
<https://sd.iso.org/projects/project/81604/overview>.
[AHAP] "Apple Haptic Audio Pattern",
<https://developer.apple.com/documentation/corehaptics/
representing_haptic_patterns_in_ahap_files>.
Muthusamy & Ullrich Expires 15 November 2021 [Page 10]
RFC XXXX The 'haptics' Top-level Media Type May 2021
[MPEG-Haptics-CfP]
"Call for Proposals on the Coded Representation of Haptics
- Phase 1", <https://www.mpegstandards.org/standards/
Explorations/40/>.
[MPEG-Haptics-Encoder-Format]
"Encoder Input Format for Haptics",
<https://www.mpegstandards.org/standards/
Explorations/40/>.
[Media-Type-Registration]
"IANA, Application for a Media Type",
<http://www.iana.org/form/media-types>.
[RIFF] "Resource Interchange File Format",
<https://www.loc.gov/preservation/digital/formats/fdd/
fdd000025.shtml>.
Authors' Addresses
Yeshwant K. Muthusamy
Immersion Corporation
330 Townsend St. Suite 234
San Francisco, CA 94107
United States of America
Phone: +1 469-583-2171
Email: ymuthusamy@immersion.com
Chris Ullrich
Immersion Corporation
330 Townsend St. Suite 234
San Francisco, CA 94107
United States of America
Phone: +1 805-320-0774
Email: cullrich@immersion.com
Muthusamy & Ullrich Expires 15 November 2021 [Page 11]