Internet DRAFT - draft-km-virt-orchestra-research-challenges
draft-km-virt-orchestra-research-challenges
Independent Submission K. Makhijani
Internet-Draft L. Dong
Intended status: Informational Futurewei
Expires: 13 January 2022 12 July 2021
Virtual Orchestra Usecase and Research Challenges
draft-km-virt-orchestra-research-challenges-00
Abstract
This document describes open research challenges for emerging media-
oriented ensemble applications. One such driving scenario is the
network delivery of virtual orchestra that imposes multi-disciplinary
challenges. Specifically, of interest are the group communication
patterns in the production, delivery and consumption as different
dimensions relating to the communication networks.
This document brings forth current research and engineering
challenges in immersive media ensembles. The network domain problems
come down to the specification of coordination of the received
content with dependency constraints. The challenges depict both real
and quasi- realtime behavior. A number of endpoint actors get
involved in delivering the ensemble aspect, the research challenges
also describe the expectations from the end points.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 13 January 2022.
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved.
Makhijani & Dong Expires 13 January 2022 [Page 1]
Internet-Draft virtual-orchestra-research-challenges July 2021
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Simplified BSD License text
as described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction and Scope . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Scenario Description . . . . . . . . . . . . . . . . . . . . 3
3.1. Multiple Streams and actors . . . . . . . . . . . . . . . 4
3.1.1. Conductor to Musicians . . . . . . . . . . . . . . . 4
3.1.2. Musicians to Stage . . . . . . . . . . . . . . . . . 5
3.2. Virtual Orchestra Scenario Challenges . . . . . . . . . . 5
4. Generlized Coordinated Service Concept . . . . . . . . . . . 6
5. Virtual Orchestra Coordination Challenges . . . . . . . . . . 7
5.1. Out-of-band Coordination . . . . . . . . . . . . . . . . 7
5.2. In-band Coordination . . . . . . . . . . . . . . . . . . 8
5.3. In-node Coordinated-forwarding . . . . . . . . . . . . . 8
6. Existing Research Work . . . . . . . . . . . . . . . . . . . 8
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
8. Security Considerations . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction and Scope
The multimedia segment has seen tremendous advancements in immersive
multimedia technologies. One of the ongoing research question is how
to deliver a complete immersion experience of the digital media.
Such media is produced from an ensemble of different actors or
multiple sources that must coordinate as they perform together in the
real environment. It translates to generating a very high volume of
generated data streams
This memo presents research and engineering challenges in multi-user
digital ensemble that need to be addressed in order to achieve these
goals, spanning from pure research and engineering/standards space.
The network related challenges are generalized as coordinated
communications and explained as group communications with explicit
dependencies. The objective of this memo is to document the
technical challenges and corresponding current approaches and to
expose requirements that should be addressed by future research and
standards work.
Makhijani & Dong Expires 13 January 2022 [Page 2]
Internet-Draft virtual-orchestra-research-challenges July 2021
2. Terminology
* Co-flows: Co dependent flows
* Coordinated service: A network capability to support
communications between co-dependent flows
3. Scenario Description
In an orchestra ensemble the multimedia streams of musicians, each in
a different place in the world come together and perform live on the
stage which may also be at a different location.
* A conductor directs the sound of the ensemble with his gestures.
These gestures must be received at the same time by the remote
musicians at different locations to play their instruments at a
specific time with a specified tempo.
* Similarly, the music transmitted from those musicians' locations
to the stage must be played together with the same beats and
tempo. Any delay or early arrival of the sound from any one
instrument can cause the ensemble to go out of tune and destroy
the entire performance.
Performing an ensemble with multiple participants separated by large
as well as varying distances (from less than a mile, to 1000 miles)
is quite difficult for applications due to varying path and latency
characteristics.
The network needs to support the coordination of directions from the
conductor to all of the musicians and the audio/visuals from
musicians to the stage. In particular, in a large-scale ensemble
when many instruments are involved, in order to to preserve the
integrity of performance, it may be necessary to allow for the
dropping of sound and hologram streams of a musician that cannot
arrive at the same time as the others and to provide mechanisms for
subsequent fast synchronization.
Virtual orchestra is a complex multi-disciplinary use case and
requires in-depth knowledge in every field to recreate the real
orchestral experience.
Makhijani & Dong Expires 13 January 2022 [Page 3]
Internet-Draft virtual-orchestra-research-challenges July 2021
+-------------+
| conductor |
+-------------+
|
| one to many
+--------------+--------------+
| | |
v v v
+-----+ +-------+ +-------+
| t0" | | t0 | | t0' |
+-(A)-+ +--(B)--+ +--(C)--+
| | |
+-----------+ | +-----------+
| | | many to one
v v v
+-----------------+
| coordinator node|
+--------+--------+
|
v
+--------+
| Stage |
+--------+
Figure 1: Virtual Orchestra Delivery over Network
3.1. Multiple Streams and actors
A virtual orchestra is a coordination of multiple flows as shown in
Figure 1. In the current network terminology this is equivalent to
multicast group of a number of endpoints and requiring to meet
cooperation between the endpoints on how to send and receive
information. An application point of view sees this as a membership
to publish/subscribe topic. In the above example, endpoint actors
are the conductor, musicians and the stage. The characteristics of
traffic are predictable and the following steps take place
3.1.1. Conductor to Musicians
* The conductor is initiator of the orchestral stream. Synchronized
reception of the gestures of a conductor are critical to the
performance.
* Musicians perform on cues or gestures received over the network.
It is necessary that all the musicians receive those cues to start
the performance.
Makhijani & Dong Expires 13 January 2022 [Page 4]
Internet-Draft virtual-orchestra-research-challenges July 2021
* The performance follows the tempo and beats from the conductor,
which must be delivered in a consistent (jitter free) manner
(incurring no jitter).
3.1.2. Musicians to Stage
Atleast one output stream per sources will be generated to create the
ensemble performance, these sources may have variable latencies.
They should be aggregated to be delivered to the stage as a unified
stream.
The two scenarios are one to many Section 3.1.1 and many to
oneSection 3.1.2 type of group communication. The coordination
constraints involve several dependencies such as of synchronization
at the start of play, maintenance of same tempo along the time scale
throughout the streaming part, description of distance for spatial
sound quality.
3.2. Virtual Orchestra Scenario Challenges
In this section we draw forth scenarios with difficulties in
delivering virtual orchestra over the network.
Note that virtual orchestra application itself maybe delivered in
different ways. Non-realtime scenarios are not relevant since, in
that case it is a non-interactive content delivery, the content does
not require aggregation from multiple sources. An application and
corresponding network can use buffering, low latency techniques and
existing transport protocols to meet the expectations of an end-user.
Specific to real-time streaming of virtual orchestra, the performance
is pseudo-real-time. It means that the synchronization of content
originating from different sources is only as fast as its slowest
path. In other words, one source-destination path of the co-flow
will cause the pace of the group stream to slow-down, even though the
other, shorter latency paths may deliver content sooner. This in a
major co-dependency challenge, since the slowest path should not have
any impact on the tempo and the beats. Thus 3 dependency
considerations for the network are: - Feasibility Dependency: Assess
and determine that with the slowest flow-member of the group if such
a flow is even feasible. - Membership Dependency (spatial): The
mechanisms to establish and determine membership and establish
relationship is needed. Corelating to publisher (conductor and
stage) and subscriber (performers) group communication model, not all
subscribers need to know about each other. - Start Time Dependency
(temporal): Each performer depends on the trigger to start from there
on time-scale, tempo and beats of the performance must be preserved.
Makhijani & Dong Expires 13 January 2022 [Page 5]
Internet-Draft virtual-orchestra-research-challenges July 2021
From a logical architectural point of view, coordination node is a
function that synchronizes all the incoming streams, it may then
either deliver all the streams or as a single stream.
4. Generlized Coordinated Service Concept
There are several examples of multi-party immersive applications (TBD
- add section) in which remote entities will be required to recreate
the behavior of being present in the same scene or environment.
Therefore, they are co-dependent on each other's spatio-temporal
behavior changes. For example, in an orchestra tempo or beats and
gestures must remain the same for all performers and position of a
musician is computed to create spatial audio.
A generalized in-network capability is introduced that consumes group
communication membership and constraints and delivers service with in
the specified constriaints.
Keeping in the network context, important terms and components of
coordinated service are introduced as below:
.-.
(---)--->
`-'---->
.---. +-----+
( )----------|Co-EP|
member `---' +-----+
+-----+ | flow Co-SN
|Co-EP|----+ | ^^
+-----+ | | .||.
| | ( ||) .-.
| v `||' ----(--)---->
.---. --------> .---. -----`-'-----> +-----+
( )-------------( )-------------------|Co-EP|
`---' -------> `---' +-----+
+-----+ | Co-SN
|Co-EP|-----+
+-----+ ------> Co-EP: coordinated service end point
Co-SN: coordinated service node
Co-Flow: coordinated flow
Member flow: member of a co-flow
* Coordinated Service: A coordinated service provides guarantee of
delivery of multiple flows in a dependent manner. It is a type of
group communication service supported by the network in which each
member of the group has a dependency on other group-members. A
coordinated service should be able to coordinate delivery of co-
flows over different categories of group communications.
Makhijani & Dong Expires 13 January 2022 [Page 6]
Internet-Draft virtual-orchestra-research-challenges July 2021
* Coordinated Service end point (Co-EP): An endpoint in a group
communication where in coordination of resources or constraints is
required.
* Coordinated service Points (Co-SN): The mechanisms to support
coordinated services in network requires new capabilities referred
to as network coordination functions. A service node is a part of
the network that understands and processes network specific
functions of coordination.
* Co-dependent Flow(Co-Flow): A set of flows that have dependencies
in relation to eachother. Each flow in the co-dependent flow set
is referred to as a member-flow. The co-flows may express
different kinds of dependencies or relationships. It may be point
to mltipoint, multipoint to point or multipoint to multipoint.
* Member flow: A single point to point member flow of a co-flow.
Coordinated services are a form of group comminication with a clearly
expressed dependencies. Possible approaches will figure out
mechanisms to manage those dependencies.
5. Virtual Orchestra Coordination Challenges
The internet is a spatial-temporal heterogeneous environment,
yielding different content delivery behaviours in time and space. No
two paths (or even different flows on the same path) can be assumed
to have identical properties in terms of latency, jitter, and
bandwidth.
Currently, any effort to support virtual orchestra in the networks is
not feasible. Managing flow dependencis entirely by the applications
on endpoints does not always guarantee absolute time constraints due
to unpredictable changes in network conditions. This necessitates
some kind of coordination with the network.
5.1. Out-of-band Coordination
The out-of-band coordination may be used to achieve distribution of
coflows in the network. The membership of co-dependent flows is
conveyed from the end-points potentially when the flows are set up,
so that the coarse-grained service (and service level objectives) can
be enabled in the network. A distribution graph of coflows and
associated dependency constraints may be constructed, and those nodes
enhance their scheduling and forwarding by factoring in the timing
information in the meta-data of packets.
Makhijani & Dong Expires 13 January 2022 [Page 7]
Internet-Draft virtual-orchestra-research-challenges July 2021
5.2. In-band Coordination
Then timestamping of transmission from sender delivery to receiver
may be conveyed as meta-data in packets transmitted from the senders.
This type of In-band signaling conveys intermediate coordination
points about the dependencies and interrelationship. To formalize
these mechanisms to carry them in data path.
5.3. In-node Coordinated-forwarding
Actual coordination effort is done on the coordination points. The
scheduling and forwarding engine should allow packets within sync
markers to be sent as per remaining δt. It needs to compare the
remaining coordination time and accordingly schedule or pace the
packet forwarding.
6. Existing Research Work
7. IANA Considerations
This document requires no actions from IANA.
8. Security Considerations
This document introduces no new security issues.
Authors' Addresses
Kiran Makhijani
Futurewei
Email: kiran.ietf@gmail.com
Lijun Dong
Futurewei
Central Expy
Santa Clara, CA 95050,
United States of America
Email: lijun.dong@futurewei.com
Makhijani & Dong Expires 13 January 2022 [Page 8]
