Internet DRAFT - draft-xiang-interdomain-science-network
draft-xiang-interdomain-science-network
ALTO WG Q. Xiang
Internet-Draft Y. Yang
Intended status: Standards Track Tongji/Yale University
Expires: June 20, 2018 December 17, 2017
Unicorn: Resource Orchestration for Large-Scale, Multi-Domain Data
Analytics
draft-xiang-interdomain-science-network-00.txt
Abstract
This document presents the design of Unicorn, a multi-domain,
geographically-distributed, data-intensive analytics system. The
setting of such a system includes edge science networks, which
provide storage and computation resources for collecting, sharing and
analyzing extremely large amounts of data, and transit networks,
which provide networking resources to connects edge science networks
for transmitting large science datasets.
The key design challenge is to accurately discover and represent
resource information from different domains. Unicorn leverages
multiple ALTO services, including ALTO-Path Vector, ALTO-Routing
State Abstraction, ALTO-Server-Side Event and ALTO-Flow Cost Service
to address this challenge. In particular, Unicorn decomposes the
resource discovery into three phases. The first phase is to identify
endpoint resource, e.g., dataset storage location, computation
resource location and output storage resource location. The second
phase is to identify the reachability information between the
locations of storage and computation resources. The third phase is
to identify the available networking resource connecting different
storage and computation resources. All information collected through
these three phases can be used by a logically centralized scheduling
system to orchestrate the resources usage.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
Xiang & Yang Expires June 20, 2018 [Page 1]
Internet-Draft Unicorn Design December 2017
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 20, 2018.
Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Settings . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4
3. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Storage and Computation Resource Discovery . . . . . . . . . 6
5. Path Discovery . . . . . . . . . . . . . . . . . . . . . . . 6
5.1. Using SDN to get flow-based site-path . . . . . . . . . . 7
5.2. Path Discovery Example . . . . . . . . . . . . . . . . . 7
6. Networking Resource Discovery . . . . . . . . . . . . . . . . 8
6.1. Networking Resource Discovery Example . . . . . . . . . . 8
6.2. A Secure Multiparty Computation Protocol to Compute
Minimal, Cross-Domain RSA . . . . . . . . . . . . . . . . 8
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 9
7.1. Normative References . . . . . . . . . . . . . . . . . . 9
7.2. Informative References . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction
As the data volume increases exponentially over time, data intensive
analytics is transiting from single-domain computing to multi-
organizational, geographically-distributed, collaborative computing,
where different organizations contribute various resources, e.g.,
computation, storage and networking resources, to collaboratively
collect, share and analyze extremely large amounts of data. One
leading example is the Large Hadron Collider (LHC) high energy
Xiang & Yang Expires June 20, 2018 [Page 2]
Internet-Draft Unicorn Design December 2017
physics (HEP) program, which aims to find new particles and
interactions in a previously inaccessible range of energies. The
scientific collaborations that have built and operate large HEP
experimental facilities at the LHC, such as the Compact Muon Solenoid
(CMS) and A Toroidal LHC ApparatuS (ATLAS), currently have more than
300 petabytes of data under management at hundreds of sites around
the world, and this volume is expected to grow to one exabyte by
approximately 2018.
This document presents Unicorn, a generic design for resource
orchestration for large-scale, multi-domain data analytics. The key
design challenge for such a resource orchestration system is to
accurately discover and represent the resource information from
different domains. Our design resorts to the Application-Layer
Traffic Optimization Protocol (ALTO) [RFC7285] to address this
challenge. In particular, several ALTO extension services, including
ALTO-Path Vector, ALTO-Routing State Abstraction, ALTO-Server-Side
Event and ALTO-Flow Cost Service, are integrated in the proposed
design.
This document focuses on the design details of Unicorn. We present
the implementation and deployment experience of Unicorn in another
document [DRAFT-UNICORN-INFO].
1.1. Settings
The targeting scenario is as follows. There are two types of
networks in the whole system. The first type is the edge science
network. An edge science networks is usually a cluster residing in a
campus network. It provides storage resources to store large
scientific datasets and computation resources to analyze these
datasets. The second type is the transit network. A transit network
does not provide any storage or computation resources. It only
provides networking resources to inter-connect different edge science
networks so that datasets can be moved and shared between different
edge science networks. Edge science networks do not directly connect
to each other, but are connected through transit networks.
Without loss of generality, a data analytics task is defined as a
3-tuple: (input dataset, program, output site). A task can be
further decomposed into a set of jobs, who have a precedence relation
defined by a directed acyclic graph (DAG). And each job can also be
defined as a 3-tuple: (input dataset, program, output site).
Xiang & Yang Expires June 20, 2018 [Page 3]
Internet-Draft Unicorn Design December 2017
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
3. Overview
The key design challenge for multi-domain data analytics system is to
accurately discover the resource information from different sites
while preserving the autonomy and privacy of each site. In order to
address this challenge, the design needs to strike a balance between
the information accuracy, the efficiency of resource discovery and
the privacy of each site. In particular, we propose the following
architecture in the Figure 1.
.---------.
| Users |
'---------'
| Tasks
.- - - - - - - - - - - - - - -|- - - - - - - - - - - - - - - - - - .
| | |
| .-----------------------. 1 .------------------------.|
| | Resource Orchestrator | -----|Storage/Computation Pool||
| '-----------------------' \ '------------------------'|
| / | | 4 \ \ |
| 2 / 3 | | 3\ \ 2 |
| .-------------. .-----------. .-------------. |
| | ALTO Server | | Execution | | ALTO Server | |
| '-------------' | Agents | '-------------' |
| | '-----------' | |
| | / \ | |
| .----------------./ \ .----------------. |
| | Site 1 | . . . | Site N | |
| '----------------' '----------------' |
'- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - '
Figure 1: Resource Orchestration for Large-Scale, Multi-Domain Data
Analytics: Architecture.
In the proposed design, each site deploys an ALTO server. Within the
site, the ALTO server collects resource information, such as dataset
locations, storage resources, computation resources, networking
resources and so on, and announces its capability, i.e., the type of
information it is willing to share with other ALTO servers or the
data analytics resource orchestrator.
Xiang & Yang Expires June 20, 2018 [Page 4]
Internet-Draft Unicorn Design December 2017
In this system, users submit analytics tasks to the resource
orchestrator. For a data analytics task, a user needs to at least
provide the analytics program. If the input dataset is not specified
by the user, it means this task does not require a dataset as input.
Similarly, if the user does not specify the computation resource or
the output dataset storage site, the system will try to allocate
default computation resources to this task, and will return the
output dataset directly to the user.
After getting a (set of) task(s) from the user(s), the orchestrator
discover the available resources for executing the submitted task(s)
in three steps, labeled in the figure. The first step is called
storage/computation resource discovery. In this step, the
orchestrator sends requests to a centralized storage and computation
resource pool to find the location of candidate input storage
resources, the computation resources and the output storage
resources.
The second step is called path discovery, in which the resource
orchestrator sends endpoint or flow cost service queries to the ALTO
servers at the site holding the candidate input storage resources and
the site holding the candidate computation resources to ask about the
connectivity from input dataset site to the computation site and that
from the computation site to the output site. The cost type of such
queries is path vector defined in [DRAFT-PV]. The response sent back
from the ALTO server to the orchestrator is a vector. Each element
in this vector is the IP address of the ingress gateway switch/router
that the candidate flow will pass along the AS-path. This vector is
called the site-path in this document.
After collecting the site-path of all the candidate (storage,
computation) flows, for each site X, the orchestrator derives F_X,
the set of candidate flows that will consume networking resources in
site X. Then the orchestrator will send endpoint/flow cost service
queries to the ALTO server at each site X to ask about the networking
resource sharing of the flow set F_X in site X. The returned
response is a set of linear inequalities called resource state
abstraction.
Using the resource information collected from the three-phase
resource discovery process, the resource orchestrator can run an
scheduling algorithm to make the resource allocation decisions to
execute the submitted tasks. The decisions include job decomposition
(DAG construction), task concatenation, job placement, network
resource allocation for input dataset movement and output movement.
These decisions will be sent to the corresponding execution agents at
different sites, which will practice these decisions and send
feedback to the orchestrator.
Xiang & Yang Expires June 20, 2018 [Page 5]
Internet-Draft Unicorn Design December 2017
When resource state changes, e.g., a network link is broken, the ALTO
server at the scene will check whether the results of existing path
discovery and networking resource discovery are affected by this
event, and sends updated resource information using the ALTO-SSE
service.
In the next few sections, we present the detailed design of the
three-phase resource discovery.
4. Storage and Computation Resource Discovery
In order to allocation resources for a (set of) data analytic tasks,
the scheduling system must first know the availability of the
resources explicitly specified in the task, i.e., the storage
resource storing the input dataset, the computation resources to run
the analytics program and the storage resource that will be used to
store the output dataset. Such resources are only provided by the
edge science networks. Therefore, a strawman design is for the
scheduling system to send requests to the resource information
servers of all the edge science networks and to get such information.
However, this solution is inefficient in that the scheduling system
needs to query all the edge science networks to get the complete
information.
This document adopts an alternative design, in which all the resource
information servers proactively send all their information about the
storage and computation resources to a centralized resource pool.
This resource pool can be a DNS server or a traditional database.
Different techniques are under investigation to improve the
scalability of this design, including sharding and distributed
hashing table (DHT).
5. Path Discovery
Having identified the locations of input dataset storage nodes, the
locations of candidate computation nodes and the locations of
candidate output dataset storage nodes, the scheduling system next
needs to find out the connectivity information between storage nodes
and computation nodes. The first connectivity information is the
reachability between storage nodes and the computation nodes. A
input storage node, a computation node and a output storage node can
be allocated to execute a job only if data movement is allowed
between the input storage node and the computation node, and between
the computation node and the output storage node.
Because edge science networks are connected through transit networks,
the data movement between candidate storage nodes and computation
nodes need to consume networking resources of multiple networks if
Xiang & Yang Expires June 20, 2018 [Page 6]
Internet-Draft Unicorn Design December 2017
these nodes are located at different edge science networks. In order
to find the networking resource sharing between different (storage,
computation) pair, the scheduling system also needs to know which
networks are involved in the data movement of each (storage,
computation) node pair.
To retrieve both the types of information, the scheduling system
issues endpoint cost service queries to the ALTO servers at edge
science networks. For the ALTO server at an edge science network X,
the scheduling system issues endpoint cost service defined in
[RFC7285] or the extension flow cost service defined in [DRAFT-FCS]
queries for all the (input storage node, computation node) pairs
where the input storage node is located in X, and all the
(computation node, output storage node) pairs where the computation
node is located in X. The cost type of such queries is the new path
vector cost type introduced in [DRAFT-PV].
For each (storage, computation) pair, the response sent by the ALTO
servers at edge science networks is a path vector providing the
information about the AS-level path for the data movement of this
pair. Different from the traditional path vector where each element
is an AS name/number, each element in the path vector sent by the
ALTO servers also includes the ingress IP address of the gateway
switch/router of the corresponding network. We call this path vector
the "site-path", to differentiate it from the traditional AS-path.
5.1. Using SDN to get flow-based site-path
ALTO servers can compute the site-path for a given (storage,
computation) pair using the information provided by BGP and
traceroute. However, BGP only supports destination-IP based routing
and limits each network's ability to make fine-grained flow-based
routing decisions. We are investigating the usage of SDN technique
to allow different networks in the multi-domain data analytics system
to exchange and make fine-grained flow-based inter-domain routing
decisions. To avoid the route advertisement explosion brought by
flow-based routing, we design use a sub/pub system that allows an
ALTO server to send routing information queries of a set of flows,
instead of the whole flow space, to other ALTO servers at other
domains.
5.2. Path Discovery Example
The following is an example of path discovery query made by the
orchestrator.
Xiang & Yang Expires June 20, 2018 [Page 7]
Internet-Draft Unicorn Design December 2017
{ "cost-type":
{ "cost-mode": "array",
"cost-metric": "ane-path" },
"endpoint-flows":
{ "srcs": [ "ipv4:172.0.0.1", "ipv4:172.0.1.1"],
"dsts": [ "ipv4:172.0.2.1", "ipv4:172.0.3.1"]}
}
And the following is the response sent from the ALTO server.
{"endpoint-cost-map":
"ipv4: 172.0.0.1 ": {
"ipv4: 172.0.2.1 ": ["ane:172.1.0.1", "ane:172.0.2.0"],
"ipv4: 172.0.3.1 ": ["ane:172.1.0.1", "ane:172.0.3.0"]},
"ipv4: 172.0.1.1 ": {
"ipv4: 172.0.2.1 ": ["ane:172.1.0.1", "ane:172.0.2.0"],
"ipv4: 172.0.3.1 ": ["ane:172.1.0.1", "ane:172.0.3.0"]}
}
6. Networking Resource Discovery
The responses from ALTO servers during the path discovery provides
the connectivity information for every pair of candidate input
dataset storage node and computation node, and that of every pair of
candidate computation node to output storage node in the form of
site-path. With such information, the scheduling system can further
discover the networking resource sharing between candidate (storage,
computation) data movement flows. In particular, for each network X,
both edge science networks and transit network, we can easily derive
the whole set of candidate data movement flows F_X that will enter
network X from the site-path information of all candidate (storage,
computation) data movement flows. After deriving F_X for each
network X, the scheduling system will send endpoint cost services or
flow cost services to retrieve the resource state abstraction
[DRAFT-RSA] for the flow set F_X.
6.1. Networking Resource Discovery Example
TBA.
6.2. A Secure Multiparty Computation Protocol to Compute Minimal,
Cross-Domain RSA
The current design of ALTO-RSA can only compute the minimal resource
state abstraction for a single network. In Unicorn, we design a
secure multiparty computation protocol to support the computation of
minimal, cross-domain routing state abstraction. This protocol
contains each network's exposure of its redundant linear inequalities
Xiang & Yang Expires June 20, 2018 [Page 8]
Internet-Draft Unicorn Design December 2017
to a small number of other networks, and ensures that the
orchestrator only gets the minimal, cross-domain resource state
abstraction. The overhead of this SMPC process is reasonable due to
the adoption of state-of-the-art secure scalar product protocol.
7. References
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
7.2. Informative References
[DRAFT-CC]
Randriamasy, S., Yang, R., Wu, Q., Deng, L., and N.
Schwan, "ALTO Cost Calendar", 2017,
<https://datatracker.ietf.org/doc/
draft-ietf-alto-cost-calendar>.
[DRAFT-DC]
Lee, Y., Bernstein, G., Dhody, D., and T. Choi, "ALTO
Extensions for Collecting Data Center Resource
Information", 2014, <https://datatracker.ietf.org/doc/
draft-lee-alto-ext-dc-resource/>.
[DRAFT-FCS]
Zhang, J., Gao, K., Wang, J., Xiang, Q., and Y. Yang,
"ALTO Extension: Flow-based Cost Query", 2017,
<https://datatracker.ietf.org/doc/draft-gao-alto-fcs/>.
[DRAFT-MC]
Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost
ALTO", 2017, <https://datatracker.ietf.org/doc/
draft-ietf-alto-multi-cost/>.
[DRAFT-NETGRAPH]
Bernstein, G., Lee, Y., Roome, W., Scharf, M., and Y.
Yang, "ALTO Topology Extensions: Node-Link Graphs", 2015,
<https://tools.ietf.org/html/draft-yang-alto-topology-06>.
[DRAFT-PM]
Roome, W. and Y. Yang, "Extensible Property Maps for the
ALTO Protocol", 2015, <https://datatracker.ietf.org/doc/
draft-roome-alto-unified-props-new/>.
Xiang & Yang Expires June 20, 2018 [Page 9]
Internet-Draft Unicorn Design December 2017
[DRAFT-PV]
Bernstein, G., Lee, Y., Roome, W., Scharf, M., and Y.
Yang, "ALTO Extension: Abstract Path Vector as a Cost
Mode", 2015, <https://tools.ietf.org/html/
draft-yang-alto-path-vector-01>.
[DRAFT-RSA]
Gao, K., Wang, X., Xiang, Q., Gu, C., Yang, Y., and G.
Chen, "A Recommendation for Compressing ALTO Path
Vectors", 2017, <https://datatracker.ietf.org/doc/
draft-gao-alto-routing-state-abstraction/>.
[DRAFT-SSE]
Roome, W. and Y. Yang, "ALTO Incremental Updates Using
Server-Sent Events (SSE)", 2015,
<https://datatracker.ietf.org/doc/
draft-ietf-alto-incr-update-sse/>.
[DRAFT-UNICORN-INFO]
Xiang, Q., Newman, H., Bernstein, G., Du, H., Gao, K.,
Mughal, A., Balcas, J., Zhang, J., and Y. Yang,
"Implementation and Deployment of A Resource Orchestration
System for Multi-Domain Data Analytics", 2017,
<https://datatracker.ietf.org/doc/
draft-xiang-alto-exascale-network-optimization/>.
[HTCondor]
Thain, D., Tannenbaum, T., and M. Livny, "Distributed
computing in practice: the Condor experience", 2005,
<http://dl.acm.org/citation.cfm?id=1064336>.
[RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S.,
Previdi, S., Roome, W., Shalunov, S., and R. Woundy,
"Application-Layer Traffic Optimization (ALTO) Protocol",
RFC 7285, DOI 10.17487/RFC7285, September 2014,
<https://www.rfc-editor.org/info/rfc7285>.
Authors' Addresses
Qiao Xiang
Tongji/Yale University
51 Prospect Street
New Haven, CT
USA
Email: qiao.xiang@cs.yale.edu
Xiang & Yang Expires June 20, 2018 [Page 10]
Internet-Draft Unicorn Design December 2017
Y. Richard Yang
Tongji/Yale University
51 Prospect Street
New Haven, CT
USA
Email: yry@cs.yale.edu
Xiang & Yang Expires June 20, 2018 [Page 11]
