ALTO-based Broker-assisted Multi-domain Orchestration
draft-lachosrothenberg-alto-brokermdo-00

Abstract

Evolving networking scenarios (e.g., 5G) demand new multiple administrative domain (aka multi-domain) orchestration models. This document proposes the use of Application-Layer Traffic Optimization (ALTO) services to offer topology and resources addressing network service discovery and provisioning by multi-domain orchestrators. The ALTO services with the proposed protocol extensions offer aggregated views on various types of resources contributing to a more simple and scalable solution for resource and service discovery in multi-domain, multi-technology environments.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on September 6, 2018.

Copyright Notice

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Terminology
• 3. Scope
• 4. Problem Statement and Challenges
• 5. Proposed Approach
• 5.1. Inter-domain Resource (IdR) Component
• 5.2. Inter-domain Topology (IdT) Component
• 5.3. ALTO Server Functionalities
• 5.4. Required ALTO Extensions
• 5.4.1. Property Map Extensions
• 5.4.2. Filtered Cost Map Extensions
• 5.5. Examples of Message Exchange
• 5.5.1. Property Map Service
• 5.5.2. Filtered Cost Map Service
• 6. Acknowledgments
• 7. IANA Considerations
• 8. Security Considerations
• 9. References
• 9.1. Normative References
• 9.2. Informative References
• Appendix A. Proof of Concept Use Case Implementation
• Authors' Addresses

1. Introduction

Envisioned 5G network architectures and related service models consider broader cooperation between stakeholders in order to provide flexible multi-operator multi-domain services. These multi-provider orchestration operations will require the information exchange across Multi-domain Orchestrators (MdOs). The key information to be exchanged between MdOs includes the abstract network topology, resource availability (e.g., CPUs, Memory, and Storage) and capability (e.g., supported network functions).

This document presents a federation networking paradigm where a broker-plane works on top of the management and orchestration plane to assist and coordinate the creation of an End-to-End Network Service (E2ENS) spanning over multi-operator multi-domain networks. Our design resorts to the Application-Layer Traffic Optimization (ALTO) protocol  [RFC7285] to address the lack of abstractions to discover and adequately represent in confidentiality-preserving fashion the resource and topology information from different administrative domains. Moreover, this draft introduces some extensions to the ALTO base protocol to support the main functionalities in our proposed architecture.

2. Terminology

We use the following definitions, as established in [ETSI-NFV-DEF]:

Administrative Domain:

Collection of systems and networks operated by a single organization or administrative authority.

Network Function (NF):

Functional block within a network infrastructure that has well-defined external interfaces and well-defined functional behaviour.

Network Functions Virtualisation (NFV):

The principle of separating network functions from the hardware they run on by using virtual hardware abstraction.

NF Forwarding Graph: (NFFG):

Graph of logical links connecting NF nodes for the purpose of describing traffic flow between these network functions.

Network Service Orchestration (NSO):

Function responsible for network service lifecycle management.

Resource Orchestration (RO):

Function responsible for global resource management governance.

Our proof of concept implementation follows the architectural proposal of the 5GEx project [H2020.5GEX]. Some additional 5GEx terms commonly used in this document are defined below:

Domain Orchestrator (DO):

Performs Resource Orchestration and/or Service Orchestration within the same administrative domain.

Multi-domain Orchestrator (MdO):

Coordinates resource and/or service orchestration at multi-domain level, where multi-domain may refer to multiple DOs or multiple administrative domains.

Resource Topology (RT):

Functional module that is responsible for keeping an updated global view of the underlying infrastructure topology exposed by DOs.

Service Graph (SG):

A high-level data model for defining flexible network services (including traffic steering primitives).

Service Access Point (SAP):

A named/tagged port supporting stitching (service to service, domain to domain, etc.)

3. Scope

Existing proposals for the network service orchestration are intrinsically conceived for single administrative domain scenarios. For example, in the standard service orchestration model described in ETSI NFV MANO framework [ETSI-NFV-MANO], one orchestrator is supposed to work within one administrative domain. The analysis of orchestration and management of network Services over multiple administrative domains have begun to be addressed by ETSI in [ETSI-NFV-MANO-MDO].

Envisioned 5G scenarios are expected to work not only with heterogeneous technologies but also across different network operators. Many ongoing initiatives and projects related are addressing the multi-provider multi-domain orchestration challenges under different approaches. For example, [H2020.5GEX] seeks to integrate multiple administrations and technologies through the collaboration between operations. Other studies are envisioned to use a centralized approach, where each domain advertises its capabilities to a federation layer which will act as a broker [VITAL][T-NOVA]. The proposed architecture in [ICAF] allows the creation of cloud services from different administrative domains, however, it is not related to the provisioning of NFV-based cross-domain network services.

All such proposals described above envision the potential introduction of new business model approaches, including federation models [PPP-5:2013] among administrative domains. In this context, this document considers each network operator involved in the community advertises its abstracted capabilities (e.g., software/hardware resources, physical/virtual network functions, etc.) to a broker (i.e., 3rd party). This latter, in its turn, provides or assists coordinate E2E network services spanning multi-domain networks.

4. Problem Statement and Challenges

The provision of value-added services through a brokerage level of orchestration requires mechanisms through which single domains can describe their network capabilities in an interoperable manner. Different interfaces, subject to potential standardization opportunities, are necessary to advertise capabilities, resources, and VNFs to trusted entities.

Network service orchestration in Multi-domain (Multi-operator/multi-technology) environments is not trivial due to series of challenges:

Scalability:

Involves the distribution of topology and resource information in a peer-to-peer fashion (MdO-to-MdO). Multi-operator multi-domain environments where the information distribution is advertised in a peer-to-peer model scales linearly. It means more MdO interconnections one has, the more it "costs" to distribute.

Flexibility:

Considers that a distributed approach does not allow domains without physical infrastructure (e.g., without BGP or BGP-LS) to advertise resource capabilities and networking resources. Such procedures consist in deploying and configuring physical peering points for these domains.

Complexity:

Refers to the discovery mechanism to pre-select candidate domains, accounting for resources and capabilities, necessary for an end-to-end network service deployment An intrinsic complexity exists in the process of assembling, logically organizing, and enabling abstraction views of different resources and capabilities in multi-domain scenarios.

5. Proposed Approach

The primary design goal for ALTO-based Broker-assisted Multi-domain Orchestration is to discover resource and topology information from different administrative domains involved in the federation, while also safeguarding the privacy and autonomy of every domain.

In the architectural proposal shown in Figure 1, a broker component is conceived to be working as coordinator of a set of MdOs, whose key components are: the Inter-domain Resource (IdR), the Inter-domain Topology (IdT) and the ALTO Server.

          

                           BROKER COMPONENT
           +--------------------------------------------+
           |                                            |
           |             +-----------------+            |
           |             |                 |            |
       XXXXXXXXXXXXXXXXXXX ALTO SERVER(s)  |            |
       X   |             |                 +            |
       X   |             +----------------+\            |
       X   |            /                   \           |
       X   |           /                     \          |
       X   |  +------+/+-------+     +---------------+  |
       X   |  | Inter-domain   |     | Inter-domain  |  |
       X   |  | Topology (IdT) |     | Resource (IdR)|  |
       X   |  +-^------^-------+     +---^-------^---+  |
       X   |    |      |                 |       |      |
       X   +--------------------------------------------+
       X        |      |                 |       |
       X        |      |                 |       |
    +--X--------+---+--------------------+       |
    |               |  |                         |
    |               |  +------------+------------+---+
    |     MdO1      |               |                |
    |               +<------------->+                +---+
    +---------------+               |      MdO2      |   |
                                    |                |   |
                                    +-+--------------+   |
                                      |                  |
Legend:                               |      MdO(n)      |
XXX ALTO Protocol                     +------------------+


          

Figure 1: Broker-assisted Multi-operator Network Architecture

5.1. Inter-domain Resource (IdR) Component

It creates a hierarchical database that contains inter-domain resource information such as resource availability (i.e., CPU, memory, and storage), Virtual Network Functions (VNFs) and Physical Network Functions (PNFs) supported and Service Access Points (SAPs) to access those resources. UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI-NFV [ETSI-NFV-MANO], among other data models can be used to create the interface between IdR and MdOs.

5.2. Inter-domain Topology (IdT) Component

A hierarchical TED (Traffic Engineering Database) that contains inter-domain network topology information including additional key parameters (e.g., throughput and latency of links). This information can be retrieved from each MdO through BGP-LS or REST interfaces.

5.3. ALTO Server Functionalities

The ALTO server component is the core of the broker layer. Multiple logically centralized ALTO servers use the information collected from IdR and IdT modules to create and provide abstract maps with a simplified view, yet enough information about MdOs involved in the federation. This information includes domain-level topology, storage resources, computation resources, networking resources and PNF/VNF capabilities.

As an ALTO client, each MdO sends ALTO service queries to the ALTO server. This server provides aggregated inter-domain information exposed as set ALTO base services defined in [RFC7285], e.g., Network Map, Cost Map and ALTO extension services, e.g., Property Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV].

For example, when a source MdO receives a customer service request, it checks whether or not it can deliver the full service. If it is unable to do so, the MdO consumes from the ALTO Server the Property Map service to have a clear global view of the resource information offered by other MdOs. This information allows discovering which candidate MdOs may be contacted to deliver the remaining requirements of a requested end-to-end service deployment. The connectivity information among discovered MdOs can be retrieved by a Cost Map service, responding, for instance, a path vector with the AS-level topology distance between the source MdO and candidate MdOs.

5.4. Required ALTO Extensions

The Property Map and the Cost Map examples defined in the previous section can be supported with some proposed extensions to the ALTO base protocol.

In this section, we introduce a non-normative overview of the Property Map and Filtered Cost Map extensions.

5.4.1. Property Map Extensions

The ALTO server MUST return multiple values for each property in the Property Map. For example, MdOs exchange (among others) a list NFs and SAPs which are supported by them. So in this scenario, an array of values can provide sufficient information that is not possible with single string values.

The specifications for the "Media Types", "HTTP Method", "Accept Input Parameters", "Capabilities" and "Uses" (described in Section 4.1, Section 4.2, Section 4.3, Section 4.4, Section 4.5 of [DRAFT-PM], respectively) remain unchanged.

The response is the same as Section 4.6 of [DRAFT-PM], except that for each property name defined in the resource's "capabilities" list, the corresponding property value MUST be encoded as JSONArray instead of JSONString.

Section 5.5.1 gives an example of a property map request and its response.

5.4.2. Filtered Cost Map Extensions

The ALTO server MUST provide connectivity information for every SG link in the SG path for an E2E requirement. This information is the AS-level topology distance in the form of path vector, and it includes all possible ways for each (source node, destination node) pair in the SG link.

In this section, we propose an extension of the Filtered Cost Map defined in Section 6.1 of [DRAFT-PV].

The specifications for the "Media Types", "HTTP method", "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV]) are unchanged.

5.4.2.1. Accept Input Parameters

The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] is extended as follow:

                
  object {
    [NFFG sg;]
  } ReqFilteredCostMap;

  object {
    JSONString nfs<1..*>;
    JSONString saps<1..*>;
    NextHops sg_links<1..*>;
    REQs reqs<1..*>;
  } NFFG;

  object {
    JSONNumber id;
    JSONString src-node;
    JSONString dst-node;
  } NextHops;

  object {
    JSONString id;
    JSONString src-node;
    JSONString dst-node;
    JSONNumber sg-path<1..*>;
  } REQs;

  
              

sg:

If present, the ALTO Server MUST allow the request input to include an SG with a formatted body as an NFFG object. An NFFG object contains NFs, SAPs, SG links representing logical connections between NFs, SAPs or both and E2E requirements as a list of ids of SG links.

It is worth noting that further versions of this draft will define a more elaborated NFFG object to support extended parameters such as monitoring parameters, resource requirements, etc.

5.4.2.2. Response

If the ALTO client includes the path vector cost mode in the "cost- type" or "multi-cost-types" field of the input parameter, the response for each SG link in each E2E requirement MUST be encoded as a JSONArray of JSONArrays of JSONStrings. Anyone of the sub-arrays indicates a potential candidate path calculated as the per-domain topological distance corresponding to the amount of traversing domains.

Moreover, as defined in Section 6.3.6 of [DRAFT-PV], If an ALTO client sends a request of the media type "application/alto-costmapfilter+json" and accepts "multipart/related", the ALTO server MUST provide path vector information along with the associated Property Map information (e.g., entry points of the corresponding foreign domains), in the same body of the response.

Section 5.5.2 gives an example of the Filtered Cost Map query and the corresponding responses.

5.5. Examples of Message Exchange

This section list a couple of examples of the Property Map and Filtered Cost Map queries and the corresponding responses. These responses are based on the information in Table 1 and Table 2 of a use case implementation described in Appendix A.

5.5.1. Property Map Service

In this example, the ALTO client wants to retrieve the entire Property Map for PID entities with the "entry-point", "cpu", "mem", "storage", "port" and "nf" properties.

        
  GET /propmap/full/inet-ucmspn HTTP/1.1
  Host: alto.example.com
  Accept: application/alto-propmap+json,application/alto-error+json

  HTTP/1.1 200 OK
  Content-Length: ###
  Content-Type: application/alto-propmap+json
  {
    "property-map": {
        "pid:AS1": {
            "entry-point": [ "http://172.25.0.10:8888/escape" ],
            "cpu": [ "50.0" ],
            "mem": [ "60.0" ],
            "storage": [ "70.0" ],
            "port": [ "SAP1" ],
            "nf": [ "NF1", "NF3" ]
        },
        "pid:AS2": {
            "entry-point": [ "http://172.26.0.10:8888/escape" ],
            "cpu": [ "10.0" ],
            "mem": [ "20.0" ],
            "storage": [ "30.0" ],
            "nf": [ "NF2" ]
        },
        "pid:AS3": {
            "entry-point": [ "http://172.27.0.10:8888/escape" ],
            "cpu": [ "80.0" ],
            "mem": [ "90.0" ],
            "storage": [ "100.0" ],
            "port": [ "SAP2" ],
            "nf": [ "NF1", "NF3" ]
        }
    }
  }
        
                    

5.5.2. Filtered Cost Map Service

The following example uses the Filtered Cost Map service to request the path vector for a given E2E requirement. The SG request information in Table 2 is used to describe the service, and it is composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and SAP2). Links connecting the NFs and SAPs ("sg_links" tag) are also included, followed by an E2E requirement ("reqs" tag) with information about the order in which NFs are traversed from SAP1 to SAP2.

Note that the request accepts "multipart/related" media type. This means the ALTO server will include associated property information in the same response.

        
    POST /costmap/pv HTTP/1.1
    Host: alto.example.com
    Accept: multipart/related, application/alto-costmap+json,
          application/alto-propmap+json, application/alto-error+json
    Content-Length: [TBD]
    Content-Type: application/alto-costmapfilter+json

    {
      "cost-type": {
        "cost-mode": "array",
        "cost-metric": "ane-path"
      },
      "sg": {
        "nfs": [ "NF1", "NF2", "NF3" ],
        "saps": [ "SAP1", "SAP2" ],
        "sg_links":[
          {
            "id": 2,
            "src-node": "SAP1",
            "dst-node": "NF1",

          },
          {
            "id": 2,
            "src-node": "NF1",
            "dst-node": "NF2",
          },
          {
            "id": 3,
            "src-node": "NF2",
            "dst-node": "NF3",
          },
          {
            "id": 4,
            "src-node": "NF3",
            "dst-node": "SAP2",
          }
        ],
        "reqs": [
          {
            "id": 1,
            "src-node": "SAP1",
            "dst-node": "SAP2",
            "sg-path": [ 1, 2, 3, 4 ]
          }
        ]
      }
    }
        
                    

The ALTO server returns connectivity information for the E2E requirement provided by the ALTO Client request of the above example. Also, the response includes Property Map information for each element in the path vector. In this case, it is retrieved a Property Map with the "entry-point" property, i.e., the URL of the MdO entry point for the corresponding network.

        

  HTTP/1.1 200 OK
  Content-Length: [TBD]
  Content-Type: multipart/related; boundary=example

  --example
  Content-Type: application/alto-endpointcost+json

  {
    "meta": {
      "cost-type": {
         "cost-mode": "array",
         "cost-metric": "ane-path"
       },
    },

    "cost-map": {
      "SAP1": {
        "SAP2": {
            "SAP1": {
                "NF1": [
                  [ "AS1" ], [ "AS1", "AS2", "AS3" ]
                ]
            },
            "NF1": {
                "NF2": [
                  [ "AS1", "AS2" ], [ "AS3", "AS2" ]
                ]
            },
            "NF2": {
                "NF3": [
                  [ "AS2", "AS1" ], [ "AS2", "AS3" ]
                ]
            },
            "NF3": {
                "SAP2": [
                  [ "AS1", "AS2", "AS3" ], [ "AS3" ]
                ]
            }
        }
      }
    }
  }

  --example
  Content-Type: application/alto-propmap+json

  {
    "property-map": {
      "pid:AS1": { "entry-point": "http://172.25.0.10:8888/escape" },
      "pid:AS2": { "entry-point": "http://172.26.0.10:8888/escape" },
      "pid:AS3": { "entry-point": "http://172.27.0.10:8888/escape" }
    }
  }

  --example--

        
                    

6. Acknowledgments

This work is supported by the Innovation Center of Ericsson S.A., Brazil (grant agreement UNI.62).

Thank you to Robert Szabo (Ericsson Research, Hungary) for the contribution and substantial feedback and suggestions in this document.

7. IANA Considerations

This document includes no request to IANA.

8. Security Considerations

TBD.

9. References

9.1. Normative References

9.2. Informative References

Appendix A. Proof of Concept Use Case Implementation

A strawman use case scenario has been implemented following the architectural proposal of the 5GEx project [H2020.5GEX]. It refers to an E2ENS orchestration involving three administrative domains.

As shown in Figure 2, each administrative domain has an MdO (MdO-AS1, MdO-AS2, and MdO-AS3) to coordinate resource and/or service orchestration at multi-operator level via interface I2 APIs. For the orchestration within the same administrative domain, each MdO uses emulated DOs with emulated I3 interfaces, since no data-plane is present. DOs use static configuration files to load local information about resources (I3-RC) and topology (I3-RT). The different MdO components are based on existing open source tools such as ESCAPE [H2020.5GEX.ESCAPE] (Service/Resource Orchestrator) and Netphony-topology [TELEFONICA.NET.TOPO] (Resource Topology) and run in Docker containers on a single computer. Besides, MdOs expose I1 interfaces to the tenants who request services and/or slices which should follow a Network Function Forwarding Graph (NFFG) [UNIFY.NFFG] format.

In case of the broker layer, the IdR and IdT components use a UNIFY Virtualizer API [UNIFY.NFFG] (broker-based I2-RC API) and a REST API (broker-based I2-RT API) respectively, in order to create the hierarchical databases. Regarding the IdT, the administrative domain 2 is a transit provider so that the domain-level topology computed is: AS1-AS2-AS3. From the inter-domain information are created the two different ALTO Map Services: (i) Property Map and (ii) Cost Map.

                    
                         +----------------------------------------+
                         |       +---------------+    BROKER LAYER|
                    XXXXXXXXXXXXXX  ALTO Server  |                |
                    X    |       +--------+----+-+                |
                    X    |               /      \                 |
                    X    | +-----------+/+--+  +-\-------------+  |
                    X    | | Inter-domain   |  | Inter-domain  |  |
    +-----------+   X    | | Topology (IdT) |  | Resource (IdR)|  |
    | Service   |   X    | +----------------+  +---------------+  |
    | Graph (SG)|   X    +---------^-^----------------^--^--------+
    | Request   |   X              * *    .............  .
    +-----+-----+   X              * *    . ..............
          |         X              * *    . .              MdO-AS3
        I1|         X              * *    . .      +--------------+
          |         X     MdO-AS1  * *    . .      |              |
    +-----|---------X-----------+  * *    . .      |     MdO-AS2  |
    |     |                     |  * *    . .   +---------------+ |
    | +---v-------------------+ |  * *    . .   | +-----------+ | |
    | |                       | |  * *    . .   | |           | | |
    | |  Network Service Orch.| |  * *    . .   | |   NSO     | | |
    | |  (NSO)                | |  * *    . .   | |           | | |
    | +-----------------------+ |  * *    . .   | +-----------+ | |
    |                           |  * *    . .   |               | |
    | +---------+               |  * *    . .   | +---+         | |
    | | Resource........................... .   | |   |         | |
    | | Topoloy |               |  * *      .......RT |         | |
    | | (RT)    | +-----------+ |  * *          | |   |         | |
    | +---------+ |Resource   | |  * *          | +---+   +---+ | |
    |             |Orch.      | |  * **********************   | | |
    |             |(RO)       ******            |         |RO | +-+
    |             +-----------+ |               |         |   | |
    |                           |<------------->|         +---+ |
    +---------------------------+      I2       +-----+---------+
               /     \                                |
            I3/       \                               |I3
    +-------+---+  +-----------+                +-----------+
    | Domain    |  | Domain    |                | Domain    |
    | Orch (DO) |  | Orch (DO) |                | Orch (DO) |
    +-----------+  +-----------+                +-----------+
                                         Legend:
                                         XXX ALTO Protocol
                                         ... broker-based I2-RT API
                                         *** broker-based I2-RC API
  
                    

Figure 2: Broker-assisted 5GEx Info Exchange

The Property Map includes property values grouped by Autonomous System (AS). Such values are SAPs, NFs and the 5GEx Entry Point (e.g., the URL of the ESCAPE orchestrator). An example of the Property Map in our prototype is:

The Cost Map defines a path vector as an array of ASes, representing the AS-level topological distance for a given E2ENS request. Path vector constraints (as described in the Multi-Cost Map [RFC8189]) can be applied to restricts the response to costs that satisfy a list of simple predicates.

Table 2 below shows a brief example of an SG request and its path vector response containing a list of potential providers to be traversed to deliver such service. Every AS path is computed from the inter-domain topology information in the IdT module. In our scenario, MdO-AS2 is a transit provider, so that the domain-level topology map is AS1<->AS2<->AS3.

Authors' Addresses