+-----------------------------+
                              | Newtork Service Orchestrator|
                              +-------+---------------+-----+
                                      |               |
                  +-----------+       |        +------+-----+
                  |   E/NMS   |       |        |  Resource  +-+
                  +-----------+       |        |Orchestrator| |
                        |             |        +------+-----+ |
                        |             |               |       |
                        |             |        +------+-----+ |
                  +-----+-----+       +--------+            | |
                  |    VNF    |----------------+    VNFM    | |
                  +-----------+                |            | |
                        |                      +------+-----+ |
                        |                             |       |
                  +-----+-----+                +------+-----+ |
                  |   NFVI    |----------------|     VIM    +-+
                  +-----------+                +------------+

Figure 3: ETSI NFV + SDN Reference Model

The model breaks the NFVO into a Network Service Orchestrator (NSO) and Resource Orchestrator (RO). This separation improves scaling of the NFVO and allows the RO to concentrate on support for multiple VNFMs and VIMs.

1.2. Challenges

Open questions exist with the NFV model around data models, scale and multi-domain support. The industry often points to the success of cloud services with respect to scaling and data models but can't agree upon how those solutions could be standardized to support NFV. The OSM model improves scaling but does not answer exactly how scalable the solution can be.

Both models assume that data is provided through the VIM to the NFVO components and data required by new VNFs is present in the system. This implies a large amount of information flow in a proactive manner from SDN Controllers through the VIM. If data is required from the Element Manager a direct interface to the VNFM is required but is not specified in either model.

Assumed access to data through the VIM or VNFM is problematic. Relevant data such as latency between sites can be measure via SDN Controllers supporting Wide Area Networking (WAN). Unfortunately WAN SDN Controllers, due to factors such as separation of concerns, scalability or security, are often not the same SDN Controllers a VIM will communicate with.

The number of SDN Controllers is often underestimated:

• SDN Controllers that support low latency signaling must be located close to the SDN switches they provide signaling on behalf of.
• SDN Controllers are separated not only by function, e.g. Wide Area, Application Specific or Data Center, but also by Security and Organization.
• The number of instances of VIMs and associated SDN Controllers varies but in general VIM Orchestrators manage a small number of racks with many VMs. Such densities produce large information that can limit the scalability of the associated SDN Controllers.

In practice, regional management models, multiple security zones and the fact that dense computing platforms can limit a system such as OpenStack to 4 to 12 racks of computing equipment per instance implies the presence of several SDN Controllers in any one network.

Regardless of a proposed framework for Orchestration, the following challenges exist:

• Access to data before a single Orchestration task occurs
• Access to data from many SDN and VIM related sources
• Access to new kinds of data required for Orchestration of a VNF
• Scalability of the data access

2. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

3. Solution Approach

A good solution that couples SDN and NFV permit Orchestration with a loose coupling of both technologies. To provide a basis for the solution a method is described that approaches how Orchestration may occur for an Orchestrator looking at a single NFV Forwarding Graph (NFV FG) may determine if a VIM has infrasctructure the meets its requirements.

3.1. Types of Information

Two kinds of information are required for the selection of a set of suitable resources to serve the VNF FG defined for a network service:

Resource Constraints

Specific constraints of a resource serving the VNFs of the VNF FG. These constraints include minimum and maximum processing, memory, storage and network requirements. They are based upon available resources, i.e. inventory, available as opposed to a Performance Constraint.

Performance Constraints

Specific constraints based upon a measured / computed metric. These measurements may occur over time and can exist outside of any specific VNF activity, e.g. the latency between two sites.

3.2. Single Candidate Solution Determination Process

Looking at the types of Orchestration information, one can evaluate a specific candidate solution for a subgraph in the VNF FG as a set of matrices.

Let VNF Requirements vector, VREQg, be the n Resource Constraints (RC) and m Performance Constraints (PC) for a VNF FG subgraph g

Let VNF Solution vector, VSOLg, be the vector representing the n Resources (R) and m Performance Metrics (PM) for a VIM's resources supporting VNF FG subgraph g

Without loss of generality satisfaction of a specific Performance Metric can be determined using subtraction, e.g. VREQg - VSOLg. For example if for latency must be less than 5 then negating the sign for the PM and associated RC. If any value in the resulting vector is less than 0 then, VSOLg does not satisfy VREQg.

Multiplying the resulting matrix by a M+N x 1 matrix of weights yields a value that can be determined as the fitness of the solution. His can be compared with VSOLg values.

3.2.1. Resource Values as Partitions

The representation of Resource Constraints includes the minimum/ maximum inventory required. However, a VIM representing a resource for a graph to a Resource Orchestrator must be careful not to over-summarize the values of the resources it has. For example, let the VNF FG subgraph G be comprised of 3 VNFs:

• VNF A requires 2 CPUs
• VNF B requires 2 CPUs and
• VNF B and VNF A have an anti-affinity, e.g. cannot be in the same hardware.

A VIM stating it has 4 CPUs of resource is insufficient to determine if it can satisfy G. Looking at the partitions (# of ways to add integers to equal the value of the integer) we see the partitions of 4 are:

• 4
• 3+1
• 2+2
• 2+1+1
• 1+1+1+1

Of the 5 partitions of 4 only one, 2+2, would satisfy G. Thus, it is important to represent the paritions and not summarize the value in order to determine if a NFVI has sufficient resources for satisfying the VNF FG.

For representing large numbers though this data can be compacted as we propose here. We let the notation for Resource values be

where X is a non-negative integer and partition of X for the entity optionally labelled "A". The sub-partitions of X are themselves partitions. The sub-partition summary is the specific sub-partition of X in comma separated values. while the options sub-partition details may also be present.

If the VIM has in Chassis X 3 blades, B1 with 2 CPU, B2 with 1 CPU and B3 with 1 CPU we represent this as

When evaluating this solution we can quickly look at the first value and determine basic satisfiability. Further inspection is possible as well.

A multi-dimensional resource representation is achieved by turning individual values into arrays. For example if we added memory for all blades of 4 Gigabytes we could represent the Resources as

Other compaction notations can be introduced as well, e.g. representing B2 and B3 as {2}[2,1] in the partition description and [ [1,2], ["B2","B3"], [1,2]]. However, it is proposed that this is the maximum amount of information for Resource values interchanged between VIMs and Orchestration functions.

When no anti-affinities are present it may be sufficient to look at the initial values of the Resource vector. As the example shows looking at specific partition information is required when anti-affinities are present in the VNF FG subgraph.

NOTE that although this document proposes a representation for Resource Values other representations exist that are valid, e.g. a tree repsresentation, and may be more efficient.

3.2.2. Performance [Metric] Constraints and Measurements

Performance Constraints for VNF FG subgraph G have value and optionally measurement constraints. Some measurement constraints are:

Staleness:

Age of the measurement that is suitable for usage; otherwise new measurements must be taken.

Granularity:

Fine or coarse grain. Fine grained measurements require direct measurements between the endpoints (resources of underlying resources such as the hardware) in question. Coarse grain measurements require values produced through measurement of entities in the same logical entity, e.g. site, chassis, rack, etc., of the proposed solution elements.

Accuracy:

The degree of closeness of measurements to its true value.

3.2.3. Data Required

Table 1 shows the data required and source(s) in order to determine VIM solution candidate suitability for a VNF FG subgraph.

                +----------------+
                |  ALTO Client   |
                +----------------+
                        |
                        | ALTO
                        |
               +--------+---------+
               | ALTO Query Tier  |
               +------------------+
                        ^
                        | ALTO SSE
                        |
               +--------+---------+
               | ALTO Aggregation |
               |      Tier        |
               +--------+---------+
                        ^
                        | ALTO SSE
                        |
                  +-----+-----+
                  |    ALTO   |
                  |  Servers  |
                  +-----+-----+

 +-----------+
 |   ALTO    |<----------------------------------------+
 |  Server   +---------------------------+             |
 +---+-------+                           | (B)         |
     |     ^                             |             |
     |     |      +-----------+          |             | (A)
     |     +------+   E/NMS   |   +------+-----+       |
     |       (C)  +-----------+   |Orchestrator+-+     |
     |                  |         +------+-----+ |     |
     |                  |                        |     |
     |            +-----+-----+   +------+-----+ |     |
     +----------->|    VNF    |---+    VNFM    | |     |
         (D)      +-----------+   |            | | +---+--------+
                        |         |            |-+-|    SDN     |
                        |         +------+-----+ | | Controller |
                        |                |       | +---+--------+
                  +-----+-----+   +------+-----+ |     |
                  |   NFVI    |---|     VIM    |-+     |
                  +-----+-----+   +------------+       |
                        |                              |
                  +-----+-----+                        |
                  |    SDN    |------------------------+
                  |   Switch  |
                  +-----+-----+

         VNF Subgraph      VIM Candidate   VIM Candidate
            Matrix            Matrix     Suitability Matrix
         +--  |   --+      +--  |   --+   +--         --+
      S1 |    |     |   C1 |    |     |   |    Any      |
      S2 |    |     |   C2 |    |     |   |  row w/o    |
      .  |  A |  B  | - .  |  C |  D  | = |  negative   |
      .  |    |     |   .  |    |     |   |   values    |
      .  |    |     |   .  |    |     |   | is as valid |
      Sn |    |     |   Cn |    |     |   |  candidate  |
      ^  +--  |   --+   ^  +--  |   --+   +--         --+
      |                 |
Each Row is a        Each row is a candidate partial solution,
VNF FG subgraph      i.e. subgraph of a specific VIM intended
                     to support the corresponding VNF FG subgraph

      +--         --+ +--      --+   +--      --+
      |    VIM      | | Weight   |   | Weighted |
      | Candidate   | | Vector   |   | Solution |
      | Suitability | | (m x 1   | = | values   |
      |   Matrix    | |  matrix) |   | (n x 1)  |
      |   (n x m)   | |          |   | vector   |
      +--         --+ +--      --+   +--      --+