Internet DRAFT - draft-veitch-nfvrg-nfv-nw-optimization
draft-veitch-nfvrg-nfv-nw-optimization
Internet Engineering Task Force A. Veitch, Ed.
Internet-Draft Netcracker, Inc.
Intended status: Informational November 30, 2016
Expires: June 3, 2017
Use Cases and Analysis on Integrated NFV and Network Optimization
draft-veitch-nfvrg-nfv-nw-optimization-00
Abstract
This is a review of issues related to the optimized deployment of
network services, aka service function chains, composed of virtual
and physical network functions, where these functions may be
instantiated in multiple distributed data centers. Criteria for
optimization are introduced, and use cases are described and
expanded, in order to establish the need for coordinated computation
of NF deployment sites and the connectivity among them. Some methods
for addressing optimization pain points are described, and potential
new requirements of the PCE and PCEP are discussed.
One point to make clear is that the exploration of specific
algorithms for NF/VNF placement and path computation is outside the
scope of this draft. We are not looking at computational or optimal
greedy search algorithms. The goals for this draft are 1) to provide
justification through use cases for more tightly integrating path and
placement computation algorithms, and 2) looking at how some of the
optimization requirements might be addressed, mostly through pre-
computation and caching, and how these might affect management and
orchestration functions and the protocols (e.g. PCEP) that are used.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 3, 2017.
Veitch Expires June 3, 2017 [Page 1]
Internet-Draft Integrated NFV and Network Optimization November 2016
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4
3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Optimization Criteria . . . . . . . . . . . . . . . . . . . . 5
5. Challenge of Network Service (or Service Function Chain)
Optimization Across Multiple Sites . . . . . . . . . . . . . 6
6. Review of Current NFV / SDN Management Solutions . . . . . . 6
7. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 9
7.2. Virtual Content Delivery Network (vCDN) - Dynamic (Flash)
Delivery . . . . . . . . . . . . . . . . . . . . . . . . 9
7.3. Video Narrowcasting Use Case . . . . . . . . . . . . . . 12
7.4. Real-Time Telemetry Use Case - Internet of Things . . . . 14
7.5. High Performance Computing (HPC) . . . . . . . . . . . . 15
8. Orchestration of Mixed Applications and Network Services . . 16
9. Generalized Architecture Options for Coordinated or Unified
SDN/NFV Deployment Planning . . . . . . . . . . . . . . . . . 17
10. Analysis of Integrated Network and Compute Optimization with
a Simple Use Case . . . . . . . . . . . . . . . . . . . . . . 18
10.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 18
10.2. Network Topology . . . . . . . . . . . . . . . . . . . . 18
10.3. Simple Orchestration . . . . . . . . . . . . . . . . . . 19
10.4. Connectivity Information in VNF Placement Planning . . . 20
10.5. Pre-computation of a Connectivity Matrix . . . . . . . . 21
10.6. Multi-valued Connectivity Matrix . . . . . . . . . . . . 22
10.7. Additional Optimization Options . . . . . . . . . . . . 23
10.8. Cached Paths Over Multiple Sites . . . . . . . . . . . . 24
10.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . 24
11. Connectivity based on Potential vs. Actual Topologies . . . . 24
12. Virtual Networks Equivalency . . . . . . . . . . . . . . . . 25
13. Implications for the Network Controllers (SDN / PCE) and
Veitch Expires June 3, 2017 [Page 2]
Internet-Draft Integrated NFV and Network Optimization November 2016
Orchestrators . . . . . . . . . . . . . . . . . . . . . . . . 26
14. Relationship to TEAS TE Topology and ACTN . . . . . . . . . . 26
15. Areas of Future Evaluation . . . . . . . . . . . . . . . . . 27
16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27
17. Security Considerations . . . . . . . . . . . . . . . . . . . 28
18. References . . . . . . . . . . . . . . . . . . . . . . . . . 28
18.1. Normative References . . . . . . . . . . . . . . . . . . 28
18.2. Informative References . . . . . . . . . . . . . . . . . 28
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 29
1. Introduction
With the advent of Network Function Virtualization (NFV), network
services, consisting of an ordered set of network functions (NFs),
may be composed of physical (PNF) and virtual network functions
(VNF). These VNF will execute in one or more virtual machines or
containers operating on standard high performance servers. The set
of network functions for a given network service might be
instantiated within a single site or the network service might be
multi-site.
This document is a brief examination of issues related to the
optimized deployment and operations of network services, and in
particular, multi-site network services. It includes a review of use
cases that reveal possible limitations in current NFV and SDN
management architectures, as defined in some standards and open
source projects, and related communications protocols.
The use cases described include the virtualization of support for
Content Delivery Networks (CDN), Internet of Things, video
narrowcasting, and high performance computing. In this analysis, an
important consideration is the shared infrastructure among a variety
of applications and services. Sharing the infrastructure has the
advantage of increasing overall utilization of the infrastructure vs.
support for dedicated solutions for each service type. One can
imagine disjoint networks for each of the services being implemented,
with the result being a very high total cost to the provider and
consumer.
Another advantage is that the resource utilization profiles for each
of these services will differ. High performance computing might
require a good deal of computational power and memory, which
translates into requiring many CPU, virtualized CDN require a large
amount of storage, and the main requirements for video services might
be low latency and jitter in the delivery of its traffic from its
source to its consumers.
Veitch Expires June 3, 2017 [Page 3]
Internet-Draft Integrated NFV and Network Optimization November 2016
An understanding of the expected resource usage for different
services and network functions will allow intelligent placement of
these services, not just to provide the user a high quality of
experience (QoE), but also provide the operator with a lower CAPEX
through smarter utilization of the resources.
One of the goals in reviewing these use cases is to understand the
common information that is necessary for an operator to effectively
provide optimal solutions for the various services to be supported,
and their disparate requirements.
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
3. Definitions
Note: This document includes concepts and terminology used in both
the IETF and in the ETSI NFV ISG. A number of concepts are shared or
have very similar counterparts between the groups. Here are some of
the key terms used and their definitions.
o Management and Orchestration (MANO) - Describes the architecture
framework to manage NFVI and orchestrate the allocation of
resources needed by the NSs and VNFs.
o Network Function (NF) - A functional block within a network
infrastructure, that has well-defined external interfaces and a
well-defined functional behavior.
o Virtual Network Function (VNF) - Implementation of an NF that can
be deployed on a Network Function Virtualisation Infrastructure
(NFVI). IETF corollary is the Service Function or SF in SFC.
o NFV Infrastructure (NFVI) - The NFV-Infrastructure is the totality
of all hardware and software components which build up the
environment in which VNFs are deployed. It may span across
several sites, e.g. where data centers operate.
o NFVI-PoP - Where a Network Function is or could be deployed as
Virtual Network Function (VNF)e
o Network Function Virtualization Orchestrator (NFVO) - functional
block that manages the Network Service (NS)lifecycle and
coordinates the management of NS lifecycle, VNF life cycle and
Veitch Expires June 3, 2017 [Page 4]
Internet-Draft Integrated NFV and Network Optimization November 2016
NFVI resources to ensure an optimized allocation of the necessary
resources and connectivity
o Network service (NS) - A composition of network functions and
defined by its functional and behavioral specification.
o VNF Forwarding Graph (VNFFG) - A NF forwarding graph where at
least one node is a VNF.
o Multi-site network service - A service that has component network
functions operating in infrastructure located in separate
geographical sites. These sites may consist of a combination of
customer sites and operator data centers. Service components are
therefore connected across wide area networks (WANs.)
4. Optimization Criteria
One question to be asked, when we mention optimization, is, what is
it that we want or need to optimize when a network service or
services are to be deployed? And the answer is that there are a
number of criteria we would like to see maximized or minimized, some
of which lead in opposite directions.
o Maximize the likelihood a requested network service will be
honored and instantiated, and not rejected due to a lack of
available resources to meet the service's SLAs.
For a multi-site service, it is important that the site of the
network functions and the connectivity between them be selected to
ensure any SLA requirements are met. The search for a solution
should not fail when adequate resources exist to support the
service request.
o Maximize the health of a network service and minimize the
likelihood that service might fail to meet its service level
agreements, or SLAs, during its lifetime.
Each network service deployed must meet an SLA, with a penalty
likely imposed should the service fail to do so. These SLAs will
include metrics such as Availability, throughput or bandwidth,
loss, latency, maximum outage time, mean time between failures,
etc. Goals include minimizing the risk of any single service
failing to meet its SLAs, as well as minimizing the overall number
of services that fail to meet their SLAs.
o Minimize the time it takes to respond to a network service request
with an instantiation of that service. Alternatively, a request
may ask only for the set of resources to support a network
Veitch Expires June 3, 2017 [Page 5]
Internet-Draft Integrated NFV and Network Optimization November 2016
service, but not to actually deploy the services. The response
time for this request should also be minimized.
o Maximize the utilization of the network and compute
infrastructure, or NFVI, to achieve OPEX and CAPEX savings.
One promise of NFV is the reduction of OPEX and CAPEX. In other
words, with greater flexibility and finer grained control, the
better a management system can utilize the NFVI. CAPEX is clearly
lower when less total NFVI is used, and OPEX should be too. For
example, excess hardware may be hibernated when not needed, saving
energy costs and management costs.
5. Challenge of Network Service (or Service Function Chain)
Optimization Across Multiple Sites
When deploying a network service, it is important to place the
component network functions (PNF/VNF) in sites to deliver an
optimized network service, as based on the criteria listed above.
Deciding how and where to deploy the VNFs is made more complex when
the network service is to be deployed across multiple sites.
A network service descriptor includes a set of NFs and links that
represent connections between pairs of the network functions.
Associated with each link might be a set of criteria, e.g. latency,
bandwidth, etc., that an instance of the service must meet. When NFs
are to be placed in different sites and separated by one or more
networks, both the sites of the NFs and the connectivity paths
between the pairs must be selected to meet the NF and connectivity
requirements defined for the network service and its components.
This means that choosing the placement of the VNFs and selecting
their connectivity are interdependent activities.
6. Review of Current NFV / SDN Management Solutions
Today a number of management and orchestration architectures for NFV
based services have been defined and are described by standards
organizations, open source projects, as well as some service
providers. The ETSI NFV ISG described the original NFV management
architecture: The Management and Orchestration of NFV based network
services, commonly referred to as MANO, was defined in phase 1 of the
ISG work program.
Since then, other standards bodies have addressed the architecture,
and a number of software projects undertaken to develop management
systems, based on, or strongly influenced, by the ETSI MANO
architecture. These include, but are not limited to:
Veitch Expires June 3, 2017 [Page 6]
Internet-Draft Integrated NFV and Network Optimization November 2016
o Lifecycle Service Orchestration (LSO) from the Metro Ethernet
Forum (MEF)
o Open Source MANO (OSM), also sponsored by ETSI
o Open Platform NFV (OPNFV) from Linux Foundation
o SONATA
o Openstack, including Tacker, and networking-sfc
o Open-O
o Enhanced Control, Orchestration, Management and Policy (ECOMP)
from ATT
o TMForum architecture
These architectures generally follow two paradigms. Each includes an
End-2-end (E2E) orchestrator that is responsible for overall
orchestration of network services, including the VNFs, PNFs and
network connectivity. This E2E orchestrator then communicates with
an NFVO, an orchestrator for the network services based on the VNFs.
Where they differ is in how the network controller, often an SDN
controller, fits. In most of these, the E2E controller communicates
directly to both the NFVO and the network controller. In others the
NFVO communicates to the network controller.
Veitch Expires June 3, 2017 [Page 7]
Internet-Draft Integrated NFV and Network Optimization November 2016
+----------------------------------------------------------------------+
| E2E Orchestration |
+----------+------------------------+------------------------+---------+
| | |
| | |
+----------+---------+ +----------+---------+ +---------+---------+
| WAN SDN Controller | | DC SDN Controller | | NFVO |
+----------+---------+ +----------+---------+ +---------+---------+
| | |
| | +---------+---------+
| | ---------+ VIM |
| | | +---------+---------+
| | | |
----------+-------- ----------+-----+-- +---------+---------+
( WAN Networks ) ( DC Network ) | VNFs |
------------------- ------------------- +-------------------+
+----------------------------------------------------------------------+
| E2E Orchestration |
+-----------------------------------+----------------------------------+
|
|
+----------+---------+
+-------------+ NFVO +--------------+
| +----------+---------+ |
| | |
| | |
+----------+---------+ +----------+---------+ +---------+---------+
| WAN SDN Controller | | DC SDN Controller +----+ VIM |
+----------+---------+ +----------+---------+ +---------+---------+
| | |
| | |
----------+-------- ----------+-------- +---------+---------+
( WAN Networks ) ( DC Network ) | VNFs |
------------------- ------------------- +-------------------+
Two typical paradigms of network service orchestration seen in open
source and SDO architectures.
Figure 1
The key point is that in both paradigms the NF deployment decisions
and connectivity decisions are controlled by separate functions. As
described in literature to date, the functionality described for
each, and the information to be exchanged, is not adequate to avoid
potential inefficiencies and unnecessary network service deployment
Veitch Expires June 3, 2017 [Page 8]
Internet-Draft Integrated NFV and Network Optimization November 2016
failures, due to the real interdependence these functions have upon
each other.
Of course, these architectures are currently undergoing active
development through open and iterative design processes. Therefore,
the state of these designs is very fluid, with new capabilities
continuously evolving. Therefore, the description provided here
represents only a moment in time. This document is intended to
provide insight into possible limitations that exist in computing
deployment solutions and to recommend updates to functional block
capabilities and communications (perhaps leading to protocol
updates), that will lead to improving these computations. It is
expected that as these projects evolve, this draft too will evolve,
further clarifying roles, communications, and needed extensions.
7. Use Cases
7.1. Introduction
The following are a set of network service examples that demonstrate
the value of coordinated or unified compute and network deployment
and configuration planning.
7.2. Virtual Content Delivery Network (vCDN) - Dynamic (Flash) Delivery
The following are a set of network service examples that demonstrate
the value of coordinated or unified compute and network deployment
and configuration planning.
ETSI and other groups have identified virtual CDN (vCDN) as an
application well suited to operate as an NFV network service. As 5G
is adopted and bandwidth greedy applications such as UHD video
proliferate, mobile data will have a huge impact as a source of
network traffic. CDNs have a proven track record as an effective way
to provide high quality and low latency delivery of content to users,
while simultaneously limiting the overall utilization of network
bandwidth. Operators need to continue to provide a high quality
service while limiting provisioning unnecessary infrastructure.
Virtual CDNs are seen as an excellent way to address these new
pressures.
There are some obvious advantages to a virtualized solution. The
costs will shift from CAPEX to OPEX as the vCDN can operate on
standard high capacity servers. More importantly, a software-based
solution enables rapid configuration of the vCDN, faster upgrades of
the software, and mobility of the vCDN vCaches. With vCDN it is
possible to dynamically and intelligently size the vCaches, and move
them to sites where they provide the best value, based upon the
Veitch Expires June 3, 2017 [Page 9]
Internet-Draft Integrated NFV and Network Optimization November 2016
current and anticipated near-term traffic volumes and patterns. More
ephemeral demands for content can be more effectively managed.
The SONATA Use Cases and Requirements document (http://www.sonata-
nfv.eu/content/d21-use-cases-and-requirements) categorizes two types
of vCDN scenarios.
First is the more traditional distribution of popular content.
Content originates at a content provider and it is distributed on
vCaches located as needed for a large number of subscribers.
The second scenario is concerned with user-generated content and is
much more dynamic. This is the one that is examined here. This is a
'flash' vCDN, one where the need for a CDN is due to a sudden burst
of content, one with a limited expected lifetime. One example might
be any large sporting event that many people attend. These attendees
will likely take and post multiple videos to the Web with a social
media application, and then they and their friends will view and
download these videos multiple times. During the event there will a
real need to support caching this content and making it available to
consumers while limiting network impact.
Veitch Expires June 3, 2017 [Page 10]
Internet-Draft Integrated NFV and Network Optimization November 2016
+-----------+
+--+ End Users | +-----------+
| +-----------+---------------------| End Users +---+
+---+------+ ------/ WAN \--- +-----------+ |
| Data |-/ \--- +-------+--+
| Center +- \ | Data |
| NFVI-PoP | \@ -+ Center |
+----------+ \@ @/ | NFVI-PoP |
| \@ @/ +----------+
-\ \@ @/ /-
--\ \@ @/ /--
---\ \@ @/ /---
---+-------------\---/--------+----
| Data Center (NFVI-PoP) |
| +---------+ +---------+ |
| | vCache | | vCache | |
---- | +---------+ +---------+ | -----
--/ +---+-------------------+--+ \--
( | Access Network \ )
--\ / | /--
--------\ | \ /--------
--/-----------------------\--
| +---------+-----+
+----------+----+ | Media Content |
| Media Content | | Creator |
| Creator | +---------------+
+---------------+
Topology of vCDN use case with a rapid short term need for content
caching.
Figure 2
The challenge for this use is for the provider to be able to rapidly
deploy and configure the a vCDN solution so that the users' QoE is
high, both the sources and the consumers, while the resource
utilization, mostly the WAN connectivity, is minimized.
In this use case the placement of the vCaches is driven by a number
of factors, including the proximity to the users posting the videos,
the proximity to users consuming the videos, the available network
capacity among the data centers, reliability (availability) depending
upon the service, the costs of the data center servers, and the cost
of the network, if connectivity requires transit across a third
party's network. This is not simply a matter of identifying the data
centers and then connecting them. Candidate sites must be
identified, connectivity options must be identified, and the
combination of these must be evaluated in the context of the factors
Veitch Expires June 3, 2017 [Page 11]
Internet-Draft Integrated NFV and Network Optimization November 2016
listed above, past and current performance measurements, and any
provider or user policies. Multiple deployment options may be
considered, with one selected to provide an optimal solution.
As this will be a dynamic vCDN deployment, frequent monitoring of
demand and delivery metrics is a major necessity, and redeployments
and upgrades, or possible reductions when demand wanes, must be
addressed with no unnecessary delay.
7.3. Video Narrowcasting Use Case
Video narrowcasting is a targeted delivery of video (and audio)
content to multiple consumers, perhaps from the tens to low
thousands, and delivery is generally expected to be broadcast
quality. This is not broadcast television, but might be used for
distance training, or education, or internal communications within
large organizations. It is also a term used to describe the delivery
of commercial television to a niche audience, e.g. from a cable
service provider. This differs from popular microcomputer based
applications such as Skype or weChat, as high quality (high
definition), high availability and very low latency are expected.
In some cases, there will be more than one video source; you can
imagine a presentation where presenters are based in different sites,
and in some of these cases, the video encodings might differ from the
different sources. In addition, lower quality video and audio might
be available for users to send feedback to the primary sources.
A number of factors must be examined when choosing how to support
this service. These include, the number of sources, their sites,
their encoding capabilities, and the bandwidth available over which
each can send. Also to be considered are the receivers, their number
and site, their bandwidth capacity over which to receive the videos,
and their codecs. How many incoming video streams can the receiver
process at once.
Today there are commercial hardware and software products and
packages to provide this capability. These, however, can be costly
and are not very flexible.
You can imagine a number of situations. The simplest would be if all
senders have the same video codec, all receiver codecs can decode as
many incoming video streams as needed, and all senders and receivers
have more than enough bandwidth for sending or receiving, and high
usage of WAN links is not a problem. When this is true there is very
little service management needed. Unfortunately, this is unlikely.
Veitch Expires June 3, 2017 [Page 12]
Internet-Draft Integrated NFV and Network Optimization November 2016
+-----------+
+--+ End Users | +-----------+
| +-----------+---------------------| End Users +---+
+---+------+ ------/ WAN \--- +-----------+ |
| Data |-/ \--- +-------+--+
| Center +- \ | Data |
| NFVI-PoP | \@@ -+ Center |
+----------+ \@ @/ | NFVI-PoP |
| \@ @/ +----------+
-\ \@ @/ /-
--\ \@ @/ /--
---\ \@ @@/ /---
+----------------\---/-------------+
| Data Center (NFVI-PoP) |
| +---------+ +-------+ +--------+ |
| | video | | video | | video | |
| | process | | mgmt | | server | |
---- | +---------+ +-------+ +--------+ | ----
--/ +-----+--------------------+-------+ \--
( | Access Network \ )
--\ / | /--
--------\ | \ /--------
--/-----------------------\--
| +---------+-----+
+----------+----+ | Video |
| Video | | Source |
| Source | +---------------+
+---------------+
Topology of video narrowcasting>.
Figure 3
More likely is that some of the following are true and must be
addressed.
There are multiple senders with different coding capabilities, or
different bandwidths available over which to send. In addition, some
of the receivers do not have the video codecs to decode some of the
sender encodings, or lack the bandwidth over which to receive the
packets of the high definition video.
When this is true, functions must be installed into the network to
manage these issues properly. Video processing can be added to mix
video of different sources, if necessary for some consumers to see
multiple senders. Consumers may even remotely control this
processing, on demand. Other processing may be needed to compress
the video, with the effect of lowering its quality, but making it
Veitch Expires June 3, 2017 [Page 13]
Internet-Draft Integrated NFV and Network Optimization November 2016
possible for consumers with limited bandwidth (e.g. mobile) to see
it. These and many other factors will drive selecting where to place
these functions, connect them to one another, and how to forward
these to the end consumers (e.g. what IP multicast groups, etc.), all
to provide the best QoE while limiting resource consumption.
The evaluation of these factors may quickly become complex, and may
be continuous as consumers join and leave. The placement of the VNFs
will depend greatly on network connectivity, capacity and latency, as
well as the potential to provide connections directly over electrical
or optical, vs. multi-hop packet.
7.4. Real-Time Telemetry Use Case - Internet of Things
The Internet of Things (IoT) is growing exponentially and will be a
huge source of information to be communicated across the Intenet in
the coming years. This use case in concerned with a large number of
sensors being activated over a short amount of time. One example of
this could include a large sporting event, e.g. the Boston Marathon,
with multiple thousands of competitors wearing health-monitoring
applications that communicate vital signs as telemetry over the
Internet. Or this could be relevant for a natural disaster (or man-
made disaster.) Sensors might be deployed, or simply activated to
report much more data, because of an incoming hurricane, or a
volcanic eruption, tornado, forest fire, or tsunami and
infrastructure emergencies. This could necessitate the creation of a
new IoT gateway and other associated components located nearer to the
sensors and with the capacity to forward the data in real-time to its
consumers.
This use case is quite similar to the flash vCDN use case described
above. Here, instead of vCaches, we have IoT gateways. Instead of
cameras taking videos to be uploaded and downloaded, we have sensors,
to be processes and forwarded in real-time to the consumers. This is
a case where low latency, and perhaps low jitter, is required, though
high throughput less so. Again, these network requirements might
impact the placement of the gateways and their connectivity.
A further complexity is introduced when we consider these sensors to
be mobile. One example will be with self-driving cars, or even human
driven cars, but where there are many more sensors. The vehicles
will generate a great deal of data which must travel over the
wireless access networks to the nodeBs, behind which will be IoT
gateways to process the data and forward the relevant data in real
time to other functions. These might automatically change driving
routes, maximum speeds for some roads, etc.
Veitch Expires June 3, 2017 [Page 14]
Internet-Draft Integrated NFV and Network Optimization November 2016
In this scenario it seems likely that different areas will have
different loads of network sensor data based upon the time of day.
There will be low volumes during the day, with greater volumes during
rush hour. And these will migrate from city or industrial areas, to
suburbs. (Just as, no doubt, today the cell phone calls managed in
different areas depend upon the time of day.) Virtual IoT gateway
deployments and the connectivity among the gateways and telemetry
consumers will there for also be varied. This is more predictable,
but deployment options must still consider connectivity, network
utilization, etc. when placing the vIoT gateways, etc. There are
also likely more dynamic mobile scenarios to be reviewed, e.g.
travelers evacuating an area due to a hurricane, etc.
7.5. High Performance Computing (HPC)
Note: A number of concepts here are taken from 'HPC-Aware VM
Placement in Infrastructure Clouds', by Gupta et al.
High performance computing (HPC) in the cloud promises to expand the
number of computing applications that can be supported and will lower
costs over all. However, HPC has a number of requirements that place
significant demands upon the compute and networking infrastructure.
Included among these are the need to 1) periodically move huge
volumes of data among distributed applications as well as 2) support
multiple processes that are tightly coupled and require frequent
communications and synchronizations. In addition, for some large
computations, there may be the need to rapidly expand to very large
numbers of collocated VNFs for large computations with very strict
synchronization time windows.
There is active research today for improving VM scheduling, hardware
utilization, how best to partition clusters for different application
types, and map functions to sites to maximize utilization.
When considering some of these demands, a few things are clear. For
communicating large volumes of data, it will be necessary to dedicate
significant bandwidth at times and limit latency. When supporting
HPC as a service a provider must anticipate these sorts of needs, and
ensure the infrastructure resources supporting other services are
utilized such that the capacity to meet this sort of demand is
possible without the need to migrate services and VNFs.
When the resource requirements of cooperating applications recommend
locating the applications in separate data centers and the
application's synchronized communications requirement demands
connectivity of a high bandwidth circuit, the provider must ensure
both the data center infrastructure and connectivity between the
Veitch Expires June 3, 2017 [Page 15]
Internet-Draft Integrated NFV and Network Optimization November 2016
datacenters meet the service requirements. In this case, while a
circuit may not exist, the function computing the connectivity should
be aware of the potential circuit, for situations such as supporting
this network service.
8. Orchestration of Mixed Applications and Network Services
The use cases listed above place a variety of disparate demands upon
the data center (NFV-IPoP) and network infrastructure. For each
individual network service instance, a service deployment algorithm
may compute the sites and connectivity for that specific instance.
These computations are made in the isolation of the context of that
specific service instance, perhaps over its lifecycle.
However, there is a need to consider and manage overall delivery and
support of these individual services. As has been described above,
it is beneficial to deploy across the data centers a balanced
distribution of network service components with different resource
requirement profiles. This will help to optimize the utilization of
the data center and network infrastructure. And in circumstances
when some of the services are known to place specific large demands
on infrastructure, for example, instantiating an ODU circuit to
provide high bandwidth real-time communications, some agent must be
aware of this sort of demand to ensure it is met.
This implies that there could be some sort of higher-level agent, one
that works with the agents (e.g. NFVO and SDN controller) computing
compute and network deployments for individual requested service
definitions. This higher layer agent must ensure that resources are
allocated properly so that those agents compute deployments that also
honor the greater goals, e.g. the balanced distribution of
heterogeneous services, resources ready for specific demands of some
of the services, general reduced energy consumption and lower costs,
etc. These should lead to the optimization goals described at the
beginning of this paper.
The exact communications that might be required to support this
should be explored. These could impact the controller NBI or the PCE
pro tocol.
Veitch Expires June 3, 2017 [Page 16]
Internet-Draft Integrated NFV and Network Optimization November 2016
+-----------------------------+
| E2E Orchestration |
| +------------+ |
| | General | |
+------+------ |Optimization|-+
| +-------+----+
| -/ |
| -/ |
+------+--+--+ +-----+------+
| | | Network |
| NFVO | | Controller |
| | |(PCE / SDN) |
+------------+ +------------+
Shows an agent that tracks usage to provide guidance for general
optimization.
Figure 4
9. Generalized Architecture Options for Coordinated or Unified SDN/NFV
Deployment Planning
To achieve the integrated NFV/SDN optimization goals requires a
software system that computes the optimizations for compute and
network.
One option would be to replace the separated agents, the NFVO and SDN
controller, with a single unified agent that processes the network
topology, configuration and utilization, as the PCE / SDN controller
does today, and computes the VNF placement, as the NFVO does today.
If we look at the E2E Orchestrator in Figure 1, one option is for the
network SDN controller and the NFVO to send full topology and NFVI
(data center topology, compute et al resources and existing use and
utilization) information to the E2E Orchestrator, where it is the
unified agent, computing the optimal deployment, computing the path
and NFVI allocations, and then directing specific actions of the PCE
/ SDN agent and NFVO to deploy the updates.
Alternatively, the functionality of the NFVO and SDN Controller
remain mostly consistent. Each fills its current set of
responsibilities, the PCE computing paths and the NFVO planning NFVI-
PoP updates with the new VNF instances. If this is done, then
enhancements must be made so these agents coordinate and compute the
best overall deployment.
Veitch Expires June 3, 2017 [Page 17]
Internet-Draft Integrated NFV and Network Optimization November 2016
10. Analysis of Integrated Network and Compute Optimization with a
Simple Use Case
10.1. Introduction
Here is presented a simple network service, composed of two VNFs
(SFs) across two data centers. This could be a vCPE service with a
remote VNF. This simple case is used here to provide a basis on
which to explore how network services (SFCs) may be orchestrated.
Solutions for the simple case are examined iteratively, each new
version presenting more advanced orchestration capabilities that
address deficiencies identified in the previous version. The purpose
here is not to examine any path computation optimization algorithms.
It is intended to demonstrate how other methods, including pre-
computation and/or memoization, might aid in meeting the optimization
goals, and what this might mean for network topology modeling and
protocol definitions.
Note: Many concepts in the TE Topology model currently being defined
in the TEAS WG, can be considered relevant to this work. The
intention of this section is to describe the general problem and
approach. How this relates to the concepts and models defined in the
TE Topology is explored in the next section of this document.
+------------------+ +-------+ +-------+
| Service Endpoint | | | | |
| (Cust. Premises) +-------+ VNF A +-----------+ VNF B |
| | | | 3GB req. | |
+------------------+ +-------+ +-------+
Shows an architecture.
Figure 5
10.2. Network Topology
The topology includes three data centers and a user enterprise site,
where the a user service endpoint is located.
Two variations are used. In the first, tunnels are already
established among the data centers. In the second, a tunnel is
established specifically to support the link between the network
functions.
Veitch Expires June 3, 2017 [Page 18]
Internet-Draft Integrated NFV and Network Optimization November 2016
+-------------+ +-------------+
+-----------------+ | | | |
|Service Endpoint | | Data Center | -------- 2 GB | Data Center |
|(Cust. Premises) +--+ (NFVI-PoP) +--( WAN )---------+ (NFVI-PoP) |
+-----------------+ | | ----+--- Available| |
+-----|-------+ | +------+------+
|
|
|
| +------+------+
| | |
| 5 GB | Data Center |
+-------------+ (NFVI-PoP) |
Available | |
+-------------+
The network topology available for establishing connectivity for the
network service.
Figure 6
10.3. Simple Orchestration
In this example the computation of the VNF deployment sites occurs
first, followed by the connectivity solution. There is only one link
property to be met, minimum bandwidth, and for the link between VNF A
and VNF B, the minimum acceptable bandwidth of 2 GB . (Note: This is
shown in the labels of the links in Figure 6.)
Here are the actions and events involved in planning the creation of
a service instance.
o Orchestration software receives the network service request.
The request includes descriptors, including resource requirements,
for each VNF (SF) and the links between them. For the VNFs, this
includes the number of needed containers (a VNF may be composed of
multiple VNFCs), compute, memory, and storage. For the links this
includes bandwidth, latency, jitter, and loss.
o Orchestrator identifies the data centers based on proximity to the
user site, and the availability of servers with the resources to
meet the VNF (SF) resource requirements and any relevant policies.
At this point, the choice of data centers for the VNF has been
finalized.
Veitch Expires June 3, 2017 [Page 19]
Internet-Draft Integrated NFV and Network Optimization November 2016
o Orchestrator requests the network controller to establish
connectivity for each virtual link connecting VNFs in the network
service description.
A path must be found between each pair of VNF, one which can
provide the minimum bandwidth associated with the virtual link
between those VNF. NOTE: If no suitable path is found, then the
network service deployment FAILS.
+-----------------+ +-------------+ +-------------+
|Service Endpoint | | Data Center | | Data Center |
|(Cust. Premises) +--+ (NFVI-PoP) | 3 GB | (NFVI-PoP) |
+-----------------+ | +---------+ | ----- Needed | +---------+ |
| | VNF A +---( WAN )----------+ VNF B | |
| +---------+ | ----- 2GB | +---------+ |
+-------------+ Avail. +-------------+
>Inadequate bandwidth for service with VNF placement computed in
advance.
Figure 7
10.4. Connectivity Information in VNF Placement Planning
An obvious deficiency of this process is in choosing VNF sites
without considering the ability of the network to provide the
necessary connectivity and bandwidth. The solution is to update the
orchestrator to consider connectivity as a factor when computing
where the VNF should be placed. Whatever the placement algorithm, it
must ensure that VNFs, connected by a virtual link in a network
service definition, have supporting network connectivity that will
meet the service requirements, e.g. bandwidth, or latency, associated
with that virtual link.
Compared to the previous solution, here the network service
deployment algorithm will place a VNF in a datacenter only if network
connectivity is acceptable. If an orchestrator chooses to place a
VNF in a datacenter, then the next VNF in the service path must be
placed in a site where connectivity between the two meets the
requirements in the virtual link description.
A simple backtracking algorithm, such as depth first search (DFS),
will start at the front of the service and check on possible next hop
data centers (including the 'current' one). It will choose one based
on proximity, available compute, storage, etc. resources, policies,
etc. It will then request of the PCE or SDN controller to compute a
path to the candidate data center, passing the virtual link
requirements, e.g. bandwidth, latency etc. The PCE/controller should
Veitch Expires June 3, 2017 [Page 20]
Internet-Draft Integrated NFV and Network Optimization November 2016
consider the topology, currently allocated bandwidth, traffic volumes
and profile, hop count, encapsulations, etc. in computing paths.
This will continue until either all options have been searched
unsuccessfully, or until a successful set of data centers and paths
between them is found. Of course, DFS will not be used, as there are
much more efficient heuristic algorithms available. But the concept
is the same regardless of the algortihm. Connectivity information is
utilized when choosing VNF sites.
One problem with this solution is the need to have the PCE or SDN
controller compute a path each time connectivity between two data
centers is to be checked. This will take a fair amount of time and
CPU cycles. Efficiency can be improved through memoization of the
path computation and its result, the first time the PCE or SDN
controller returns a value. Before asking the PCE to compute a path,
a check of the cache, for the result of any earlier computation, can
be made. Keys would be the endpoint data centers, the network
service ID and the virtual link ID.
10.5. Pre-computation of a Connectivity Matrix
One problem identified in the previous solution is the number of
delays that accumulate due to repeated path requests to the PCE/
controller. Memoization is identified as a method for reducing the
impact. Nevertheless, the response time to instantiate a service
will be longer than it need be. For a network service with m VNF
(SF), if each is in a separate site, then, at best, m-1 requests to
the PCE/controller are required. If there are problems finding
paths, the count of path computations rises. This has the potential
to be computationally costly and require a lot of time before
providing a satisfactory solution.
A better solution would be to pre-compute connectivity among the
sites and make this available for the algorithm. This will likely
increase the overall compute load, but it would shorten the time to
respond to service requests. In this case a connection is computed
for each pair of data centers and possible endpoints (user sites).
It is maintained through continuous requests of the PCE, both
triggered by a baseline frequency as well as reported events that
could impact connection. Assuming a total of n data centers and
other sites, this results in a connectivity matrix that is n x n. It
may be sparse, as some sites may not connect to others, and of
course, a site does not connect to itself. Each entry in the matrix
represents a path between the sites, and each has relevant bandwidth,
latency, etc. attributes.
Here now the computation, whatever the algorithm, is much more
efficient. Connectivity information for any pair of data centers or
Veitch Expires June 3, 2017 [Page 21]
Internet-Draft Integrated NFV and Network Optimization November 2016
possible endpoints is locally available. There is no need to request
information from another process, the PCE or other SDN controller.
10.6. Multi-valued Connectivity Matrix
Unfortunately, as described above, this solution still has
limitations. Only one path, perhaps selected based on greatest
available bandwidth, is computed for each entry in the n x n
connectivity matrix. This path is not necessarily engineered to
maximize or weight the path metrics in any way. If a path between
two datacenters needs a latency of 40 msec and the path in the
connectivity matrix has latency of 80 msec, that option will be
rejected. This is a problem because a perfectly acceptable path with
the needed bandwidth might exist along a different route.
A possible solution here is to compute more than one path for each
pair of sites, where each is based on a different set of criteria
priorities. These could include maximum bandwidth, minimum latency,
fewest encapsulations (tunnels, tunnels in tunnels, packet layer to
optical and back), fewest hops, fewest committed tunnels, least
dynamic traffic patterns, etc.) Others might include different
balances of these values, and anti-affinity rules, e.g. do not share
the same underlying optical paths, something important for
identifying connectivity with backup paths.
The result, therefore, is an n x n x m array, or simply an n x n
array with each cell containing a list of path options, where the
list length may vary for different sets of endpoints. Each list
entry (or cell in the m dimension for a 3 dimensional array) will
represent a unique path supporting the connectivity between the two
endpoints, where each has been computed for a different set of
service priorities.
There is still the concern that a network service might be requested
for which a needed path's requirements (from a virtual link) are not
satisfied by any entry in the connectivity matrix. In other words,
no connection with the relevant service requirements has been pre-
computed. The implication is that this could result in an
unnecessary rejection of the requested network service. To address
this, it might be appropriate to include an option for the path or
the orchestrator. If the deplyment algorithm finds no satisfactory
path in the matrix, the orchestrator could request directly to the
PCE to see if it can find connectivity to meet its needs.
Note: In the matrix, there may be endpoint pairs for which no path
has been found. The value in the matrix would be FAIL, or perhaps
include something with a bit more useful information. If a request
Veitch Expires June 3, 2017 [Page 22]
Internet-Draft Integrated NFV and Network Optimization November 2016
fails because the requirements do not match any of the cached paths,
it is an option to ask the PCE to re-check, or not.
There are a number of ways the orchestrator could approach doing
this. The orchestrator might request the PCE to compute a point to
point path that meets the service requirements, while itself, IN
PARALLEL, continues to compute a different VNF placement and path
solution. If the PCE/controller finds a possible path to support the
connection, or perhaps even if not, it would also make sense to add
it to the connectivity matrix and change the path request options so
that it is continuously computed and checked.
10.7. Additional Optimization Options
To this point the connectivity matrix has been described as n x n,
with each possible endpoint, e.g. a data center or customer network,
being represented. In other words, there are n possible endpoints.
For any pair of endpoints, if there is some connectivity between
them, even if across multiple physical networks and domains, one or
more paths between them could be continuously computed and stored.
However, this is probably unnecessary. Based on data center sites,
physical and virtual connectivity, available functions, and customer
interests, the sites that might connect with other sites are likely a
significant subset of the full n x n possible connections.
Therefore, it would make sense to weight the site-2-site connections.
A very simple method is to compute paths only for site-to-site pairs
that commonly connect as part of network services. Other site-to-
site paths would remain empty. There are likely a number of ways to
improve this based upon other factors, including learned behavior.
One option is to initially compute a full n x n connectivity matrix,
with mutiple paths for different priorities. Subsequent refreshing
of any connection depends upon the frequency of the use of that
connection. The frequency of updates would depend upon the frequency
of the connection use. Lower frequency connections will be assigned
a lower weight. The weight is refreshed whenever the connection is
used, with the weight raised and the 'time last assigned' updated.
On the other hand, another agent will execute periodically, and it
will reduce each weight by a certain amount (though no lower than
zero.)
When a new service is to be placed, if the orchestrator finds a lower
weight connection to be useful, it might choose to use it. If the
age is older than some configured threshold, it might ask the PCE to
re-compute the path before it uses it.
Veitch Expires June 3, 2017 [Page 23]
Internet-Draft Integrated NFV and Network Optimization November 2016
10.8. Cached Paths Over Multiple Sites
In addition to the pre-computed site-2-site paths, another
optimization step is to cache full or partial multi-site or multi
data center paths. Is is likely there will be a fixed number of
network services and each with a fixed, or mostly fixed, order of
network functions. Even if some network services are dynamic in
order, size and content (NFs), these will likely be a small minority,
at the outset of NFV based network service delivery. Therefore, it
will be possible to identify a finite number of network services,
including their VNF-FGs, i.e. the ordering of the NFs, and needed
compute and network resources. The frequencies and instantiation
patterns of these services and their locations can be used to drive
how often and how many cached instances should be computed, in order
to ensure resources are available. Computing for multiple links in a
network service also provides the opportunity to sychronize the
computations of available resources. This should avoid the
possibility of any connections in a network service unintentionally
relying on the availability of bandwidth on the same physical link.
10.9. Summary
The purpose of this section has been to present options for computing
optimized network service deployment that might drive requirements
for a network SDN controller or PCE. The organization of this
section has been intended to build and make clear how the use of pre-
computed and cached paths between the data center sites can benefit
in the delivery of optimzed services. It has described the creation
and management of computing multiple paths for each pairwise set of
data centers, based on different weights of factors, e.g. latency,
bandwidth, etc. It also has outlined how pre-computed full service
paths might also work. The result should be a greater likelihood of
being able to deliver a service, as well as a significant reduction
in the time to respond to service requests.
11. Connectivity based on Potential vs. Actual Topologies
Something missing here is how the orchestrator and PCE should
consider 'potential connectivity' as part of the VNF placement and
connectivity process. For example, in the HPC use case, we know
there will likely be a need for a high bandwidth low latency
connection. This is probably something that is not available in the
connectivity matrix as described so far. The matrix includes only
information from the existing network topology configuration,
including the configuration of existing tunnels, and performance
metrics.
Veitch Expires June 3, 2017 [Page 24]
Internet-Draft Integrated NFV and Network Optimization November 2016
This calls out that there is a need for an agent providing
connectivity information, e.g. PCE or SDN controller, to also
present potential connectivity, should it be needed. This is not a
simply issue. For example, updating or introducing a new underlying
optical network circuit will affect the topology of the packet
network running on top of it. This means that existing paths between
sites would no longer be valid.
An orchestrator's deployment or placement algorithm should have the
option to request this sort of 'potential' connectivity for any pair
of sites, and the PCE or network controller that is computing these
paths should be able to provide this.
This functionality could be used for near term needs, perhaps
dedicating that circuit for a single network service, such as the HPC
service. In the long term, other advantages might be realized. For
example, if the system is creating a large number of multi-hop
connections between two data centers, the orchestrator or PCE may
recognize this and propose a possible direct circuit connection to
reduce costs and latency. This is an area for more analysis. It is
something that will drive new information exchange between functional
blocks, e.g. the PCE or SDN controller, via its NBI and perhaps SBI,
if hierarchical architectures are used.
12. Virtual Networks Equivalency
To this point the description has been for a relatively simple
network, with paths computed for different data center connections.
Another perspective on this is to consider each data center and user
endpoint as a node in the network, where these nodes are connected by
one or more virtual links. This is a virtual network then, where
there is an abstraction layer between the physical and virtual
networks. Each entry in the connectivity matrix described above
represents a link between any two data centers. There may be more
than one link, as these multiple links represent the multiple entries
per matrix cell, with each created based on different priorities e.g.
bandwidth, latency, etc. In fact, the connectivity matrix described
effectively describes this sort of virtual network. As noted
earlier, the connectivity matrix may be sparse. Equivalently, the
virtual network may not be show a link between every two nodes (data
centers, etc.)
One thing to consider is the possibility of multiple virtual
networks, where each supports a different customer or some type of
independent domain. Multiple virtual networks (connectivity
matrices) created, for each per customer or domain. There might even
be layering. For example, a customer may use virtual networks (plus
compute resources) from multiple providers, weave these together, and
Veitch Expires June 3, 2017 [Page 25]
Internet-Draft Integrated NFV and Network Optimization November 2016
then offer services to its own customers. This is equivalent to
having specific connectivity matrices from each separate network and
then combining these to create per customer connectivity matrices
that combine information from the separate network matrices.
13. Implications for the Network Controllers (SDN / PCE) and
Orchestrators
First, the network connectivity matrix (virtual network) must be
continuously computed and the connection (link between sites)
information passed to the NFV network service orchestrator as updates
are generated. The computations must consider topology, current
resource allocations, measured utilization, analytics, e.g. trending,
baselines, patterns, and policies. Path computation must compute
multiple paths concurrenty, for the same endpoints, but different
priorities, policies, etc. The computations are not for one set of
service parameters, but to identify for a link the available
resources (typically bandwidth) for meeting certain service
requirements.
A PCE path request should configure path re-computation and updates
at a constant rate, e.g. every 5 minutes, as well as compute updates
whenever an event triggers it. Such an event might be a change in
bandwidth, or perhaps latency, due to allocation over a path
currently in the connectivity matrix, or congestion due to traffic
volumes. Each time the PCE computes a path it must do so with the
latest information regarding connectivity, link state, reservations,
and utilization and performance metrics.
Note, unfortunatelypre-computation can create other problems. For
example, when computing a path, it is possible that the same physical
(or virtual) link could be used in more than one NF-NF connection for
a single network service. A conflict might exist resulting in
inadequate resources, and this would not be recognized. For smaller
services this might not be significant; however, for services that
might require a lot of bandwidth, e.g. a large vCDN deployment, this
might make a difference.
Finally, for the cached multi-site and connection solution, this will
require support for requesting and then computing the hop by hop
paths, synchronized, and caching them such that a full soluton can be
delivered quickly.
14. Relationship to TEAS TE Topology and ACTN
In the IETF there is a lot of relevant ongoing related to enhancing
the PCE capabilities, the PCE Protocol, and the introduction of the
Abstraction and Control of Transport Networks (ACTN) framework. A
Veitch Expires June 3, 2017 [Page 26]
Internet-Draft Integrated NFV and Network Optimization November 2016
number of concepts described here are really relevant to transport
network services (including MPLS, segment routing, and technologies
other than optical), whther supporting NFV network services or not.
Some of these are described in the TEAS WG TE Topology, currently in
development. These include the description and definitions of
network nodes, TE virtual links, and multiple layers of abstraction.
PCE enhancements are defined to support this, and the ACTN framework
address management and control from the service layer.
15. Areas of Future Evaluation
This is an early draft of this Internet draft. Its intention has
been to highlight, through use cases, the need for co-ordinated
optimization of network and compute resources for NFV based network
services delivery and to expand upon some solution options that lead
to possible new requirements of the network controller (SDN / PCE.)
There are a number of topics raised here, or which are related, that
require further study. These include:
o Testbed to evaluate effectiveness of pre-computation of
connectivity and full network service paths.
o Evaluate requirements to support forecast or pre-sechedule network
services.
o Determine any controller / PCE (PCEP) NBI / SBI messaging updates
that might be needed
o Evaluate implications of non-ACID computations vs. deployments and
methods to reduce risk.
o Centralized vs. Distributed computation of optimization
o Review work in other IETF and standardswork - ONF Transport, MEF,
FORCES, ALTO, NFVRG, SDNRG, SFC WG, etc.
16. IANA Considerations
This memo includes no request to IANA.
All drafts are required to have an IANA considerations section (see
Guidelines for Writing an IANA Considerations Section in RFCs
[RFC5226] for a guide). If the draft does not require IANA to do
anything, the section contains an explicit statement that this is the
case (as above). If there are no requirements for IANA, the section
will be removed during conversion into an RFC by the RFC Editor.
Veitch Expires June 3, 2017 [Page 27]
Internet-Draft Integrated NFV and Network Optimization November 2016
17. Security Considerations
All drafts are required to have a security considerations section.
See RFC 3552 [RFC3552] for a guide.
18. References
18.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
18.2. Informative References
[I-D.ietf-pce-hierarchy-extensions]
Zhang, F., Zhao, Q., Dios, O., Casellas, R., and D. King,
"Extensions to Path Computation Element Communication
Protocol (PCEP) for Hierarchical Path Computation Elements
(PCE)", draft-ietf-pce-hierarchy-extensions-03 (work in
progress), July 2016.
[I-D.ietf-pce-pcep-yang]
Dhody, D., Hardwick, J., Beeram, V., and j.
jefftant@gmail.com, "A YANG Data Model for Path
Computation Element Communications Protocol (PCEP)",
draft-ietf-pce-pcep-yang-01 (work in progress), October
2016.
[I-D.ietf-pce-stateful-pce]
Crabbe, E., Minei, I., Medved, J., and R. Varga, "PCEP
Extensions for Stateful PCE", draft-ietf-pce-stateful-
pce-17 (work in progress), November 2016.
[I-D.ietf-teas-actn-framework]
Ceccarelli, D. and Y. Lee, "Framework for Abstraction and
Control of Traffic Engineered Networks", draft-ietf-teas-
actn-framework-01 (work in progress), October 2016.
[I-D.ietf-teas-pce-central-control]
Farrel, A., Zhao, Q., Li, Z., and C. Zhou, "An
Architecture for Use of PCE and PCEP in a Network with
Central Control", draft-ietf-teas-pce-central-control-00
(work in progress), September 2016.
Veitch Expires June 3, 2017 [Page 28]
Internet-Draft Integrated NFV and Network Optimization November 2016
[I-D.ietf-teas-yang-te]
Saad, T., Gandhi, R., Liu, X., Beeram, V., Shah, H.,
Bryskin, I., Chen, X., Jones, R., and B. Wen, "A YANG Data
Model for Traffic Engineering Tunnels and Interfaces",
draft-ietf-teas-yang-te-05 (work in progress), October
2016.
[I-D.ietf-teas-yang-te-topo]
Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and
O. Dios, "YANG Data Model for TE Topologies", draft-ietf-
teas-yang-te-topo-06 (work in progress), October 2016.
[RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC
Text on Security Considerations", BCP 72, RFC 3552,
DOI 10.17487/RFC3552, July 2003,
<http://www.rfc-editor.org/info/rfc3552>.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", BCP 26, RFC 5226,
DOI 10.17487/RFC5226, May 2008,
<http://www.rfc-editor.org/info/rfc5226>.
[RFC5441] Vasseur, JP., Ed., Zhang, R., Bitar, N., and JL. Le Roux,
"A Backward-Recursive PCE-Based Computation (BRPC)
Procedure to Compute Shortest Constrained Inter-Domain
Traffic Engineering Label Switched Paths", RFC 5441,
DOI 10.17487/RFC5441, April 2009,
<http://www.rfc-editor.org/info/rfc5441>.
Author's Address
Andrew Veitch (editor)
Netcracker, Inc.
95 Sawyer Road
Waltham 02453
US
Email: andrew.veitch@netcracker.com
Veitch Expires June 3, 2017 [Page 29]