Internet DRAFT - draft-unify-nfvrg-devops
draft-unify-nfvrg-devops
NFVRG C. Meirosu
Internet Draft Ericsson
Intended status: Informational A. Manzalini
Expires: September 2016 Telecom Italia
R. Steinert
SICS
G. Marchetto
Politecnico di Torino
I. Papafili
Hellenic Telecommunications Organization
K. Pentikousis
EICT
S. Wright
AT&T
March 20, 2016March 18, 2016
DevOps for Software-Defined Telecom Infrastructures
draft-unify-nfvrg-devops-04.txt
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on September 20, 2016.
Meirosu, et al. Expires September 20,18, 2016 [Page 1]
Internet-Draft DevOps Challenges March 2016
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Abstract
Carrier-grade network management was optimized for environments built
with monolithic physical nodes and involves significant deployment,
integration and maintenance efforts from network service providers.
The introduction of virtualization technologies, from the physical
layer all the way up to the application layer, however, invalidates
several well-established assumptions in this domain. This draft opens
the discussion in NFVRG about challenges related to transforming the
telecom network infrastructure into an agile, model-driven production
environment for communication services. We take inspiration from data
center DevOps regarding how to simplify and automate management
processes for a telecom service provider software-defined
infrastructure (SDI). Among the identified challenges, we consider
scalability of observability processes and automated inference of
monitoring requirements from logical forwarding graphs, as well as
initial placement (and re-placement) of monitoring functionality
following changes in flow paths enforced by the controllers. In
another category of challenges, verifying correctness of behavior for
network functions where flow rules are no longer necessary and
sufficient for determining the forwarding state (for example,
stateful firewalls or load balancers) is very difficult with current
technology. Finally, we introduce challenges associated with
operationalizing DevOps principles at scale in software-defined
telecom networks in three areas related to key monitoring,
verification and troubleshooting processes.
Table of Contents
1. Introduction...................................................3
Meirosu, et al. Expires September 20,18, 2016 [Page 2]
Internet-Draft DevOps Challenges March 2016
2. Software-Defined Telecom Infrastructure: Roles and DevOps
principles........................................................5
2.1. Service Developer Role....................................5
2.2. VNF Developer role........................................6
2.3. System Integrator role....................................6
2.4. Operator role.............................................6
2.5. Customer role.............................................6
2.6. DevOps Principles.........................................7
3. Continuous Integration.........................................8
4. Continuous Delivery............................................9
5. Consistency, Availability and Partitioning Challenges..........9
6. Stability Challenges..........................................10
7. Observability Challenges......................................12
8. Verification Challenges.......................................14
9. Troubleshooting Challenges....................................16
10. Programmable network management..............................17
11. DevOps Performance Metrics...................................18
12. Security Considerations......................................19
13. IANA Considerations..........................................19
14. References...................................................19
14.1. Informative References..................................19
15. Contributors.................................................22
16. Acknowledgments..............................................22
17. Authors' Addresses...........................................23
1. Introduction
Carrier-grade network management was developed as an incremental
solution once a particular network technology matured and came to be
deployed in parallel with legacy technologies. This approach requires
significant integration efforts when new network services are
launched. Both centralized and distributed algorithms have been
developed in order to solve very specific problems related to
configuration, performance and fault management. However, such
algorithms consider a network that is by and large functionally
static. Thus, management processes related to introducing new or
maintaining functionality are complex and costly due to significant
efforts required for verification and integration.
Network virtualization, by means of Software-Defined Networking (SDN)
and Network Function Virtualization (NFV), creates an environment
where network functions are no longer static or strictly embedded in
physical boxes deployed at fixed points. The virtualized network is
dynamic and open to fast-paced innovation enabling efficient network
management and reduction of operating cost for network operators. A
significant part of network capabilities are expected to become
available through interfaces that resemble the APIs widespread within
Meirosu, et al. Expires September 20,18, 2016 [Page 3]
Internet-Draft DevOps Challenges March 2016
datacenters instead of the traditional telecom means of management
such as the Simple Network Management Protocol, Command Line
Interfaces or CORBA. Such an API-based approach, combined with the
programmability offered by SDN interfaces [RFC7426], open
opportunities for handling infrastructure, resources, and Virtual
Network Functions (VNFs) as code, employing techniques from software
engineering.
The efficiency and integration of existing management techniques in
virtualized and dynamic network environments are limited, however.
Monitoring tools, e.g. based on simple counters, physical network
taps and active probing, do not scale well and provide only a small
part of the observability features required in such a dynamic
environment. Although huge amounts of monitoring data can be
collected from the nodes, the typical granularity is rather coarse.
Debugging and troubleshooting techniques developed for software-
defined environments are a research topic that has gathered interest
in the research community in the last years. Still, it is yet to be
explored how to integrate them into an operational network management
system. Moreover, research tools developed in academia (such as
NetSight [H2014], OFRewind [W2011], FlowChecker [S2010], etc.) were
limited to solving very particular, well-defined problems, and
oftentimes are not built for automation and integration into carrier-
grade network operations workflows.
The topics at hand have already attracted several standardization
organizations to look into the issues arising in this new
environment. For example, IETF working groups have activities in the
area of OAM and Verification for Service Function Chaining
[I-D.aldrin-sfc-oam-framework] [I-D.lee-sfc-verification] for Service
Function Chaining. At IRTF, [RFC7149] asks a set of relevant
questions regarding operations of SDNs. The ETSI NFV ISG defines the
MANO interfaces [NFVMANO], and TMForum investigates gaps between
these interfaces and existing specifications in [TR228]. The need for
programmatic APIs in the orchestration of compute, network and
storage resources is discussed in [I-D.unify-nfvrg-challenges].
From a research perspective, problems related to operations of
software-defined networks are in part outlined in [SDNsurvey] and
research referring to both cloud and software-defined networks are
discussed in [D4.1].
The purpose of this first version of this document is to act as a
discussion opener in NFVRG by describing a set of principles that are
relevant for applying DevOps ideas to managing software-defined
telecom network infrastructures. We identify a set of challenges
related to developing tools, interfaces and protocols that would
Meirosu, et al. Expires September 20,18, 2016 [Page 4]
Internet-Draft DevOps Challenges March 2016
support these principles and how can we leverage standard APIs for
simplifying management tasks.
2. Software-Defined Telecom Infrastructure: Roles and DevOps principles
Agile methods used in many software focused companies are focused on
releasing small interactions of code to implement VNFs with high
velocity and high quality into a production environment. Similarly,
Service providers are interested to release incremental improvements
in the network services that they create from virtualized network
functions. The cycle time for devops as applied in many open source
projects is on the order of one quarter year or 13 weeks.
The code needs to undergo a significant amount of automated testing
and verification with pre-defined templates in a realistic setting.
From the point of view of infrastructure management, the verification
of the network configuration as result of network policy
decomposition and refinement, as well as the configuration of virtual
functions, is one of the most sensitive operations. When
troubleshooting the cause of unexpected behavior, fine-grained
visibility onto all resources supporting the virtual functions
(either compute, or network-related) is paramount to facilitating
fast resolution times. While compute resources are typically very
well covered by debugging and profiling toolsets based on many years
of advances in software engineering, programmable network resources
are a still a novelty and tools exploiting their potential are
scarce.
2.1. Service Developer Role
We identify two dimensions of the "developer" role in software-
defined infrastructure (SDI). One dimension relates to determining
which high-level functions should be part of a particular service,
deciding what logical interconnections are needed between these
blocks and defining a set of high-level constraints or goals related
to parameters that define, for instance, a Service Function Chain.
This could be determined by the product owner for a particular family
of services offered by a telecom provider. Or, it might be a key
account representative that adapts an existing service template to
the requirements of a particular customer by adding or removing a
small number of functional entities. We refer to this person as the
Service Developer and for simplicity (access control, training on
technical background, etc.) we consider the role to be internal to
the telecom provider.
Meirosu, et al. Expires September 20,18, 2016 [Page 5]
Internet-Draft DevOps Challenges March 2016
2.2. VNF Developer role
Another dimension of the "developer" role is a person that writes the
software code for a new virtual network function (VNF). Depending on
the actual VNF being developed, this person might be internal or
external (e.g. a traditional equipment vendor) to the telecom
provider. We refer to them as VNF Developers.
2.3. System Integrator role
The System Integrator role is to some extent similar to the Service
Developer: people in this role need to identify the components of the
system to be delivered. However, for the Service Developer, the
service components are pre-integrated meaning that they have the
right interfaces to interact with each other. In contrast, the
Systems Integrator needs to develop the software that makes the
system components interact with each other. As such, the Systems
Integrator role combines aspects of the Developer roles and adds yet
another dimension to it. Compared to the other Developer roles, the
System Integrator might face additional challenges due to the fact
that they might not have access to the source code of some of the
components. This limits for example how fast they could address
issues with components to be integrated, as well as uneven workload
depending on the release granularity of the different components that
need to be integrated.
2.4. Operator role
The role of an Operator in SDI is to ensure that the deployment
processes were successful and a set of performance indicators
associated to a service are met while the service is supported on
virtual infrastructure within the domain of a telecom provider.
2.5. Customer role
A Customer contracts a telecom operator to provide one or more
services. In SDI, the Customer may communicate with the provider
through an online portal. Compared to the Service Developer, the
Customer is external to the operator and may define changes to their
own service instance only in accordance to policies defined by the
Service Developer. In addition to the usual per-service utilization
statistics, in SDI the portal may enable the customer to trigger
certain performance management or troubleshooting tools for the
service. This, for example, enables the Customer to determine whether
the root cause of certain error or degradation condition that they
observe is located in the telecom operator domain or not and may
facilitate the interaction with the customer support teams.
Meirosu, et al. Expires September 20,18, 2016 [Page 6]
Internet-Draft DevOps Challenges March 2016
2.6. DevOps Principles
In line with the generic DevOps concept outlined in [DevOpsP], we
consider that these four principles as important for adapting DevOps
ideas to SDI:
* Deploy with repeatable, reliable processes: Service and VNF
Developers should be supported by automated build, orchestrate and
deploy processes that are identical in the development, test and
production environments. Such processes need to be made reliable and
trusted in the sense that they should reduce the chance of human
error and provide visibility at each stage of the process, as well as
have the possibility to enable manual interactions in certain key
stages.
* Develop and test against production-like systems: both Service
Developers and VNF Developers need to have the opportunity to verify
and debug their respective SDI code in systems that have
characteristics which are very close to the production environment
where the code is expected to be ultimately deployed. Customizations
of Service Function Chains or VNFs could thus be released frequently
to a production environment in compliance with policies set by the
Operators. Adequate isolation and protection of the services active
in the infrastructure from services being tested or debugged should
be provided by the production environment.
* Monitor and validate operational quality: Service Developers, VNF
Developers and Operators must be equipped with tools, automated as
much as possible, that enable to continuously monitor the operational
quality of the services deployed on SDI. Monitoring tools should be
complemented by tools that allow verifying and validating the
operational quality of the service in line with established
procedures which might be standardized (for example, Y.1564 Ethernet
Activation [Y1564]) or defined through best practices specific to a
particular telecom operator.
* Amplify development cycle feedback loops: An integral part of the
DevOps ethos is building a cross-cultural environment that bridges
the cultural gap between the desire for continuous change by the
Developers and the demand by the Operators for stability and
reliability of the infrastructure. Feedback from customers is
collected and transmitted throughout the organization. From a
technical perspective, such cultural aspects could be addressed
through common sets of tools and APIs that are aimed at providing a
shared vocabulary for both Developers and Operators, as well as
simplifying the reproduction of problematic situations in the
development, test and operations environments.
Meirosu, et al. Expires September 20,18, 2016 [Page 7]
Internet-Draft DevOps Challenges March 2016
Network operators that would like to move to agile methods to deploy
and manage their networks and services face a different environment
compared to typical software companies where simplified trust
relationships between personnel are the norm. In software companies,
it is not uncommon that the same person may be rotating between
different roles. In contrast, in a telecom service provider, there
are strong organizational boundaries between suppliers (whether in
Developer roles for network functions, or in Operator roles for
outsourced services) and the carrier's own personnel that might also
take both Developer and Operator roles. How DevOps principles reflect
on these trust relationships and to what extent initiatives such as
co-creation could transform the environment to facilitate closer Dev
and Ops integration across business boundaries is an interesting area
for business studies, but we could not for now identify a specific
technological challenge.
3. Continuous Integration
Software integration is the process of bringing together the software
component subsystems into one software system, and ensuring that the
subsystems function together as a system. Software integration can
apply regardless of the size of the software components. The
objective of Continuous Integration is to prevent integration
problems close to the expected release of a software development
project into a production (operations) environment. Continuous
Integration is therefore closely coupled with the notion of DevOps as
a mechanism to ease the transition from development to operations.
Continuous integration may result in multiple builds per day. It is
also typically used in conjunction with test driven development
approaches that integrate unit testing into the build process. The
unit testing is typically automated through build servers. Such
servers may implement a variety of additional static and dynamic
tests as well as other quality control and documentation extraction
functions. The reduced cycle times of continuous enable improved
software quality by applying small efforts frequently.
Continuous Integration applies to developers of VNF as they integrate
the components that they need to deliver their VNF. The VNFs may
contain components developed by different teams within the VNF
Provider, or may integrate code developed externally - e.g. in
commercial code libraries or in open source communities.
Service providers also apply continuous integration in the
development of network services. Network services are comprised of
Meirosu, et al. Expires September 20,18, 2016 [Page 8]
Internet-Draft DevOps Challenges March 2016
various aspects including VNFs and connectivity within and between
them as well as with various associated resource authorizations. The
components of the networks service are all dynamic, and largely
represented by software that must be integrated regularly to maintain
consistency. Some of the software components that Service Providers
may be sourced from VNF Providers or from open source communities.
Service Providers are increasingly motivated to engage with open
Source communities [OSandS]. Open source interfaces supported by open
source communities may be more useful than traditional paper
interface specifications. Even where Service Providers are deeply
engaged in the open source community (e.g. OPNFV) many service
providers may prefer to obtain the code through some software
provider as a business practice. Such software providers have the
same interests in software integration as other VNF providers.
4. Continuous Delivery
The practice of Continuous Delivery extends Continuous Integration by
ensuring that the software (either a VNF code or code for SDI)
checked in on the mainline is always in a user deployable state and
enables rapid deployment by those users. For critical systems such as
telecommunications networks, Continuous Delivery has the advantage of
including a manual trigger before the actual deployment in the live
system, compared to the Continuous Deployment methodology which is
also part of DevOps processes in software companies.
5. Consistency, Availability and Partitioning Challenges
The CAP theorem [CAP] states that any networked shared-data system
can have at most two of following three properties: 1) Consistency
(C) equivalent to having a single up-to-date copy of the data; 2)
high Availability (A) of that data (for updates); and 3) tolerance to
network Partitions (P).
Looking at a telecom SDI as a distributed computational system
(routing/forwarding packets can be seen as a computational problem),
just two of the three CAP properties will be possible at the same
time. The general idea is that 2 of the 3 have to be chosen. CP favor
consistency, AP favor availability, CA there are no partition. This
has profound implications for technologies that need to be developed
in line with the "deploy with repeatable, reliable processes"
principle for configuring SDI states. Latency or delay and
partitioning properties are closely related, and such relation
becomes more important in the case of telecom service providers where
Devs and Ops interact with widely distributed infrastructure.
Meirosu, et al. Expires September 20,18, 2016 [Page 9]
Internet-Draft DevOps Challenges March 2016
Limitations of interactions between centralized management and
distributed control need to be carefully examined in such
environments. Traditionally connectivity was the main concern: C and
A was about delivering packets to destination. The features and
capabilities of SDN and NFV are changing the concerns: for example in
SDN, control plane Partitions no longer imply data plane Partitions,
so A does not imply C. In practice, CAP reflects the need for a
balance between local/distributed operations and remote/centralized
operations.
Furthermore to CAP aspects related to individual protocols,
interdependencies between CAP choices for both resources and VNFs
that are interconnected in a forwarding graph need to be considered.
This is particularly relevant for the "Monitor and Validate
Operational Quality" principle, as apart from transport protocols,
most OAM functionality is generally configured in processes that are
separated from the configuration of the monitored entities. Also,
partitioning in a monitoring plane implemented through VNFs executed
on compute resources does not necessarily mean that the dataplane of
the monitored VNF was partitioned as well.
6. Stability Challenges
The dimensions, dynamicity and heterogeneity of networks are growing
continuously. Monitoring and managing the network behavior in order
to meet technical and business objectives is becoming increasingly
complicated and challenging, especially when considering the need of
predicting and taming potential instabilities.
In general, instability in networks may have primary effects both
jeopardizing the performance and compromising an optimized use of
resources, even across multiple layers: in fact, instability of end-
to-end communication paths may depend both on the underlying
transport network, as well as the higher level components specific to
flow control and dynamic routing. For example, arguments for
introducing advanced flow admission control are essentially derived
from the observation that the network otherwise behaves in an
inefficient and potentially unstable manner. Even with resources over
provisioning, a network without an efficient flow admission control
has instability regions that can even lead to congestion collapse in
certain configurations. Another example is the instability which is
characteristic of any dynamically adaptive routing system. Routing
instability, which can be (informally) defined as the quick change of
network reachability and topology information, has a number of
possible origins, including problems with connections, router
Meirosu, et al. Expires September 20,18, 2016 [Page 10]
Internet-Draft DevOps Challenges March 2016
failures, high levels of congestion, software configuration errors,
transient physical and data link problems, and software bugs.
As a matter of fact, the states monitored and used to implement the
different control and management functions in network nodes are
governed by several low-level configuration commands (today still
done mostly manually). Further, there are several dependencies among
these states and the logic updating the states (most of which are not
kept aligned automatically). Normally, high-level network goals (such
as the connectivity matrix, load-balancing, traffic engineering
goals, survivability requirements, etc) are translated into low-level
configuration commands (mostly manually) individually executed on the
network elements (e.g., forwarding table, packet filters, link-
scheduling weights, and queue-management parameters, as well as
tunnels and NAT mappings). Network instabilities due to configuration
errors can spread from node to node and propagate throughout the
network.
DevOps in the data center is a source of inspiration regarding how to
simplify and automate management processes for software-defined
infrastructure. Although the low-level configuration could be
automated by DevOps tools such as CFEngine [C2015], Puppet [P2015]
and Ansible [A2015], the high-level goal translation towards tool-
specific syntax is still a manual process. In addition, while
carrier-grade configuration tools using the NETCONF protocol support
complex atomic transaction management (which reduces the potential
for instability), Ansible requires third-party components to support
rollbacks and the Puppet transactions are not atomic.
As a specific example, automated configuration functions are expected
to take the form of a "control loop" that monitors (i.e., measures)
current states of the network, performs a computation, and then
reconfigures the network. These types of functions must work
correctly even in the presence of failures, variable delays in
communicating with a distributed set of devices, and frequent changes
in network conditions. Nevertheless cascading and nesting of
automated configuration processes can lead to the emergence of non-
linear network behaviors, and as such sudden instabilities (i.e.
identical local dynamic can give rise to widely different global
dynamics).
Meirosu, et al. Expires September 20,18, 2016 [Page 11]
Internet-Draft DevOps Challenges March 2016
7. Observability Challenges
Monitoring algorithms need to operate in a scalable manner while
providing the specified level of observability in the network, either
for operation purposes (Ops part) or for debugging in a development
phase (Dev part). We consider the following challenges:
* Scalability - relates to the granularity of network observability,
computational efficiency, communication overhead, and strategic
placement of monitoring functions.
* Distributed operation and information exchange between monitoring
functions - monitoring functions supported by the nodes may perform
specific operations (such as aggregation or filtering) locally on the
collected data or within a defined data neighborhood and forward only
the result to a management system. Such operation may require
modifications of existing standards and development of protocols for
efficient information exchange and messaging between monitoring
functions. Different levels of granularity may need to be offered for
the data exchanged through the interfaces, depending on the Dev or
Ops role. Modern messaging systems, such as Apache Kafka [AK2015],
widely employed in datacenter environments, were optimized for
messages that are considerably larger than reading a single counter
value (typical SNMP GET call usage) - note the throughput vs record
size from [K2014]. It is also debatable to what extent properties
such as message persistence within the bus are needed in a carrier
environment, where MIBs practically offer already a certain level of
persistence of management data at the node level. Also, they require
the use of IP addressing which might not be needed when the monitored
data is consumed by a function within the same node.
* Common communication channel between monitoring functions and
higher layer entities (orchestration, control or management systems)
- a single communication channel for configuration and measurement
data of diverse monitoring functions running on heterogeneous hard-
and software environments. In telecommunication environments,
infrastructure assets span not only large geographical areas, but
also a wide range of technology domains, ranging from CPEs, access-,
aggregation-, and transport networks, to datacenters. This
heterogeneity of hard- and software platforms requires higher layer
entities to utilize various parallel communication channels for
either configuration or data retrieval of monitoring functions within
these technology domains. To address automation and advances in
monitoring programmability, software defined telecommunication
infrastructures would benefit from a single flexible communication
channel, thereby supporting the dynamicity of virtualized
environments. Such a channel should ideally support propagation of
Meirosu, et al. Expires September 20,18, 2016 [Page 12]
Internet-Draft DevOps Challenges March 2016
configuration, signalling, and results from monitoring functions;
carrier-grade operations in terms of availability and multi-tenant
features; support highly distributed and hierarchical architectures,
keeping messages as local as possible; be lightweight, topology
independent, network address agnostic; support flexibility in terms
of transport mechanisms and programming language support.
Existing popular state-of-the-art message queuing systems such as
RabbitMQ [R2015] fulfill many of these requirements. However, they
utilize centralized brokers, posing a single point-of-failure and
scalability concerns within vastly distributed NFV environment.
Furthermore, transport support is limited to TCP/IP. ZeroMQ [Z2015]
on the other hard lacks any advanced features for carrier-grade
operations, including high-availability, authentication, and tenant
isolation.
* Configurability and conditional observability - monitoring
functions that go beyond measuring simple metrics (such as delay, or
packet loss) require expressive monitoring annotation languages for
describing the functionality such that it can be programmed by a
controller. Monitoring algorithms implementing self-adaptive
monitoring behavior relative to local network situations may employ
such annotation languages to receive high-level objectives (KPIs
controlling tradeoffs between accuracy and measurement frequency, for
example) and conditions for varying the measurement intensity. Steps
in this direction were taken by the DevOps tools such as Splunk
[S2015], whose collecting agent has the ability to load particular
apps that in turn access specific counters or log files. However,
such apps are tool specific and may also require deploying additional
agents that are specific to the application, library or
infrastructure node being monitored. Choosing which objects to
monitor in such environment means deploying a tool-specific script
that configures the monitoring app.
* Automation - includes mapping of monitoring functionality from a
logical forwarding graph to virtual or physical instances executing
in the infrastructure, as well as placement and re-placement of
monitoring functionality for required observability coverage and
configuration consistency upon updates in a dynamic network
environment. Puppet [P2015] manifests or Ansible [A2015] playbooks
could be used for automating the deployment of monitoring agents, for
example those used by Splunk [S2015]. However, both manifests and
playbooks were designed to represent the desired system configuration
snapshot at a particular moment in time - they would now need to be
generated automatically by the orchestration tools instead of a
DevOps person.
* Actionable data
Meirosu, et al. Expires September 20,18, 2016 [Page 13]
Internet-Draft DevOps Challenges March 2016
Data produced by observability tools could be utilized in a wide
category of processes, ranging from billing and dimensioning to real-
time troubleshooting and optimization. In order to allow for data-
driven automated decisions and actuations based on these decisions,
the data needs to be actionable. We define actionable data as being
representative for a particular context or situation and an adequate
input towards a decision. Ensuring actionable data is challenging in
a number of ways, including: defining adaptive correlation and
sampling windows, filtering and aggregation methods that are adapted
or coordinated with the actual consumer of the data, and developing
analytical and predictive methods that account for the uncertainty or
incompleteness of the data.
* Data Virtualization
Data is key in helping both Developers and Operators perform their
tasks. Traditional Network Management Systems were optimized for
using one database that contains the master copy of the operational
statistics and logs of network nodes. Ensuring access to this data
from across the organization is challenging because strict privacy
and business secrets need to be protected. In DevOps-driven
environments, data needs to be made available to Developers and their
test environments. Data virtualization collectively defines a set of
technologies that ensure that restricted copies of the partial data
needed for a particular task may be made available while enforcing
strict access control. Further than simple access control, data
virtualization needs to address scalability challenges involved in
copying large amounts of operational data as well as automatically
disposing of it when the task authorized for using it has finished.
8. Verification Challenges
Enabling ongoing verification of code is an important goal of
continuous integration as part of the data center DevOps concept. In
a telecom SDI, service definitions, decompositions and configurations
need to be expressed in machine-readable encodings. For example,
configuration parameters could be expressed in terms of YANG data
models. However, the infrastructure management layers (such as
Software-Defined Network Controllers and Orchestration functions)
might not always export such machine-readable descriptions of the
runtime configuration state. In this case, the management layer
itself could be expected to include a verification process that has
the same challenges as the stand-alone verification processes we
outline later in this section. In that sense, verification can be
considered as a set of features providing gatekeeper functions to
Meirosu, et al. Expires September 20,18, 2016 [Page 14]
Internet-Draft DevOps Challenges March 2016
verify both the abstract service models and the proposed resource
configuration before or right after the actual instantiation on the
infrastructure layer takes place.
A verification process can involve different layers of the network
and service architecture. Starting from a high-level verification of
the customer input (for example, a Service Graph as defined in
[I-D.unify-nfvrg-challenges]), the verification process could go more
in depth to reflect on the Service Function Chain configuration. At
the lowest layer, the verification would handle the actual set of
forwarding rules and other configuration parameters associated to a
Service Function Chain instance. This enables the verification of
more quantitative properties (e.g. compliance with resource
availability), as well as a more detailed and precise verification of
the abovementioned topological ones. Existing SDN verification tools
could be deployed in this context, but the majority of them only
operate on flow space rules commonly expressed using OpenFlow syntax.
Moreover, such verification tools were designed for networks where
the flow rules are necessary and sufficient to determine the
forwarding state. This assumption is valid in networks composed only
by network functions that forward traffic by analyzing only the
packet headers (e.g. simple routers, stateless firewalls, etc.).
Unfortunately, most of the real networks contain active network
functions, represented by middle-boxes that dynamically change the
forwarding path of a flow according to function-local algorithms and
an internal state (that is based on the received packets), e.g. load
balancers, packet marking modules and intrusion detection systems.
The existing verification tools do not consider active network
functions because they do not account for the dynamic transformation
of an internal state into the verification process.
Defining a set of verification tools that can account for active
network functions is a significant challenge. In order to perform
verification based on formal properties of the system, the internal
states of an active (virtual or not) network function would need to
be represented. Although these states would increase the verification
process complexity (e.g., using simple model checking would not be
feasible due to state explosion), they help to better represent the
forwarding behavior in real networks. A way to address this challenge
is by attempting to summarize the internal state of an active network
function in a way that allows for the verification process to finish
within a reasonable time interval.
Meirosu, et al. Expires September 20,18, 2016 [Page 15]
Internet-Draft DevOps Challenges March 2016
9. Troubleshooting Challenges
One of the problems brought up by the complexity introduced by NFV
and SDN is pinpointing the cause of a failure in an infrastructure
that is under continuous change. Developing an agile and low-
maintenance debugging mechanism for an architecture that is comprised
of multiple layers and discrete components is a particularly
challenging task to carry out. Verification, observability, and
probe-based tools are key to troubleshooting processes, regardless
whether they are followed by Dev or Ops personnel.
* Automated troubleshooting workflows
Failure is a frequently occurring event in network operation.
Therefore, it is crucial to monitor components of the system
periodically. Moreover, the troubleshooting system should search for
the cause automatically in the case of failure. If the system follows
a multi-layered architecture, monitoring and debugging actions should
be performed on components from the topmost layer to the bottom layer
in a chain. Likewise, the result of operations should be notified in
reverse order. In this regard, one should be able to define
monitoring and debugging actions through a common interface that
employs layer hopping logic. Besides, this interface should allow
fine-grained and automatic on-demand control for the integration of
other monitoring and verification mechanisms and tools.
* Troubleshooting with active measurement methods
Besides detecting network changes based on passively collected
information, active probes to quantify delay, network utilization and
loss rate are important to debug errors and to evaluate the
performance of network elements. While tools that are effective in
determining such conditions for particular technologies were
specified by IETF and other standardization organization, their use
requires a significant amount of manual labor in terms of both
configuration and interpretation of the results.
In contrast, methods that test and debug networks systematically
based on models generated from the router configuration, router
interface tables or forwarding tables, would significantly simplify
management. They could be made usable by Dev personnel that have
little expertise on diagnosing network defects. Such tools naturally
lend themselves to integration into complex troubleshooting workflows
that could be generated automatically based on the description of a
particular service chain. However, there are scalability challenges
associated with deploying such tools in a network. Some tools may
poll each networking device for the forwarding table information to
Meirosu, et al. Expires September 20,18, 2016 [Page 16]
Internet-Draft DevOps Challenges March 2016
calculate the minimum number of test packets to be transmitted in the
network. Therefore, as the network size and the forwarding table size
increase, forwarding table updates for the tools may put a non-
negligible load in the network.
10. Programmable network management
The ability to automate a set of actions to be performed on the
infrastructure, be it virtual or physical, is key to productivity
increases following the application of DevOps principles. Previous
sections in this document touched on different dimensions of
programmability:
- Section 5 approached programmability in the context of developing
new capabilities for monitoring and for dynamically setting
configuration parameters of deployed monitoring functions
- Section 7 reflected on the need to determine the correctness of
actions that are to be inflicted on the infrastructure as result
of executing a set of high-level instructions
- Section 8 considered programmability in the perspective of an
interface to facilitate dynamic orchestration of troubleshooting
steps towards building workflows and for reducing the manual steps
required in troubleshooting processes
We expect that programmable network management - along the lines of
[RFC7426] - will draw more interest as we move forward. For example,
in [I-D.unify-nfvrg-challenges], the authors identify the need for
presenting programmable interfaces that accept instructions in a
standards-supported manner for the Two-way Active Measurement
Protocol (TWAMP)TWAMP protocol. More specifically, an excellent
example in this case is traffic measurements, which are extensively
used today to determine SLA adherence as well as debug and
troubleshoot pain points in service delivery. TWAMP is both widely
implemented by all established vendors and deployed by most global
operators. However, TWAMP management and control today relies solely
on diverse and proprietary tools provided by the respective vendors
of the equipment. For large, virtualized, and dynamically
instantiated infrastructures where network functions are placed
according to orchestration algorithms proprietary mechanisms for
managing TWAMP measurements have severe limitations. For example,
today's TWAMP implementations are managed by vendor-specific,
typically command-line interfaces (CLI), which can be scripted on a
platform-by-platform basis. As a result, although the control and
Meirosu, et al. Expires September 20,18, 2016 [Page 17]
Internet-Draft DevOps Challenges March 2016
test measurement protocols are standardized, their respective
management is not. This hinders dramatically the possibility to
integrate such deployed functionality in the SP-DevOps concept. In
this particular case, recent efforts in the IPPM WG
[I-D.cmzrjp-ippm-twamp-yang] aim to define a standard TWAMP data
model and effectively increase the programmability of TWAMP
deployments in the future.
Data center DevOps tools, such as those surveyed in [D4.1], developed
proprietary methods for describing and interacting through interfaces
with the managed infrastructure. Within certain communities, they
became de-facto standards in the same way particular CLIs became de-
facto standards for Internet professionals. Although open-source
components and a strong community involvement exists, the diversity
of the new languages and interfaces creates a burden for both vendors
in terms of choosing which ones to prioritize for support, and then
developing the functionality and operators that determine what fits
best for the requirements of their systems.
11. DevOps Performance Metrics
Defining a set of metrics that are used as performance indicators is
important for service providers to ensure the successful deployment
and operation of a service in the software-defined telecom
infrastructure.
We identify three types of considerations that are particularly
relevant for these metrics: 1) technical considerations directly
related to the service provided, 2) process-related considerations
regarding the deployment, maintenance and troubleshooting of the
service, i.e. concerning the operation of VNFs, and 3) cost-related
considerations associated to the benefits from using a Software-
Defined Telecom Infrastructure.
First, technical performance metrics shall be service-dependent/-
oriented and may address inter-alia service performance in terms of
delay, throughput, congestion, energy consumption, availability, etc.
Acceptable performance levels should be mapped to SLAs and the
requirements of the service users. Metrics in this category were
defined in IETF working groups and other standardization
organizations with responsibility over particular service or
infrastructure descriptions.
Second, process-related metrics shall serve a wider perspective in
the sense that they shall be applicable for multiple types of
Meirosu, et al. Expires September 20,18, 2016 [Page 18]
Internet-Draft DevOps Challenges March 2016
services. For instance, process-related metrics may include: number
of probes for end-to-end QoS monitoring, number of on-site
interventions, number of unused alarms, number of configuration
mistakes, incident/trouble delay resolution, delay between service
order and deliver, or number of self-care operations.
Third, cost-related metrics shall be used to monitor and assess the
benefit of employing SDI compared to the usage of legacy hardware
infrastructure with respect to operational costs, e.g. possible man-
hours reductions, elimination of deployment and configuration
mistakes, etc.
Finally, identifying a number of highly relevant metrics for DevOps
and especially monitoring and measuring them is highly challenging
because of the amount and availability of data sources that could be
aggregated within one such metric, e.g. calculation of human
intervention, or secret aspects of costs.
12. Security Considerations
TBD
13. IANA Considerations
This memo includes no request to IANA.
14. References
14.1. Informative References
[NFVMANO] ETSI, "Network Function Virtualization (NFV) Management
and Orchestration V0.6.1 (draft)", Jul. 2014
[I-D.aldrin-sfc-oam-framework] S. Aldrin, R. Pignataro, N. Akiya.
"Service Function Chaining Operations, Administration and
Maintenance Framework", draft-aldrin-sfc-oam-framework-02,
(work in progress), July 2015.
Meirosu, et al. Expires September 20,18, 2016 [Page 19]
Internet-Draft DevOps Challenges March 2016
[I-D.lee-sfc-verification] S. Lee and M. Shin. "Service Function
Chaining Verification", draft-lee-sfc-verification-00,
(work in progress), February 2014.
[RFC7426] E. Haleplidis (Ed.), K. Pentikousis (Ed.), S. Denazis, J.
Hadi Salim, D. Meyer, and O. Koufopavlou, "Software Defined
Networking (SDN): Layers and Architecture Terminology",
RFC 7426, January 2015
[RFC7149] M. Boucadair and C Jaquenet. "Software-Defined Networking:
A Perspective from within a Service Provider Environment",
RFC 7149, March 2014.
[TR228] TMForum Gap Analysis Related to MANO Work. TR228, May 2014
[I-D.unify-nfvrg-challenges] R. Szabo et al. "Unifying Carrier and
Cloud Networks: Problem Statement and Challenges", draft-
unify-nfvrg-challenges-03 (work in progress), October 2016
[I-D.cmzrjp-ippm-twamp-yang] Civil, R., Morton, A., Zheng, L.,
Rahman, R., Jethanandani, M., and K. Pentikousis, "Two-Way
Active Measurement Protocol (TWAMP) Data Model", draft-
cmzrjp-ippm-twamp-yang-02 (work in progress), October 2015.
[D4.1] W. John et al. D4.1 Initial requirements for the SP-DevOps
concept, universal node capabilities and proposed tools,
August 2014.
[SDNsurvey] D. Kreutz, F. M. V. Ramos, P. Verissimo, C. Esteve
Rothenberg, S. Azodolmolky, S. Uhlig. "Software-Defined
Networking: A Comprehensive Survey." To appear in
proceedings of the IEEE, 2015.
[DevOpsP] "DevOps, the IBM Approach" 2013. [Online].
[Y1564] ITU-R Recommendation Y.1564: Ethernet service activation
test methodology, March 2011
[CAP] E. Brewer, "CAP twelve years later: How the "rules" have
changed", IEEE Computer, vol.45, no.2, pp.23,29, Feb. 2012.
[H2014] N. Handigol, B. Heller, V. Jeyakumar, D. Mazieres, N.
McKeown; "I Know What Your Packet Did Last Hop: Using
Packet Histories to Troubleshoot Networks", In Proceedings
of the 11th USENIX Symposium on Networked Systems Design
and Implementation (NSDI 14), pp.71-95
Meirosu, et al. Expires September 20,18, 2016 [Page 20]
Internet-Draft DevOps Challenges March 2016
[W2011] A. Wundsam, D. Levin, S. Seetharaman, A. Feldmann;
"OFRewind: Enabling Record and Replay Troubleshooting for
Networks". In Proceedings of the Usenix Anual Technical
Conference (Usenix ATC '11), pp 327-340
[S2010] E. Al-Shaer and S. Al-Haj. "FlowChecker: configuration
analysis and verification of federated Openflow
infrastructures" In Proceedings of the 3rd ACM workshop on
Assurable and usable security configuration (SafeConfig
'10). Pp. 37-44
[OSandS] S. Wright, D. Druta, "Open Source and Standards: The Role
of Open Source in the Dialogue between Research and
Standardization" Globecom Workshops (GC Wkshps), 2014 ,
pp.650,655, 8-12 Dec. 2014
[C2015] CFEngine. Online: http://cfengine.com/product/what-is-
cfengine/, retrieved Sep 23, 2015.
[P2015] Puppet. Online: http://puppetlabs.com/puppet/what-is-puppet,
retrieved Sep 23, 2015.
[A2015] Ansible. Online: http://docs.ansible.com/ , retrieved Sep
23, 2015.
[AK2015] Apache Kafka. Online:
http://kafka.apache.org/documentation.html, retrieved Sep
23, 2015.
[S2015] Splunk. Online: http://www.splunk.com/en_us/products/splunk-
light.html , retrieved Sep 23, 2015.
[K2014] J. Kreps. Benchmarking Apache Kafka: 2 Million Writes Per
Second (On Three Cheap Machines). Online:
https://engineering.linkedin.com/kafka/benchmarking-apache-
kafka-2-million-writes-second-three-cheap-machines,
retrieved Sep 23, 2015.
[R2015] RabbitMQ. Online: https://www.rabbitmq.com/ , retrieved Oct
13, 2015
[Z2015] ZeroMQ. Online: http://zeromq.org/ , retrieved Oct 13, 2015
Meirosu, et al. Expires September 20,18, 2016 [Page 21]
Internet-Draft DevOps Challenges March 2016
15. Contributors
W. John (Ericsson), J. Kim (Deutsche Telekom), S. Sharma (iMinds)
16. Acknowledgments
The research leading to these results has received funding from the
European Union Seventh Framework Programme FP7/2007-2013 under grant
agreement no. 619609 - the UNIFY project. The views expressed here
are those of the authors only. The European Commission is not liable
for any use that may be made of the information in this document.
We would like to thank in particular the UNIFY WP4 contributors, the
internal reviewers of the UNIFY WP4 deliverables and Russ White and
Ramki Krishnan for their suggestions.
This document was prepared using 2-Word-v2.0.template.dot.
Meirosu, et al. Expires September 20,18, 2016 [Page 22]
Internet-Draft DevOps Challenges March 2016
17. Authors' Addresses
Catalin Meirosu
Ericsson Research
S-16480 Stockholm, Sweden
Email: catalin.meirosu@ericsson.com
Antonio Manzalini
Telecom Italia
Via Reiss Romoli, 274
10148 - Torino, Italy
Email: antonio.manzalini@telecomitalia.it
Juhoon Kim
Deutsche Telekom AG
Winterfeldtstr. 21
10781 Berlin, Germany
Email: J.Kim@telekom.de
Rebecca Steinert
SICS Swedish ICT AB
Box 1263, SE-16429 Kista, Sweden
Email: rebste@sics.se
Sachin Sharma
Ghent University-iMinds
Research group IBCN - Department of Information Technology
Zuiderpoort Office Park, Blok C0
Gaston Crommenlaan 8 bus 201
B-9050 Gent, Belgium
Email: sachin.sharma@intec.ugent.be
Guido Marchetto
Politecnico di Torino
Corso Duca degli Abruzzi 24
10129 - Torino, Italy
Email: guido.marchetto@polito.it
Ioanna Papafili
Hellenic Telecommunications Organization
Measurements and Wireless Technologies Section
Laboratories and New Technologies Division
2, Spartis & Pelika str., Maroussi,
GR-15122, Attica, Greece
Buidling E, Office 102
Meirosu, et al. Expires September 20,18, 2016 [Page 23]
Internet-Draft DevOps Challenges March 2016
Email: iopapafi@oteresearch.gr
Kostas Pentikousis
EICT GmbH
Torgauer Strasse 12-15
Berlin 10829
Germany
Email: k.pentikousis@eict.de
Steven Wright
AT&T Services Inc.
1057 Lenox Park Blvd NE, STE 4D28
Atlanta, GA 30319
USA
Email: sw3588@att.com
Wolfgang John
Ericsson Research
S-16480 Stockholm, Sweden
Email: wolfgang.john@ericsson.com
Meirosu, et al. Expires September 20,18, 2016 [Page 24]
