o--------------------------------o
              |                                |
              | +-------------+   +----------+ |
              | | Application |   |  Service | |
              | +-------------+   +----------+ |
              |       Application Plane        |
              o---------------Y----------------o
                              |              
*-----------------------------Y---------------------------------*
|           Network Services Abstraction Layer (NSAL)           |
*------Y------------------------------------------------Y-------*
       |                                                |
       |               Service Interface                |
       |                                                |
o------Y------------------o       o---------------------Y------o
|      |    Control Plane |       | Management Plane    |      |
| +----Y----+   +-----+   |       |  +-----+       +----Y----+ |
| | Service |   | App |   |       |  | App |       | Service | |
| +----Y----+   +--Y--+   |       |  +--Y--+       +----Y----+ |
|      |           |      |       |     |               |      |
| *----Y-----------Y----* |       | *---Y---------------Y----* |
| | Control Abstraction | |       | | Management Abstraction | |
| |     Layer (CAL)     | |       | |      Layer (MAL)       | |
| *----------Y----------* |       | *----------Y-------------* |
|            |            |       |            |               |
o------------|------------o       o------------|---------------o
             |                                 |
             | CP                              | MP
             | Southbound                      | Southbound 
             | Interface                       | Interface
             |                                 |
*------------Y---------------------------------Y----------------*
|         Device and resource Abstraction Layer (DAL)           |
*------------Y---------------------------------Y----------------*
|            |                                 |                |
|    o-------Y----------o   +-----+   o--------Y----------o     |
|    | Forwarding Plane |   | App |   | Operational Plane |     |
|    o------------------o   +-----+   o-------------------o     |
|                       Network Device                          |
+---------------------------------------------------------------+

Figure 1: SDN Layer Architecture

3.1. Overview

This document follows a network device centric approach: Control refers to the device packet handling capability, while Management refers to the overall device operation aspects. We view a network device as a complex resource which contains and is part of multiple resources similar to [DIOPR]. Resources can be simple, single components of a network device, for example a port or a queue of the device, and can also be aggregated into complex resources, for example a network device.

The reader should keep in mind throughout this document that we make no distinction between "physical" and "virtual" resources, as we do not delve into implementation or performance aspects. In other words, a resource can be implemented fully in hardware, fully in software, or any hybrid combination in between. Further, we do not distinguish on whether a resource is implemented as an overlay or as a part/component of some other device. Finally, network device software can run on so-called "bare metal" or on a virtualized substrate.

SDN spans multiple planes as illustrated in Figure 1. Starting from the bottom part of the figure and moving towards the upper part, we identify the following planes:

• Forwarding Plane - Responsible for handling packets in the datapath. Actions of the forwarding plane include, but are not limited to, forwarding, dropping and changing packets. The forwarding plane is usually the termination point for control plane services and applications. The forwarding plane can contain forwarding resources such as classifiers.
• Operational Plane - Responsible for managing the operational state of the Network Device, e.g. active/inactive, number of ports, port status, etc. The Operational Plane is usually the termination point for management plane services and applications. The operational plane relates to (operational aspects of) Network Device resources such as ports, memory, and so on. Members of the SDNRG have different opinions in regards to the operational plane, with the argument being that it does not constitute a plane by itself but an amalgamation of functions in the data plane. For others it is considered a plane similar to the rest of the planes as a plane may be considered as a distinction between different areas of operations.
• Control Plane - Responsible for taking decisions on how packets should be forwarded by one or more Network Devices and pushing such decisions down to the Network Devices to be executed. The control plane usually focuses mostly on the forwarding plane and less on the operational plane of the device. The control plane may be interested in operational plane information which could include, for example, the current state of a particular port or its capabilities. The control plane's main job is to fine-tune the forwarding tables that reside in the forwarding plane, based on the network topology or external service requests.
• Management Plane - Responsible for monitoring, configuring and maintaining network devices, e.g. taking decisions regarding the state of a Network Device. The management plane usually focuses mostly on the operational plane of the device and less on the forwarding plane. The management plane may be used to configure the forwarding plane, but it does so infrequently and through a more wholesale approach than the control plane. For instance, the management plane may set up all or part of the forwarding rules at once, although such action would be expected to be taken sparingly.
• Application Plane - The plane where applications that rely on the network to provide services for end users and processes reside. Applications that directly (or primarily) support the operation of the forwarding plane (such as routing processes within the control plane) are not considered part of the application plane. Note that applications may be implemented in a modular and distributed fashion and, therefore, can often span multiple planes in Figure 1.

All planes mentioned above are connected via Interfaces (as indicated with "Y" in Figure 1. An Interface may take multiple roles depending on whether the connected planes reside on the same (physical or virtual) device. If the respective planes are designed so that they do not have to reside in the same device, then the Interface can only take the form of a protocol. If the planes are co-located on the same device, then the Interface could be implemented via an open/proprietary protocol, an open/proprietary software inter-process communication API, or operating system kernel system calls.

Applications, i.e. software programs that perform specific computations that consume services without providing access to other applications, can be implemented natively inside a plane or can span multiple planes. For instance, applications or services can span both the control and management plane and, thus, be able to use both the CPSI and MPSI. An example of such a case would be an application that uses both [OpenFlow] and [OF-CONFIG].

Services, i.e. software programs that provide APIs to other applications or services, can also be natively implemented in specific planes. Services that span multiple planes belong to the application plane as well.

While not shown in Figure 1, services, applications and entire planes, can be placed in a recursive manner thus providing overlay semantics to the model. For example, application plane services can provide through NSAL services to other applications or services. Additional examples include virtual resources that are realized on top of a physical resources and hierarchical control plane controllers [KANDOO].

It must be noted, however, that in Figure 1 we present an abstract view of the various planes, which is devoid of implementation details. Many implementations tend to place the management plane on top of the control plane, which may be interpreted as having the control plane acting as a service to the management plane. Traditionally, the control plane was tightly coupled with the device. When taken as whole, the control plane was distributed network-wide. On the other hand, the management plane has been traditionally centralized and was responsible for managing the control plane and the devices. However, with the adoption of SDN principles, this distinction is no longer so clear-cut.

Additionally, this document considers four abstraction layers:

The Device and resource Abstraction Layer (DAL) abstracts the device's forwarding and operational plane resources to the control and management plane. Variations of DAL may abstract both planes or either of the two and may abstract any plane of the device to either the control or management plane.

The Control Abstraction Layer (CAL) abstracts the CP southbound interface and the DAL from the applications and services of the Control Plane.

The Management Abstraction Layer (MAL) abstracts the MP southbound interface and the DAL from the applications and services of the Management Plane.

The Network Services Abstraction Layer (NSAL) provides service abstractions for use by applications and other services.

We observe that the view presented in this document is quite well-aligned with recently published work by the ONF; see [ONFArch]. A key difference, however, is that the ONF architecture does not include the management plane in its scope.

SDN related activities have begun in many other SDO's, such as:

• ITU: Architectural work [ITUSG13] but have not been published at the time this document was written.

3.2. Network Devices

A Network Device is an entity that receives packets on its ports and performs one or more network functions on them. For example, the network device could forward a received packet, drop it, alter the packet header (or payload) and forward the packet, and so on. A Network Device is an aggregation of multiple resources such as ports, cpu, memory and queues. Resources are either simple or can be aggregated to form complex resources that can be viewed as one resource. The Network Device is in itself a complex resource.

Network devices can be implemented in hardware or software and can be either physical or virtual. As has already been mentioned before, this document makes no such distinction. Each network device has both a Forwarding Plane and an Operational Plane.

The Forwarding Plane, commonly referred to as the "data path", is responsible for handling and forwarding packets. The Forwarding Plane provides switching, routing transformation and filtering functions. Resources of the forwarding plane include but are not limited to filters, meters, markers and classifiers.

The Operational Plane is responsible for the operational state of the network device, for instance, with respect to status of network ports and interfaces. Operational plane resources include, but are not limited to, memory, CPU, ports, interfaces and queues.

The Forwarding and the Operational Planes are exposed via the Device and resource Abstraction Layer (DAL), which may be expressed by one or more abstraction models. Examples of Forwarding Plane abstraction models are ForCES [RFC5812] and OpenFlow [OpenFlow]. Examples of the Operational Plane abstraction model include the ForCES model [RFC5812], the YANG model [RFC6020] and SNMP MIBs [RFC3418].

Examples of Network Devices include switches and routers. Additional examples include network elements that may operate at a layer above IP, such as firewalls, load balancers and video transcoders.

Note that applications can also reside in a network device. Examples of such applications include event monitoring, and handling (offloading) topology discovery or ARP [RFC0826] in the device itself instead of forwarding such traffic to the control plane.

3.3. Control Plane

The control plane is usually distributed and is responsible mainly for the configuration of the forwarding plane using a Control Plane Southbound Interface (CPSI) with DAL as a point of reference. CP is responsible for instructing FP about how to handle network packets.

Communication between control planes, colloquially referred to as the "east-west" interface, is usually implemented through gateway protocols like BGP [RFC4271]. However, the corresponding protocol messages are in fact exchanged in-band and subsequently redirected by the forwarding plane to the control plane for further processing. Examples in this category include [RCP], [SoftRouter] and [RouteFlow].

Control Plane functionalities usually include:

• Topology discovery and maintenance
• Packet route selection and instantiation
• Path failover mechanisms

The CPSI is usually defined with the following characteristics:

• time-critical interface which requires low latency and sometimes high bandwidth in order to perform many operations in short order.
• oriented towards wire efficiency and device representation instead of human readability

Examples include fast- and high-frequency of flow or table updates, high throughput and robustness for packet handling and events.

CPSI can be implemented using a protocol, an API or even interprocess communication. If the Control Plane and the Network Device are not collocated, then this interface is certainly a protocol. Examples of CPSIs are ForCES [RFC5810] and the Openflow protocol [OpenFlow].

The Control Abstraction Layer (CAL) provides access to control applications and services to various CPSIs. The Control Plane may support more than one CPSIs.

Control applications can use CAL to control a network device without providing any service to upper layers. Examples include applications that perform control functions, such as OSPF, BGP, etc.

Control Plane service examples include a virtual private LAN service, service tunnels, topology services, etc.

3.4. Management Plane

The Management Plane is usually centralized and aims to ensure that the network, which consists of network devices, is running optimally by communicating with the network devices's Operational Plane using a Management Plane Southbound Interface (MPSI) with DAL as a point of reference.

Management plane functionalities are typically initiated, based on an overall network view, and traditionally have been human-centric. However, lately algorithms are replacing most human intervention. Management plane functionalities [FCAPS] [RFC3535] usually include:

• Fault and Monitoring management
• Configuration management

Normally MSPI, in contrast to the CPSI, is not a time-critical interface and does not share the CPSI requirements.

MSPI is [RFC3535] typically closer to human interaction than the control plane and therefore the MSPI usually has the following characteristics:

• It is oriented more towards usability, with optimal wire performance being a secondary concern.
• Messages tend to be less frequent than in the CPSI

As an example of usability versus performance, we refer to the consensus of the 2002 IAB Workshop [RFC3535], as mentioned in [RFC6632], where textual configuration files should be able to contain international characters. Human-readable strings should utilize UTF-8, and protocol elements should be in case-insensitive ASCII which require more processing capabilities to parse.

The MPSI can range from a protocol, to an API or even interprocess communication. If the Management Plane is not embedded in the network device, the MSPI is certainly a protocol. Examples of MPSIs are ForCES [RFC5810], NETCONF [RFC6241], OVSDB [RFC7047] and SNMP [RFC3411].

The Management Abstraction Layer (MAL) provides access to management applications and services to various MPSIs. The Management Plane may support more than one MPSI.

Management Applications can use MAL to manage the network device without providing any service to upper layers. Examples of management applications include network monitoring and fault detection and recovery applications.

Management Plane Services provide access to other services or applications above the Management Plane.

3.5. Control vs Management plane

During the SDNRG meetings as well via the list, one of the most commonly discussed topic, in regards to this document, was the clear distinction between control and management. We have identified the following characteristics that together or each one may provide the necessary differentiator between the planes.

3.5.1. Timescale

Timescale refers to how fast an application in the respective plane react or need to manipulate the forwarding or operational plane of the device. In general, the control plane needs to send updates very often within the range of milliseconds and that requires a high bandwidth and low latency links. In contrast the management plane reacts generally at very slow timeframes, minutes, hours or even days, e.g. in the case of changing the configuration state of the device, or and thus do not need to be very efficient on the wire.

3.5.2. Ephemeral/Persistent

Another distinction that discussed was the distinction between ephemeral versus persistent state. Ephemeral state is state that may have a very limited lifespan, such as routing decisions, and thus is usually associated with the control plane. On the other hand, persistent state is state that may have a larger and extended lifespan which may range from hours to days and months and is usually associated with the management plane. Persistent state is also usually associated with data store of the state.

3.5.3. Locality

Before the concept of centralizing the controller, usually the control plane is local to device and distributed whilst the management plane is usually centralized and remote from the device. However, as has been noted before, centralizing, or "locally centralizing" the controller tends to muddle the distinction of the control and management plane on locality.

3.5.4. CAP Theorem

An additional distinction was introduced in the 89th IETF and refers to the CAP theorem.

The CAP theorem views a distributed computing system as composed of multiple computational resources (i.e., CPU, memory, storage) that are connected via a communications network and together perform a task and identifies three characteristics of distributed systems that are universally desirable:

• Consistency, meaning that the system responds identically to a query no matter which node receives the request (or does not respond at all)
• Availability, i.e., that the system always responds to a request (although the response may not be consistent or correct)
• Partition tolerance, namely that the system continues to function even when nodes or the communications network fail.

In 2000 Eric Brewer [CAPBR] conjectured that a distributed system can satisfy any two of these guarantees at the same time, but not all three. This conjecture was later proven by Gilbert and Lynch [CAPGL] and is now usually called the CAP theorem

Correctly forwarding a packet through a network, is a computational problem. One of the major abstractions that SDN posits - all network elements are computational resources that perform the single computational task of inspecting fields in an incoming packet and deciding how to forward it.

Since the task of forwarding a packet from network ingress to network egress is obviously carried out by a large number of forwarding elements, the network of forwarding devices is a distributed computational system. Hence, the CAP theorem applies to forwarding of packets.

In the context of the CAP theorem, control plane operations are usually local and fast (available), while management plane operations are usually centralized (consistent) and slow.

The CAP theorem provides insights on SDN performance. For example, in regards to locality, although not explicitly stated, modern SDN philosophy, centralizing the controller, stresses consistency. The controller acts as a consistent global database, and specific mechanisms ensure that a packet entering the network is handled consistently by all switches and acts like a management entity. The issue of tolerance to loss of connectivity to the controller is not addressed by the basic SDN model. When an SDN switch can't reach its controller the flow will be unavailable until the connection is restored. The use of multiple non-collocated SDN controllers has been proposed (e.g., by configuring the SDN switch with a list of controllers); this improves partition tolerance, but at the cost of loss of absolute consistency

3.6. Network Services Abstraction Layer

The Network Services Abstraction Layer (NSAL) provides access from services of the control, management and application planes to services and applications of the application plane. Note that the term SAL is overloaded, as it is often used in several contexts ranging from system design to service-oriented architectures therefore we prefixed it with "Network" to emphasize that this term relates to Figure 1 and we map it accordingly in Section 4 to prominent SDN approaches.

Service Interfaces can take many forms pertaining to their specific requirements. Examples of service interfaces include but are not limited to, RESTful APIs, open or proprietary protocols such as NETCONF, inter-process communications, CORBA interfaces, etc.

Two leading standards of service interface are RESTful interfaces and RPC interfaces. Both follow a client-server architecture and use XML or JSON to pass messages but each have some slightly different characteristics.

RESTful interfaces, designed with the Representational state transfer design paradigm [REST], have the following characteristics:

• Resource identification - individual resources are identified using a resource identifier, for example a URI.
• Manipulation of resources through representations - Resources are represented in a format like JSON, XML or HTML.
• Self-descriptive messages - Each message has enough information to describe how the message is to be processed.
• Hypermedia as the engine of application state - a client needs no prior knowledge of how to interact with a server, not through a fixed interface.

Remote procedure calls (RPC), e.g. [RFC5531], XML-RPC etc., have the following characteristics:

• Individual procedures are identified using an identifier
• A client needs to know the procedure name and the parameters

3.7. Application Plane

Applications and services that use services from the control and/or management plane form the Application Plane.

Additionally, services residing in the Application Plane may provide services to other services and applications that reside in the application plane via the service interface.

Examples of applications include network topology discovery, network provisioning, path reservation, etc.

4. SDN Model View

We advocate that the SDN southbound interface should encompass both CSPI and MSPI.

The SDN northbound interface is implemented in the Network Services Abstraction Layer of Figure 1.

The above model can be used to describe in a concise manner all prominent SDN-enabling technologies, as we explain in the following subsections.

4.1. ForCES

The IETF-standardized Forwarding and Control Element Separation (ForCES [RFC5810]) framework consists of one model and two protocols. ForCES separates the Forwarding from the Control Plane via an open interface, namely the ForCES protocol which operates on entities of the forwarding plane that have been modeled using the ForCES model.

The ForCES model is based on the fact that a network element is composed of numerous logically separate entities that cooperate to provide a given functionality -such as routing or IP switching- and yet appear as a normal integrated network element to external entities and secondly with a protocol to transport information.

ForCES models the Forwarding Plane using Logical Functional Blocks (LFBs) which are connected in a graph, composing the Forwarding Element (FE). LFBs are described in an XML language, based on an XML schema.

LFB definitions include:

• Base and custom-defined datatypes
• Metadata definitions
• Input and Output ports
• Operational parameters, or components
• Capabilities
• Event definitions

The ForCES model can be used to define LFBs from fine- to coarse-grained as needed irrelevant of whether they are physical or virtual.

The ForCES protocol is agnostic to the model and can be used to monitor, configure and control any ForCES-modeled element. The protocol has very simple commands: Set, Get and Del(ete). ForCES is a protocol designed for high throughput and fast updates.

ForCES [RFC5810] can be mapped to the framework illustrated in Figure 1 as follows:

• The ForCES model can be used to describe DAL, both for the Operational and the Forwarding Plane, using LFBs.
• The ForCES protocol can then be both the CPSI and the MPSI. ForCES is inherently specified for the CPSI and satisfies its requirements, however it can also be utilized for the MPSI.
• CAL and MAL must be able to utilize the ForCES protocol.

4.2. NETCONF

The Network Configuration Protocol (NETCONF [RFC6241]), is an IETF-standardized network management protocol [RFC6632]. NETCONF provides mechanisms to install, manipulate, and delete the configuration of network devices.

NETCONF protocol operations are realized as remote procedure calls (RPCs). The NETCONF protocol uses an Extensible Markup Language (XML) based data encoding for the configuration data as well as the protocol messages. Recent studies, such as [ESNet] and [PENet], have shown that NETCONF performs better than SNMP [RFC3411].

Additionally, the YANG data modeling language [RFC6020] has been developed for specifying NETCONF data models and protocol operations. YANG is a data modeling language used to model configuration and state data manipulated by NETCONF, NETCONF remote procedure calls, and NETCONF notifications.

YANG models the hierarchical organization of data as a tree, in which each node has either a value or a set of child nodes. Additionally, YANG structures data models into modules and submodules allowing reusability and augmentation. YANG models can describe constraints to be enforced on the data. Additionally YANG has a set of base datatype and allows custom defined datatypes as well.

YANG allows the definition of NETCONF RPCs allowing the protocol to have an extensible number of commands. For RPC definition, the operations names, input parameters, and output parameters are defined using YANG data definition statements.

NETCONF can be mapped to the framework illustrated in Figure 1 as follows:

• The YANG model [RFC6020] is suitable for specifying DAL for the operational plane and NETCONF [RFC6241] for the MPSI.
• Technically, the YANG model [RFC6020] can be used to specify DAL for the Forwarding plane as well. That said, in principle NETCONF [RFC6241] is a management protocol which was not (originally) designed for fast CP updates, and it might not be suitable for addressing the requirements of CPSI.

4.3. OpenFlow

[OpenFlow] is a framework originally developed by Standford, and currently under active standards development through the Open Networking Foundation. Initially, the goal was to provide a way for researchers to run experimental protocols in a production network [OFSIGC]. OpenFlow provides a protocol with which a controller may manage a static model of an OpenFlow switch.

An OpenFlow switch consists of one or more flow tables which perform packet lookups, actions on a success packet lookup and forwarding, a group table and an OpenFlow channel to an external controller. The switch communicates with the controller which manages the switch via the OpenFlow protocol.

OpenFlow has undergone many revisions. The current version is 1.4 [OpenFlow] and supports amongst others, multiple controllers for high availability and extensible flow match field protocol messages to support arbitraty match fields. Efforts to define OpenFlow 2.0 [PPIPP] are already underway aiming to provide an abstract forwarding model to provide protocol independence and device programmability.

OpenFlow can be mapped to the framework illustrated in Figure 1 as follows:

• The Openflow switch specifications [OpenFlow] covers DAL for the Forwarding Plane and provides the specification for CPSI.
• The OF-CONFIG protocol [OF-CONFIG] based on the YANG model [RFC6020], provides DAL for the Operational Plane and specifies NETCONF [RFC6241] as the MPSI. OF-CONFIG overlaps with the OpenFlow DAL, but with NETCONF [RFC6241] as the transport protocol it shares the limitations described in the previous section.
• CAL must be able to utilize the OpenFlow protocol.
• MAL must be able to utilize the NETCONF protocol.

4.4. I2RS

I2RS is currently developed by a recently-established IETF working group. The intention is to provide a standard interface to the routing system for real-time or event-driven interaction through a collection of protocol-based control or management interfaces. Essentially, I2RS aims to make the routing information base (RIB) programmable thus enabling new kinds of network provisioning and operation.

I2RS does not initially intend to create new interfaces, but rather leverage or extend existing ones and define informational models for the routing system. For example, the latest I2RS problem statement [I-D.ietf-i2rs-problem-statement] discusses previously-defined IETF protocols and data models such as ForCES, YANG, NETCONF, and SNMP.

Currently the I2RS working group is developing an Information Model [I-D.ietf-i2rs-rib-info-model] in regards to the Network Services Abstraction Layer for the I2RS agent.

I2RS can be mapped to the framework illustrated in Figure 1 as follows:

• The I2RS architecture [I-D.ietf-i2rs-architecture] encompasses the Control and Application Planes and uses any CPSI and DAL that is available, whether that may be ForCES, OpenFlow or another Interface.
• The I2RS agent is a Control Plane Service. All services or applications on top of that belong to either the Control, Management or the Application plane. In the I2RS documents, management access to the agent may be provided by management protocols like SNMP and NETCONF. The I2RS protocol may also be mapped to the Service Interface as it will provide access even to other than control applications.

4.5. BFD

Bidirectional Forwarding Detection (BFD) [RFC5880], is an IETF network protocol designed for detecting communication failures between two forwarding elements which are directly connected. It is intended to be implemented in some component of the forwarding engine of a system, in cases where the forwarding and control engines are separated.

BFD provides low-overhead detection of faults even on physical media that do not support failure detection of any kind, such as Ethernet, virtual circuits, tunnels and MPLS Label Switched Paths.

BFD could be mapped to the framework illustrated in Figure 1 either as:

1. A control plane service or application that would use the CPSI towards the forwarding plane to send/receive BFD packets.
2. Or, better, as it was intended for, i.e. as an application that runs on the device itself and uses the forwarding plane to send/receive BFD packets and update the operational plane resources accordingly.

4.6. SNMP

The Simple Network Management Protocol (SNMP) is an IETF management protocol and is currently at the third version called SNMPv3 and described in STD 62, RFC 3417 [RFC3417], RFC 3412 [RFC3412] and RFC 3414 [RFC3414]. It consists of a set of standards for network management, including an application layer protocol, a database schema, and a set of data objects. SNMP exposes management data (managed objects) in the form of variables on the managed systems, which describe the system configuration. These variables can then be queried and set by managing applications.

SNMP uses an extensible design for describing data, defined by management information bases (MIBs). MIBs describe the structure of the management data of a device subsystem. MIBs use a hierarchical namespace containing object identifiers (OID). Each OID identifies a variable that can be read or set via SNMP. MIBs use the notation defined by Structure of Management Information Version 2 SMIv2 [RFC2578]

SNMP could be mapped to the framework illustrated in Figure 1 as:

1. SNMP MIBs are usually used to describe DAL for the Operational Plane.
2. SNMP is usually used for the MPSI.

5. Acknowledgements

The authors would like to acknowledge Salvatore Loreto and Sudhir Modali for the initial discussion on the SDNRG mailing list as well as their draft-specific comments that helped put this document in a better shape.

Additionally the authors would like to acknowledge Russ White, Linda Dunbar, Robert Raszuk, Pedro Martinez-Julia, Lee Young, Yaakov Stein, Shivleela Arlimatti, Gurkan Deniz, Scott Brim, Carlos Pignataro, Ramki Krishnan, Bless Roland, Tim Copley, Francisco Javier Ros Munoz, Sriganesh Kini, Alan Clark, Erik Nordmark, Scott Mansfield, Dirk Kutscher, Roland Bless, David E Mcdysan, Bhumip Khasnabish and Georgios Karagiannis for their critical comments and discussions at the IETF 88 and 89 meetings (and the SDNRG mailing list), which we took into consideration while revising this document.

Special thanks to Yaakov Stein for providing text related to the CAP theorem and Scott Mansfield for information regarding ITU status on SDN

6. IANA Considerations

This memo makes no requests to IANA.

7. Security Considerations

TBD

8. Informative References

[A4D05]	Greenberg, Albert, et al., "A clean slate 4D approach to network control and management", ACM SIGCOMM Computer Communication Review 35.5 (2005): 41-54 , 2005.
[CAPBR]	Eric A. Brewer, "Towards robust distributed systems.", Symposium on Principles of Distributed Computing (PODC). 2000 , 2000.
[CAPGL]	Seth Gilbert, and Nancy Ann Lynch., "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services", ACM SIGACT News 33.2 (2002): 51-59. , 2002.
[DIOPR]	Denazis, Spyros, Kazuho Miki, John Vicente, and Andrew Campbell., "Designing interfaces for open programmable routers.", In Active Networks, pp. 13-24. Springer Berlin Heidelberg, 1999 , 1999.
[ESNet]	Yu, James, and Imad Al Ajarmeh., "An empirical study of the NETCONF protocol.", In Networking and Services (ICNS), 2010 Sixth International Conference on, pp. 253-258. IEEE, 2010. , 2010.
[FCAPS]	International Telecommunication Union, "X.700: Management Framework For Open Systems Interconnection (OSI) For CCITT Applications", September 1992.
[I-D.ietf-i2rs-architecture]	Atlas, A., Halpern, J., Hares, S., Ward, D. and T. Nadeau, "An Architecture for the Interface to the Routing System", Internet-Draft draft-ietf-i2rs-architecture-04, June 2014.
[I-D.ietf-i2rs-problem-statement]	Atlas, A., Nadeau, T. and D. Ward, "Interface to the Routing System Problem Statement", Internet-Draft draft-ietf-i2rs-problem-statement-04, June 2014.
[I-D.ietf-i2rs-rib-info-model]	Bahadur, N., Folkes, R., Kini, S. and J. Medved, "Routing Information Base Info Model", Internet-Draft draft-ietf-i2rs-rib-info-model-03, May 2014.
[ITUATM]	CCITT, Geneva, Switzerland, "CCITT Recommendation 1.361, B-ISDN ATM Layer Specification", 1990.
[ITUSG13]	Telecommunication Standardization sector of ITU, "ITU, Study group 13", 2013.
[ITUSS7]	Telecommunication Standardization sector of ITU, "ITU, Q.700 : Introduction to CCITT Signalling System No. 7", 1993.
[KANDOO]	Hassas Yeganeh, Soheil, and Yashar Ganjali., "Kandoo: a framework for efficient and scalable offloading of control applications.", In Proceedings of the first workshop on Hot topics in software defined networks, pp. 19-24. ACM SIGCOMM, 2012. , 2012.
[NV09]	Chowdhury, NM Mosharaf Kabir, and Raouf Boutaba, "Network virtualization: state of the art and research challenges", Communications Magazine, IEEE 47.7 (2009): 20-26 , 2009.
[OF-CONFIG]	Open Networking Foundation, "OpenFlow Management and Configuration Protocol 1.1.1", March 2013.
[OF08]	McKeown, Nick, et al., "OpenFlow: enabling innovation in campus networks", ACM SIGCOMM Computer Communication Review 38.2 (2008): 69-74 , 2008.
[OFSIGC]	McKeown, Nick, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner., "OpenFlow: enabling innovation in campus networks.", ACM SIGCOMM Computer Communication Review 38, no. 2 (2008): 69-74. , 1998.
[ONFArch]	Open Networking Foundation, "SDN Architecture Overview", December 2013.
[OpenFlow]	Open Networking Foundation, "The OpenFlow 1.4 Specification.", October 2013.
[P1520]	Biswas, Jit, Aurel A. Lazar, J-F. Huard, Koonseng Lim, Semir Mahjoub, L-F. Pau, Masaaki Suzuki, Soren Torstensson, Weiguo Wang, and Stephen Weinstein., "The IEEE P1520 standards initiative for programmable network interfaces.", Communications Magazine, IEEE 36, no. 10 (1998): 64-70. , 1998.
[PENet]	Hedstrom, Brian, Akshay Watwe, and Siddharth Sakthidharan, "Protocol Efficiencies of NETCONF versus SNMP for Configuration Management Functions", PhD dissertation, Master's thesis, University of Colorado, 2011 , 2011.
[PNSurvey99]	Campbell, Andrew T., et al, "A survey of programmable networks", ACM SIGCOMM Computer Communication Review 29.2 (1999): 7-23 , September 1992.
[PPIPP]	Bosshart, Pat, Dan Daly, Martin Izzard, Nick McKeown, Jennifer Rexford, Dan Talayco, Amin Vahdat, George Varghese, and David Walker., "Programming Protocol-Independent Packet Processors.", arXiv preprint arXiv:1312.1719 (2013). , 2013.
[RCP]	Caesar, Matthew, Donald Caldwell, Nick Feamster, Jennifer Rexford, Aman Shaikh, and Jacobus van der Merwe., "Design and implementation of a routing control platform.", In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2, pp. 15-28. USENIX Association, 2005. , 2005.
[REST]	Fielding, Roy, "Fielding Dissertation: Chapter 5: Representational State Transfer (REST).", 2000.
[RFC0826]	Plummer, D., "Ethernet Address Resolution Protocol: Or converting network protocol addresses to 48.bit Ethernet address for transmission on Ethernet hardware", STD 37, RFC 826, November 1982.
[RFC1953]	Newman, P., Edwards, W., Hinden, R., Hoffman, E., Ching Liaw, F., Lyon, T. and G. Minshall, "Ipsilon Flow Management Protocol Specification for IPv4 Version 1.0", RFC 1953, May 1996.
[RFC2297]	Newman, P., Edwards, W., Hinden, R., Hoffman, E., Liaw, F., Lyon, T. and G. Minshall, "Ipsilon's General Switch Management Protocol Specification Version 2.0", RFC 2297, March 1998.
[RFC2578]	McCloghrie, K., Perkins, D. and J. Schoenwaelder, "Structure of Management Information Version 2 (SMIv2)", STD 58, RFC 2578, April 1999.
[RFC3411]	Harrington, D., Presuhn, R. and B. Wijnen, "An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks", STD 62, RFC 3411, December 2002.
[RFC3412]	Case, J., Harrington, D., Presuhn, R. and B. Wijnen, "Message Processing and Dispatching for the Simple Network Management Protocol (SNMP)", STD 62, RFC 3412, December 2002.
[RFC3414]	Blumenthal, U. and B. Wijnen, "User-based Security Model (USM) for version 3 of the Simple Network Management Protocol (SNMPv3)", STD 62, RFC 3414, December 2002.
[RFC3417]	Presuhn, R., "Transport Mappings for the Simple Network Management Protocol (SNMP)", STD 62, RFC 3417, December 2002.
[RFC3418]	Presuhn, R., "Management Information Base (MIB) for the Simple Network Management Protocol (SNMP)", STD 62, RFC 3418, December 2002.
[RFC3535]	Schoenwaelder, J., "Overview of the 2002 IAB Network Management Workshop", RFC 3535, May 2003.
[RFC4271]	Rekhter, Y., Li, T. and S. Hares, "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, January 2006.
[RFC5531]	Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, May 2009.
[RFC5743]	Falk, A., "Definition of an Internet Research Task Force (IRTF) Document Stream", RFC 5743, December 2009.
[RFC5810]	Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, W., Dong, L., Gopal, R. and J. Halpern, "Forwarding and Control Element Separation (ForCES) Protocol Specification", RFC 5810, March 2010.
[RFC5812]	Halpern, J. and J. Hadi Salim, "Forwarding and Control Element Separation (ForCES) Forwarding Element Model", RFC 5812, March 2010.
[RFC5880]	Katz, D. and D. Ward, "Bidirectional Forwarding Detection (BFD)", RFC 5880, June 2010.
[RFC6020]	Bjorklund, M., "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, October 2010.
[RFC6241]	Enns, R., Bjorklund, M., Schoenwaelder, J. and A. Bierman, "Network Configuration Protocol (NETCONF)", RFC 6241, June 2011.
[RFC6632]	Ersue, M. and B. Claise, "An Overview of the IETF Network Management Standards", RFC 6632, June 2012.
[RFC7047]	Pfaff, B. and B. Davie, "The Open vSwitch Database Management Protocol", RFC 7047, December 2013.
[RFC7149]	Boucadair, M. and C. Jacquenet, "Software-Defined Networking: A Perspective from within a Service Provider Environment", RFC 7149, March 2014.
[RINA]	John Day, Ibrahim Matta, and Karim Mattar., "Networking is IPC: a guiding principle to a better internet.", In Proceedings of the 2008 ACM CoNEXT Conference, p. 67. ACM, 2008. , 2008.
[RouteFlow]	Nascimento, Marcelo R., Christian E. Rothenberg, Marcos R. Salvador, Carlos NA Correa, Sidney C. de Lucena, and Mauricio F. Magalhaes., "Virtual routers as a service: the routeflow approach leveraging software-defined networks.", In Proceedings of the 6th International Conference on Future Internet Technologies, pp. 34-37. ACM, 2011. , 2011.
[SDNACS]	Diego Kreutz, Fernando M. V. Ramos, Paulo Verissimo, Christian Esteve Rothenberg, Siamak Azodolmolky, Steve Uhlig, "Software-Defined Networking: A Comprehensive Survey.", arXiv preprint arXiv:1406.0440 , 2014.
[SDNHistory]	Feamster, Nick, Jennifer Rexford, and Ellen Zegura., "The Road to SDN", ACM Queue11, no. 12 (2013): 20. , 2013.
[SLTSDN]	Yosr Jarraya, Taous Madi, and Mourad Debbabi, "A Survey and a Layered Taxonomy of Software-Defined Networking", To be published in Communications Surveys and Tutorials, IEEE Issue: 99 , 2014.
[SoftRouter]	Lakshman, T. V., T. Nandagopal, R. Ramjee, K. Sabnani, and T. Woo., "The softrouter architecture.", In Proc. ACM SIGCOMM Workshop on Hot Topics in Networking. 2004. , 2004.
[Tempest]	Rooney, Sean, Jacobus E. van der Merwe, Simon A. Crosby, and Ian M. Leslie., "The Tempest: a framework for safe, resource assured, programmable networks.", Communications Magazine, IEEE 36, no. 10 (1998): 42-53 , 1998.

Authors' Addresses