Internet DRAFT - draft-khasnabish-vmmi-problems
draft-khasnabish-vmmi-problems
Network Working Group Bhumip Khasnabish
Internet-Draft ZTE USA,Inc.
Intended status: Informational Bin Liu
Expires: July 3, 2013 ZTE Corporation
Baohua Lei
Feng Wang
China Telecom
Dec 30, 2012
Mobility and Interconnection of Virtual Machines and Virtual Network
Elements
draft-khasnabish-vmmi-problems-03.txt
Abstract
In this draft, we discuss the challenges and requirements related to
the migration, mobility, and interconnection of Virtual Machines
(VMs)and Virtual Network Elements (VNEs). VM migration scheme across
IP subnets is needed to implement virtual computing resources sharing
across multiple network administrative domains. Many technologies
are involved in the VM migration across DCs. These technologies are
classified and discussed according to their different locations in
the inter-DC and intra-DC network. For the seamless online migration
in various scenarios, many problems need to be resolved in the
control plane. The VM migration process should be adapted to these
aspects. We also describe the limitations of various types of
virtual local area(VLAN) networking technologies and virtual private
networking (VPN) technologies that are traditionally expected to
support such migration, mobility, and interconnections.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 3, 2013.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 1]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1. Conventions Used in this Document . . . . . . . . . . . . 4
2. Terminology and Concepts . . . . . . . . . . . . . . . . . . . 4
3. Control & Mobility Related Problem Specifications . . . . . . 6
3.1. Summarization of Mobility in Virtualized Environments . . 7
3.2. VM Migration Mobility Problems across IP Subnets/WAN . . . 7
3.2.1. IP Tunnel Problems . . . . . . . . . . . . . . . . . . 9
3.2.2. IP Allocation Strategy Problems . . . . . . . . . . . 10
3.2.3. Routing Synchronization Strategy Problems . . . . . . 12
3.2.4. The Migration Protocol State Machine of VM Online
Migration across Subnets . . . . . . . . . . . . . . . 12
3.2.5. Resource Gateway Problems . . . . . . . . . . . . . . 13
3.2.6. Optimized Location of Default Gateway . . . . . . . . 13
3.2.7. Other Problems . . . . . . . . . . . . . . . . . . . . 13
3.3. VM Mobility Problems Implemented on the VR Device . . . . 13
3.4. Security and Authentication of VMMI . . . . . . . . . . . 13
3.5. The Virtual Network Model . . . . . . . . . . . . . . . . 14
3.6. The Processing Flow . . . . . . . . . . . . . . . . . . . 14
3.7. The NVE/OBP Location Problems . . . . . . . . . . . . . . 15
3.7.1. NVE/OBP on the Server . . . . . . . . . . . . . . . . 16
3.7.2. NVE/OBP on the ToR . . . . . . . . . . . . . . . . . . 17
3.7.3. Hybrid Scenario . . . . . . . . . . . . . . . . . . . 19
4. Technology Problems involved in the VM Migration . . . . . . . 19
4.1. TRILL . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2. SPB . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.3. Data Center Interconnection Fabric Related Problems . . . 20
4.4. Types and Applications of VPN Interconnections between
DCs which Provide DCI . . . . . . . . . . . . . . . . . . 21
4.4.1. Types of VPNs . . . . . . . . . . . . . . . . . . . . 21
4.4.2. Applications of L2VPN in DCs . . . . . . . . . . . . . 21
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 2]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
4.4.3. Applications of L3VPN in DCs . . . . . . . . . . . . . 22
4.5. The Actual Number of Available Isolated Domains . . . . . 22
4.6. Problems of the Number of Management
Devices/Management Domains of DC in the NVO3 Network . . . 22
4.7. Limitation of TCAM capacity . . . . . . . . . . . . . . . 23
4.8. SDN . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.9. LISP . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5. Others Problems Related with IETF and IEEE . . . . . . . . . . 23
5.1. Review of VXLAN, NVGRE, and NVO3 . . . . . . . . . . . . . 23
5.2. The East-West Traffic Problem . . . . . . . . . . . . . . 25
5.3. The MAC, IP, and ARP Explosion Problems . . . . . . . . . 26
5.4. Suppressing Flooding within a VLAN . . . . . . . . . . . . 27
5.5. Packet Encapsulation Problems . . . . . . . . . . . . . . 27
6. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 27
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27
8. Security Considerations . . . . . . . . . . . . . . . . . . . 28
9. IANA Consideration . . . . . . . . . . . . . . . . . . . . . . 28
10. Normative References . . . . . . . . . . . . . . . . . . . . . 28
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 3]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
1. Introduction
There are many challenges related to the VM migration and their
interconnections among two or more data centers (DCs). The
technologies that can be used for VM migration and DC interconnection
should support the required level of performance, security,
scalability, along with simplicity and cost-effective management,
operations and maintenance.
In this draft, the issues and requirements for moving the virtual
machines are summarized with reference to the necessary conditions
for migration, business needs, state classification, security, and
efficiency.
In this draft, the requirements for VMMI technologies that are useful
on large-scale Layer-2 network and on segmented IP network/WAN are
discussed. VM migration scheme across IP subnets/WAN is therefore
needed to implement virtual computing resources sharing across
multiple network administrative domains. This will make a wider
range of VM migration possible, and allow for migration of VMs to
different types of DCs. It can be adapted to different types of
physical networks, different topological networks, and various
protocols. For the seamless online migration in these scenarios, a
very intelligent seamless VM migration orchestration is needed in the
control plane. We summarize the requirements of virtual networks for
VM migration, virtual networking, and operations in DCI/overlay
modes.
1.1. Conventions Used in this Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
2. Terminology and Concepts
o ARP: Address Resolution Protocol
o DC: Data Center
o DC GW: Data Center Gateway
o DCI: Data Center Interconnection
o DCS: Data Center Switch
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 4]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
o FDB: Forwarding DataBase
o HPC: High-Performance Computing
o IDC: Internet Data Center
o IP: Internet Protocol
o IP VPN: Layer 3 VPN, defined in L3VPN working group
o LISP: Locator ID Separation Protocol
o NVO3: Network Virtualization Overlays (Over Layer-3)
o OBP: Overlay network boundary point
o OTV: Overlay Transport Virtualization
o PBB: Provider Backbone Bridge
o PM: Physical Machine
o QoS: Quality of Service
o STP: Spanning Tree Protocol
o TNI: Tenant Network Identifier
o ToR: Top of the Rack
o TRILL: Transparent Interconnection of Lots of Links
o VLAN: Virtual Local Area Networking
o VM: Virtual Machine
o VMMI: Virtual Machine Mobility and Interconnection
o VN: Virtual Network
o VNI: Virtual Network Identifier
o VNE: Virtual Network Entity.(a virtualized laye-3/network entity
with associated virtualized port and virtualized processing
capabilities)
o VPN: Virtual Private Network
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 5]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
o VPLS: Virtual Private LAN Service
o VR: Virtual Router(a logical device which realizes simulation
capabilities of the physical router on the software using hardware
layer's resources and capabilities)
o VRRP: Virtual Router Redundancy Protocol
o VSE: Virtual Switching Entity (a virtualized laye-2/switch entity
with associated virtualized port and virtualized processing
capabilities)
o VSw: Virtual Switch
o WAN: Wide Area Network
o communication agent: NVO3 communication agent(an entity which
forwards traffic between NVO3 and non-NVO3 environments)
3. Control & Mobility Related Problem Specifications
Overall, the requirements of VM migration bring in the following
challenges in the forefront of data center operations and management:
(A)How can the existing technologies be compatible with each other,
including various multi-tenant network technologies, network
interface technologies between VMs and ToRs, Access layer network
technologies, networking technologies within the DC and across DCs?
So that these technologies can work seamlessly with overlay network
technologies through virtualization technologies. It may accomplish
the accommodation of a large number of individual tenant networks in
the DC and the communication between the isolated domains. It makes
the VMs can be seamlessly online migrated in the management domain.
(B)When the VMs are migrated from one DC to another within one
administrative domain, (i) how to ensure that the necessary
conditions of migration are satisfied, (ii) how to ensure that a
successful migration occurs without service disruption, and (iii) how
to ensure successful rollback when any unforeseen problem occurs in
the migration process.
(C)When the VMs are migrated from one administrative domain to
another, how to solve the problems of seamless communications between
the domains. There are several different solutions(such as VXLAN,
NVGRE,etc) to the current Layer-2 (L2) based DC interconnection
technology, and each can solve different problems in different
scenarios. If the unification of packet encapsulation mapping rules
in different solutions can be achieved, it is bound to promote
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 6]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
seamless migration of VMs among DCs along with the desired
integration in cloud computing and networking.
(D) How to utilize IP based technologies to accomplish the migration
of VMs over layer-3 (L3) network? For example, VPN technology can be
used to carry L2 and L3 traffic across the IP/MPLS core network.
We discuss the above in more details in the following sections. A
related draft [DCN Ops Req] discusses data center network and
operations requirements.
3.1. Summarization of Mobility in Virtualized Environments
Mobility refers to the movement of a VM from one server to another
server within one DC or across DCs, while maintaining the VM's
original IP and MAC address throughout the process. When a VM is
migrated to a new location, it should maintain the existing client
sessions, and the state of the VM sessions should be copied to the
new location. VM mobility does not change the VLAN/subnet connection
to the VM, and it requires that the serving VLAN is extended to the
new location of the VM.
In order to support VM mobility, it is required to allow VMs to be
migrated easily and repeatedly -- that is as often as needed by the
applications and services -- among a large (more than two) number of
DCs. Seamless migration of VMs in mixed IPv4 and IPv6 VPN
environments should be supported by using appropriate DC GWs.
Some widely used VM migration tools require that management programs
on the source server and destination server are directly connected
via an L2 network. The objective is to facilitate the implementation
of smooth VM migration.
The participating source server and destination server in the VM
migration process may be located in different DCs. It may be
required to extend the Layer-2 network beyond what is covered by the
L2 network of the source DC. This may create islands of the same
VLAN in different (geographically dispersed) DCs.
Besides, the optimal forwarding in a VLAN that supports VM mobility
may involve traffic management over multiple DCs. The support of
seamless mobility of VM across DCs may not necessarily always achieve
optimal intra-VLAN forwarding and routing.
3.2. VM Migration Mobility Problems across IP Subnets/WAN
There are many existing implementable solutions for migrating VM
within a traditional LAN(non-large-scale Layer-2 network). These
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 7]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
solutions include Xen, KVM, and VMWare, which all implement VM image
file sharing based on NFS, and only CPU and memory status are
migrated. These are layer-2 VM migration technologies. The
advantage of the implementation is that VM's IP addresses don't need
to be changed after the VM migration. With the development and
popularization of the DCs and virtualization technologies, the number
of servers and network environment in a single LAN will limit the
scalability of the virtual computing environment.
In addition, when re-configuring the VLAN in the traditional DC
network, STP (MSTP) will lead to the VLAN isolation. It is a very
serious problem in the DC network, especially in the storage network,
because the storage network is very demanding for uninterrupted
service.
With the evolution of new technologies, such as various
virtualization technologies, the large-scale Layer-2 technology, DCI
technology and overlay network technology, the network environment in
the VM migration will become more complex. In order to realize
virtual computing resource sharing and online VM migration across
multiple management domains using these technologies as the
foundation, VM migration scheme based on Nvo3 network technology is
needed to adapt to the complex network environment. This will make a
wider range of VM migration possible, and can allow for migration of
VMs to different types of DCs. It can be adapted to different types
of physical networks, different topological networks, and various
protocols.
For example, in the process of VM migration in DC, there are
scenarios that VM in the traditional three-tier topological network
is migrated through WAN to Fat-Tree topological network, or to a
variety of other topological networks. For the seamless online
migration in these scenarios, a very intelligent seamless VM online
migration is needed to be implemented in the control plane.
If VM migration is only implemented in the L2 domain, people are
concerned about the expansion of the number of VLANs or isolated
domains, such as the 16,000,000 isolated domains in PBB.
Now the limitless and seamless VM online migration across overlay
based IP subnets means that the following issues need to be
addressed, in order to achieve our goal: to create a true virtual
network environment that is separated from the physical network.
Isolation domain mapping rules.
Migration across IP subnets.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 8]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
VM migration in the overlay network needs to be adapted to the
heterogeneous network topology.
How the source network environment(i.e. the network between the VM or
its server and the connected ToR) adapts its configuration to the
destination network environment?
The network redirection technology, IP-in-IP technology and dynamic
IP tunnel configuration will be used to allow online VM migration
across subnets.
A module which allocates IP addresses to VMs is needed, which can
manage each IP allocation in the virtual network. The IP allocation
should not conflict with each other, and make the path cost of
routing forwarding as small as possible. It is necessary to know the
DC network topology, its routing protocols, and real-time results of
the path cost to realize minimum path cost. We know that the network
topologies of different DCs are not necessarily the same. For
example, the network topologies and routing protocols of traditional
DC and Fat-Tree network DC are different. The addition of related
protocol processing in the control plane is needed for seamless VM
migration between them. Otherwise, online VM migration cannot be
implemented across DCs or across IP subnets. The scheme of IPinIP
tunneling resolves the contradiction between unchanged IP addresses
during the VM migration and changed IP addresses when VMs are
migrated across IP subnets. Therefore, the VM's mobility problem can
be solved only after the above mentioned problems have been solved.
Service providers can implement VM migration by upgrading its
software to support new protocols, and the hardware devices don't
need to be upgraded.
These problems are described as below:
3.2.1. IP Tunnel Problems
During the VM migration, it is required to establish the IP-in-IP
tunnels . The purpose is to make the user/application have no
perception of the migration process, and their IP addresses on the
related layer should be the same. The scheme of IPinIP tunneling
resolves the contradiction between unchanged VM IP addresses during
the VM migration and changed server IP addresses when VMs migrate
across IP subnets. OBP is involved in setting up IP tunnels.
According to nvo3 control plane protocols, there are two positions
for OBP (NVE / VTEP) in the DC: on the Server and on ToR. Placing
OBP on the server can minimize its correlation with network elements
in the specific network topology. It will face more problems if OBP
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 9]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
is placed on ToR. NVE is preferred to be placed on the Server
(unless there are other stronger reasons). It will create a virtual
network for VM communications. The traffic between VMs will not
directly be exposed on the wire and switches.
However, OBP on the Server can reduce the weak coupling with DC
topology to a certain extent, but they cannot be completely
unrelated.
The disadvantage of network connection solutions for online VM
migration across different subnets is that the network configuration
of VM needs to be changed after the migration, and the migration
process is opaque. So the transparent migration of VM needs to be
implemented, and network connection redirection technology needs to
be considered.
Since users cannot utilize the VMs due to changes of the network
access point during the online migration of VMs across subnets, the
scheme of network connection redirection system based on Proxy Mobile
IP (PM IP) can be used. VM which is migrated to the external subnet
is regarded as a mobile node and the IP address isn't changed. All
the packets to/from the VM is transmitted through the bi-directional
tunnel between the external network and the home network, in order to
implement online transparent migration across subnets. After the
necessary data has been migrated, the tunnel directly connected to
the location of the VM's new server is re-established at the
preferably switching speed.
The source VM and the destination VM need to be activated
simultaneously and must be dynamically configured with IP tunnel. In
order to make the VM migration process completely transparent
(including transparent to the VMs' applications and the outside
users), the migration environment of the VMs should be regarded as a
mobile network environment, and the migrated VM is regarded as a
mobile node. The mobile agent function of the host should be taken
full advantage to communicate with the external network.
3.2.2. IP Allocation Strategy Problems
In the encapsulation of packets described as above, the IP address of
the VM is a critical entity. Its allocation is based on DHCP in the
small network. With the expansion of the network scale, IP address
conflict is more likely to occur. When the VM is migrated to another
network, its IP address may possibly conflict with IP address of the
VM or physical host in the destination network. For example, the
duplicate IP addresses will causes confusion and migration failure.
Therefore, a management module allocating IP addresses to VMs is
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 10]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
needed, which can manage each IP allocation in the virtual network.
The IP allocation should not conflict with each other, and should
make the path cost of routing and forwarding as small as possible.
When allocating IP, it should not conflict with the currently
assigned IP network segments of the VM clusters. In addition, it
should not conflict with the IP network segments where the physical
hosts are located. It also should not conflict with the destination
IP network segments after the migration. So the synchronization of
IP address allocation information needs to be done. Of course, the
synchronization in the whole network is not necessary as long as
there are ways to ensure no conflict exists. Moreover, the
allocation method needs to consider the introduction of network
overhead as small as possible, as well as insufficient IP issues in
the destination network segments.
When allocating IP addresses to the hosts based on DHCP protocol, the
IP addresses in the IP address pool are allocated from small to
large. The insufficient number of addresses in the pool may lead to
conflict with assigned VMs' IP addresses, which hinders VM migration.
The IP address allocation from small to large makes assigned VMs' IP
addresses may affect routing protocol to choose the better path.
Especially for the specific architectures like Fat-Tree, specific
network topology and the protocol architecture of specific routing
strategy (such as OSPF) should be utilized. The VM migration process
must be adapted to these aspects, and cannot be copied for purely
Layer-2 migration approach. So VM migration is inherently related to
network topologies and network routing protocols.
In the Fat-tree topology, IP addressing and IP allocation methods of
network servers and switches are related to the routing protocols.
Two routing methods can be chosen: OSPF protocol (OSPF domain cannot
be too large), and the fixed routing configuration.
As VM in the destination DC is needed to assign an IP, in order to
prevent IP conflict, routing protocols used in the destination DC
need to be known. For example, OSPF routing protocol (in this case,
the new added network node is assigned IP address by using DHCP), or
the fixed configuration IP routing protocol is used in the Fat-tree
topology. If the former, the number and distribution of reserved IP
addresses in the IP address pool are different from the latter.
Therefore a scheme is required to know the adopted network topology
and address allocation strategy, IP usage for each segment, the
remaining number of IP addresses, etc. This information cannot be
acquired purely by the existing DHCP protocol.
Different routing strategies have different routing management
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 11]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
mechanism for VM migration across the DCs for the following reasons:
(a) It involves the uniqueness problem of the IP address assignment
and IP tunnel establishment, and (b) it involves the global unified
management issues. These problems will be discussed later.
In the addressing method of the fixed routing protocol, the IP
address assigned to the device located within DC actually contains
the location information. The type of the corresponding device can
easily be determined through IP, and the location of the device in
the topology can also be visually judged.
So these addresses must be avoided in the automated allocation of IP
addresses to the VMs.
The function of DHCP protocol needs to be greatly enhanced, or the
protocols and tools of IP address allocation need to be re-designed.
Moreover, negotiation is needed before migration. Because it may be
required to migrate the VMs back after the migration process has been
finished, in order to make the migration process smoothly, the IP
addresses of source /destination communications agents should be
considered for reservation. The reserved IP address relates to
network topology and IP address allocation strategy in the source/
destination network environment.
For the control plane protocols in the nvo3 network, reasonable
allocation of IP address is used according to the adopted network
topology and routing protocols in the source and destination DC, in
order to achieve the seamless VM migration and the optimal path as
much as possible.
In addition, the above mentioned problems are also involved in
PortLand network topology, similar to Fat Tree network topology.
Future server-centric network topologies, such as Dcell/Bcube network
topology, also need to achieve compatibility in the control plane.
3.2.3. Routing Synchronization Strategy Problems
In order to ensure the normal data forwarding after the VM migration,
the routing synchronization between the source network and
destination network is needed.
3.2.4. The Migration Protocol State Machine of VM Online Migration
across Subnets
As for the routing strategy discussed earlier, compared to the
migration in the same IP subnet, the IP allocation strategy and
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 12]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
routing synchronization strategy will be changed. So the state and
handling of routing updates must be included in the state machine of
VM migration across subnets at the preparation phase before the VM
migration.
Therefore, if the VM is allowed to span cross subnets, the network
redirection technology should be used. For IP-in-IP technology, the
advantage of it is good compatibility with network equipments, as
long as upgrading their software.
3.2.5. Resource Gateway Problems
A resource gateway is needed to record IP address resources that have
been used, and IP network segments which the used IP addresses belong
to.
3.2.6. Optimized Location of Default Gateway
The VM's default gateway should be in a close topological proximity
to the ToR that is connected to the server presently hosting that VM.
3.2.7. Other Problems
Migration across domains has proposed new requirements for network
protocols, for example, the ARP response packet mechanism is no
longer applicable in the WAN. In addition, some packets will be lost
during the migration, which does not apply to parallel computing.
There are also problems such as computing resources sharing across
multiple administrative domains, etc.
3.3. VM Mobility Problems Implemented on the VR Device
Each virtual router (VR) has logically independent routing table and
forwarding table. It supports the overlay of private IP addresses
and public IP addresses. Multiple VRs can be logically formed on one
physical router. Each VR individually runs its own instances of the
routing protocols, and has its dedicated I/O ports, the cache, the
address space, routing and forwarding tables and network management
software. But it isn't in a way that supports multi-tenancy.
Therefore it does not support the live migration of VM.
3.4. Security and Authentication of VMMI
During the VM migration process, it is required to give proper
considerations to the security related matters; this includes solving
traffic roundabout issues, ensuring that the firewall functionalities
are appropriately enacted, and so on.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 13]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
Therefore, in addition to authorization and authentication,
appropriate policies and measures to check/enforce the security level
must be in place while migrating VMs from one DC to another,
especially from a private DC to a public DC in the Cloud [NIST 800-
145, Cloud/DataCenter SDO Survey].
For example, when a VM is migrated to the destination DC network, the
corresponding switch port connected to the VM and its host server
should utilize the port strategy of the source switch. The end time
of the VM migration and the issue time of the strategy must be
synchronized. If the former is earlier than the latter, the services
may not get a timely response, and if the former is later than the
latter, it may not have exact level of network security for a time
period.
What may be helpful in such environment is the creation and
maintenance of a reasonable interactive state machine.
3.5. The Virtual Network Model
Based on the above problems, two requirements will be added on the
virtual network model: Firstly, the routing information is adjusted
automatically according to the physical location of VM after the VM
is migrated to a new subnet; Secondly, a logical entity, namely
"virtual network communications agent", is added, which is
responsible for data routing, storage and forwarding in the inter-
subnets communications. The agent can be dynamically created and
revoked, and can be running on each server.
The communication nodes in the overlay layer are called overlay
entities, which are composed by all VMs and communication agents.
Each VN on the overlay layer can be customized as required. The VN
is composed by the specified VMs and communication agents. VMs and
communication agents may come from different physical networks. They
are connected through private tunnels established by the
communication agents. The positions of communication agents in the
physical network can be divided into two categories: on the server or
on the network device (such as on the ToR).
3.6. The Processing Flow
During the process, VM migration messages will trigger the topology
updates of the VMs' clusters in the source virtual network and
destination virtual network. It is therefore required to acquire the
network topology, the routing protocols, and the IP address
assignment rules for each other on both ends, so the VM can be
assigned an unique IP address. The routing information of the
communications agents is updated. The communications agent captures
the corresponding VM's packets, encapsulates them into the data
section of the packets, and adds the necessary control information
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 14]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
(such as self-defined forwarding rules). After the encapsulation,
these packets are transferred to the destination network through the
tunnels between the communications agents. The communications agent
in the destination network de-capsulates the packets and processes
the information, and then delivers the packets to the destination
network. The data transfer process across subnets is now completed.
The modules which need to be modified are as follows:
According to the above processing flow, the modules can be divided by
function as follows: Routing management, MAC capture, Tunnel packet
encapsulation, Tunnel forwarding, Tunnel packet de-capsulation, and
Forwarding in the destination network.
3.7. The NVE/OBP Location Problems
VMs communicate with each other through the interconnected network
either within the same domain, or between different domains.
According to various NVE / OBP position, the processing is different.
As it is transparent to network topology and L2/L3 protocol, NVE /
OBP on the server should be the default configuration mode.
Assume that a set of VMs and the network that interconnects them are
allowed to communicate with each other, MAC source and destination
addresses in the Ethernet header of the packets exchanged among these
VMs are preserved. This is L2-based VM communication within a LAN.
Any VM should have its own IP. If a VM belongs to more than one
domain, this VM will have multiple IP addresses and multiple logical
interfaces, which is similar to the model of L3 switches.
Different VM clusters are distinguished by VLAN mechanism in the same
L2 physical domain. In the case of VM communications across IP
subnets, the packets are encapsulated in NVE, and directly delivered
to the peer NVE, and then transferred to the destination VM.
Once migration across L3 network occurs, some scenarios will cause
the MAC source address to be modified.
It is also possible that a VM may belong to different VM cluster
networks at the same time, and the two VM clusters are distinguished
by the VLANs(VN ID).
In the above case, from the perspective of the overlay network, VLAN
is mapped to VNI. Different VNI domains are isolated from each
other. If you want to communicate between different VNI domains, the
packets should be routed according to the outer layer addresses, and
then the outer headers are stripped. The packets are looked up
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 15]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
according to the inner layer addresses.
As NVEs may belong to different domains, if a NVE communicates with
the other NVE in the same domain, the VLAN-ID of packets exchanged
should be the same. In order to simplify the process, the VLAN-IDs
are allowed to be removed. But once a NVE communicates with the
other NVE in the different domain, the VLAN-ID of packets exchanged
may be different.
Note that 'Two VMs' in the following scenarios refers to 'Two VMs
which are communicating with each other'.
3.7.1. NVE/OBP on the Server
Scenario classification:
Note that 'belonging to the same VN ID' refers to 'within the same
subnet'.
(1)Two VMs are on the same server and belong to the same VN ID.
In this case, the packets are forwarded with the Layer-2 mechanism.
VM migration processing scenario: this scenario is not in the scope
of VM migration.
(2)Two VMs are on the same server, but belong to different VN IDs.
In this case, the packets are forwarded with the Layer-3 mechanism.
VM migration processing scenario: this scenario is not in the scope
of VM migration.
(3)Two VMs are on different servers, but belong to the same VN ID.
In this case, the packets are encapsulated on the local NVE, and
forwarded according to the outer layer addresses with the L2
mechanism. After they are delivered to the peer NVE, the outer layer
addresses are stripped, and then the packets are forwarded according
to the inner layer addresses.
VM migration processing scenario: it can be processed in the way as
across IP subnets, and IP-in-IP tunnel is needed.
(4)Two VMs are on different servers, and belong to different VN IDs.
In this case, the packets are encapsulated on the local NVE, and
forwarded according to the outer layer addresses with the L3
mechanism. After they are delivered to the peer NVE, the outer layer
addresses are stripped, and then the packets are forwarded according
to the inner layer addresses.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 16]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
VM migration processing scenario: it is similar with the third
scenario.
In the above cases, the processing of the outer layer information of
the packets, including L2 information( such as destination MAC,
source MAC, VLAN ID) and L3 information, follows the existing
mechanism, because the underlying network is transparent to the
overlay layer.
In the case of NVE on the server, the routing protocols (unicast and
multicast) may not be changed on the underlying network, but the
routing design of the overlay layer is subjected to considerable
restriction.
3.7.2. NVE/OBP on the ToR
In case of NVE on the ToR, NVE needs to handle the VIDs of various
packets. Once the VM is migrated, the rules of source network also
need to be migrated, causing physical network configuration changes.
Therefore, it is required to develop a set of rules to deal with it.
In this case, the source of the VIDs used by the VM please refer to
Section 4.5 (The Actual Number of Available Isolated Domains).
Various rules and usage range of the VIDs are required to be set.
The VLAN-ID used by a given VM refers to the VLAN-ID carried by the
traffic that is originated by that VM and within the same L2 physical
domain.
When a VM is communicating with the VM in a different VLAN, there are
two ways to implement before their communication messages enter into
the processing module in the overlay layer. Firstly, the
implementation is on Layer-2. One possible solution is that the port
on which the server hosting the VM is connected to ToR belongs to two
or more VLANs. Secondly, the implementation is on Layer-3. The VM
has multiple IP addresses and logical interfaces. The number of
addresses and interfaces are related to the number of communication
parties. These methods are similar to the approaches of conventional
Layer-2 forwarding and Layer-3 routing. After the processing, the
packets enter into the processing module in the overlay layer, and
then the header information is processed.
Scenario classification:
(1)Two VMs are on the servers connected to the ports on the same ToR
and these ports belong to the same VLAN.
In this case, the packets are forwarded according to the underlying
network address with the L2 mechanism.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 17]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
VM migration processing scenario: VM is migrated in the way as within
the same VLAN.
(2)Two VMs are on the servers connected to the ports on the same ToR
and these ports belong to different VLANs.
In this case, the packets are forwarded according to the underlying
network address with the L3 mechanism.
VM migration processing scenario: it can be processed in the way as
across IP subnets, and IP-in-IP tunnel is needed.
(3)Two VMs are on the servers connected to the ports on different
ToRs and these ports belong to the same VLAN.
In this case, the packets are encapsulated on the ToRs, then enter
into the processing in the underlying network. If the ToRs are
directly L2 connected, the packets are forwarded according to the
outer layer addresses with the L2 mechanism; if the ToRs are not
directly L2 connected, the packets are routed according to the outer
layer addresses with the L3 mechanism. After they are delivered to
the peer NVE, the outer layer addresses are stripped, and then the
packets are forwarded according to the inner layer addresses.
VM migration processing scenario: it is similar with the second
scenario.
(4)Two VMs are on the servers connected to the ports on different
ToRs and these ports belong to different VLANs.
In this case, the packets are encapsulated on the ToRs, and processed
in the underlying network. Since the ToRs are not directly L2
connected, the packets are routed according to the outer layer
addresses with the L3 mechanism. After they are delivered to the
peer NVE, the outer layer addresses are stripped, and then the
packets are forwarded according to the inner layer addresses.
VM migration processing scenario: it is similar with the second
scenario.
When a VM is communicating with the VM in a different VLAN, there are
two implementations before their communication messages enter into
the processing module in the overlay layer. Firstly, the
implementation is on Layer-2. One possible solution is that the port
belongs to two or more VLANs, on which the server hosting the VM is
connected to ToR. Secondly, the implementation is on Layer-3. The
VM has multiple IP addresses and logical interfaces. The number of
addresses and interfaces are related to the number of communication
parties. These methods are similar to the approaches of conventional
Layer-2 forwarding and Layer-3 routing. After the processing, the
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 18]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
packets enter into the processing module in the overlay layer, and
then the header information is processed.
3.7.3. Hybrid Scenario
The hybrid scenario should be considered in the communication model
of NVO3 network. This may include the situation when NVE is on the
Server in some part of the network, and NVE is on ToR in other part
of the network. The normal communication between two parts of the
network should be covered under the hybrid scenario discussion.
4. Technology Problems involved in the VM Migration
As mentioned above, when VMs are migrated from one server to another,
the source and destination server may be within the same DC, or in
the DCs at different geographic locations. The remote data centers
can be interconnected through different DCI technologies. Overall,
the technologies involved in the VM migration across DCs include
those within the DC. The key technologies involved in the VM
migration are broadly categorized according to different locations of
the elements in DCs as follows:
1)a) VM/Network-aware VM migration technologies, which makes the
network automatically perceive the network behavior of VM/Host (such
as QoS / VLAN, etc.). The network behavior is mainly perceived by
the switch which the server hosting VM is directly connected to.
Such technologies include IEEE EVB / VDP, PortExtender (802.1Qbh),
and so on.
b) Service-aware VM migration technologies, e.g., a distributed
cluster of VMs may be used for Unified Communications Services (UCS)
, and when any UCS related VMs need to be migrated, a set of VMs from
the clusters of the same category can be used.
2) The network interconnection driven technologies within the DC,
including IETF TRILL, IEEE 802.1Q/SPBV, and PBB / SPBM.
3) The internal exit device interconnection based technologies across
the DCs, including IETF VPLS, PBB-EVPN.
4) The ToR / Host interconnection technologies within the DC as well
as across the DCs, including the NVO3 which IETF is currently
standardizing.
The above-mentioned classification is according to the locations of
various elements in DC. The technologies in the second class or
third class can logically become one of the foundations of the
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 19]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
technologies in the fourth class.
The above-mentioned VM migration technologies may be considered to be
within the scope of NVO3 WG.
4.1. TRILL
The large-scale Layer-2 and multi-path interconnection in the DC
network can be implemented through TRILL. The transparent TRILL
network can be used to ensure that the MAC and IP remain unchanged in
the VM migration. However, in order to guarantee service continuity
during the VM migration, the control plane of NOV3 exchanges messages
with the control plane of TRILL, so that the position state can be
quickly updated after the VM migration. Business traffic can be
quickly pointed to the new network address of the VM after the
migration and tenants may not have any perception of the VM
migration.
4.2. SPB
As with TRILL, the large-scale Layer-2 and multi-path interconnection
in the DC network can be implemented through SPB. The transparent
SPB network can be used to ensure that the MAC and IP remain
unchanged in the VM migration. However, in order to guarantee
service continuity during the VM migration, the control plane of NOV3
exchanges messages with the control plane of SPB, so that the
position state can be quickly updated after the VM migration.
Business traffic can be quickly pointed to the new network address of
the VM after the migration and tenants may not have any perception of
the VM migration.
4.3. Data Center Interconnection Fabric Related Problems
One of the most important factors that directly impact the VMMI is
connectivity among the relevant data centers. There are many
features that determine this required connectivity. These features
of connectivity include bandwidth, security, quality of service, load
balancing capability, etc. These are frequently utilized to make
decision on whether a VM can join a host in real-time or it needs to
join VRF in certain unit of VM.
The requirements related to the above are as follows:
o The negative impact of ARP, MAC and IP entry explosion on the
individual network which contains a large number of tenants should be
minimized by DC and DC-interconnection technologies.
o The link capacity of both intra-DC and inter-DC network should be
effectively utilized. Efficient utilization of the link capacity
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 20]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
requires traffic forwarding on the shortest path between two VMs both
within the DC and across DCs. Therefore, Traffic should be forwarded
on the shortest path between two VMs within the DC or across DCs.
o Support of east-west traffic between tenants'applications located
in different DCs.
Many mature VPN technologies can be utilized to provide connectivity
between DCs. The extension of VLAN and virtual domain between DCs
may also be utilized for this purpose.
4.4. Types and Applications of VPN Interconnections between DCs which
Provide DCI
4.4.1. Types of VPNs
Related technologies of layer3 VPN: BGP / MPLS IP Virtual Private
Networks (VPNs), RFC 4364, etc.
Related technologies of layer2 VPN: PBB + L2VPN, TRILL + L2VPN, VLAN
+ L2VPN, NVGRE [draft-sridharan-virtualization-nvgre-00], PBB VPLS,
E-VPN, PBB-EVPN, VPLS, VPWS, etc.
4.4.2. Applications of L2VPN in DCs
It is a very common practice to use L2 interconnection technologies
for DC interconnection across geographical regions. Note that VPN
technology is also used to carry L2 and L3 traffic across the IP/MPLS
core network. This technology can be used in the same DC to support
scalability or interconnection across L3 domains. VPLS is commonly
used for IP/MPLS connection over WAN and it supports transparent LAN
services. IP VPN, including BGP / MPLS IP VPN and IPSec VPN, has
been used in a common IP/MPLS core network to provide virtual IP
routing instances.
The implementation of PBB plus L2-VPN can take advantage of some of
the existing technologies. It is flexible to use VPN network in the
cloud computing environment and can support a sufficient number of
VPN connections/sessions (networking resources), which is much larger
than the 4K VLAN mode of L2VPN. Therefore, the resulting effect is
similar to that of VXLAN.
Note that PBB can not only support access to more than 16M virtual
LAN instances, it can also separate the tenants and provide different
domains through isolated MAC address spaces.
The use of PBB encapsulation has one major advantage. Note that
since VM's MAC address will not be processed by ToRs and Core SWs,
MAC table size of ToRs and Core SWs may be reduced by two orders of
magnitude; the specific number is related with the number of VMs in
each server and VMs' virtual interfaces.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 21]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
One solution to solve problems in DC is to deploy other technologies
in the existing DC network. A service provider can separate its
domains of VLAN into different VLAN islands, in this way each island
can support up to 4K VLANs. Domains of VLAN can be interconnected
via VPLS, at the same time, DC GWs can be used as VPLS PEs.
If retaining the existing VLAN-based solutions only in VSw, while the
number of tenants in some VLAN islands is more than 4K, the service
provider needs to deploy VPLS deeper in the DC network. This is
equivalent to supporting L2VPN from the ToRs, and using the existing
VPLS solutions to enable MPLS for the ToR and core DC elements.
4.4.3. Applications of L3VPN in DCs
IP VPN technology can also be used for DC network virtualization.
For example, multi-tenant L3 virtualization can be achieved by
assigning a different IP VPN instance to each tenant who needs L3
virtualization in a DC network.
There are many advantages of using IP VPN as an L3 virtualization
solution within DC compared to using existing virtual routing
technology. Some of the advantages are as mentioned below:
(1) It supports many VRF-to-VRF tunneling options containing
different operational models: BGP/MPLS IP VPN, IP or L3 VPN GRE, etc.
(2) The connections of IP VPN instances used in Cloud services below
the WAN can be IP VPN that is directly involved in the WAN.
4.5. The Actual Number of Available Isolated Domains
The isolation of VMs is achieved through VNI. One way to acquire the
value of the VNI is through tag mapping in the underlying network.
VNI may be derived from the 12-bit VLAN ID, or from other related 24-
bit information (e.g. 24-bit PBB I-SID tag mapping, or other
information which can be mapped to VNI), It relates to the technical
solutions adopted by the Provider Edge Bridge and the degree of
chip's support. If VNI is from 12-bit VLAN ID, the actual number of
available VN IDs is 4096 in the application; if it is from 24-bit
VLAN ID, the actual number of available VN IDs is 16M in the
application. This is the compatibility issue between the isolated
domains in the overlay layer and the isolated domains in the
underlying network.
4.6. Problems of the Number of Management Devices/Management Domains of
DC in the NVO3 Network
The settings of management domains should be done in concert with the
control plane of NVO3 network. It relates to the management domain
of VM. The range of VM's management domain can be considered from
two aspects.
Firstly, it relates to whether only one management center (e.g.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 22]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
vCenter) is required. With VCenter as an example, a single vCenter
under the 32-bit OS can control 200 ESX Hosts and 2,000 VMs. If the
number of the ESX Hosts and VMs in the DC exceeds the quantity, there
should be multiple vCenters in place.
Secondly, it relates to the institutional settings. If the company
needs operations across the country or across the WAN, a single
vCenter can manage through the connections across the WAN, However,
multiple vCenters may be needed for hierarchical settings of global
organizational structure or the required security settings.
4.7. Limitation of TCAM capacity
Regardless of the locations of communication agents in the overlay
network, the number of 16M isolated domains is clearly beyond the
maximum TCAM capacity which hardware currently can support, so it is
necessary to consider the hierarchical virtual network.
4.8. SDN
If SDN technology is used for VM migration in Nvo3 network, it also
has to face the problem of limited TCAM capacity which hardware can
support, and the problem is more prominent.
4.9. LISP
If LISP is used an option in Nvo3 control plane, it will face the
problem of notify the ITR of fast switching to new Locator IP after
the migration. For example, a VM has just migrated from site A to
site B. The ITR do not know, and still use the old Locator IP to send
the packets. The applications are certainly interrupted and won't be
restored until the ITR acquires the new Locator IP. However, the
length of the interruption time has a great influence on the time-
sensitive operations.
5. Others Problems Related with IETF and IEEE
5.1. Review of VXLAN, NVGRE, and NVO3
In order to solve the problem of insufficient number of VLANs in DC,
the technologies like VXLAN and NVGRE have adopted two major
strategies; one of the strategies is the encapsulation and the other
is tunneling.
Both VXLAN and NVGRE use encapsulation and tunneling to create a
number of VLAN subnets, which can be extended to the Layer-2 and
Layer-3 networks. This solves the problem of limitation of the
number of VLAN as defined by IEEE802.1Q, and helps achieve shared
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 23]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
load-balancing in multi-tenant environment in both public and private
networks.
The VXLAN technology is introduced in 2011, and it is designed to
address the number restrictions of 802.1Q VLAN. The technologies
like MAC in MAC, MAC in GRE also extend the number of VLANs.
However, VXLAN attempts to address the issues related to inadequate
utilization of link resources, monitoring of packets after re-
encapsulation of header more effectively.
The frame format of VXLAN is the same as that of OTV and LISP,
although these three solutions solve different problems of DC
Interconnection and VM migration. Also, in VXLAN, the packet is
encapsulated in MAC in UDP, and addressing is extended to 24-bit,
which is the effective solution to the restrictions of VLAN number.
UDP encapsulation enables the logical virtual network extension to
different subnets. It also supports the migration of VMs across
subnets. The change of the frame's structure increases the field for
extending the VLAN.
Note that VXLAN solves different problem compared to OTV. OTV solves
the problem of DC interconnection, which builds an IP tunnel between
different data centers through MAC in IP. VXVLAN mainly solves the
problem of limitation of VLAN resources in DCs due to the increase in
the number of tenants. The key is the expansion of the VNI field to
increase the number of VLANs. Both technologies can be applied to VM
migration, since the two packet formats are almost the same and
completely compatible.
NVGRE specifies the 24-bit Tenant Network Identifier (TNI) and
resolves some issues related to supporting multiple tenants in DC
network. It uses GRE to create an independent virtual Layer-2
network, and limits physical Layer-2 network to expand across subnet
borders. Terminals supporting NVGRE insert the TNI indicators in the
GRE headers to separate the TNIs.
NVGRE and VXLAN solve the same problem. The two technologies were
proposed almost at the same time. However, there are some
differences between them:
VXLAN not only increases VXLAN header(VNI), but also increases the
outer UDP encapsulation on the packet, which facilitates live
migration of VMs across subnets. In addition, differentiated
services can be supported to the tenants in the same subnet because
of the use of UDP. Both proposals are built on the assumption that
load-balancing is the necessary condition to achieve efficient
operation. VXLAN randomly assigns port number to achieve load-
balancing, while NVGRE uses the retained 8-bit in the key GRE field.
However, there may be opportunity to improve the capability of the
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 24]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
control plane for both mechanisms in future.
5.2. The East-West Traffic Problem
Let us discuss the background of East-West traffic problem first.
There are a variety of applications in the DC, such as distributed
computing, distributed storage, and distributed search. These
applications and services need frequent exchanges of transactions
between the business servers across the DCs. According to the
traditional three-tier network model, the data stream first flows
north-south and then finally flows east-west. In order to improve
the forwarding efficiency of the data stream, it is necessary to
update the existing network model and network forwarding technology.
Among others, the Layer-2 multi-path technology being studied is one
of the directions to solve this problem.
Distributed computing is the basis of transformation of the existing
IT services. This allows scalable and efficient use of sometimes
underutilized computing and storage resources scattered across the
data centers.
In typical data centers, the average server utilization is often low
in the existing network. The concept of virtualization and
distributed computing can perfectly solve the problem of capacity
limitation of a single server in demanding environments in certain
DCs via on-demand utilization of resources and without impacting the
performance. This revolutionary technology of distributed computing
and services using resources in the DCs also produces several
horizontal flows of traffic. The application of distributed
computing technology on the servers produces a large number of
interactive traffic streams between servers. In addition, the types
of DC would influence the traffic model both within and across data
centers.
The first type of DC is telecom operators who usually not only
operate DCs, but also supply bandwidth for the Level-2 ISP providers.
The second type is the traditional ISP companies with strong power.
The third type is some IT enterprises which invest in the
construction of DCs. The fourth type is high-performance computing
(HPC) centers that are built by universities, research institutes and
organizations. Note that in these types of DCs, the south-north
traffic flow is significantly smaller compared to the horizontal
flow, and this brings greatest challenges to the network design and
installation. In addition to the normal flow of traffic due to the
distributed computing, storage, communications, and management, hot
backup and live VM migration produce a sudden lateral flow of traffic
and associated challenges.
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 25]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
There are two potential solutions to the distributed horizontal flow
of traffic, as described below.
A. The first one is to solve the problem of east-west traffic within
the server clusters by exploiting representative technologies such as
vswitch, Dcell, B-cube, and DCTCP .
B. The second solution is the network-based solution. The tree
structure of the traditional DC network is not inherently efficient
for horizontal flow of traffic. The problems can be solved in two
ways: (i)in the direction of radical changes: radical deformations in
changing the tree structure to multi-path, and (ii) in the direction
of mild improvement: changing L2 big trees to L2 small trees and
meeting the requirements by expanding the interconnection capacity of
the upper node, clustering/stacking system, and link trunking.
The requirements related to the above are as follows: Stacking
technology across the data center requires specialized interfaces,
and the length of feasible transmission distance is limited.
The problems related to the above statement include the following:
(a) although TRILL resolves the multi-path problem of Layer-2
protocol, it negatively impacts the multi-path properties of Layer-3
protocol. This is because only one active default router supports
Virtual Router Redundancy Protocol (VRRP), and this means that the
multi-path characteristics cannot be fully utilized in Layer-3
protocol. (b)TRILL does not define how to deal with the problem of
overlapping namespace.
5.3. The MAC, IP, and ARP Explosion Problems
Network devices within data centers encounter many problems for
supporting conventional communication framework because they need to
accommodate a huge number of IP, MAC addresses and ARP.
Each blade server in a network device usually supports at least 16-40
VMs, and each VM has its own MAC address and IP address. The
entities like Disk, memory, FDB table, MAC table, etc, cause an
increase in convergence time. In order to accommodate this large
number of the servers, different options for the network topology,
for example, fat tree topology or a conventional network topology may
be considered.
The number of ARP packets grows not only with the number of virtual
L2 domains or ELANs, which is instantiated on server, but also with
the number of VMs in that domain. Therefore, scenarios like overload
of ARP entries on the servers/hypervisors, exhaustion of ARP entries
on the routers/PEs, and processing overload of L3 service appliances,
must be efficiently resolved. Otherwise, these problems will easily
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 26]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
propagate throughout the layer-2 switching network.
Consequently, what are needed to resolve these problems include (a)
automated management of MAC/IP/ARP in DCs, and (b) network deployment
that will reduce the explosion in MAC number requirements in DCs.
5.4. Suppressing Flooding within a VLAN
Efficient operations of DCs in nvo3 network require that flooding of
broadcast, multicast and unknown unicast frames within a VLAN (that
may be caused by the improper configuration) should be reduced.
5.5. Packet Encapsulation Problems
In order to achieve seamless migration of VMs across DCs that support
different VLAN expansion mechanisms, unification of packet
encapsulation methods is required.
6. Acknowledgement
The following experts have provided valuable comments on the earlier
version of this draft: Thomas Narten, Christopher LILJENSTOLPE,
Steven Blake, Ashish Dalela, Melinda Shore, David Black, Joel M.
Halpern, Vishwas Manral, Lizhong Jin, Juergen Schoenwaelder, Donald
Eastlake, and Truman Boyes. We express our sincere thanks to them,
and expect that they will continue to provide suggestions in future.
7. References
[PBB-VPLS] Balus, F. et al. "Extensions to VPLS PE model for
Provider
Backbone Bridging", draft-ietf-l2vpn-pbb-vpls-pe-model-
04.txt (work in progress), October 2011.
[DCN Ops Req] A. Dalela. "Datacenter Network and Operations
Requirements",
draft-dalela-dc-requirements-00.txt, December 30, 2011
[VPN Applicability] Nabil Bitar. "Cloud Networking: Framework and
VPN Applicability",
draft-bitar-datacenter-vpn-applicability-01.txt, October 2011
[VXLAN] M.Mahalingam. "VXLAN: A Framework for Overlaying Virtualized
Layer 2 Networks over Layer 3 Networks",
draft-mahalingam-dutt-dcops-vxlan-01.txt, February 24, 2012
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 27]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
[NIST 800-145] NIST Special Publication 800-145, Peter Mell and
Timothy Grance, The NIST definition of cloud computing,
http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf,
September 2011
[Cloud/DataCenter SDO Survey] B. Khasnabish and C. JunSheng. "Cloud/
DataCenter SDO Activities Survey and Analysis",
draft-khasnabish-cloud-sdo-survey-02.txt, December 28, 2011
[NVGRE] M. Sridharan. "NVGRE: Network Virtualization using Generic
Routing Encapsulation",
draft-sridharan-virtualization-nvgre-00.txt, September 2011
[NVO3] Thomas Narten. " NVO3: Network Virtualization", l2vpn-9.pdf,
November 2011
8. Security Considerations
To be added later, on as-needed basis.
9. IANA Consideration
The extensions that are discussed in this draft are related to DC
operations environment.
10. Normative References
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006.
Authors' Addresses
Bhumip Khasnabish
ZTE USA,Inc.
55 Madison Avenue, Suite 160 Morristown, NJ 07960
USA
Phone: +001-781-752-8003
Email: vumip1@gmail.com, bhumip.khasnabish@zteusa.com
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 28]
Internet-Draft Mobility and Interconnection of VM & VNE Dec 2012
Bin Liu
ZTE Corporation
15F, ZTE Plaza, No.19 East Huayuan Road,Haidian District
Beijing 100191
P.R.China
Phone: +86-10-59932098
Email: richard.bohan.liu@gmail.com,liu.bin21@zte.com.cn
Baohua Lei
China Telecom
118, St. Xizhimennei, Office 709, Xicheng District
Beijing
P.R.China
Phone: +86-10-58552124
Email: leibh@ctbri.com.cn
Feng Wang
China Telecom
118, St. Xizhimennei, Office 709, Xicheng District
Beijing
P.R.China
Phone: +86-10-58552866
Email: wangfeng@ctbri.com.cn
Bhumip Khasnabish, et al. Expires July 3, 2013 [Page 29]
