Internet DRAFT - draft-liu-nvo3-ps-vxlan-perfomance
draft-liu-nvo3-ps-vxlan-perfomance
Network Working Group Vic Liu
Internet Draft China Mobile
Intended status: Informational Bob Mandeville
Iometrix
Brooks Hickman
Spirent Communications
Weiguo Hao
Huawei Technologies
Zu Qiang
Ericsson
Expires: January 2015 July 3, 2014
Problem Statement for VxLAN Performance Test
draft-liu-nvo3-ps-vxlan-perfomance-00.txt
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be modified,
and derivative works of it may not be created, and it may not be
published except as an Internet-Draft.
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be modified,
and derivative works of it may not be created, except to publish it
as an RFC and to translate it into languages other than English.
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
<Lastname> Expires January 3, 2015 [Page 1]
Internet-Draft PS for VxLAN Performance July 2014
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on January 3,2015.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document.
Abstract
As the number of data center tenant increased, 4K VLANs, mobility,
broadcasting issues have become the network bottleneck. VxLAN has
being take into consideration in China Mobile IDC.
Vic Liu Expires January 3, 2015 [Page 2]
Internet-Draft PS for VxLAN Performance July 2014
There are two implementation solutions for VXLAN. The first one is
that NVE resides in TOR (top of rack switch), another one is that
NVE resides in V-Switch established in hypervisor of physical server.
For virtualized network, it's much better to implement NVE in V-
switch because it's directly connect with Virtual Machines and
easier for business to carry on. As we research in VxLAN solution,
some problems are take into our considerations. How much resources
will be consumed by VxLAN in a virtualized network environment. This
draft introduces the problem for VxLAN performance by test.
There is no methodology which can effectively evaluate VxLAN
forwarding performance. This draft also attempts to address this
issue, give a VXLAN performance evaluation method, especially when
VxLAN resides in the virtual switch.
Table of Contents
1. Introduction ................................................ 3
2. Consideration on VxLAN performanc ........................... 4
2.1. Test methodology for VxLAN performance in virtual network 4
2.2. Large-scale VxLAN test issues ........................... 4
2.3. Key index in VxLAN performance .......................... 5
2.4. Test Bed Setup ......................................... 5
2.5. Benchmark test on virtualized network ................... 8
3. Problem statement on VxLAN performance ....................... 9
3.1. VxLAN performance on test bed ........................... 9
3.2. VxLAN Scalable test issues ............................. 11
4. Security Considerations ..................................... 11
5. IANA Considerations ........................................ 11
6. References ................................................. 11
6.1. Normative References ................................... 11
6.2. Informative References ................................. 11
7. Acknowledgments ............................................ 12
1. Introduction
As the number of data center tenant increased, 4K VLANs, mobility,
broadcasting issues have become the network bottleneck. VxLAN has
being take into consideration in China Mobile IDC.
There are two implementation solutions for VXLAN. The first one is
that NVE resides in TOR (top of rack switch), another one is that
NVE resides in V-Switch established in hypervisor of physical server.
For virtualized network, it's much better to implement NVE in V-
switch because it's directly connect with Virtual Machines and
Vic Liu Expires January 3, 2015 [Page 3]
Internet-Draft PS for VxLAN Performance July 2014
easier for business to carry on. As we research in VxLAN solution,
some problems are take into our considerations. How much resources
will be consumed by VxLAN in a virtualized network environment. This
draft introduces the problem for VxLAN performance by test.
There is no methodology which can effectively evaluate VxLAN
forwarding performance. This draft also attempts to address this
issue, give a VXLAN performance evaluation method, especially when
VxLAN resides in the virtual switch.
2. Consideration on VxLAN performance
While we testing performance on virtualized network, some issues and
key index should be considered clearly.
2.1. Test methodology for VxLAN performance in virtual network
It's different from test for physical switch. Because firstly in
virtual network, the DUT (VxLAN on V-switch), hypervisor and virtual
test center (it's a VM) is all in one physical server. Secondly,
it's not like RFC 2544 that the test center generate line rate
traffic(usually 1G or 10G) and test the physical server's
performance. As we generate traffic from one server to another
(model A below), it has a fold point during traffic increase from 1G
to 10G because the vCPU is overloading. For example, server A
generate 1G traffic and server B can receive 100%, but server A
generate 10G traffic and server B can only receive 530Mb traffic.
So in this test, the test process is designed as follows:
a) Firstly use the server to connect with a physical test center.
b) Make a traffic benchmark of 128, 256, 512, 1024, 1518bytes.
c) Setup the test bed this the benchmark to get performance
without VxLAN.
d) Setup VxLAN and running the same performance test.
2.2. Large-scale VxLAN test issues
When test the scale of VLANs, it can be simulate 4K VLAN on the test
center. But, in virtual network, the virtual center is a virtual
machine. And virtual machine can only establish one VxLAN with
VSwitch. So it can't test the large scale VxLAN performance.
Vic Liu Expires January 3, 2015 [Page 4]
Internet-Draft PS for VxLAN Performance July 2014
2.3. Key index in VxLAN performance
a) CPU: CPU utilization is very important for VxLAN. However, vCPU
can be allocated for VM. But it cannot allocated for hypervisor
and VSwitch. We use the test methodology to evaluate the CPU
performance for VxLAN.
b) Memory: Memory is not sensitive from the test result. But we
still think it should be listed as one VxLAN performance index.
c) Latency: When traffic is forwarded between VM to VM across two
different physical server. Latency should be an index.
d) Throughput: We use the benchmark as the traffic throughput.
e) Packet-lost: Virtual network may have few packet-lost because of
unstable of vCPU. Less than 2% of packet-lost is acceptable.
2.4. Test Bed Setup
The test bed includes two physical server with 10GE NIC, a test
center, a 10GE TOR switch for test traffic and a 1GE TOR switch for
management.
----------------------
|Test Center PHY 10GE*2|
----------------------
||
||
----------
=====| 10GE TOR |=======
|| ---------- ||
|| ||
|| ||
------------------- -------------------
| -------------- | | -------------- |
| |V-switch(VTEP)| | | |V-switch(VTEP)| |
| -------------- | | -------------- |
| | | | | | | |
| ----- ----- | | ----- ----- |
| |TCVM1| |TCVM2|| | |TCVM1| |TCVM2||
| ----- ----- | | ----- ----- |
Vic Liu Expires January 3, 2015 [Page 5]
Internet-Draft PS for VxLAN Performance July 2014
------------------- -------------------
Server1 Server2
Vic Liu Expires January 3, 2015 [Page 6]
Internet-Draft PS for VxLAN Performance July 2014
Two Dell server are R710XD (CPU: E5-2460) and R710 (CPU: E5-2430)
with a pair of 10GE NIC. And in the server we allocate 2 vCPU and 8G
memory to each Test Center Virtual Machine (TCVM).
In traffic model A: We use a physical test center connect to each
server to verify the benchmark of each server.
----------------------
|Test Center PHY 10GE*2|
----------------------
||
||
-------------------
| -------------- |
| |V-switch(VTEP)| |
| -------------- |
| | | |
| ----- ----- |
| |TCVM1| |TCVM2||
| ----- ----- |
-------------------
Server1
In traffic model B: We use the traffic model A benchmark to test the
performance of VxLAN.
----------
=====| 10GE TOR |=======
|| ---------- ||
|| ||
|| ||
------------------- -------------------
| -------------- | | -------------- |
| |V-switch(VTEP)| | | |V-switch(VTEP)| |
| -------------- | | -------------- |
| | | | | | | |
| ----- ----- | | ----- ----- |
| |TCVM1| |TCVM2|| | |TCVM1| |TCVM2||
| ----- ----- | | ----- ----- |
------------------- -------------------
Server1 Server2
Vic Liu Expires January 3, 2015 [Page 7]
Internet-Draft PS for VxLAN Performance July 2014
2.5. Benchmark test on virtualized network
The reason we need a benchmark test is we realized that the
virtualized network is different from physical network device. We
cannot use test methodology like RFC 2544. The performance is not
linear growth with traffic we generate. It has an inflection point.
To get the benchmark, we use traffic model A and get the result
table below:
Server 1: CPU E5-2430
------------------------------------------------------------
| Byte| Rate(GE)| Server CPU MHZ |Server Mem| VM CPU| VM Mem|
------------------------------------------------------------
| 0 | 0 | 505 | 3022 | 372 | 695 |
------------------------------------------------------------
| 128 | 0.46 | 6085 | 3021 | 5836 | 695 |
------------------------------------------------------------
| 256 | 0.84 | 6365 | 3021 | 6143 | 696 |
------------------------------------------------------------
| 512 | 1.56 | 6330 | 3021 | 6099 | 696 |
------------------------------------------------------------
| 1024| 2.88 | 5922 | 3021 | 5726 | 696 |
------------------------------------------------------------
| 1518| 4.00 | 5713 | 3023 | 5441 | 696 |
------------------------------------------------------------
Server 2: CPU E5-2620
------------------------------------------------------------
| Byte| Rate(GE)| Server CPU MHZ |Server Mem| VM CPU| VM Mem|
------------------------------------------------------------
| 0 | 0 | 505 | 2900 | 239 | 698 |
------------------------------------------------------------
| 128 | 0.61 | 5631 | 2900 | 5117 | 698 |
------------------------------------------------------------
| 256 | 0.94 | 5726 | 2896 | 5157 | 698 |
------------------------------------------------------------
| 512 | 2.02 | 5786 | 2901 | 5217 | 698 |
------------------------------------------------------------
| 1024| 4.02 | 5884 | 2901 | 5097 | 698 |
------------------------------------------------------------
| 1518| 5.61 | 5856 | 2901 | 5197 | 698 |
Vic Liu Expires January 3, 2015 [Page 8]
Internet-Draft PS for VxLAN Performance July 2014
------------------------------------------------------------
As we get the benchmark, we use the lower benchmark (server1) in
traffic model B and test VxLAN performance.
3. Problem statement on VxLAN performance
3.1. VxLAN performance on test bed
We use the lower benchmark (server 1) to generate traffic from
server 1 to server 2 with VxLAN encapsulation and get the
performance result of the two servers. And because of VxLAN
encapsulation increases the packet length, to avoid MTU problem we
use 1450 instead 1518 as original packet length.
Server 1 with VxLAN: CPU E5-2430
------------------------------------------------------------
| Byte| Rate(GE)| Server CPU MHZ |Server Mem| VM CPU| VM Mem|
------------------------------------------------------------
| 0 | 0 | 515 | 3042 | 374 | 696 |
------------------------------------------------------------
| 128 | 0.46 | 6395 | 3040 | 5748 | 696 |
------------------------------------------------------------
| 256 | 0.84 | 6517 | 3042 | 5923 | 696 |
------------------------------------------------------------
| 512 | 1.56 | 6668 | 3041 | 5857 | 696 |
------------------------------------------------------------
| 1024| 2.88 | 6280 | 3043 | 5506 | 696 |
------------------------------------------------------------
| 1450| 4.00 | 6233 | 3045 | 5309 | 696 |
------------------------------------------------------------
Server 2: CPU E5-2620
------------------------------------------------------------
| Byte| Rate(GE)| Server CPU MHZ |Server Mem| VM CPU| VM Mem|
------------------------------------------------------------
| 0 | 0 | 450 | 2905 | 239 | 698 |
------------------------------------------------------------
| 128 | 0.46 | 6203 | 2905 | 5897 | 698 |
------------------------------------------------------------
| 256 | 0.84 | 5937 | 2906 | 5797 | 698 |
------------------------------------------------------------
| 512 | 1.56 | 5993 | 2909 | 5737 | 698 |
------------------------------------------------------------
| 1024| 2.88 | 5710 | 2912 | 5697 | 698 |
Vic Liu Expires January 3, 2015 [Page 9]
Internet-Draft PS for VxLAN Performance July 2014
------------------------------------------------------------
| 1450| 4.00 | 5863 | 2902 | 5697 | 698 |
------------------------------------------------------------
By analyzing the testing result, we have conclusion as follows:
a) CPU: VxLAN function resided in VSwitch increases physical CPU
usage. The table below shows the increasing percentage of CPU
usage after using one VxLAN ID. The average increase is 6.51% in
server 1 and 4.07% in server 2. This increase is cost by one VxLAN.
We will still evaluate increase in large scale VxLAN scenario.
----------------------------
| Byte| Server 1 | Server 2 |
----------------------------
| 0 | 4.04% | 24.65% |
----------------------------
| 128 | 7.57% | 7.26% |
----------------------------
| 256 | 1.15% | 7.59% |
----------------------------
| 512 | 8.66% | 2.41% |
----------------------------
| 1024| 4.49% | 0.14% |
----------------------------
| 1450| 11.61% | 1.84% |
****************************
|*AVG | 6.51% | 4.07% |
----------------------------
b) Memory: Memory of both physical server and virtual machine are
not sensitive during VxLAN test.
c) Packet-loss: Because virtual network is based on X86 architecture.
When vCPU utilizing rate reaches over 90%, there will be about 2%
packet-loss. It is different with VXLAN on physical switch that no
pack-lost forwarding is a necessary requirement.
d) Line-rate forwarding: It is well known to us, in virtual network,
traffic become unstable as CPU goes overload. Whatever we try, we
can't reach line rate using any packet length. Finally, we reach
10Gb using 1518 byte without VxLAN between two server by add 3
pair of TCVM and each TCVM allocated 2 vCPU and 8G memory. While
we add VxLAN and decrease 1518 to 1450(in case of fragment), on
same network, we can only get 5.6 Gb unstable throughput.
Vic Liu Expires January 3, 2015 [Page 10]
Internet-Draft PS for VxLAN Performance July 2014
3.2. VxLAN Scalable test issues
All the tests above are based on one VN. As we considering Multi-VN
scenario. One problem we can't overlook is, the VSwitch can only
recognize (or study) the VN ID from VNIC of VM (TCVM). As we
generate thousands of VxLAN by the TCVM, none can be studied by
VSwitch except the VxLAN to VNIC. We calculate, one VM can provide
10 VNIC (MAX) which allocate 10 VxLAN, and one physical server
install 20 VM. If we make a 5000VxLAN scale performance test, there
will be at least 25 server.
4. Security Considerations
5. IANA Considerations
6. References
6.1. Normative References
[1] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[2] Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Syntax Specifications: ABNF", RFC 2234, Internet Mail
Consortium and Demon Internet Ltd., November 1997.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2234] Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Syntax Specifications: ABNF", RFC 2234, Internet Mail
Consortium and Demon Internet Ltd., November 1997.
6.2. Informative References
[3] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP
and Its Effect on Busy Servers", Proc. Infocom 1999 pp. 1573-
1583.
Vic Liu Expires January 3, 2015 [Page 11]
Internet-Draft PS for VxLAN Performance July 2014
[Fab1999] Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in
TCP and Its Effect on Busy Servers", Proc. Infocom 1999 pp.
1573-1583.
7. Acknowledgments
<Add any acknowledgements>
Authors' Addresses
Vic Liu
China Mobile
32 Xuanwumen West Ave, Beijing, China
Email: liuzhiheng@chinamobile.com
Bob Mandeville
Iometrix
3600 Fillmore Street
Suite 409
San Francisco, CA 94123
USA
bob@iometrix.com
Brooks Hickman
Spirent Communications
1325 Borregas Ave
Sunnyvale, CA 94089
USA
Brooks.Hickman@spirent.com
Weiguo Hao
Huawei Technologies
101 Software Avenue, Nanjing 210012, China
Email: haoweiguo@huawei.com
Zu Qiang
Ericsson
8400, boul. Decarie
Ville Mont-Royal, QC,
Canada
Email: Zu.Qiang@Ericsson.com
Vic Liu Expires January 3, 2015 [Page 12]
