Considerations for Benchmarking Network Performance in Containerized Infrastructures
draft-dcn-bmwg-containerized-infra-00

Abstract

This draft describes benchmarking considerations for a containerized infrastructure. In a containerized infrastructure, Virtualized Network Functions(VNFs) are deployed on operating-system-level virtualization platform by abstracting the user namespace as opposed to virtualization using a hypervisor. Leveraging this, the system configurations and networking scenarios for VNF benchmarking will be partially changed by way of resource allocation and network port binding between a physical host and VNFs. In this draft we compare the state of the art in container networking architecture with networking on VM-based virtualized systems, and provide several test scenarios for network performance in containerized infrastructure.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on September 8, 2019.

Copyright Notice

Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Terminology
• 3. Benchmarking Consideration
• 3.1. Comparison with VM based Infrastructure
• 3.2. Additional Considerations for Container Networking
• 4. Test Scenarios
• 5. Security Considerations
• 6. Informative References
• Authors' Addresses

1. Introduction

The Benchmarking Methodology Working Group(BMWG) has recently expanded its benchmarking scope from Physical Network Function(PNF) running on dedicated hardware system to Network Function Virtualization(NFV) infrastructure and Virtualized Network Function(VNF). [RFC8172] described considerations for configuring NFV infrastructure and benchmarking metrics, and [RFC8204] gives guidelines for benchmarking virtual switch which connects VNFs in Open Platform for NFV(OPNFV).

Recently NFV infrastructure has evolved to include a lightweight virtualized platform called the containerized infrastructure, where VNFs share the same host Operating System(OS) and they are logically isolated by using a different namespace. While previous NFV infrastructure uses a hypervisor to allocate resources for Virtual Machine(VMs) and instantiate VNFs on it, the containerized infrastructure virtualizes resources without a hypervisor, therefore making containers very lightweight and more efficient in infrastructure resource utilization compared to a VM based NFV infrastructure. When we consider benchmarking for VNFs in the containerized infrastructure, it may have a different Device Under Test(DUT) configuration compared with both black-box benchmarking and VM-based NFV infrastructure as described in [RFC8172]. Accordingly, additional configuration parameters and testing strategies may be required.

In the containerized infrastructure, a VNF network is implemented by running both switch and router functions in the host system. For example, the internal communication between VNFs in the same host uses the L2 bridge function, while communication with external node(s) uses the L3 router function. For container networking, the host system may use a virtual switch(vSwitch), but other options exist. In the [ETSI-TST-009], they describe differences in networking structure between VM-based and container-based infrastructure. Occasioned by these differences, deployment scenarios for testing network performance described in [RFC8204] may be partially applied to the containerized infrastructure, but other scenarios may be required.

In this draft, we describe differences and additional considerations for benchmarking containerized infrastructure based on [RFC8172] and [RFC8204]. In particular, we focus on differences in system configuration parameters and networking configurations of the containerized infrastructure compared with VM-based NFV infrastructure. Note that, although the detailed configurations of both infrastructures differ, the new benchmarks and metrics defined in [RFC8172] can be equally applied in containerized infrastructure from a generic-NFV point of view, and therefore defining additional metrics or methodologies is out of scope.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document is to be interpreted as described in [RFC2119]. This document uses the terminology described in [RFC8172], [RFC8204], [ETSI-TST-009].

3. Benchmarking Consideration

3.1. Comparison with VM based Infrastructure

For benchmarking of containerized infrastructure, as mentioned in [RFC8172], the basic approach is to reuse existing benchmarks developed within the BMWG. Various network function specifications already defined in BMWG should still be applied to containerized VNFs for performance comparison with physical network functions and VM based VNFs.

+---------------------------------+  +--------------------------------+
|+--------------+ +--------------+|  |+------------+    +------------+|
||   Guest VM   | |   Guest VM   ||  || Container  |    | Container  ||
||+------------+| |+------------+||  ||+----------+|    |+----------+||
|||     APP    || ||     APP    |||  |||   APP    ||    ||   APP    |||
||+------------+| |+------------+||  ||+----------+|    |+----------+||
||+------------+| |+------------+||  ||+----------+|    |+----------+||
|||Guest Kernel|| ||Guest Kernel|||  ||| Bin/Libs ||    || Bin/Libs |||
||+------------+| |+------------+||  ||+----------+|    |+----------+||
|+------^-------+ +-------^------+|  |+-----^------+    +------^-----+|
|+------|-----------------|------+|  |+-----|------------------|-----+|
||      |    Hypervisor   |      ||  ||     |+----------------+|     ||
|+------|-----------------|------+|  ||     ||Container Engine||     ||
|+------|-----------------|------+|  ||     |+----------------+|     ||
||      | Host OS Kernel  |      ||  ||     |  Host OS Kernel  |     ||
|+------|-----------------|-----+||  |+-----|------------------|-----+|
|    +--v-----------------v--+    |  |  +---v------------------v---+  |
+----|    physical network   |----+  +--|    physical network      |--+
     +-----------------------+            +--------------------------+   
    (a) VM-Based Infrastructure       (b) Containerized Infrastructure
                        

Figure 1: Comparison of NFV Infrastructures

In Figure 1, we describe two different NFV architectures: VM-based and Containerized. A major distinction between containerized infrastructure and VM based infrastructure is that with the former, all VNFs share the same host resources including but not limited to computing, storage and networking resources, as well as the host Operating System(OS), kernel and libraries. The absence of the guest OS and the hypervisor, necessitates the following considerations that occur in the test environment:

o Concerning hardware for containerized infrastructure, all components described in [RFC8172] can be part of the test setup. While the capabilities of servers and storage should meet the minimum requirements for testing, it is possible to deploy a test environment with less capabilities than in a VM based infrastructure.

o About configuration parameters, containerized infrastructure needs specified management system instead of hypervisor(e.g. Linux Container, Docker Engine).

o In the VM based infrastructure, each VM has packet processing in the kernel of the guest OS through its own CPU threads, virtualized and assigned by hypervisor. On the other hand, containerized VNFs use the host CPU without virtualization. Different CPU resource assignment methods may have different CPU utilization perspectives for VNF performance benchmarking.

o From a Memory Management Unit(MMU) point of view, there is a difference in how the paging process is conducted between two environments. The main difference lies in the isolated nature of the OS for VM-based VNFs. In the containerized infrastructure, memory paging which processes conversion between physical address and virtual address is affected by the host resource directly. Thus, memory usage of each VNFs is more dependent on the host resource capabilities than in VM-based VNFs.

o Some network drivers may have varying dependencies for each environment. For example, a vhost-net driver used in a guest OS cannot be used for a container; on the other hand, a veth driver can be only applicable within a containerized infrastructure.

3.2. Additional Considerations for Container Networking

In the containerized infrastructure, there are various network architectures depending on the deployment environment and models. Since container networking typically involves using virtual switch functions, base network configuration parameters for container networking benchmarks are mostly similar with VM based VNF networking described in [RFC8204]. Additional considerations for container networking are described as follows:

o Networking depends on deployment models: Containerized VNFs have several deployment models. Containerized VNFs can be deployed as a cluster called POD by Kubernetes, otherwise each VNF can be deployed separately using Docker. In former case, there is only one external network interface for a POD which contains more than one VNF. An alternative deployment model considers a scenario in which containerized VNFs or PODs are running on VM-based infrastructure. Figure 2 shows briefly differences of network architectures based on deployment models. [ETSI-TST-009] describes in more detail the differences between them. Other deployment models are classified bases on whether containerized VNFs are deployed on baremetal or inside of the VM.

+---------------------------------------------------------------------+
|                          Baremetal Node                             |
|                                                                     |
| +--------------+ +--------------+ +-------------- + +-------------+ |
| |              | |     POD      | |      VM       | |     VM      | |
| |              | |+------------+| |+-------------+| |  +------+   | |
| |   Container  | ||  Container || ||Container VNF|| |  | PODs |   | |
| |     VNF      | ||    VNFs    || |+-----^-------+| |  +---^--+   | |
| |              | |+------------+| |      |        | |      |      | |
| |   +------+   | |   +------+   | |   +--v---+    | |  +---v--+   | |
| +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ |
|     +--^---+         +---^--+         +--^---+         +---^--+     |
|        |                 |               |                 |        |
|        |                 |            +--v---+         +---v--+     |
| +------|-----------------|------------|vhost |---------|vhost |---+ |
| |      |                 |            +--^---+         +---^--+   | |
| |      |                 |               |                 |      | |
| |   +--v---+         +---v--+         +--v---+         +---v--+   | |
| | +-| veth |---------| veth |---------| Tap  |---------| Tap  |-+ | |
| | | +--^---+         +---^--+         +--^---+         +---^--+ | | |
| | |    |                 |    vSwitch    |                 |    | | |
| | | +--|-----------------|---------------|-----------------|--+ | | |
| | +-|  |                 |    Bridge     |                 |  |-+ | |
| |   +--|-----------------|---------------|-----------------|--+   | |
| |      |   +---------+   |            +--|-----------------|---+  | |
| |      |   |Container|   |            |  |    Hypervisor   |   |  | |
| |      |   | Engine  |   |            |  |                 |   |  | |
| |      |   +---------+   |            +--|-----------------|---+  | |
| |      |                 |  Host Kernel  |                 |      | |
| +------|-----------------|---------------|-----------------|------+ |
|     +--v-----------------v---------------v-----------------v--+     |
+-----|                      physical network                   |-----+
      +---------------------------------------------------------+      
                        

Figure 2: Examples of Networking Architecture based on Deployment Models

o Network Plug-ins: In the containerized infrastructure, specific networking functions can be supported by attaching various plug-ins. Container Network Model(CNM) and Container Network Interface(CNI) are currently the most popular network plug-ins. According each network plug-in, they have different runtime structure or accessibilities to namespace. Actual testing results may vary depending on plug-in types and its supporting drivers.

o Network Types: To enhance forwarding capabilities, similar to the VM based infrastructure, the containerized infrastructure can also employ use of specific networking technologies such as SR-IOV.

4. Test Scenarios

TBD

5. Security Considerations

TBD

6. Informative References

Authors' Addresses