VNF Benchmarking Methodology
draft-rosa-bmwg-vnfbench-01
This document describes a common methodology for benchmarking Virtualized Network Functions (VNFs) in general-purpose hardware. Specific cases of benchmarking methodologies for particular VNFs can be derived from this document. An open source reference implementation called Gym is reported as a running code embodiment of the proposed methodology for VNFs.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 4, 2018.
Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Benchmarking Methodology Working Group (BMWG) initiated efforts, approaching considerations in [RFC8172], to develop methodologies for benchmarking VNFs. Similarly described in [RFC8172], VNF benchmark motivating aspects define: (i) pre-deployment infrastructure dimensioning to realize associated VNF performance profiles; (ii) comparison factor with physical network functions; (iii) and output results for analytical VNF development.
Having no strict and clear execution boundaries, different from earlier self-contained black-box benchmarking methodologies described in BMWG, a VNF depends on underlying virtualized environment parameters [ETS14a], intrinsic considerations for analysis when addressing performance. This document stands as a ground methodology guide for VNF benchmarking. It addresses the state-of-the-art publications and the current developments in similar standardization efforts (e.g., [ETS14c] and [RFC8204]) towards bechmarking VNFs.
Common benchmarking terminology contained in this document is derived from [RFC1242]. Also, the reader is assumed to be familiar with the terminology as defined in the European Telecommunications Standards Institute (ETSI) NFV document [ETS14b]. Some of these terms, and others commonly used in this document, are defined below.
- NFV:
- Network Function Virtualization - The principle of separating network functions from the hardware they run on by using virtual hardware abstraction.
- NFVI PoP:
- NFV Infrastructure Point of Presence - Any combination of virtualized compute, storage and network resources.
- NFVI:
- NFV Infrastructure - Collection of NFVI PoPs under one orchestrator.
- VIM:
- Virtualized Infrastructure Manager - functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one operator's Infrastructure Domain (e.g. NFVI-PoP).
- VNFM:
- Virtualized Network Function Manager - functional block that is responsible for controlling and managing the VNF life-cycle.
- NFVO:
- NFV Orchestrator - functional block that manages the Network Service (NS) life-cycle and coordinates the management of NS life-cycle, VNF life-cycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity.
- VNF:
- Virtualized Network Function - a software-based network function.
- VNFD:
- Virtualised Network Function Descriptor - configuration template that describes a VNF in terms of its deployment and operational behaviour, and is used in the process of VNF on-boarding and managing the life cycle of a VNF instance.
- VNF-FG:
- Virtualized Network Function Forwarding Graph - an ordered list of VNFs creating a service chain.
This document assumes VNFs as black boxes when defining VNF benchmarking methodologies. White box approaches are assumed and analysed as a particular case under proper considerations of internal VNF instrumentation.
VNF benchmarking considerations are defined in [RFC8172]. Additionally, VNF pre-deployment testing considerations are well explored in [ETS14c].
Following the ETSI's model in [ETS14c], we distinguish three methods for VNF evaluation:
- Benchmarking:
- Where parameters (e.g., cpu, memory, storage) are provided and the corresponding performance metrics (e.g., latency, throughput) are obtained. Note, such request might create multiple reports, for example, with minimal latency or maximum throughput results.
- Verification:
- Both parameters and performance metrics are provided and a stimulus verify if the given association is correct or not.
- Dimensioning:
- Where performance metrics are provided and the corresponding parameters obtained. Note, multiple deployment interactions may be required, or if possible, underlying allocated resources need to be dynamically altered.
Note: Verification and Dimensioning can be reduced to Benchmarking. Therefore, we detail Benchmarking in what follows.
A generic VNF benchmarking setup is shown in Figure 1, and its components are explained below. Note here, not all components are mandatory, and VNF benchmarking scenarios, further explained, can dispose components in varied settings.
+---------------+
| Manager |
Control | (Coordinator) |
Interface +---+-------+---+
+--------+-----------+ +-------------------+
| | |
| | +-------------------------+ |
| | | System Under Test | |
| | | | |
| | | +-----------------+ | |
| +--+------- + | | |
| | | VNF | | |
| | | | | |
| | +----.---------.--+ | |
+-----+---+ | Monitor | : : | +-----+----+
| Agent | |{listeners}|----^---------V--+ | | Agent |
|(Sender) | | | Execution | | |(Receiver)|
| | | | Environment | | | |
|{Probers}| +-----------| | | |{Probers} |
+-----.---+ | +----.---------.--+ | +-----.----+
: +---------^---------V-----+ :
V : : :
:................>.....: :............>..:
Stimulus Traffic Flow
Figure 1: Generic VNF Benchmarking Setup
- Agent --
- executes active stimulus using probers, benchmarking tools, to benchmark and collect network and system performance metrics. While a single Agent is capable of performing localized benchmarks (e.g., stress tests on CPU, memory, disk I/O), the interaction among distributed Agents enable the generation and collection of end-to-end metrics (e.g., frame loss rate, latency). In a deployment scenario, one Agent can create the benchmark stimuli and the other end be the VNF itself where, for example, one-way latency is evaluated. A prober defines a software/hardware-based tool able to generate traffic specific to a VNF (e.g., sipp) or generic to multiple VNFs (e.g., pktgen). An Agent can be defined by a physical or virtual network function.
- Monitor --
- when possible, it is instantiated inside the target VNF or NFVI PoP (e.g., as a plug-in process in a virtualized environment) to perform passive monitoring, using listeners, for metrics collection based on benchmark tests evaluated according to Agents` stimuli. Different from the active approach of Agents that can be seen as generic benchmarking VNFs, monitor observes particular properties according to NFVI PoPs and VNFs capabilities. A listener defines one or more interfaces for the extraction of particular metrics monitored in a target VNF and/or execution environment. Logically, a Monitor is defined by as a virtual network function.
- Manager --
- in a VNF benchmarking deployment scenario, is responsible for (i) the coordination and synchronization of activities of Agents and Monitors, (ii) collecting and parsing all VNF benchmarking results, and (iii) aggregating the inputs and parset benchmark outputs to construct a VNF performance profile, report that correlates the VNF stimuli and the monitored metrics. A Manager executes the main configuration, operation and management actions to deliver the VNF benchmarking results. A Manager can be defined by a physical or virtual network function.
- Virtualized Network Function (VNF) --
- consists of one or more software components adequate for performing a network function according to allocated virtual resources and satisfied requirements in an execution environment. A VNF can demand particular configurations for benchmarking specifications, demonstrating variable performance profiles based on available virtual resources/parameters and configured enhancements targetting specific technologies.
- Execution Environment --
- defines a virtualized and controlled composition of capabilities necessary for the execution of a VNF. An execution environment stands as a general purpose level of virtualization with abstracted resources available for one or more VNFs. It can also define specific technology habilitation, incurring in viable settings for enhancing VNF performance profiles.
A VNF benchmark deployment scenario establishes the physical and/or virtual instantiation of components defined in a VNF benchmarking setup.
Based on a generic VNF benchmarking setup, the following considerations hold for deployment scenarios:
- Components can be composed in a single entity and defined as black or white boxes. For instance, Manager and Agent could jointly define a software entity to perform a VNF benchmark and present results.
- Monitor is not a mandatory component and must be considered only when performed white box benchmarking approaches for a VNF and/or its execution environment.
- Monitor can be defined by multiple instances of software components, each addressing a VNF or execution environment and their respective open interfaces for the extraction of metrics.
- Agents can be disposed in varied topology setups, included the possibility of multiple input and output ports of a VNF being directly connected each in one Agent.
- All benchmarking components defined in a deployment scenario must perform the synchronization of clocks to an international time standard.
In general, VNF benchmarks must capture relevant causes of performance variability. Examples of VNF performance influencing aspects can be observed in:
- Deployment Scenario Topology:
- The orchestrated disposition of components can define particular interconnections among them composing a specific case/method of VNF benchmarking.
- Execution Environment:
- The availability of generic and specific capabilities satisfying VNF requirements define a skeleton of opportunities for the allocation of VNF resources. In addition, particular cases can define multiple VNFs interacting in the same execution environment of a benchmarking setup.
- VNF:
- A detailed description of functionalities performed by a VNF sets possible traffic forwarding and processing operations it can perform on packets, added to its running requirements and specific configurations, which might affect and compose a benchmarking setup.
- Agent:
- The toolset available for benchmarking stimulus for a VNF and its characteristics of packets format, disposition, and workload can interfere in a benchmarking setup. VNFs can support specific traffic format as stimulus.
- Monitor:
- In a particular benchmarking setup where measurements of VNF and/or execution environment metrics are available for extraction, an important analysis consist in verifying if the Monitor components can impact performance metrics of the VNF and the underlying execution environment.
- Manager:
- The overall composition of VNF benchmarking procedures can determine arrangements of internal states inside a VNF, which can interfere in observed benchmark metrics.
Portability as a intrinsic characteristic of VNFs, allow them to be deployed in multiple environments, enabling, even parallel, benchmarking procedures in varied deployment scenarios. A VNF benchmarking methodology must be described in a clear and objective manner in order to allow effective repeatability and comparability of the test results.
For the sake of clarity and generalization of VNF benchmarking tests, consider the following definitions.
- VNF Benchmarking Layout (VNF-BL) --
- a setup that specifies a method of how to measure a VNF Performance Profile. The specification includes structural and functional instructions, and variable parameters at different abstractions (e.g., topology of the deployment scenario, benchmarking target metrics, parameters of benchmarking components). VNF-BL may be specific to a VNF or applicable to several VNF types. A VNF-BL can be used to elaborate a VNF benchmark deployment scenario aiming the extraction of particular VNF performance metrics.
- VNF Performance Profile: (VNF-PP) --
- defines a mapping between VNF allocated capabilities (e.g., cpu, memory) and the VNF performance metrics (e.g., throughput, latency between in/out ports) obtained in a benchmarking test elaborated based on a VNF-BL. Logically, packet processing metrics are presented in a specific format addressing statistical significance where a correspondence among VNF parameters and the delivery of a measured/qualified VNF performance exists.
In addition to a VNF-BL, all the items listed below, added their associated, and not limited to, settings must be contained in annotations describing a VNF benchmark deployment scenario. Ideally, any person in possession of such annotations and the necessary/associated skeleton of hardware and software components should be able to reproduce the same deployment scenario and VNF benchmarking test.
- VNF:
- type, model, version/release, allocated resources, specific parameters, technology requirements, software details.
- Execution Environment:
- type, model, version/release, available resources, technology capabilities, software details.
- Agents:
- toolset of available probers and related benchmarking metrics, workload, traffic formats, virtualization layer (if existent), hardware capabilities (if existent).
- Monitors:
- toolset of available listeners and related monitoring metrics, monitoring target (VNF and/or execution environment), virtualization layer (if existent), hardware capabilities (if existent).
- Manager:
- utilized procedures during the benchmark test, set of events and settings exchanged with Agents/Monitors, established sequence of possible states triggered in the target VNF.
Consider the following definitions:
- Trial:
- Consists in a single process or iteration to obtain VNF benchmarking metrics as a singular measurement.
- Test:
- Defines strict parameters for benchmarking components perform one or more trials.
- Method:
- Consists of a VNF-BL targeting one or more Tests to achieve VNF benchmarking measurements. A Method explicits ranges of parameter values for the configuration of benchmarking components realized in a Test.
The following sequence of events compose basic general procedures that must be performed for the execution of a VNF benchmarking test.
- 1.
- The sketch of a VNF benchmarking setup must be defined to later be translated into a deployment scenario. Such sketch must contain all the structural and functional settings composing a VNF-BL. At the end of this step the complete Method of benchmarking the target VNF is defined.
- 2.
- Via an automated orchestrator or in a manual process, all the components of the VNF benchmark setup must be allocated and interconnected. VNF and the execution environment must be configured to properly address the VNF benchmark stimuli.
- 3.
- Manager, Agent(s) and Monitor(s) (if existent), must be started and configured to execute the benchmark stimuli and retrieve expected/target metrics captured during and at the end of the VNF benchmarking test. One or more trials realize the measurement of VNF performance metrics.
- 4.
- Output results from each obtained benchmarking test must be received by Manager. In an automated or manual process, intended metrics to be extracted defined in the VNF-BL must compose a VNF-PP, resulting in a VNF benchmark report.
Configurations and procedures concerning particular cases of VNF benchmarks address testing methodologies proposed in [RFC8172]. In addition to the general description previously defined, some details must be taken into consideration in the following VNF benchmarking cases.
- Noisy Neighbor:
- An Agent can detain the role of a noisy neighbor, generating a particular workload in synchrony with a benchmarking procedure over a VNF. Adjustments of the noisy workload stimulus type, frequency, virtualization level, among others, must be detailed in the VNF-BL.
- Representative Capacity:
- An average value of workload must be specified as an Agent stimulus. Considering a long-term analysis, the VNF must be configured to properly address a desired average behavior of performance in comparison with the value of the workload stimulus.
- Flexibility and Elasticity:
- Having the possibility of a VNF be composed by multiple components, internal events of the VNF might trigger variated behaviors activating functionalities associated with elasticity, such as load balancing. In this terms, a detailed characterization of a VNF must be specified and be contained in the VNF-PP and benchmarking report.
- On Failures:
- Similarly to the case before, benchmarking setups of VNF must also capture the dynamics involved in the VNF behavior. In case of failures, a VNF would restart itself and possibly result in a off-line period. A VNF-PP and benchmarking report must clearly capture such variation of VNF states.
- White Box VNF:
- A benchmarking setup must define deployment scenarios to be compared with and without monitor components into the VNF and/or the execution environment, in order to analyze if the VNF performance is affected. The VNF-PP and benchmarking report must contain such analysis of performance variability, together with all the targeted VNF performance metrics.
On the extraction of VNF and execution environment performance metrics various trials must be performed for statistical significance of the obtained benchmarking results. Each trial must be executed following a particular deployment scenario composed by a VNF-BL.
A VNF Benchmarking Report correlates structural and functional parameters of VNF-BL with targeted/extracted VNF benchmarking metrics of the obtained VNF-PP.
A VNF performance profile must address the combined set of classified items in the 3x3 Matrix Coverage defined in [RFC8172].
The software, named Gym, is a framework for automated benchmarking of Virtualized Network Functions (VNFs). It was coded following the initial ideas presented in a 2015 scientific paper entitled “VBaaS: VNF Benchmark-as-a-Service” [Rosa-a]. Later, the evolved design and prototyping ideas were presented at IETF/IRTF meetings seeking impact into NFVRG and BMWG.
Gym was built to receive high-level test descriptors and execute them to extract VNFs profiles, containing measurements of performance metrics – especially to associate resources allocation (e.g., vCPU) with packet processing metrics (e.g., throughput) of VNFs. From the original research ideas [Rosa-a], such output profiles might be used by orchestrator functions to perform VNF lifecycle tasks (e.g., deployment, maintenance, tear-down).
The proposed guiding principles, elaborated in [Rosa-b], to design and build Gym can be compounded in multiple practical ways for multiple VNF testing purposes:
- Comparability: Output of tests shall be simple to understand and process, in a human-read able format, coherent, and easily reusable (e.g., inputs for analytic applications).
- Repeatability: Test setup shall be comprehensively defined through a flexible design model that can be interpreted and executed by the testing platform repeatedly but supporting customization.
- Configurability: Open interfaces and extensible messaging models shall be available between components for flexible composition of test descriptors and platform configurations.
- Interoperability: Tests shall be ported to different environments using lightweight components.
In [Rosa-b] Gym was utilized to benchmark a decomposed IP Multimedia Subsystem VNF. And in [Rosa-c], a virtual switch (Open vSwitch - OVS) was the target VNF of Gym for the analysis of VNF benchmarking automation. Such articles validated Gym as a prominent open source reference implementation for VNF benchmarking tests. Such articles set important contributions as discussion of the lessons learned and the overall NFV performance testing landscape, included automation.
Gym stands as the open source reference implementation that realizes the VNF Benchmarking Methodologies presented in this document. Gym is being released open source at [Gym]. The code repository includes also VNF Benchmarking Layout (VNF-BL) examples on the vIMS and OVS targets as described in [Rosa-b] and [Rosa-c].
TBD
This document does not require any IANA actions.
The authors would like to thank the support of Ericsson Research, Brazil.
11. References
11.1. Normative References
| [ETS14a] |
ETSI, "Architectural Framework - ETSI GS NFV 002 V1.2.1", Dec 2014. |
| [ETS14b] |
ETSI, "Terminology for Main Concepts in NFV - ETSI GS NFV 003 V1.2.1", Dec 2014. |
| [ETS14c] |
ETSI, "NFV Pre-deployment Testing - ETSI GS NFV TST001 V1.1.1", April 2016. |
| [RFC1242] |
S. Bradner, "Benchmarking Terminology for Network Interconnection Devices", July 1991. |
| [RFC8172] |
A. Morton, "Considerations for Benchmarking Virtual Network Functions and Their Infrastructure", July 2017. |
| [RFC8204] |
M. Tahhan, B. O'Mahony, A. Morton, "Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV)", September 2017. |
11.2. Informative References
| [Gym] |
"Gym Home Page" |
| [Rosa-a] |
R. V. Rosa, C. E. Rothenberg, R. Szabo, "VBaaS: VNF Benchmark-as-a-Service", Fourth European Workshop on Software Defined Networks , Sept 2015. |
| [Rosa-b] |
R. Rosa, C. Bertoldo, C. Rothenberg, "Take your VNF to the Gym: A Testing Framework for Automated NFV Performance Benchmarking", IEEE Communications Magazine Testing Series , Sept 2017. |
| [Rosa-c] |
R. V. Rosa, C. E. Rothenberg, "Taking Open vSwitch to the Gym: An Automated Benchmarking Approach", IV Workshop pré-IETF/IRTF, CSBC Brazil, July 2017. |