Internet DRAFT - draft-hz-ippm-cei
draft-hz-ippm-cei
IP Performance Measurement S. Liu
Internet-Draft Huawei Cloud
Intended status: Standards Track Y. Wang
Expires: 2 September 2024 Huawei
W. Sun
X. Huang
S. Zhou
Huawei Cloud
H. Huang, Ed.
T. Zhou, Ed.
Huawei
1 March 2024
Customer Experience Index for Evaluating Network Quality for Cloud
Applications
draft-hz-ippm-cei-00
Abstract
This document outlines a unified Customer Experience Index (CEI)
designed to assist cloud vendors in assessing network quality,
reflecting the customer experience with cloud applications when
accessed via the public network.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 2 September 2024.
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
Liu, et al. Expires 2 September 2024 [Page 1]
Internet-Draft Customer Experience Index for Evaluating March 2024
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
2. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Customer Experience Index . . . . . . . . . . . . . . . . . . 4
3.1. Observation . . . . . . . . . . . . . . . . . . . . . . . 4
3.2. Unified Index . . . . . . . . . . . . . . . . . . . . . . 5
3.3. Parameter Tuning . . . . . . . . . . . . . . . . . . . . 6
3.3.1. Weight Proportion . . . . . . . . . . . . . . . . . . 6
3.3.2. Parameter a, b . . . . . . . . . . . . . . . . . . . 6
4. Security Considerations . . . . . . . . . . . . . . . . . . . 6
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6
6. References . . . . . . . . . . . . . . . . . . . . . . . . . 6
6.1. Normative References . . . . . . . . . . . . . . . . . . 6
6.2. Informative References . . . . . . . . . . . . . . . . . 7
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7
1. Introduction
This document introduces a unified Customer Experience Index (CEI)
designed to assist cloud vendors in assessing the network quality
that mirrors the customer experience of cloud applications when
accessed via the public network. The CEI, once quantified, empowers
cloud vendors to proactively enhance network services, aiding in
network planning and construction. Furthermore, it enables cloud
customers to distinguish the service quality of various cloud
vendors, allowing them to select cost-effective services tailored to
their applications.
Cloud vendors and cloud enterprises focus on different network
indicators (Key Performance Index) used to anticipate the quality of
customer experience regarding various applications(e.g., gaming,
audio and video, online stores). However, KPIs only provide implicit
information and cannot directly reflect the customers' perceived
experience. Moreover, there is no unified evaluation method of
customer experience based on common network KPIs in the industry. On
the other hand, it is difficult for cloud vendors to directly access
Liu, et al. Expires 2 September 2024 [Page 2]
Internet-Draft Customer Experience Index for Evaluating March 2024
application-level Key Quality Index (KQI) data though it may
explicitly imply customer experience. As the number of enterprises
who deploy the service in the cloud gradually increases, there is
growing demand for deriving authentic customer experience from basic
network metrics to facilitate network optimizations.
A significant gap persists between network KPIs and customer
experience. The primary network KPIs accessible to most cloud
vendors—network latency, packet-loss rate, and jitter—encompass three
categories. Considering multiple dimensions of network quality
proves beneficial for end-users.
[I-D.teigen-ippm-app-quality-metric-reqs] Customers' demands for
experience quality vary across different cloud services and are
linked to specific KPIs. For instance, those accessing real-time
interactive games prioritize network latency; those utilizing video-
on-demand services are more concerned with packet-loss rate than
latency; and those engaging with cloud storage services consider both
latency and packet-loss rate. No single KPI can provide an accurate
reflection of the experience for diverse services. Both cloud
vendors and customers seek unified evaluation standards for
experience quality when accessing cloud services.
This document accounts for a range of key network-observable
indicators, offering a unified, objective, and comprehensive CEI to
help enterprises evaluate customer experience through measurable
network KPIs in a reasonable and fair manner. Predominantly based on
three network KPIs—network latency, packet-loss rate, and jitter—the
CEI aims to thoroughly assess network quality. The allocation of
weights to these KPIs within the CEI can be customized to suit
different application scenarios.
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
1.2. Terminology
CEI: Customer Experience Index
KPI: Key Performance Indicator
KQI: Key Quality Indicator
Liu, et al. Expires 2 September 2024 [Page 3]
Internet-Draft Customer Experience Index for Evaluating March 2024
2. Motivation
Cloud service providers aim to precisely evaluate the network
quality, crucial to the customer experience of their cloud services,
and implement targeted improvements to their network infrastructure.
Similarly, cloud customers seek a unified and fair scoring standard
to guide their selection of superior cloud services. But they
currently face some challenges:
* Cloud service providers often find it challenging to gather direct
feedback on customer experiences.
* Although obtaining network monitoring data, such as Key
Performance Indicators (KPIs), is relatively straightforward in
real-time, a clear correlation between the customer experience's
Key Quality Indicators (KQI) and network KPIs remains elusive.
* Network quality is primarily assessed using three KPIs: network
latency, packet-loss rate, and jitter. Cloud service providers
support a wide array of applications, including those sensitive to
latency (like gaming applications) and packet loss (such as audio
and video applications). This diversity necessitates different
KPIs to gauge the experience quality accurately, leading to a
fragmented evaluation approach.
3. Customer Experience Index
This document introduces the Customer Experience Index (CEI), a
measure reflecting customer experience with cloud services. It
enables cloud service providers to swiftly evaluate their service
quality through a synthesis of key network metrics.
3.1. Observation
Customer experience often exhibits distinct zones—sensitive and
smooth—based on their response to changes in specific indicators.
For instance, in scenarios sensitive to latency, such as cloud
gaming, customer satisfaction remains high within an acceptable
latency range (smooth zone). However, exceeding a certain latency
threshold leads to a sharp decline in experience (sensitive zone).
Accordingly, the CEI employs an S-Curve for its calculation, a method
prevalent in biostatistics and sociology for modeling ecosystems and
urbanization trends. The S-Curve, particularly through the Sigmoid
function, effectively maps values to a (0,1) interval, delineating
two smooth zones and a sensitive zone, mirroring the nuanced nature
of customer experience.
Liu, et al. Expires 2 September 2024 [Page 4]
Internet-Draft Customer Experience Index for Evaluating March 2024
^
1-| '''---... ┓
| ''--.. ┃- smooth zone
| '. ┛
| - ┓
| . ┃- sensitive zone
| . ┃
| - ┛
| ┏ '.
| smooth zone -┃ ''--..
0-| ┗ '''---...
+-------------------------------------------->
Figure 1: S-curve Example
Therefore, this document designs the following formula to evaluate
customer experience for each network KPI:
f(x) = (1+e^b)/(1+e^(a*x+b))
* x is the value measured by network KPI, a and b are tunable
parameters, and f(x) represents S-curve for certain KPI.
* Parameter a represents the overall slope of the curve, mainly
affecting the range of the central sensitive area.
* Parameter b represents the offset and scaling of the curve. The
initial smoothing area can be shielded via tuning b, which can
express KPIs that immediately enter the sensitive area from the
very beginning.
3.2. Unified Index
Each KPI is represented by a distinct S-curve to ensure independence
among the indicators. Specifically, unique S-curves for network
latency, packet-loss rate, and jitter are created by assigning
specific parameters (a and b), offering tailored indexes for
applications sensitive to these different metrics. The comprehensive
CEI score is then derived by aggregating these three S-curves, each
weighted appropriately:
CEI(x, y, z) = w1 * f1(x) + w2 * f2(y) + w3 * f3(z)
* x, y, and z respectively indicate values of the three major
network KPIs: network latency, packet-loss rate, and jitter.
* f1, f2, f3 represent the three individual S-curves.
Liu, et al. Expires 2 September 2024 [Page 5]
Internet-Draft Customer Experience Index for Evaluating March 2024
* w1, w2, w3 represent the empirical weights.
3.3. Parameter Tuning
3.3.1. Weight Proportion
The CEI's flexibility allows for fine-tuning to meet specific
application needs by adjusting its weight values (w1, w2, w3),
enabling precise adaptation for various application categories.
Typically, cloud customers engage in scenarios that are either
sensitive to latency—like gaming applications—or to packet loss, such
as audio and video streaming. For instance, in latency-sensitive
scenarios, the weights for latency, packet-loss rate, and jitter
could be adjusted to a ratio of 7:2:1 (w1:w2:w3); whereas for packet-
loss-sensitive scenarios, a ratio of 2:7:1 (w1:w2:w3) might be more
appropriate. This tailored approach allows the CEI to accurately
assess network quality for different types of applications from a
specific viewpoint (e.g., a fixed test point) across various cloud
vendors.
3.3.2. Parameter a, b
The parameters a and b of the CEI formula can be fine-tuned via:
* Determine the initial values of parameters a and b by fitting each
KPI CEI curve based on a large amount of operational data.
* Parameters a and b can be further tuned based on preferences of
certain application class. For example, when the packet-loss rate
is no higher than 𝑘, it is desired that CEI goes up as the network
latency lowers. CEI can set tuning goals according to such
preferences and fine-tune parameters a and b.
4. Security Considerations
TBD.
5. IANA Considerations
This document has no IANA actions.
6. References
6.1. Normative References
Liu, et al. Expires 2 September 2024 [Page 6]
Internet-Draft Customer Experience Index for Evaluating March 2024
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
6.2. Informative References
[I-D.teigen-ippm-app-quality-metric-reqs]
Teigen, B. I. and M. Olden, "Requirements for a Network
Quality Framework Useful for Applications, Users, and
Operators", Work in Progress, Internet-Draft, draft-
teigen-ippm-app-quality-metric-reqs-02, 18 October 2023,
<https://datatracker.ietf.org/doc/html/draft-teigen-ippm-
app-quality-metric-reqs-02>.
Authors' Addresses
Sifa Liu
Huawei Cloud
China
Email: liusifa@huawei.com
Yaojing Wang
Huawei
China
Email: wangyaojing1@huawei.com
Wei Sun
Huawei Cloud
China
Email: james.sunwei@huawei.com
Xiang Huang
Huawei Cloud
China
Email: terran.huang@huawei.com
Shuai Zhou
Huawei Cloud
China
Liu, et al. Expires 2 September 2024 [Page 7]
Internet-Draft Customer Experience Index for Evaluating March 2024
Email: zhoushuai@huawei.com
Hongyi Huang (editor)
Huawei
China
Email: hongyi.huang@huawei.com
Tianran Zhou (editor)
Huawei
China
Email: zhoutianran@huawei.com
Liu, et al. Expires 2 September 2024 [Page 8]
