| Large-Scale Measurement of Broadband Performance | A. Bovo |
| Internet-Draft | independent |
| Intended status: Informational | R. Marks |
| Expires: January 21, 2016 | EthAirNet Associates |
| July 20, 2015 |
Mobile LMAP Use Cases
draft-bovo-lmap-mobile-01
This document discusses the use cases for broadband measurements applied to the mobile domain as an adjunct to such scenarios for the landline broadband domain. The specifics related to the mobile domain are discussed considering them as possible extensions of IETF LMAP measurements.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 21, 2016.
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Networks and services must accommodate service profiles with exponentially increasing mobile traffic. Such traffic has significantly changed the network architectures as well as the way the networks are controlled and measured. The same is true of services; service providers have to deal with specific problems related to traffic profiles coming from the mobile domain, with specific requirements different those of landline broadband access.
This draft describes a set of use cases for broadband measurements applied to the mobile domain. Some of these use cases apply to landline networks as well. The primary aim at this initial stage is to detail specific use cases that are important in the mobile domain and that are aimed to complement measurements originating from line-based networks. Mobile considerations as documented here may lead to an expanded scope for the LMAP Working Group.
This document is drawn from work done within the IEEE Project 802.16.3 on "Mobile Broadband Network Performance Measurements" [P802.16.3]. That project considers a measurement framework for mobile broadband measurement scenarios applicable to multiple stakeholders. Such models support flexible measurements and provide the basis for a standardized framework facilitating measurement comparison and validation.
The following abbreviations are used in this document:
| Abbreviation | Expansion |
|---|---|
| KPI | Key Performance Indicator |
| NO | Network Operator |
| SIM | subscriber identification module |
| SLA | Service Level Agreement |
| SON | Self-Organizing Networks |
| UE | User Equipment |
Specifically mobile scenarios for broadband measurements are considered here for reasons including the following:
Considering specifics of the mobile domain allows completing the overall picture for broadband measurements, resulting in a more comprehensive standardization. In the absence of a standardized mobile broadband measurements framework efforts by each stakeholder to compare network performance will continue to be complex as well as difficult to ascertain quantitatively or to compare fairly with other results.
Some measurement purposes/applications related to the mobile domain are itemized in Table 1.
| Item # | Measurement Purposes / Applications |
|---|---|
| 1 | Assess overall data on Quality of Experience of set of networks available to consumers |
| 2 | Assess quality of Experience of a specific network |
| 3 | Identify limitations in deployment of a specific network |
| 4 | Monitor for changes in operation of a specific network |
| 5 | Diagnose problems in a specific network |
| 6 | Improve knowledge of system performance |
| 7 | Lead the market toward more effective networks |
| 8 | Encourage the redeployment of scarce spectrum using efficient technologies and implementations |
| 9 | Compare measured performance data to simulated results |
| 10 | Assess theoretical models |
| 11 | Assess technology elements proposed during standards development |
| 12 | Assess service measurements geo-located |
Various use cases are summarized in Table 2 below, with cross reference measurement purposes listed in Table 1 and to relevant stakeholders.
| Use case | Stakeholder | Measurement Purposes |
|---|---|---|
| POLICY CHECK | Governmental policy maker | 1 |
| COMPETITION CHECK | Governmental policy maker | 1 |
| STRATEGIC DIRECTION | Governmental policy maker | 1,12 |
| EFFICIENCY IMPROVEMENT | Governmental policy maker | 1,12 |
| SERVICE CHECK | User (individual or enterprise) | 1,3,4,12 |
| TRENDS | User (individual or enterprise) | 1,12 |
| COMPARISONS | User (individual or enterprise) | 2,12 |
| AUTONOMOUS NETWORK PERFORMANCE ACCESS | Cellular tower operator | 2,4,5 |
| AUTONOMOUS RADIO NETWORK ACCESS | Cellular tower operator | 2,5,8 |
| HANDOVER | Wireless carrier / Network operator | 2,6 |
| SERVICE LEVEL AGREEMENT | Wireless carrier / Network operator | 1,2 |
| RESOURCE USAGE | Wireless carrier / Network operator | 2,5 |
| TEST OF NEW RELEASE INTRODUCTION | Wireless carrier / Network operator | 2,5 |
| CHECK MODELS | Researcher | 1,2,6,10 |
| ACCESS TO REAL IN-FIELD DATA | Researcher | 1,2,6,10 |
| METRICS AVAILABILITY | Standards developer | 2,10 |
| CHECK OF OPTIONS | Standards developer | 2,6,11 |
| UE CHARACTERIZATION | User device vendor | 1,2 |
| APPLICATION CHARACTERIZATION | Application developer | 1,2 |
| MOBILE SERVICE CHARACTERIZATION | Mobile Application Service Provider | 1,2,8 |
These use cases are detailed below.
Use cases for the "User (individual or enterprise)" stakeholder include the following:
Naturally, individual or enterprise users, as well as virtual network operators, are interested in getting the best service from the mobile network and from the service provider. The performance data collected using specific services can be compared by the stakeholders in relation to the measurement conditions (operator, radio interface, end service host, type of service, location, mobility, etc.). In some cases, the User is an enterprise supporting many individual end user devices over a long period of time, over a large geographic region (which may be limited to, for example, specific service routes). Standardized measurements could be enable agreement on an SLA between the enterprise and a mobile service provider.
Mobility and radio conditions affect service delivery, so a standardized characterization of services is incomplete without consideration of those conditions.
Measurement applications that are useful for these purposes are overall data on quality of experience measured by UEs but also service measurements geo-located or correlated to time-of-day, time-of-week, etc. Even measurements of changes on a specific network behaviour or on a specific service can be interesting and aid in identification of limitations on a specific network.
A simple example of network limitation could be the absence of connection continuity within a certain geographical location, so connections are dropped as soon as the UE moves into a region, possibly at predictable times.
Analyzing day-to-day trends is a use case relevant for example to enterprise organizations that need to assure certain connection reliability over time to their associated or customers. Checking trends is also useful for profiling the customer access and identifying bottlenecks in the service or overload conditions.
An example of such a use case is an enterprise that has to size and maintain the network resources for customers accessing its network, for example for e-commerce or customer support, and desire to measure the service experience of mobile access over time, to identify bottleneck conditions.
Measurement application that can be used for such purposes are overall data on quality of experience measured by UEs but also service measurements geo-located. All these measurements have to be analyzed as trends over time correlating also with enterprise network conditions and host behaviour.
An enterprise or individual user could be interested in understand the relevance of a specific issue that it is experiencing, is comparison with other UEs (perhaps located in the same area) and accessing similar services. This could be used to check, for example, UE configuration for correct behaviour. Measurements application useful for these purposes could be an enterprise or public (anonymous) repository of end-user measurements toward specific services, geo-located and providing the type of device as possible aggregation criteria for measurements. Comparison use cases are also relevant to inter-comparison of network operators in the enterprise's service areas.
Use cases for the "Cellular tower operator" stakeholder include the following:
A network operator builds and develops new tower sites based significantly on customer demand; the operator has good access to data regarding such demand. The success of an access cellular tower operator, on the other hand, is dependent on building and developing new tower locations for use by multiple network operators, none of which may share operational information. Therefore, a cellular tower operator may seek access to a broad set of public mobile measurements, possible over a broad set of network operators, in order to inform tower development activities that will serve multiple network operators, ideally without expensive drive tests.
Measurement applications useful for these use cases could be the quality of experience of the specific network, the network diagnosis and the change of operation in the specific network.
A cellular tower operator is interested in getting the most from each site, checking the correctness of the configuration against the radio conditions and suggesting improvements to NO that operate the rest of the network. Getting autonomous access to the end-user experience correlated with radio conditions and cell identification helps the cell tower operator to be more proactive in suggesting network improvements. Measurements could be also used to suggest migration to other radio access technologies and otherwise upgrading the cellular site.
Measurement applications useful for these use cases could be the quality of experience of the specific network, the diagnosis of problems, and the encouragement of deployment of more efficient radio access technologies.
Use cases for the "Wireless carrier/Network operator" include the following:
As usual in the mobile domain, it is necessary to characterize the broadband services during packet switched handover events. Such characterization can be correlated to specific cells and/or specific services. The adoption of measurements at end user premises can be reused even for setting proper values for handover settings.
Measurement applications are quality of experience of a specific network, correlated with radio conditions, location information, and device information. All these measurements have the consequence that the stakeholder can improve knowledge of its network performance, useful also to optimize and design better updates of the network.
Mobility characterization can be used even to characterize the radio access technology behaviour during service lifetime. Such characterization could suggest improvements of the mobile network or additional features to support better mobility in connected state.
The existence of a Service Level Agreement is predicated on an understanding between the parties on accurate measurement methods. Currently, SLAs are rare in the mobile domain, partially because such an understanding is difficult to reach. Standardized measurements will help enable mobile SLAs.
Checking the Service Level Agreement is needed not only to be sure that the contract to customers is satisfied but also could be useful to check the performance of different NOs homogeneously. This could imply understanding limitations and trigger analysis to improve the core network or redesign radio deployment to make it more efficient or to move to new radio access technology.
Measurement applications include the quality of experience of a specific network and also overall data on quality of experience of set of networks available to customers.
Traffic and performance measurements are useful to optimize network parameter configuration, feeding for example optimization systems. Even if this job is typically done at the network level, the availability of the UE perspective can help especially for network technologies that do not provide a complete reporting of the UE measurements. For example, spectrum allocation and configuration parameters can be modified according to the traffic and performance measurements provided by UEs, for example to avoid network overload and congestion and reduce loss of radio contact.
Measurement applications are the quality of experience of a specific network and diagnosis of problems in a specific network.
Measurements provided by an end-user can be used to characterize the performance changes following new network equipment.
Aggregated per cell and radio technology, such measurements can provide KPIs that can be used to compare the network performance before and after new release changes at the access network level, from the end user perspective.
Measurement applications are the quality of experience of a specific network and diagnosis of problems in a specific network.
Use cases for the "Researcher" stakeholder include the following:
A researcher can be interested in checking the correctness of some hypothesis or theoretical models, starting from in-field data. For example, the statistical model for service request arrival rate by customers can be checked against real conditions. Therefore, measurement applications useful for researchers can be quality of experience of a specific network or quality of experience on a set of networks available to the user. Other relevant applications include improving knowledge of system performance and checking the network behaviour against theoretical models.
Another possible use case is getting access to real data in order to achieve additional information useful for researchers, as for example test set or traffic profiles and consistency.
Measurement applications useful for this use case are similar to the previous one.
Use cases for the "Standards developer" include the following:
End user measurements are useful to understand actual mobile performance and so make decisions on improved standards based on measured facts, as obtained in a standardized manner. For example, latency metrics available on specific radio access technology can support upgraded standards.
Measurement application useful for standards developer can be quality of experience of a specific network or comparing measured results with theoretical models.
Performance measurements allow comparison of expected results with current results, in order to validate technical choices. An example could be the adoption of specific codec for voice or the adoption of specific protocols for data services. Another example is the impact of security on performance, because authentication and ciphering techniques could affect overall performance.
Measurement applications include performance assessment, correlated to radio, location, and technology details. Other measurement applications include the assessment of technology elements proposed during the standard development.
Use cases for the "User device vendor" include the following:
User device manufacturers are generally interested in the range of possible network performance that their users may experience, particularly in correlation to factors such as device features. Such information informs the design decisions of the manufacturer, for example providing guidance as to which features will be most relevant to user experience.
User device manufacturers can be interested in the adoption of UE broadband measurements to characterize the interoperability of the device against real networks. The adoption of KPIs divided per user device is helpful for this type of characterization. In fact, different network settings can be the reason for a variable interoperability between the network and the UE.
So, measurement applications useful for user device vendors can be quality of experience of a specific network or quality of experience on a set of networks available to the user.
Use cases for the "Application developer" stakeholder include the following:
Mobile application (app) developers are generally interested in the range of possible network performance that their users may experience. Such information informs the design decisions of the developers, for example providing guidance as to which app features will enhance, or detract from, user experience.
In case of mobile applications, it is important to characterize how well the app is performing through the network. This characterization can be also the trigger for app changes that minimize the drawbacks with the network interaction. It is possible for mobile application developers also to include monitoring callbacks that can be useful for passive measurements. Other use cases for app developer are a sort of usage profiling, understanding when and how much an app is used by customers. Timing the app's access to services from the network is important, for example, to correlate usage drop by the user to poor quality of service experience.
Measurement application useful for app developer can be quality of experience of a specific network or quality of experience on a set of networks available to the user.
Use cases for "Mobile application service provider" stakeholder include the following:
Mobile application service providers are interested in mobile broadband measurement characterization, particularly correlating the measurements to the specific invoked application. In fact, the service accessed by mobile has to satisfy certain conditions not relevant to services accessed from landline networks. Assessment is possible by measuring the volume of mobile application transactions and also by correlating such usage with the actual performance level. A consequence of such analysis could be encouragement of users to adopt specific radio technologies with some critical mobile apps.
In case of networks that do not support the bandwidth needed for a specific application, it may be useful to trigger modifications either in the application itself or in the network. The trade-off decision can be driven even by end user measurements.
Measurement application useful for the app developer can be quality of experience of a specific network or quality of experience on a set of networks available to the user. Measurements can be used to encourage the network operator to adopt more efficient radio technologies to support services that could be limited by network efficiency so far.
The Governmental policy maker stakeholder includes several use cases:
In order to check the behaviour of network players against current policies, the behaviour of networks must be measured from an end-user perspective. So, as an example, if current policies require that emergency calls be supported by all network operators even when the UE is SIM-less or belonging to a different operator, then measurements are required to check the compliance against this requirement.
Measurement application that can be useful for this use case are overall data on quality of experience measured by UEs across different networks. It is important to characterize the measurements with dimensions that support the policy check under test. As in the previous example, it is necessary to associate the measurements to the home network users, roamers, and SIM-less UEs.
Governmental policy makers could encourage competition between NOs. In case of resources shared by different actors (e.g. shared sites, shared network entities, radio spectrum, etc.) it could be relevant to measure current behaviour for different NOs. An example could be the support of legacy services that the governmental policy maker requires be maintained (e.g. technically obsolete radio technologies to be maintained for a certain period of time with specific level of service).
In this use case the overall data on quality of experience could be measured by UEs across different networks.
Governmental policy makers could encourage the adoption of technologies with a strategic plan, for example to achieve a better environmental sustainability. Measurements with this perspective can be related for example to battery usage per services, spectrum needs, or specific geographical areas. For example, it could be specified that some specific geographical region will support at least a certain number of concurrent calls per location area. So, admission control and network allocated resources have to deal with these requirements.
Another example relates to technological evolution, such as measuring the amount of traffic for specific services encouraging the deployment of new technologies to support better usage of scarce spectrum, for example with a better spectral efficiency or operation with reduced radio signal power. In general, users may be hesitant to convert to new UE technology due to cost without unbiased reassurance of the technical benefits to the individual users of making the transition; unbiased measurements may thereby encourage individual equipment upgrades that would contribute toward a more efficient network operation.
Measurement application for these use cases include the overall data on quality of experience measured by UEs and also service measurements geo-located or correlated to radio conditions, even associated to other UE parameters (e.g. type of device, radio access technology, battery consumption,radio signal strength).
The authors acknowledge the participants in the IEEE 802.16 Working Group on Broadband Wireless Access and in IEEE 802.16's Project 802.16.3 on Mobile Broadband Network Performance Measurements.
This memo includes no request to IANA.
Candidate Control and Report protocols are required to meet security requirements identified in [I-D.ietf-lmap-framework] .
| [I-D.ietf-lmap-framework] | Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., Aitken, P. and A. Akhter, "A framework for Large-Scale Measurement of Broadband Performance (LMAP)", Internet-Draft draft-ietf-lmap-framework-11, February 2015. |
| [P802.16.3] | IEEE 802.16 Working Group on Broadband Wireless Access, A. Bovo (editor), "IEEE 802.16.3 Architecture and Requirements for Mobile Broadband Network Performance Measurements", Oct 2014. |