Internet DRAFT - draft-feamster-livingood-iab-covid19-workshop
draft-feamster-livingood-iab-covid19-workshop
Internet Architecture Board F. Bronzino
Internet-Draft Universite Savoie Mont Blanc
Intended status: Informational E. Culley
Expires: 1 May 2021 Comcast
N. Feamster
S. Liu
University of Chicago
J. Livingood
Comcast
P. Schmitt
Princeton University
28 October 2020
IAB COVID-19 Workshop: Interconnection Changes in the United States
draft-feamster-livingood-iab-covid19-workshop-01
Abstract
During the early weeks and months of the COVID-19 pandemic,
significant changes to Internet usage occurred as a result of a
sudden global shift to people working, studying and quarantining at
home. One aspect that this affected was interconnection between
networks, which this paper studies. This paper explores some of the
effects of these changes on Internet interconnection points, in terms
of utilization, traffic ratios, and other performance characteristics
such as latency.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 1 May 2021.
Bronzino, et al. Expires 1 May 2021 [Page 1]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Long-Term Interconnection Data . . . . . . . . . . . . . . . 3
3. Comcast's COVID-19-Related Experiences . . . . . . . . . . . 4
4. Detailed Statistical Observations . . . . . . . . . . . . . . 6
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 6
6. Security Considerations . . . . . . . . . . . . . . . . . . . 6
7. Normative References . . . . . . . . . . . . . . . . . . . . 6
8. Informative References . . . . . . . . . . . . . . . . . . . 6
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7
1. Introduction
During the early weeks and months of the COVID-19 pandemic
[WHO-Declaration], significant changes to Internet usage occurred as
a result of a sudden global shift to people working, studying and
quarantining at home. One aspect that this affected was
interconnection between networks, which this paper studies.
In 2016, the Interconnection Measurement Project [Tinker-Blog] was
launched. The IMP platform initially collected interconnection-
related data from seven U.S.-based cable-based Internet Service
Providers (ISPs) and an [ArXiv-Paper] shared some of the details and
findings. The initial focus of the project was to explore
utilization characteristics of interconnect links during a period of
time when video traffic was steadily increasing. The project
concluded that there was ample aggregate capacity on interconnect
links between ISPs and peers (including content providers), and that
ISPs continually added capacity to their interconnects to keep pace
with the growth in traffic.
Bronzino, et al. Expires 1 May 2021 [Page 2]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
This IAB paper looks specifically at the long-term interconnection
data from one of those ISPs, Comcast. We examine the long-term pre-
COVID-19 trend as well as what occurred as COVID-19 impacted the
Internet from February 2020 through August 2020. We also include
observations from Comcast concerning interconnection changes during
this timeframe.
We hope that this information will be useful to the IAB workshop and
the Internet community more broadly. It may serve as an interesting
and useful historical reference in the future.
2. Long-Term Interconnection Data
The IMP platform collected interconnection data starting in 2016,
roughly four years prior to the COVID-19-driven shift in Internet
usage. This provides an interesting capability for a before and
after view of interconnection. A full explanation of the data can be
found in Section 3 of the [ArXiv-Paper]. At a high level, the
maindata collected encompasses:
* Timestamp (representing a five-minute interval)
* Region (representing an aggregated link group)
* Anonymized partner network
* Access ISP
* Total ingress bytes
* Total egress bytes
* Capacity
Utilization is captured based on sampled IPFIX records, with a packet
sampling rate of 1/1,000. SNMP polling data yields information about
the capacity on each link. The IMP platform does not have direct
access to partner network identities---this dataset only includes an
anonymous identifier corresponding to that particular partner.
However, in collaboration with Comcast, IMP has worked to identify
specific partner networks in the dataset for specific peers in the
interest of detail COVID-19-related study.
Focusing on Comcast within this dataset reveals several trends, in
both aggregate capacity and utilization, as well as how utilization
and capacity changed during the period of the COVID-19 pandemic. The
data also reveals how traffic volumes changed for specific peers
during this time period.
Bronzino, et al. Expires 1 May 2021 [Page 3]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
The IMP data that tracks capacity shows Comcast steadily adding
capacity from mid-2018 to present, with a significant increase in the
rate of additional capacity from the beginning of 2020, increasing
further in the second and third quarters of 2020. Specifically, we
analyzed the rate at which capacity was added during these periods,
on a month-by-month basis. We found that Comcast was adding
aggregate capacity on its interconnects at *nearly twice* the rate as
it was being added during 2019.
Over a long timeframe, Comcast's daily peak hour interconnection port
utilization on its busiest links has remained consistently around
90%. For our analysis, we consider the links that represent the 99th
percentile in terms of utilization. During the period at the end of
March and the beginning of April 2020, daily peak utilization briefly
increased to about 97% but steadily returned to normal levels in a
matter of weeks by the end of April 2020, as Comcast quickly
increased the rate at which it added capacity to keep pace with
growing traffic demands.
We also explored traffic volumes associated with each Comcast peer,
comparing the ranks and volumes of each individual peer as measured
on September 1, 2020 as compared to January 1, 2020, in both the
upstream and downstream directions. Doing so allowed us to
understand both the magnitude of changes in traffic volumes and
ratios, as well as how specific individual peers deviated from normal
baseline behavior, in terms of both upstream and downstream traffic
rates and ratios. The data suggests that both the upstream and
downstream directions saw some changes, although the deviations in
upstream traffic patterns were greater: the Pearson coefficient for
downstream traffic was 0.977, whereas for upstream traffic the
Pearson correlation coefficient was only 0.935, indicating a more
significant shift in upstream traffic ratios during this timeframe.
3. Comcast's COVID-19-Related Experiences
Comcast observed a wide range of significant changes in Internet
usage as residential users remained at home and shifted to working
and studying from home. Changes in usage patterns observed in the
access network in sum led to changes in the traffic flowing to
interconnected networks. As the pandemic developed, there was a wide
variety of changes in traffic volumes. At some locations in the
network little change was detected while other locations saw a huge
growth in the volume of traffic.
At the peak of the surge, in March and April 2020, the average amount
of growth observed across locations and types was roughly 33%. Voice
& video conferencing (conferencing hereafter) jumped as much as 285%
and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers
Bronzino, et al. Expires 1 May 2021 [Page 4]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
[June-Blog] [July-Blog]. As this continued into May 2020 we observed
conferencing remained up roughly 210-285%, VPN traffic up 30-40% and
gaming downloads up 20-80%, and web-based streaming video consumption
up 20-40% [May-Blog]. In this several week period, traffic
essentially grew at or more than it had in the prior year, which was
significant growth in a short period of time.
In the months following the onset of the pandemic Comcast observed:
* Overall average downstream peak growth is up 13% (up as much as
20% at times).
* Overall average upstream peak growth is up 36%.
* In the access network, an average of 771 network augments per week
were performed, peaking at over 1,800 in a single week and over
with over 7 weeks with more than 1,000 per week. For comparison,
the average earlier in the year was roughly 350 per week.
* In the core network, over 500 augments were made in order to add
146 Tbps in capacity.
* On a daily basis roughly 700,000 automated speed tests from
customer homes were conducted in order to gauge the customer
experience during this time. Average speeds to customers (both
downstream and upstream) have generally remained at or above 105%
of advertised speeds since March 1, 2020 in all regions. National
average speeds have remained between 110% - 115% of advertised
speeds over the same period. [NF-Paper-1] and [NF-Paper-2]
* The share of streaming video as a percentage of total traffic is
declined slightly from 67% to 63%. Despite strong growth,
conferencing occupies a small share of total traffic and grew from
1% share to 4%. But gaming software released have driven
significant download spikes since late April 2020.
* For interconnection, peering coordinators across operators worked
cooperatively and quickly to cut through any red tape and add new
capacity as quickly as possible.
* In 2019, settlement free interconnection capacity [SFI-Policy] - a
subset of overall interconnect types - grew by 15%. Between
January and August 2020, driven by COVID-19 changes, there was an
overall 37% increase in capacity from that prior 2019 level. And
between March and October 2020 one Settlement Free Peer alone
increased 115%.
Bronzino, et al. Expires 1 May 2021 [Page 5]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
* Between March and October 2020 from Comcast observed other notable
per-peer traffic increases of 245% and 3,900%.
4. Detailed Statistical Observations
As briefly mentioned in previous sections, downstream traffic rates
from many partners remained stable---the Pearson correlation
coefficient for peak download rates between January 1, 2020 and
September 2, 2020 is 0.977, indicating that the peak download rates
to most peers was similar between these two time periods. On the
other hand, certain peers experienced either a significant increase
or decrease in peak download rates---often by two or three orders of
magnitude. Similarly, other peers experienced a decrease in peak
downstream rates by several orders of magnitude.
On the other hand, upstream traffic rates were far less stable: In
contrast, the Pearson coefficient for upload rates between January 1,
2020 and September 2, 2020 was only 0.935, suggesting more a more
significant deviation in peak upstream rates. As with peak download
rates, some peers experienced significant decreases, as well: in one
outlier case, peak rate decreased by almost five orders of magnitude.
A small handful of peers saw similar decreases. Yet, a far greater
number of peers saw increases in peak upload rates by two to three
orders of magnitude.
5. IANA Considerations
This document includes no request to IANA.
6. Security Considerations
This document includes no security considerations.
7. Normative References
8. Informative References
[ArXiv-Paper]
Feamster, NF., "Revealing Utilization at Internet
Interconnection Points", 5 September 2016,
<https://arxiv.org/pdf/1603.03656.pdf>.
[July-Blog]
Nafshi, EN., "COVID-19 Network Report: How A Smart Network
Delivered Speed and Stability When it Mattered", 13 July
2020, <https://corporate.comcast.com/stories/covid-19-
network-report-smart-network-speed-and-stability>.
Bronzino, et al. Expires 1 May 2021 [Page 6]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
[June-Blog]
Werner, TW., "Cresting the Wave: The Factors that Powered
our Network Through the COVID-19 Surge", 15 June 2020,
<https://corporate.comcast.com/press/releases/cresting-
the-wave-how-our-network-thrived-what-comes-next>.
[May-Blog] Comcast, "COVID-19 Network Update", 20 May 2020,
<https://corporate.comcast.com/covid-19/network/may-
20-2020>.
[NF-Paper-1]
Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast Design
Audit Report of Comcast's Network Performance Measurement
System", April 2020, <https://www.netforecast.com/
netforecast-design-audit-report-of-comcasts-network-
performance-measurement-system/>.
[NF-Paper-2]
Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast's
Report on Comcast's Network Performance Measurement System
Results Data", May 2020, <https://www.netforecast.com/
netforecasts-report-on-comcasts-network-performance-
measurement-system-results-data/>.
[SFI-Policy]
Comcast Cable Communications Management, LLC, "Comcast
Settlement-Free Interconnection (SFI) Policy", October
2013, <https://www.xfinity.com/peering/>.
[Tinker-Blog]
Feamster, NF., "Interconnection Measurement Project
Website", 9 May 2016, <https://freedom-to-
tinker.com/2016/05/09/the-interconnection-measurement-
project/>.
[WHO-Declaration]
Adhanom Ghebreyesus, TAG., "WHO Director-General's opening
remarks at the media briefing on COVID-19 - 11 March
2020", 11 March 2020,
<https://www.who.int/dg/speeches/detail/who-director-
general-s-opening-remarks-at-the-media-briefing-on-covid-
19---11-march-2020>.
Authors' Addresses
Bronzino, et al. Expires 1 May 2021 [Page 7]
Internet-Draft COVID-19 Interconnection Traffic Effects October 2020
Francesco Bronzino
Universite Savoie Mont Blanc
Annecy-le-Vieux
France
Email: francesco.bronzino@univ-smb.fr
Elizabeth Culley
Comcast
Mount Laurel, NJ
United States of America
Email: elizabeth_culley@comcast.com
Nick Feamster
University of Chicago
Chicago, IL
United States of America
Email: feamster@uchicago.edu
Shinan Liu
University of Chicago
Chicago, IL
United States of America
Email: shinanliu@uchicago.edu
Jason Livingood
Comcast
Philadelphia, PA
United States of America
Email: jason_livingood@comcast.com
Paul Schmitt
Princeton University
Princeton, NJ
United States of America
Email: pschmitt@cs.princeton.edu
Bronzino, et al. Expires 1 May 2021 [Page 8]
