Internet DRAFT - draft-white-distoptflood
draft-white-distoptflood
Network Working Group R. White
Internet-Draft S. Hegde
Intended status: Informational Juniper Networks
Expires: January 28, 2021 S. Zandi
LinkedIn
July 27, 2020
IS-IS Optimal Distributed Flooding for Dense Topologies
draft-white-distoptflood-04
Abstract
In dense topologies, such as data center fabrics based on the Clos
and butterfly fabric topologies, flooding mechanisms designed for
sparse topologies, when used in these dense topologies, can
"overflood," or carry too many copies of topology and reachability to
fabric devices. This results in slower convergence times and higher
resource utilization. The modifications to the flooding mechanism in
the Intermediate System to Intermediate System (IS-IS) link state
protocol described in this document reduce resource utilization to a
minimum, while increaseing convergence performance in dense
topologies.
Note that a Clos fabric is used as the primary example of a desne
flooding topology throughout this document. However, the flooding
optimizations described in this document apply to any dense topology.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 28, 2021.
White, et al. Expires January 28, 2021 [Page 1]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2. Contributors . . . . . . . . . . . . . . . . . . . . . . 3
1.3. Experience . . . . . . . . . . . . . . . . . . . . . . . 3
1.4. Sample Network . . . . . . . . . . . . . . . . . . . . . 3
2. Flooding Modifications . . . . . . . . . . . . . . . . . . . 5
2.1. Optimizing Flooding . . . . . . . . . . . . . . . . . . . 5
2.2. Flooding Failures . . . . . . . . . . . . . . . . . . . . 6
3. Use of Flooding Leaders and Flooding Mechanism Advertisements 6
4. Security Considerations . . . . . . . . . . . . . . . . . . . 6
5. References . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.1. Normative References . . . . . . . . . . . . . . . . . . 7
5.2. Informative References . . . . . . . . . . . . . . . . . 8
Appendix A. Flooding Optimization Operation . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12
1. Introduction
1.1. Goals
The goal of this draft is to solve one specific set of problems
involved in operating a link state protocol in a dense mesh topology.
The problem with such topologies is the connectivity density, which
causes too much information to be flooded (or too much repeated state
to be flooded). Analysis and experiment show, for instance, that in
a butterfly fabric of around 2500 intermediate systems, each
intermediate system will receive 40+ copies of any changed LSP
fragment. This not only wastes bandwidth and processor time, this
dramatically slows convergence speed.
White, et al. Expires January 28, 2021 [Page 2]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
This document describes a set of modifications to existing IS-IS
flooding mechanisms which minimize the number of LSP framgents
received by individual intermediate systems, potentially to one copy
per intermediate system. The mechanisms described in this document
are similar to those implemented in OSPF to support mobile ad-hoc
networks, as described in [RFC5449], [RFC5614], and [RFC7182]. These
mechanisms have been widely deployed and tested.
1.2. Contributors
The following people have contributed to this draft: Nikos
Triantafillis, Ivan Pepelnjak, Christian Franke, Hannes Gredler, Les
Ginsberg, Naiming Shen, Uma Chunduri, Nick Russo, and Rodny Molina.
1.3. Experience
The modifications described in this draft have been implemented in
the FR Routing open source routing stack, and hence are available for
testing and modification. The implementation is part of the
openfabric daemon, which can be conditionally compiled from isisd.
Note openfabricd has further modifications are not described in this
document.
Lab testing shows these modifications reduce flooding in a large
scale emulated butterfly network topology; without these
modifications, intermediate systems receive, on average, 40 copies of
any changed LSP fragment. With these modifications, intermediate
systems recieve, on average, two copies of any changed LSP fragment.
In many cases, each intermediate system receives one copy of each
changed LSP. In terms of performance, the modifications described
here reduce convergence times by around 50%. A network that converges
in about 30-40 seconds without these modifications converged in 15-20
seconds with these modifications. Processor load times were not
checked, as this was an emulated environment.
1.4. Sample Network
The following spine and leaf fabric will be used to describe these
modifications.
White, et al. Expires January 28, 2021 [Page 3]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
+----+ +----+ +----+ +----+ +----+ +----+
| 1A | | 1B | | 1C | | 1D | | 1E | | 1F | (T0)
+----+ +----+ +----+ +----+ +----+ +----+
+----+ +----+ +----+ +----+ +----+ +----+
| 2A | | 2B | | 2C | | 2D | | 2E | | 2F | (T1)
+----+ +----+ +----+ +----+ +----+ +----+
+----+ +----+ +----+ +----+ +----+ +----+
| 3A | | 3B | | 3C | | 3D | | 3E | | 3F | (T2)
+----+ +----+ +----+ +----+ +----+ +----+
+----+ +----+ +----+ +----+ +----+ +----+
| 4A | | 4B | | 4C | | 4D | | 4E | | 4F | (T1)
+----+ +----+ +----+ +----+ +----+ +----+
+----+ +----+ +----+ +----+ +----+ +----+
| 5A | | 5B | | 5C | | 5D | | 5E | | 5F | (T0)
+----+ +----+ +----+ +----+ +----+ +----+
Figure 1
To reduce confusion (spine and leaf fabrics are difficult to draw in
plain text art), this diagram does not contain the connections
between devices. The reader should assume that each device in a
given layer is connected to every device in the layer above it. For
instance:
o 5A is connected to 4A, 4B, 4C, 4D, 4E, and 4F
o 5B is connected to 4A, 4B, 4C, 4D, 4E, and 4F
o 4A is connected to 3A, 3B, 3C, 3D, 3E, 3F, 5A, 5B, 5C, 5D, 5E, and
5F
o 4B is connected to 3A, 3B, 3C, 3D, 3E, 3F, 5A, 5B, 5C, 5D, 5E, and
5F
o etc.
The tiers or stages of the fabric are also marked for easier
reference. T0 is assumed to be connected to application servers, or
rather they are Top of Rack (ToR) intermediate systems. The
remaining tiers, T1 and T2, are connected only to the fabric itself.
White, et al. Expires January 28, 2021 [Page 4]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
2. Flooding Modifications
Flooding is perhaps the most challenging scaling issue for a link
state protocol running on a dense, large scale fabric. This section
describes modifications to the IS-IS flooding process to reduce
flooding load on a dense or mesh topology.
2.1. Optimizing Flooding
To reduce the flooding of link state information in the form of Link
State Protocol Data Units (LSPs), the following tables are required
to compute a set of reflooders:
o Neighbor List (NL) list: The set of neighbors
o Neighbor's Neighbors (NN) list: The set of neighbor's neighbors;
this can be calculated by running SPF truncated to two hops
o Do Not Reflood (DNR) list: The set of neighbors who should have
LSPs (or fragments) who should not reflood LSPs
o Reflood (RF) list: The set of neighbors who should flood LSPs (or
fragments) to their adjacent neighbors to ensure synchronization
NL is set to contain all neighbors, and sorted deterministically (for
instance, from the highest IS identifier to the lowest). All
intermediate systems within a single fabric SHOULD use the same
mechanism for sorting the NL list. NN is set to contain all
neighbor's neighbors, or all intermediate systems that are two hops
away, as determined by performing a truncated SPF. The DNR and RF
tables are initially empty. To begin, the following steps are taken
to reduce the size of NN and NL:
o Remove all intermediate systems from NL and NN that in the
shortest path to the IS that originated the LSP
Then, for every IS in NL:
o If the current entry in NL is connected to any entries in NN:
* Move the IS to RF
* Remove the intermediate systems connected to the IS from NN
o Else move the IS to DNR
The calculation terminates when the NL is empty.
White, et al. Expires January 28, 2021 [Page 5]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
When flooding, LSPs transmitted to adjacent neighbors on the RF list
will be transmitted normally. Adjacent intermediate systems on this
list will reflood received LSPs into the next stage of the topology,
ensuring database synchronization. LSPs transmitted to adjacent
neighbors on the DNR list, however, MUST be transmitted using a
circuit scope PDU as described in [RFC7356].
2.2. Flooding Failures
It is possible in some failure modes for flooding to be incomplete
because of the flooding optimizations outlined. Specifically, if a
reflooder fails, or is somehow disconnected from all the links across
which it should be reflooding, it is possible an LSP is only
partially flooded through the fabric. To prevent such situations,
any IS receiving an LSP transmitted using DNR SHOULD:
o Set a short timer; the default should be less than one second
o When the timer expires, send a Complete Sequence Number Packet
(CSNP) to all neighbors
o Process any Partial Sequence Number Packets (PSNPs) as required to
resynchronize
o If a resynchronization is required, notify the network operator
through a network management system
3. Use of Flooding Leaders and Flooding Mechanism Advertisements
[I-D.ietf-lsr-dynamic-flooding], section 5.1.1, describes the
election of a flooding domain leader, which can advertise the kind of
flooding reduction mechanism used in the flooding domain.
Implementations of this draft MAY implement the election and
advertisement of a flooding domain leader as described in section
5.1.1 of [I-D.ietf-lsr-dynamic-flooding]. If the election of a
flooding domain leader is implemented, implementations SHOULD also
advertise the flooding mechanism using the IS-IS Dynamic Flooding
Sub-TLV described in section 5.1.2 of
[I-D.ietf-lsr-dynamic-flooding], using Algorithm number (TBD).
4. Security Considerations
This document outlines modifications to the IS-IS protocol for
operation on high density network topologies. Implementations SHOULD
implement IS-IS cryptographic authentication, as described in
[RFC5304], and should enable other security measures in accordance
with best common practices for the IS-IS protocol.
White, et al. Expires January 28, 2021 [Page 6]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
5. References
5.1. Normative References
[I-D.ietf-lsr-dynamic-flooding]
Li, T., Psenak, P., Ginsberg, L., Chen, H., Przygienda,
T., Cooper, D., Jalil, L., Dontula, S., and G. Mishra,
"Dynamic Flooding on Dense Graphs", draft-ietf-lsr-
dynamic-flooding-07 (work in progress), June 2020.
[ISO10589]
International Organization for Standardization,
"Intermediate system to Intermediate system intra-domain
routeing information exchange protocol for use in
conjunction with the protocol for providing the
connectionless-mode Network Service (ISO 8473)", ISO/
IEC 10589:2002, Second Edition, Nov 2002.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC2629] Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629,
DOI 10.17487/RFC2629, June 1999,
<https://www.rfc-editor.org/info/rfc2629>.
[RFC5120] Przygienda, T., Shen, N., and N. Sheth, "M-ISIS: Multi
Topology (MT) Routing in Intermediate System to
Intermediate Systems (IS-ISs)", RFC 5120,
DOI 10.17487/RFC5120, February 2008,
<https://www.rfc-editor.org/info/rfc5120>.
[RFC5301] McPherson, D. and N. Shen, "Dynamic Hostname Exchange
Mechanism for IS-IS", RFC 5301, DOI 10.17487/RFC5301,
October 2008, <https://www.rfc-editor.org/info/rfc5301>.
[RFC5303] Katz, D., Saluja, R., and D. Eastlake 3rd, "Three-Way
Handshake for IS-IS Point-to-Point Adjacencies", RFC 5303,
DOI 10.17487/RFC5303, October 2008,
<https://www.rfc-editor.org/info/rfc5303>.
[RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic
Engineering", RFC 5305, DOI 10.17487/RFC5305, October
2008, <https://www.rfc-editor.org/info/rfc5305>.
White, et al. Expires January 28, 2021 [Page 7]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
[RFC5308] Hopps, C., "Routing IPv6 with IS-IS", RFC 5308,
DOI 10.17487/RFC5308, October 2008,
<https://www.rfc-editor.org/info/rfc5308>.
[RFC5309] Shen, N., Ed. and A. Zinin, Ed., "Point-to-Point Operation
over LAN in Link State Routing Protocols", RFC 5309,
DOI 10.17487/RFC5309, October 2008,
<https://www.rfc-editor.org/info/rfc5309>.
[RFC5311] McPherson, D., Ed., Ginsberg, L., Previdi, S., and M.
Shand, "Simplified Extension of Link State PDU (LSP) Space
for IS-IS", RFC 5311, DOI 10.17487/RFC5311, February 2009,
<https://www.rfc-editor.org/info/rfc5311>.
[RFC5316] Chen, M., Zhang, R., and X. Duan, "ISIS Extensions in
Support of Inter-Autonomous System (AS) MPLS and GMPLS
Traffic Engineering", RFC 5316, DOI 10.17487/RFC5316,
December 2008, <https://www.rfc-editor.org/info/rfc5316>.
[RFC7356] Ginsberg, L., Previdi, S., and Y. Yang, "IS-IS Flooding
Scope Link State PDUs (LSPs)", RFC 7356,
DOI 10.17487/RFC7356, September 2014,
<https://www.rfc-editor.org/info/rfc7356>.
[RFC7981] Ginsberg, L., Previdi, S., and M. Chen, "IS-IS Extensions
for Advertising Router Information", RFC 7981,
DOI 10.17487/RFC7981, October 2016,
<https://www.rfc-editor.org/info/rfc7981>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
5.2. Informative References
[I-D.ietf-isis-segment-routing-extensions]
Previdi, S., Ginsberg, L., Filsfils, C., Bashandy, A.,
Gredler, H., and B. Decraene, "IS-IS Extensions for
Segment Routing", draft-ietf-isis-segment-routing-
extensions-25 (work in progress), May 2019.
[RFC3277] McPherson, D., "Intermediate System to Intermediate System
(IS-IS) Transient Blackhole Avoidance", RFC 3277,
DOI 10.17487/RFC3277, April 2002,
<https://www.rfc-editor.org/info/rfc3277>.
White, et al. Expires January 28, 2021 [Page 8]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
[RFC3719] Parker, J., Ed., "Recommendations for Interoperable
Networks using Intermediate System to Intermediate System
(IS-IS)", RFC 3719, DOI 10.17487/RFC3719, February 2004,
<https://www.rfc-editor.org/info/rfc3719>.
[RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
Border Gateway Protocol 4 (BGP-4)", RFC 4271,
DOI 10.17487/RFC4271, January 2006,
<https://www.rfc-editor.org/info/rfc4271>.
[RFC5304] Li, T. and R. Atkinson, "IS-IS Cryptographic
Authentication", RFC 5304, DOI 10.17487/RFC5304, October
2008, <https://www.rfc-editor.org/info/rfc5304>.
[RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation
Element (PCE) Communication Protocol (PCEP)", RFC 5440,
DOI 10.17487/RFC5440, March 2009,
<https://www.rfc-editor.org/info/rfc5440>.
[RFC5449] Baccelli, E., Jacquet, P., Nguyen, D., and T. Clausen,
"OSPF Multipoint Relay (MPR) Extension for Ad Hoc
Networks", RFC 5449, DOI 10.17487/RFC5449, February 2009,
<https://www.rfc-editor.org/info/rfc5449>.
[RFC5614] Ogier, R. and P. Spagnolo, "Mobile Ad Hoc Network (MANET)
Extension of OSPF Using Connected Dominating Set (CDS)
Flooding", RFC 5614, DOI 10.17487/RFC5614, August 2009,
<https://www.rfc-editor.org/info/rfc5614>.
[RFC5820] Roy, A., Ed. and M. Chandra, Ed., "Extensions to OSPF to
Support Mobile Ad Hoc Networking", RFC 5820,
DOI 10.17487/RFC5820, March 2010,
<https://www.rfc-editor.org/info/rfc5820>.
[RFC5837] Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen,
N., and JR. Rivers, "Extending ICMP for Interface and
Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837,
April 2010, <https://www.rfc-editor.org/info/rfc5837>.
[RFC6232] Wei, F., Qin, Y., Li, Z., Li, T., and J. Dong, "Purge
Originator Identification TLV for IS-IS", RFC 6232,
DOI 10.17487/RFC6232, May 2011,
<https://www.rfc-editor.org/info/rfc6232>.
[RFC7182] Herberg, U., Clausen, T., and C. Dearlove, "Integrity
Check Value and Timestamp TLV Definitions for Mobile Ad
Hoc Networks (MANETs)", RFC 7182, DOI 10.17487/RFC7182,
April 2014, <https://www.rfc-editor.org/info/rfc7182>.
White, et al. Expires January 28, 2021 [Page 9]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
[RFC7921] Atlas, A., Halpern, J., Hares, S., Ward, D., and T.
Nadeau, "An Architecture for the Interface to the Routing
System", RFC 7921, DOI 10.17487/RFC7921, June 2016,
<https://www.rfc-editor.org/info/rfc7921>.
Appendix A. Flooding Optimization Operation
Recent testing has shown that flooding is largely a "non-issue" in
terms of scaling when using high speed links connecting intermediate
systems with reasonable processing power and memory. However,
testing has also shown that flooding will impact convergence speed
even in such environments, and flooding optimization has a major
impact on the performance of a link state protocol in resource
constrained environments. Some thoughts on flooding optimization in
general, and the flooding optimization contained in this document,
follow.
There are two general classes of flooding optimization available for
link state protocols. The first class of optimization relies on a
centralized service or server to gather the link state information
and redistribute it back into the intermediate systems making up the
fabric. Such solutions are attractive in many, but not all,
environments; hence these systems compliment, rather than compete
with, the system described here. Systems relying on a service or
server necessarily also rely on connectivity to that service or
server, either through an out-of-band network or connectivity through
the fabric itself. Because of this, these mechanisms do not apply to
all deployments; some deployments require underlying reachability
regardless of connectivity to an outside service or server.
The second possibility is to create a fully distributed system that
floods the minimal amount of information possible to every
intermediate system. The system described in this draft is an
example of such a system. Again, there are many ways to accomplish
this goal, but simplicity is a primary goal of the system described
in this draft.
The system described here divides the work into two different parts;
forward and reverse optimization. The forward optimization begins by
finding the set of intermediate systems two hops away from the
flooding device, and choosing a subset of connected neighbors that
will successfully reach this entire set of intermediate systems, as
shown in the diagram below.
White, et al. Expires January 28, 2021 [Page 10]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
G
|
A B C--+
| | | |
+--D--+ E H
| | |
+----F--+--+
Figure 2
If F is flooding some piece of information, then it will find the
entire set of intermediate systems within two hops by discovering its
neighbors and their neighbors from the local LSDB. This will include
A, B, C, D, and E--but not G. From this set, F can determine that D
can reach A and B, while a single flood to either E or H will reach
C. Hence F can flood to D and either E or H to reach C. F can
choose to flood to D and E normally. Because H still needs to
receive this new LSP (or fragment!), but does not need to reflood to
C, F can send the LSP using link local signaling. In this case, H
will receive and process the new LSP, but not reflood it.
Rather than carrying the information necessary through hello
extensions, as is done in [RFC5820], the neighbors are allowed to
complete initial synchronization, and then a truncated shortest path
tree is built to determine the "two hop neighborhood." This has the
advantage of using mechanisms already used in IS-IS, rather than
adding new processes. The risk with this process is any LSPs flooded
through the network before this initial calculation takes place will
be suboptimal. This "two hop neighborhood" process has been used in
OSPF deployments for a number of years, and has proven stable in
practice.
Rather than setting a timer for reflooding, the implementation
described here uses IS-IS' ability to describe the entire database
using a CSNP to ensure flooding is successful. This adds some small
amount of overhead, so there is some balance between optimal flooding
and ensuring flooding is complete.
The reverse optimization is simpler. It relies on the observation
that any intermediate system between the local IS and the origin of
the LSP, other than in the case of floods removing an LSP from the
shared LSDB, should have already received a copy of the LSP. For
instance, if F originates an LSP in the figure above, and E refloods
the LSP to C, C does not need to reflood back to F if F is on its
shortest path tree towards F. It is obvious this is not a "perfect"
optimization. A perfect optimization would block flooding back along
a directed acyclic graph towards the originator. Using the SPT,
White, et al. Expires January 28, 2021 [Page 11]
Internet-DraIS-IS Optimal Distributed Flooding for Dense Topo July 2020
however, is a quick way to reduce flooding without performing more
calculations.
The combination of these two optimizations have been seen, in
testing, to reduce the number of copies any IS receives from the tens
to precisely one.
Authors' Addresses
Russ White
Juniper Networks
Email: russ@riw.us
Shraddha Hegde
Juniper Networks
Email: shraddha@juniper.net
Shawn Zandi
LinkedIn
Email: szandi@linkedin.com
White, et al. Expires January 28, 2021 [Page 12]
