+---------+             +--------+
              | Spine1  |             | Spine2 |
              +-+-+-+-+-+             +-+-+-+-++
                | | | |                 | | | |
                | | | +-----------------|-|-|-|-+
                | | +------------+      | | | | |
       +--------+ +-+            |      | | | | |
       |   +----------------------------+ | | | |
       |   |        |  +------------------+ | +----+
       |   |        |  |         |  +-------+   |  |
       |   |        |  |         |  |           |  |
     +-+---+-+   +--+--+-+     +-+--+--+     +--+--+-+
     | Leaf1 |   | Leaf2 |     | Leaf3 |     | Leaf4 |
     +-------+   +-------+     +-------+     +-------+

Figure 2: A Fat Tree Topology

3.2. Applicability Statement

This extension assumes the network is a Spine-Leaf topology, and it should not be applied in an arbitrary network setup. The spine nodes can be viewed as the aggregation layer of the network, and the leaf nodes as the access layer of the network. The leaf nodes use a load sharing algorithm with spine nodes as nexthops in routing and forwarding.

This extension works when the spine nodes are inter-connected, and it works with a pure CLOS or Fat Tree topology based network where the spines are NOT interconnected.

Although the example diagram in Figure 1 shows a fully meshed Spine-Leaf topology, this extension also works in the case where they are partially meshed. For instance, leaf1 through leaf10 may be fully meshed with spine1 through spine5 while leaf11 through leaf20 is fully meshed with spine4 through spine8, and all the spines are inter-connected in a redundant fashion.

This extension also works with a topology with more than the typical two layers of spine and leaf. For instance, in example diagrams Figure 1 and Figure 2, there can be another Core layer of routers/switches on top of the aggregation layer. From an IS-IS routing point of view, the Core nodes are not affected by this extension and will have the complete topology and routing information just like the spine nodes. To make the network even more scalable, the Core layer can operate as a level-2 IS-IS sub-domain while the Spine and Leaf layers operate as stays at the level-1 IS-IS domain.

This extension also supports the leaf nodes having local connections to other leaf nodes, in the example diagram Figure 1 there is a connection between 'Leaf1' node and 'Leaf2' node, and an external host can be dual homed into both of the leaf nodes.

This extension assumes the link between the spine and leaf nodes are point-to-point, or point-to-point over LAN [RFC5309]. The links connecting among the spine nodes or the links between the leaf nodes can be any type.

3.3. Extension Encoding

This extension introduces one TLV which may be used in IS-IS Hello (IIH) PDUs or in Circuit Scoped Link State PDUs (CS-LSP) [RFC7356]. It is used by both spine and leaf nodes in this Spine-Leaf mechanism.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |      Type     |     Length    |            SL Flag            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         .. Optional Sub-TLVs
    +-+-+-+-+-+-+-+-+-.... 

The fields of this TLV are defined as follows:

   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Reserved         |B|R|L|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
              

• Type:

  1 octet Suggested value 150 (to be assigned by IANA)

  Length:

  1 octet (2 + length of sub-TLVs).

  Flags

  16 bits

  • L bit (0x01):

    Only leaf node sets this bit. If the L bit is set in the SL flag, the node indicates it is in 'Leaf-Mode'.

    R bit (0x02):

    Only Spine node sets this bit. If the R bit is set, the node indicates to the leaf neighbor that it can be used as the default route gateway.

    B bit (0x04):

    Only leaf node sets this bit on Leaf-Leaf link, in additional to the 'L' bit setting. If the B bit is set, the node indicates to its leaf neighbor that it can be used as the backup default route gateway.

  Optional Sub-TLV:

  Not defined in this document, for future extension

  sub-TLVs MAY be included when the TLV is in a CS-LSP. sub-TLVs MUST NOT be included when the TLV is in an IIH

3.3.1. Spine-Leaf Sub-TLVs

If the data center topology is a pure CLOS or Fat Tree, there are no link connections among the spine nodes. If we also assume there is not another Core layer on top of the aggregation layer, then the traffic from one leaf node to another may have a problem if there is a link outage between a spine node and a leaf node. For instance, in the diagram of Figure 2, if Leaf1 sends data traffic to Leaf3 through Spine1 node, and the Spine1-Leaf3 link is down, the data traffic will be dropped on the Spine1 node.

To address this issue spine and leaf nodes may send/request specific reachability information via the sub-TLVs defined below.

Two Spine-Leaf sub-TLVs are defined. The Leaf-Set sub-TLV and the Info-Req sub-TLV.

3.3.1.1. Leaf-Set Sub-TLV

This sub-TLV is used by spine nodes to optionally advertise Leaf neighbors to other Leaf nodes. The fields of this sub-TLV are defined as follows:

• Type:

  1 octet Suggested value 1 (to be assigned by IANA)

  Length:

  1 octet MUST be a multiple of 6 octets.

  Leaf-Set:

  A list of IS-IS System-ID of the leaf node neighbors of this spine node.

3.3.1.2. Info-Req Sub-TLV

This sub-TLV is used by leaf nodes to request more specific prefix information from a selected spine node, upon detecting one of the spine node has lost the connection to a leaf node. The fields of this sub-TLV are defined as follows:

• Type:

  1 octet Suggested value 2 (to be assigned by IANA)

  Length:

  1 octet. It MUST be a multiple of 6 octets.

  Info-Req:

  List of IS-IS System-IDs of leaf nodes for which connectivity information is being requested.

3.3.2. Advertising IPv4/IPv6 Reachability

In cases where connectivity between a leaf node and a spine node is down, the leaf node MAY request reachability information from a spine node as described in Section 3.3.1.2. The spine node utilizes TLVs 135 [RFC5305] and TLVs 236 [RFC5308] to advertise this information. These TLVs MAY be included either in IIHs or CS-LSPs sent from the spine to the requesting leaf node. Sending such information in IIHs has limited scale - all reachability information MUST fit within a single IIH. It is therefore recommended that CS-LSPs be used.

3.4. Mechanism

Each leaf node is provisioned by network operators as an IS-IS 'Leaf-Node'. A spine node does not need explicit configuration. A leaf node inserts the Spine-Leaf TLV in IIHs it originates. The TLV has the 'L' bit set in the flags field.

When a spine node receives an IIH with the SL TLV and 'L' bit set, it labels the point-to-point interface and adjacency to be a 'Leaf-Peer'. IIHs sent by the spine node on a link to a Leaf-Peer includes the Spine-Leaf TLV with the 'R' bit set in the flags field. The 'R' bit indicates to the 'Leaf-Peer' neighbor that the spine node can be used as a default routing nexthop.

There is no change to the IS-IS adjacency bring-up mechanism for Spine-Leaf peers.

For the spine node with 'Leaf-Peer' adjacencies, the IS-IS LSP flooding is blocked to the 'Leaf-Peer' interface, except for the LSP PDUs in which the IS-IS System-ID matches the System-ID of the 'Leaf-Peer' adjacency. This exception is needed since when the leaf node reboots, the spine node needs to forward to the leaf node its previous generation of LSP. No other LSP PDU needs to be flooded over this 'Leaf-Peer' interface.

The leaf node will perform IS-IS LSP flooding as normal over all of its IS-IS adjacencies, this means the leaf node will flood its own LSPs over to spine nodes since those are all the LSPs in its LSP database.

The spine node will receive all the LSP PDUs in the network, including all the spine nodes and leaf nodes. It will perform Shortest Path First (SPF) as normal IS-IS node does. There is no change to the route calculation and forwarding on the spine nodes.

But the leaf node does not have any LSP in the network except for its own, and there is no need to perform SPF algorithm on the system. It only needs to download the default route with the nexthops of those 'Spine-Peer' which has the 'R' bit set in the Spine-Leaf TLV in IIH PDUs. IS-IS can perform equal cost or unequal cost load sharing while using the spine nodes as nexthops. The aggregated metric of the outbound interface and the 'Reverse Metric' [REVERSE-METRIC] can be used for this purpose.

In summary, this extension requires leaf node to insert Spine-Leaf TLV in IIH, and set the 'L' bit in the SL flag, and download IS-IS default route using the spine nodes as nexthops where the 'Spine-Peer' set the 'R' bit in its IIH PDU; It requires spine node to respond from 'Leaf-Peer' by inserting Spine-Leaf TLV in its IIH, setting the 'R' bit in the SL flag, and blocking the LSP flooding with the exception that it will set SRMflag on the LSPs that belong to the 'Leaf-Peer' over that interface.

3.4.1. Pure CLOS Topology

In a data center where the topology is pure CLOS or Fat Tree, there is no interconnection among the spine nodes, and there is not another Core layer above the aggregation layer, when the link between a spine and a leaf goes down, there is a possibility of black holing the data traffic in the network.

As in the diagram Figure 2, if the link Spine1-Leaf3 goes down, there needs to be a way for Leaf1, Leaf2 and Leaf4 to avoid the Spine1 if the destination of data traffic is to Leaf3 node.

In the above example, the Spine1 and Spine2 are provisioned to advertise the Spine-Leaf sub-TLV of Leaf-Set. Originally both Spines will advertise Leaf1 through Leaf4 as their Leaf-Set. When the Spine1-Leaf3 link is down, Spine1 will only have Leaf1, Leaf2 and Leaf4 in its Leaf-Set. This allows the other leaf nodes to know that Spine1 has lost the leaf node of Leaf3.

Each leaf node can select another spine node to request for some prefix information associated with the lost leaf node. In this diagram of Figure 2, there are only two spine nodes (Spine-Leaf topology can have more than two spine nodes in general). Each leaf node can independently select a spine node for the leaf information. The leaf nodes will include the Info-Req sub-TLV in the Spine-Leaf TLV towards that spine node, Spine2 in this case.

The spine node, upon receiving the request from one or more leaf nodes, it will find the associated IPv6/IPv6 prefixes for this requested client node, and the spine node will include the IPv6/IPv4 Info-Advertise sub-TLV when sending message towards the leaf nodes. For instance, it will include the IPv4 loopback prefix of the leaf3 based on the policy configured or administrative tag attached to the prefixes. When the leaf nodes receive the more specific prefixes, they will install the advertised prefixes towards the other spine nodes (Spine2 in this example).

For instance in the data center overlay scenario, when any IP destination or MAC destination uses the leaf3's loopback as the tunnel nexthop, the overlay tunnel from leaf nodes will only select Spine2 as the gateway to reach leaf3 as long as the Spine1-Leaf3 link is still down.

3.5. Implementation and Operation

3.5.1. CSNP PDU

In Spine-Leaf extension, Complete Sequence Number PDU (CSNP) does not need to be transmitted over the Spine-Leaf link. Some IS-IS implementation sends CSNPs after the initial adjacency bring-up over point-to-point interface. There is no need for this optimization here since the Leaf does not need to receive any other LSPs from the network, and the only LSPs transmitted across the Spine-Leaf link is the leaf node LSP.

Also in the graceful restart case[RFC5306], for the same reason, there is no need to send the CSNPs over the Spine-Leaf interface. It only needs to set the SRMflag on the LSPs belonging to the 'Leaf-Peer' on the spine node, and set the SRMflag on its own LSPs on the leaf node.

3.5.2. Leaf to Leaf connection

Leaf to leaf node links are useful in host redundancy cases in switching networks, and normally there is no special requirement of mechanism is needed for this case. Each leaf node will set the 'L' bit in its IIH of the Spine-Leaf flag. LSP will be exchanged over this link. In the example diagram Figure 1, the Leaf1 will get Leaf2's LSP and Leaf2 will get Leaf1's LSP. They will install more specific routes towards each other using this local Leaf-Leaf link. SPF will be performed in this case just like when the entire network only involves with those two IS-IS nodes. This does not affect the normal Spine-Leaf mechanism they perform toward the spine nodes.

Besides the local leaf-to-leaf traffic, the leaf node can serve as a backup gateway for its leaf neighbor. It needs to remove the 'Overload-Bit' setting in its LSP, and it sets both the 'L' bit and the 'B' bit in the SL-flag with a high 'Reverse Metric' value.

3.5.3. Overload Bit

The leaf node SHOULD set the 'overload' bit on its LSP PDU, since if the spine nodes were to forward traffic not meant for the local node, the leaf node does not have the topology information to prevent a routing/forwarding loop.

3.5.4. Spine Node Hostname

This extension creates a non-reciprocal relationship between the spine node and leaf node. The spine node will receive leaf's LSP and will know the leaf's hostname, but the leaf does not have spine's LSP. This extension allows the Dynamic Hostname TLV [RFC5301] to be optionally included in spine's IIH PDU when sending to a 'Leaf-Peer'. This is useful in troubleshooting cases.

3.5.5. IS-IS Reverse Metric

This metric is part of the aggregated metric for leaf's default route installation with load sharing among the spine nodes. When a spine node is in 'overload' condition, it should use the IS-IS Reverse Metric TLV in IIH [REVERSE-METRIC] to set this metric to maximum to discourage the leaf using it as part of the loadsharing.

In some cases, certain spine nodes may have less bandwidth in link provisioning or in real-time condition, and it can use this metric to signal to the leaf nodes dynamically.

In other cases, such as when the spine node loses a link to a particular leaf node, although it can redirect the traffic to other spine nodes to reach that destination leaf node, but it MAY want to increase this metric value if the inter-spine connection becomes over utilized, or the latency becomes an issue.

In the leaf-leaf link as a backup gateway use case, the 'Reverse Metric' SHOULD always be set to very high value.

3.5.6. Spine-Leaf Traffic Engineering

Besides using the IS-IS Reverse Metric by the spine nodes to affect the traffic pattern for leaf default gateway towards multiple spine nodes, the IPv6/IPv4 Info-Advertise sub-TLVs can be selectively used by traffic engineering controllers to move data traffic around the data center fabric to alleviate congestion and to reduce the latency of a certain class of traffic pairs. By injecting more specific leaf node prefixes, it will allow the spine nodes to attract more traffic on some underutilized links.

3.5.7. Other End-to-End Services

Losing the topology information will have an impact on some of the end-to-end network services, for instance, MPLS TE or end-to-end segment routing. Some other mechanisms such as those described in PCE [RFC4655] based solution may be used. In this Spine-Leaf extension, the role of the leaf node is not too much different from the multi-level IS-IS routing while the level-1 IS-IS nodes only have the default route information towards the node which has the Attach Bit (ATT) set, and the level-2 backbone does not have any topology information of the level-1 areas. The exact mechanism to enable certain end-to-end network services in Spine-Leaf network is outside the scope of this document.

3.5.8. Address Family and Topology

IPv6 Address families[RFC5308], Multi-Topology (MT)[RFC5120] and Multi-Instance (MI)[RFC6822] information is carried over the IIH PDU. Since the goal is to simplify the operation of IS-IS network, for the simplicity of this extension, the Spine-Leaf mechanism is applied the same way to all the address families, MTs and MIs.

3.5.9. Migration

For this extension to be deployed in existing networks, a simple migration scheme is needed. To support any leaf node in the network, all the involved spine nodes have to be upgraded first. So the first step is to migrate all the involved spine nodes to support this extension, then the leaf nodes can be enabled with 'Leaf-Mode' one by one. No flag day is needed for the extension migration.

4. IANA Considerations

A new TLV codepoint is defined in this document and needs to be assigned by IANA from the "IS-IS TLV Codepoints" registry. It is referred to as the Spine-Leaf TLV and the suggested value is 150. This TLV is only to be optionally inserted either in the IIH PDU or in the Circuit Flooding Scoped LSP PDU. IANA is also requested to maintain the SL-flag bit values in this TLV, and 0x01, 0x02 and 0x04 bits are defined in this document.

   Value  Name                   IIH  LSP  SNP  Purge  CS-LSP
   -----  ---------------------  ---  ---  ---  -----  -------
   150    Spine-Leaf              y    n    n    n        y
            

This extension also proposes to have the Dynamic Hostname TLV, already assigned as code 137, to be allowed in IIH PDU.

   Value  Name                   IIH  LSP  SNP  Purge
   -----  ---------------------  ---  ---  ---  -----
   137    Dynamic Name            y    y    n    y
            

Two new sub-TLVs are defined in this document and needs to be added assigned by IANA from the "IS-IS TLV Codepoints". They are referred to in this document as the Leaf-Set sub-TLV and the Info-Req sub-TLV. It is suggested to have the values 1 and 2 respectively.

5. Security Considerations

Security concerns for IS-IS are addressed in [ISO10589], [RFC5304], [RFC5310], and [RFC7602]. This extension does not raise additional security issues.

6. Acknowledgments

TBD.

7. Document Change Log

7.1. Changes to draft-shen-isis-spine-leaf-ext-03.txt

• Submitted March 2017.
• Added the Spine-Leaf sub-TLVs to handle the case of data center pure CLOS topology and mechanism.
• Added the Spine-Leaf TLV and sub-TLVs can be optionally inserted in either IIH PDU or CS-LSP PDU.
• Allow use of prefix Reachability TLVs 135 and 236 in IIHs/CS-LSPs sent from spine to leaf.

7.2. Changes to draft-shen-isis-spine-leaf-ext-02.txt

• Submitted October 2016.
• Removed the 'Default Route Metric' field in the Spine-Leaf TLV and changed to using the IS-IS Reverse Metric in IIH.

7.3. Changes to draft-shen-isis-spine-leaf-ext-01.txt

• Submitted April 2016.
• No change. Refresh the draft version.

7.4. Changes to draft-shen-isis-spine-leaf-ext-00.txt

• Initial version of the draft is published in November 2015.

8. References

8.1. Normative References

[ISO10589]	ISO "International Organization for Standardization", "Intermediate system to Intermediate system intra-domain routeing information exchange protocol for use in conjunction with the protocol for providing the connectionless-mode Network Service (ISO 8473), ISO/IEC 10589:2002, Second Edition.", Nov 2002.
[REVERSE-METRIC]	Shen, N., Amante, S. and M. Abrahamsson, "IS-IS Routing with Reverse Metric", Internet-Draft draft-ietf-isis-reverse-metric-04, 2016.
[RFC2119]	Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC5120]	Przygienda, T., Shen, N. and N. Sheth, "M-ISIS: Multi Topology (MT) Routing in Intermediate System to Intermediate Systems (IS-ISs)", RFC 5120, DOI 10.17487/RFC5120, February 2008.
[RFC5301]	McPherson, D. and N. Shen, "Dynamic Hostname Exchange Mechanism for IS-IS", RFC 5301, DOI 10.17487/RFC5301, October 2008.
[RFC5304]	Li, T. and R. Atkinson, "IS-IS Cryptographic Authentication", RFC 5304, DOI 10.17487/RFC5304, October 2008.
[RFC5305]	Li, T. and H. Smit, "IS-IS Extensions for Traffic Engineering", RFC 5305, DOI 10.17487/RFC5305, October 2008.
[RFC5306]	Shand, M. and L. Ginsberg, "Restart Signaling for IS-IS", RFC 5306, DOI 10.17487/RFC5306, October 2008.
[RFC5308]	Hopps, C., "Routing IPv6 with IS-IS", RFC 5308, DOI 10.17487/RFC5308, October 2008.
[RFC5310]	Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R. and M. Fanto, "IS-IS Generic Cryptographic Authentication", RFC 5310, DOI 10.17487/RFC5310, February 2009.
[RFC6822]	Previdi, S., Ginsberg, L., Shand, M., Roy, A. and D. Ward, "IS-IS Multi-Instance", RFC 6822, DOI 10.17487/RFC6822, December 2012.
[RFC7356]	Ginsberg, L., Previdi, S. and Y. Yang, "IS-IS Flooding Scope Link State PDUs (LSPs)", RFC 7356, DOI 10.17487/RFC7356, September 2014.
[RFC7602]	Chunduri, U., Lu, W., Tian, A. and N. Shen, "IS-IS Extended Sequence Number TLV", RFC 7602, DOI 10.17487/RFC7602, July 2015.

8.2. Informative References

[RFC4655]	Farrel, A., Vasseur, J. and J. Ash, "A Path Computation Element (PCE)-Based Architecture", RFC 4655, DOI 10.17487/RFC4655, August 2006.
[RFC5309]	Shen, N. and A. Zinin, "Point-to-Point Operation over LAN in Link State Routing Protocols", RFC 5309, DOI 10.17487/RFC5309, October 2008.

Authors' Addresses