EVPN multi-homing support for L3 services

1.1. Problems with unicast load-balancing from core to CE

The layer-2 hashing performed by CE over its LAG port means that its possible for only one service PE to populate its ARP/ND cache. Take for example PE1 and PE2 from Figure 1. If CE1 ARP/ND response happens to always hash over I1 towards PE1, then PE2 ARP/ND table will be empty. Since unicast traffic from remote PEs can be received by either service PE, traffic that reaches the service PE2 will not find an ARP entry matching the host IP address and traffic will drop until ARP/ND resolves the adjacency.¶

If the CEs hash implementation always calculates the ARP/ND response towards PE1, the resolution on PE2 will never happen and traffic load balanced to PE2 will black-hole.¶

The route sync solution is described in Section 2.4¶

1.2. Problems with multicast from core to CE

Similar to the unicast behavior above, multicast IGMP join messages from CE to LAG link may always hash to a single PE.¶

When PIM runs on both redundant layer-3 PEs that both service multicast for the same access segment, PIM elects only one of the PEs as a PIM Designated Router (DR) using PIM DR election algorithm [RFC7761]. The PIM DR is responsible for tracking local multicast listeners and forwarding traffic to those listeners. The PIM DR is also responsible for sending local Join/Prune messages towards the RP or source.¶

For example, if in Figure 1 PE2 is designated PIM-RP, but CE IGMP join messages are hashed to I1 towards PE1, then multicast traffic will not be attracted to this service pair as PE2 will not send PIM Join on behalf of CE.¶

In order to ensure that the PIM DR always has all the MCAST route(s) and able to forward PIM Join/Prune message towards RP, BGP-EVPN multicast route-sync will be leveraged to synchronize MCAST route(s) learned to the DR.¶

When a fail-over occurs, multicast states would be pre-programmed on the newly elected DR service PE and assumes responsibility for the routing and forwarding of all the traffic.¶

The multicast route sync solution is described in Section 2.5¶

1.3. Problems with IGP adjacencies over the LAG port

A layer-3 CE device/router that connects to the redundant PEs may establish an IGP adjacency on the bundle port. In this case, the adjacency will be formed to one of the PEs and IGP customer route(s) will only be present on that PE.¶

This prevents the load-balancing benefits of redundant PEs from supporting this use case, as only one PE will be aware and advertising the customer routes to the core.¶

                  <---------+
                            | IGP Adj
    +-------+               |
    |       | 1.1.1.1/24    |
    | PE1   +-----------+   |
    |       |           |   |
    |       |           |   +
    +-------+           |
                        |
        +               |  +------+
  RT5   |             L |  | CE   +------>H1
  Sync  |             A +->+      |
        v             G |  |      |
                        |  |      +------>R1
    +-------+           |  +------+
    |       |           |    1.1.1.2/2
    | PE2   +-----------+
    |       | 1.1.1.1/24
    |       |
    +-------+

Figure 2 provides an example of this use case, where CE forms an IGP adjacency with PE1 (example: ISIS or OSPF), and advertises its H1 and R1 routes into the IP-VRF of PE1. PE1 may then redistribute this IGP route into the core as an L3 service. Any remote PEs will only be aware of the service from PE1, and cannot load balance through PE2 as well.¶

Further study is required in order to support the case of BGP PE to CE protocols.¶

A solution to this is described in Section 2.6¶

1.4. Problems with supporting multiple subnets on same ES in all active mode

In the case where the L3 service is L3VPN such as [RFC4364], it is likely the CE device could be a layer-2 switch supporting multiple subnets through the use of VLANs. In addition, each VLAN may be associated with a different customer VRF.¶

When ARP/ND routes are synchronized between the PEs for ARP proxy support using RT-2, a similar problem is encountered as described by Section 1.1 of [I-D.sajassi-bess-evpn-ac-aware-bundling]. The PE receiving RT-2 is unable to determine which sub-interface the ARP/ND entry is associated with.¶

When IGMP routes are synchronized between the PEs using RT-7 and RT-8, a similar problem is encountered as described by Section 1.2 of [I-D.sajassi-bess-evpn-ac-aware-bundling]. The PE receiving RT-7 and RT-8 is unable to determine which sub-interface the IGMP join is associated with.¶

This document proposes to use the solution defined by Section 4 of [I-D.sajassi-bess-evpn-ac-aware-bundling] to solve both these cases. All route sync messages (RT-2, RT-5, RT-7, RT-8) will carry an Attachment Circuit Identifier Extended Community to signal which sub-interface the routes were learnt on.¶

1.5. Acronyms

BD:

Broadcast Domain. As per [RFC7432], an EVI consists of a single or multiple BDs. In case of VLAN-bundle and VLAN-aware bundle service model, an EVI contains multiple BDs.¶

DF:

Designated Forwarder¶

DR:

Designated Router¶

EC:

BGP Extended Community¶

ES:

Ethernet Segment. When a customer site (device or network) is connected to one or more PEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet Segment'.¶

ESI:

Ethernet Segment Identifier. A unique non-zero identifier that identifies an Ethernet Segment is called an 'Ethernet Segment Identifier'.¶

ETAG:

Ethernet Tag. An Ethernet tag identifies a particular broadcast domain, e.g., a VLAN. An EVPN instance consists of one or more broadcast domains.¶

EVI:

An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN¶

ICL:

Inter Chassis Link¶

IGMP:

Internet Group Management Protocol¶

IP-VRF:

A VPN Routing and Forwarding table for IP routes on an PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a layer 3 VPN in an PE.¶

L3AA

All-Active Redundancy Mode for Layer 3 services. When all PEs attached to an Ethernet segment are allowed to forward known unicast traffic to/from that Ethernet segment for a given VLAN, then the Ethernet segment is defined to be operating in All-Active redundancy mode.¶

MAC-VRF:

A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on a PE. A MAC-VRF is also an instantiation of an EVI in a PE¶

MC-LAG:

Multi-Chassis Link Aggregation Group (MC-LAG).¶

PE:

Provider Edge.¶

PIM:

Protocol Independent Multicast¶

RT-2:

EVPN route type 2, i.e., MAC/IP advertisement route, as defined in [RFC7432].¶

RT-5:

EVPN route type 5, i.e., IP Prefix route, as defined in Section 3 of [I-D.ietf-bess-evpn-prefix-advertisement]¶

RT-7:

EVPN route type 7, i.e., Multicast Join Synch Route, as defined in Section 9.2 of [I-D.ietf-bess-evpn-igmp-mld-proxy]¶

RT-8:

EVPN route type 8, i.e., Multicast Leave Synch Route, as defined in Section 9.3 of [I-D.ietf-bess-evpn-igmp-mld-proxy]¶

1.6. Requirements

1. The multi-homing solution MUST support Layer-3 access interface¶
2. The multi-homing solution MUST support Layer-3 access sub-interface¶
3. The solution MUST support unicast and multicast VPN services¶
4. The solution SHOULD support igp synchronization¶
5. The solution SHOULD support unicast and multicast GRT services¶
6. The solution MUST support all-active load-balancing mode¶
7. The solution MAY support single-active load-balancing mode¶
8. The solution MUST support port-active load-balancing mode¶

2.1. Mapping of L3VRF to EVPN EVI

A separate EVPN instance will be configured to each layer-3 VRF and be marked for route-sync only. Each L3-VRF will have a unique associated EVI ID. The multi-homed peer PEs MUST have the same configured EVI to layer-3 VRF mapping. This mapping also extends to the GRT, where a unique EVI ID can be assigned to support non VPN layer-3 services. Mis-configuration detection across peering PEs are left for further study.¶

When an EVPN instance is created as route-sync only, a MAC-VRF table is created to store all advertised routes. Local MAC learning may be disabled as this feature does not require MAC-only RT-2 advertisements.¶

This EVI is applicable to the multi-homed peer PEs only¶

The EVPN instance will be responsible for populating the following layer-3 VRF tables from remotely synced routes from peer PE¶

• ARP/ND¶
• IGMP¶
• IP (for customer subnets learned from IGP adjacency)¶

In the example Figure 3, route-syncs from VRF CUST1 will have EVI-RT BGP Extended Community (EC) with EVI 1, and VRF CUST2 will have EVI 2.¶

2.2. Mapping for L3 Interface to ESI

The ESI represents the L3 LAG interface between PE and CEs. This ESI is signalled using RT-4 with the ES-Import Route Target as described in Section 8.1.1 of [RFC7432] so that the service PE peers can discover each others common ES.¶

In the example Figure 3, route-syncs from interface BE1 have ES-Import RT EC with ESI 1¶

2.3. Mapping for L3 Sub-Interface to Attachment Circuit ID

The Attachment Circuit ID represens the sub-interface subnet on the L3 LAG interface between PE and CEs. The AC-ID is signalled using RT-2, RT-5, RT-7 and RT-8 by attaching Attachment Circuit ID Extended community as described in Section 6.1 of [I-D.sajassi-bess-evpn-ac-aware-bundling].¶

In the example Figure 3, route-syncs from sub-interface BE1.1 (VLAN1) have Attachment-Circuit-ID EC with ID 1¶

2.4. Route sync for ARP/ND

This document proposes solving the issue described in Section 1.1 using RT-2 IP/MAC route sync as described in Section 10 of [RFC7432] with a modification described below.¶

2.5. Route sync for IGMP

This document proposes solving the issue described in Section 1.2 using RT-7 and RT-8 route sync as described by [I-D.ietf-bess-evpn-igmp-mld-proxy].¶

Local IGMP join and leave will trigger a RT-7/8 route sync to peer PE.¶

2.6. Customer Subnet Route sync using Route-type(5)

Section 3 of [I-D.ietf-bess-evpn-prefix-advertisement] provides a mechanism to synchronize layer-3 customer subnets between the PEs in order to solve problem described in Section 1.3.¶

Using Figure 2 as example, if PE1 forms the IGP adjacency with CE, it will be the only PE with knowledge of the customer subnet R1. BGP on PE1 will then advertise R1 to remote PEs using L3-VPN signalling.¶

Although PE2 has the same ES connection to the CE, and could provide load balancing to remote PEs, due to it not having formed an IGP adjacency with CE it is not aware of the customer subnet R1.¶

This can be solved by PE1 signaling R1 to PE2 using a RT-5 synch route. BGP on PE2 can then advertise this customer subnet R1 towards the core is if it was locally learned through IGP, and provide load-balancing from the remote PEs.¶

The route-type(5) will carry the ESI as well as the gateway address GW (prefix next-hop address).¶

The same mapping mechanism will be used as for Route and IGMP sync, where EVI will determine the L3-VRF, ESI carried with route-type(5) will provide the main interface, and the gateway address will provide the nexthop.¶

2.7. Mapping for VLAN to ETAG

Another possible signalling of VLAN/sub-interface between service PE peers is to use the Ethernet Tag (ETAG) ID value in RT-2, RT-5, RT-7 and RT-8 as apposed to the Attachment Circuit Extended Community.¶

This will not work with vlan-aware bundling mode, but as that is a layer2 mode this should not prevent ETAGs use for L3 services.¶