ptb -> router ecmp -> nexthop L4/L7 load balancer -> destination

  router --> load balancer 1 --->
       \\--> load balancer 2 ---> load-balanced service
        \--> load balancer N --->

            

Figure 1

The router ECMP decision is used because it is part of the forwarding architecture, can be performed at line rate, and does not depend on shared state or coordination across a distributed forwarding architecture which may include multiple routers. The ECMP routing decision is deterministic with respect to packets having the same computed hash.

The typical case where ICMPv6 PTB messages are received at the load balancer is where the path MTU from the client to the load balancer is limited by a tunnel in which the client itself is not aware of. In particular, in the case of a TCP connection where TLS is employed, the first packet that is likely to exceed a tunnel MTU lower than that specified by the MSS on the client and the load balancer/server is the TLS ServerHello and certificate.

Direct experience says that the frequency of PTB messages is small compared to total flows. This says a lot about native vs. tunneled IPv6 deployment and the relative maturity of production IPv6 deployment. Techniques such as happy-eyeballs may actually contribute some amelioration to the IPv6 client experience. Still, the expectation is that PMTUD should work and that unnecessary breakage of client traffic should be avoided.

Some final observations are that it is typically not possible even if potentially desirable to be able to independently set the TCP MSS for different address families on end-systems.

The problem as described does also impact IPv4; however, the ability to fragment on wire and the relative rarity of sub-1500 byte MTUs that are not coupled to changes in client behavior (for example, endpoint VPN clients set the tunnel interface MTU accordingly for performance reasons) makes the problem sufficiently rare that some deployments simply choose to ignore it.

3. Mitigation

Mitigation of the described issue involves ensuring that an ICMPv6 error message is distributed to the same anycast server responsible for the flow for which the error is generated. Mitigation could be done by looking into the payload of the ICMPv6 message (to determine which TCP flow it was associated with) before making a forwarding decision.

Alternative mitigation is predicated upon distributing the PTB message to all anycast servers under the assumption that the one that needs it will be able to match it to a flow and the others can update their route cache with the new MTU. Such distribution has significant implications for resource consumption and the potential for self-inflicted denial-of-service if not carefully employed. Fortunately, the number of flows for which this problem occurs is relatively small (10 or fewer pps on 1Gb/s or more worth of https traffic) and sensible ingress rate limits can protect the anycast server tiers with potential fallout only under circumstances of deliberate duress.

3.1. Alternatives

As an alternative it is assumed to be appropriate to lower the TCP MSS to 1220 in order to accommodate 1280 byte MTU. We consider this undesirable as hosts may not be able to independently set TCP MSS by address-family, or alternatively that it relies on a middle-box to clamp the MSS independently from the end-systems.

3.2. Implementation

1. Filter-based-forwarding matches next-header ICMPv6 type-2 and matches a next-hop on a particular subnet directly attached to both border routers. (Filter is policed to reasonable limits, we chose 1000pps)
2. Filter is applied on input side of all external interfaces
3. A proxy located at the next hop forwards ICMPv6 type-2 packets received at the next-hop to an Ethernet broadcast address (example ff:ff:ff:ff:ff:ff) on all specified subnets. This was necessitated by the routers inability (in IPv6) to forward the same packet to multiple next-hops.
4. Anycast servers receive the PTB error and process packet as needed

A simple Python scapy script can be used to perform the ICMPv6 proxy reflection.

      #!/usr/bin/python

      from scapy.all import *

      IFACE_OUT = ["p2p1", "p2p2"]

      def icmp6_callback(pkt):
          if pkt.haslayer(IPv6) and (ICMPv6PacketTooBig in pkt) and pkt[Ether].dst != 'ff:ff:ff:ff:ff:ff':
              del(pkt[Ether].src)
              pkt[Ether].dst = 'ff:ff:ff:ff:ff:ff'
              pkt.show()
              for iface in IFACE_OUT:
                  sendp(pkt, iface=iface)

      def main():
          sniff(prn=icmp6_callback, filter="icmp6 and (ip6[40+0] == 2)", store=0)

      if __name__ == '__main__':
          main()
 

This example script listens on all interfaces for IPv6 PTB errors being forwarded using filter-based-forwarding. It removes the existing Ethernet source and rewrites a new Ethernet destination of the Ethernet broadcast address. It then sends the resulting frame out the p2p1 and p2p2 interfaces where our anycast servers reside.

4. Improvements

There are several ways to imagine that improvements could be made to the situation with respect to ECMP load balancing of ICMPv6 PTB.

1. Routers with sufficient capacity within the lookup process could parse all the way through the L3 or L4 header in the ICMPv6 payload beginning at bit offset 32 of the ICMP header. By reordering the elements of the hash to match the inward direction of the flow, the PTB error could be directed to the same next-hop as the incoming packets in the flow.
2. The FIB could be programmed with a multicast distribution tree that included all of the necessary next-hops.
3. Ubiquitous implementation of RFC 4821 [RFC4821] Packetization Layer Path MTU Discovery would probably go a long way towards reducing dependence on ICMPv6 PTB.

5. Acknowledgements

The authors would like to thank Ray Hunter for review.

6. IANA Considerations

This memo includes no request to IANA.

7. Security Considerations

The employed mitigation has the potential to greatly amplify the impact of a deliberately malicious sending of ICMPv6 PTB messages. Sensible ingress rate limiting can reduce the potential for impact; however, legitimate traffic may be lost in the process.

The proxy replication results in devices not associated with the flow that generated the PTB being recipients of an ICMPv6 message which contains a fragment of a packet. This could arguably result in information disclosure. Recipient machines should be in a common administrative domain.

8. Informative References

[RFC4821]

Mathis, M. and J. Heffner, "Packetization Layer Path MTU Discovery", RFC 4821, March 2007.

Authors' Addresses