___      ___
    /   \    /   \
 ..| AS1 |..| AS2 |..
:   \___/    \___/   :
:      \      /      :
:       \    /       :
:        \__/        :
: IXP   /    \       :
:      |  RS  |      :
:       \____/       :
:        /  \        :
:       /    \       :
:    __/      \__    :
:   /   \    /   \   :
 ..| AS3 |..| AS4 |..
    \___/    \___/
        

Figure 2: IXP-based Interconnection with Route Server

As illustrated in Figure 2, each router on the IXP fabric requires only a single BGP session to the route server, from which it can receive reachability information for all other routers on the IXP which also connect to the route server.

4. Operational Considerations for Route Server Installations

4.1. Path Hiding

"Path hiding" is a term used in [I-D.ietf-idr-ix-bgp-route-server] to describe the process whereby a route server may mask individual paths by applying conflicting routing policies to its Loc-RIB. When this happens, route server clients receive incomplete information from the route server about network reachability.

There are several approaches which may be used to mitigate against the effect of path hiding; these are described in [I-D.ietf-idr-ix-bgp-route-server]. However, the only method which does not require explicit support from the route server client is for the route server itself to maintain a individual Loc-RIB for each client which is the subject of conflicting routing policies.

4.2. Route Server Scaling

While deployment of multiple Loc-RIBs on the route server presents a simple way to avoid the path hiding problem noted in Section 4.1, this approach requires significantly more computing resources on the route server than where a single Loc-RIB is deployed for all clients. As the [RFC4271] BGP decision process must be applied to all Loc-RIBs deployed on the route server, both CPU and memory requirements on the host computer scale approximately according to O(P * N), where P is the total number of unique paths received by the route server and N is the number of route server clients which require a unique Loc-RIB. As this is a super-linear scaling relationship, large route servers may derive benefit from deploying per-client Loc-RIBs only where they are required.

Regardless of whether any Loc-RIB optimization technique is implemented, the route server's theoretical upper-bound network bandwidth requirements will scale according to O(P_tot * N), where P_tot is the total number of unique paths received by the route server and N is the total number of route server clients. In the case where P_avg (the arithmetic mean number of unique paths received per route server client) remains roughly constant even as the number of connected clients increases, the total number of prefixes will equal the average number of prefixes multiplied by the number of clients. Symbolically, this can be written as P_tot = P_avg * N. If we assume that in the worst case, each prefix is associated with a different set of BGP path attributes, so must be transmitted individually, the network bandwidth scaling function can be rewritten as O((P_avg * N) * N) or O(N^2). This quadratic upper bound on the network traffic requirements indicates that the route server model may not scale well for larger numbers of clients.

In practice, most prefixes will be associated with a limited number of BGP path attribute sets, allowing more efficient transmission of BGP routes from the route server than the theoretical analysis suggests. In the analysis above, P_tot will increase monotonically according to the number of clients, but will have an upper limit of the size of the full default-free routing table of the network in which the IXP is located. Observations from production route servers have shown that most route server clients generally avoid using custom routing policies and consequently the route server may not need to deploy per-client Loc-RIBs. These practical bounds reduce the theoretical worst-case scaling scenario to the point where route-server deployments are manageable on even on larger IXPs.

4.2.1. Tackling Scaling Issues

The problem of scaling route servers still presents serious practical challenges and requires careful attention. Scaling analysis indicates problems in three key areas: route processor CPU overhead associated with BGP decision process calculations, the memory requirements for handling many different BGP path entries, and the network traffic bandwidth required to distribute these BGP routes from the route server to each route server client.

4.2.1.1. View Merging and Decomposition

View merging and decomposition, outlined in [RS-ARCH], describes a method of optimising memory and CPU requirements where multiple route server clients are subject to exactly the same routing policies. In this situation, multiple Loc-RIB views can be merged into a single view.

There are several variations of this approach. If the route server operator has prior knowledge of interconnection relationships between route server clients, then the operator may configure separate Loc-RIBs only for route server clients with unique routing policies. As this approach requires prior knowledge of interconnection relationships, the route server operator must depend on each client sharing their interconnection policies, either in a internal provisioning database controlled by the operator, or else in an external data store such as an Internet Routing Registry Database.

Conversely, the route server implementation itself may implement internal view decomposition by creating virtual Loc-RIBs based on a single in-memory master Loc-RIB, with delta differences for each prefix subject to different routing policies. This allows a more fine-grained and flexible approach to the problem of Loc-RIB scaling, at the expense of requiring a more complex in-memory Loc-RIB structure.

Whatever method of view merging and decomposition is chosen on a route server, pathological edge cases can be created whereby they will scale no better than fully non-optimised per-client Loc-RIBs. However, as most route server clients connect to a route server for the purposes of reducing overhead, rather than implementing complex per-client routing policies, edge cases tend not to arise in practice.

4.2.1.2. Destination Splitting

Destination splitting, also described in [RS-ARCH], describes a method for route server clients to connect to multiple route servers and to send non-overlapping sets of prefixes to each route server. As each route server computes the best path for its own set of prefixes, the quadratic scaling requirement operates on multiple smaller sets of prefixes. This reduces the overall computational and memory requirements for managing multiple Loc-RIBs and performing the best-path calculation on each.

In practice, the route server operator would need all route server clients to send a full set of BGP routes to each route server. The route server operator could then selectively filter these prefixes for each route server by using either BGP Outbound Route Filtering [RFC5291] or else inbound prefix filters configured on client BGP sessions.

4.2.1.3. NEXT_HOP Resolution

As route servers are usually deployed at IXPs where all connected routers are on the same layer 2 broadcast domain, recursive resolution of the NEXT_HOP attribute is generally not required, and can be replaced by a simple check to ensure that the NEXT_HOP value for each received BGP route is a network address on the IXP LAN's IP address range.

4.3. Prefix Leakage Mitigation

Prefix leakage occurs when a BGP client unintentionally distributes BGP routes to one or more neighboring BGP routers. Prefix leakage of this form to a route server can cause serious connectivity problems at an IXP if each route server client is configured to accept all BGP routes from the route server. It is therefore RECOMMENDED when deploying route servers that, due to the potential for collateral damage caused by BGP route leakage, route server operators deploy prefix leakage mitigation measures in order to prevent unintentional prefix announcements or else limit the scale of any such leak. Although not foolproof, per-client inbound prefix limits can restrict the damage caused by prefix leakage in many cases. Per-client inbound prefix filtering on the route server is a more deterministic and usually more reliable means of preventing prefix leakage, but requires more administrative resources to maintain properly.

If a route server operator implements per-client inbound prefix filtering, then it is RECOMMENDED that the operator also builds in mechanisms to automatically compare the Adj-RIB-In received from each client with the inbound prefix lists configured for those clients. Naturally, it is the responsibility of the route server client to ensure that their stated prefix list is compatible with what they announce to an IXP route server. However, many network operators do not carefully manage their published routing policies and it is not uncommon to see significant variation between the two sets of prefixes. Route server operator visibility into this discrepancy can provide significant advantages to both operator and client.

4.4. Route Server Redundancy

As the purpose of an IXP route server implementation is to provide a reliable reachability brokerage service, it is RECOMMENDED that exchange operators who implement route server systems provision multiple route servers on each shared Layer-2 domain. There is no requirement to use the same BGP implementation or operating system for each route server on the IXP fabric; however, it is RECOMMENDED that where an operator provisions more than a single server on the same shared Layer-2 domain, each route server implementation be configured equivalently and in such a manner that the path reachability information from each system is identical.

4.5. AS_PATH Consistency Check

[RFC4271] requires that every BGP speaker which advertises a BGP route to another external BGP speaker prepends its own AS number as the last element of the AS_PATH sequence. Therefore the leftmost AS in an AS_PATH attribute should be equal to the autonomous system number of the BGP speaker which sent the BGP route.

As [I-D.ietf-idr-ix-bgp-route-server] suggests that route servers should not modify the AS_PATH attribute, a consistency check on the AS_PATH of an BGP route received by a route server client would normally fail. It is therefore RECOMMENDED that route server clients disable the AS_PATH consistency check towards the route server.

4.6. Export Routing Policies

Policy filtering is commonly implemented on route servers to provide prefix distribution control mechanisms for route server clients. A route server "export" policy is a policy which affects prefixes sent from the route server to a route server client. Several different strategies are commonly used for implementing route server export policies.

4.6.1. BGP Communities

Prefixes sent to the route server are tagged with specific standard [RFC1997] or extended [RFC4360] BGP community attributes, based on pre-defined values agreed between the operator and all clients. Based on these community tags, BGP routes may be propagated to all other clients, a subset of clients, or none. This mechanism allows route server clients to instruct the route server to implement per-client export routing policies.

As both standard and extended BGP community values are currently restricted to 6 octets or fewer, it is not possible for both the global and local administrator fields in the BGP community to fit a 4-octet autonomous system number. Bearing this in mind, the route server operator SHOULD take care to ensure that the predefined BGP community values mechanism used on their route server is compatible with [RFC4893] 4-octet ASNs.

4.6.2. Internet Routing Registries

Internet Routing Registry databases (IRRDBs) may be used by route server operators to construct per-client routing policies. [RFC2622] Routing Policy Specification Language (RPSL) provides an comprehensive grammar for describing interconnection relationships, and several toolsets exist which can be used to translate RPSL policy description into route server configurations.

4.6.3. Client-accessible Databases

Should the route server operator not wish to use either BGP community tags or the public IRRDBs for implementing client export policies, they may implement their own routing policy database system for managing their clients' requirements. A database of this form SHOULD allow a route server client operator to update their routing policy and provide a mechanism for allowing the client to specify whether they wish to exchange all their prefixes with any other route server client. Optionally, the implementation may allow a client to specify unique routing policies for individual prefixes over which they have routing policy control.

4.7. Layer 2 Reachability Problems

Layer 2 reachability problems on an IXP can cause serious operational problems for IXP participants which depend on route servers for interconnection. Ethernet switch forwarding bugs have occasionally been observed to cause non-transitive reachability. For example, given a route server and two IXP participants, A and B, if the two participants can reach the route server but cannot reach each other, then traffic between the participants may be dropped until such time as the layer 2 forwarding problem is resolved. This situation does not tend to occur in bilateral interconnection arrangements, as the routing control path between the two hosts is usually (but not always, due to IXP inter-switch connectivity load balancing algorithms) the same as the data path between them.

Problems of this form can be partially mitigated by using [RFC5881] bidirectional forwarding detection. However, as this is a bilateral protocol configured between routers, and as there is currently no protocol to automatically configure BFD sessions between route server clients, BFD does not currently provide an optimal means of handling the problem. Even if automatic BFD session configuration were possible, practical problems would remain. If two IXP route server clients were configured to run BFD between each other and the protocol detected a non-transitive loss of reachability between them, each of those routers would internally mark the other's prefixes as unreachable via the BGP path announced by the route server. As the route server only propagates a single best path to each client, this could cause either sub-optimal routing or complete connectivity loss if there were no alternative paths learned from other BGP sessions.

4.8. BGP NEXT_HOP Hijacking

Section 5.1.3(2) of [RFC4271] allows eBGP speakers to change the NEXT_HOP address of a received BGP route to be a different internet address on the same subnet. This is the mechanism which allows route servers to operate on a shared layer 2 IXP network. However, the mechanism can be abused by route server clients to redirect traffic for their prefixes to other IXP participant routers.

         ____
        /    \
       | AS99 |
        \____/
         /  \
        /    \ 
     __/      \__
    /   \    /   \
 ..| AS1 |..| AS2 |..
:   \___/    \___/   :
:      \      /      :
:       \    /       :
:        \__/        :
: IXP   /    \       :
:      |  RS  |      :
:       \____/       :
:                    :
 ....................