+--------+     +--------+
       | Client |     | Client |
       +--------+     +--------+
               ^       ^
               |       | ALTO protocol
              +---------+
              |  ALTO   |
              | Server  |
              +---------+
               ^   ^   ^    Potential
               |   |   |  data sources
      +--------+   |   +--------+
      |            |            |
 +---------+  +---------+  +---------+
 |   BGP   |  |   I2RS  |  |   NMS   |
 | Speaker |  |  Client |  |   OSS   |
 +---------+  +---------+  +---------+
      ^            ^            ^
      |            |            |
 Link-State      I2RS      SNMP/NETCONF,
  NLRI for       data      traffic statistics,
  IGP/BGP                  IPFIX, etc.

Figure 9: Potential data sources for ALTO

As illustrated in Figure 9, the topology data used by an ALTO server can originate from different data sources:

• The document [I-D.ietf-idr-ls-distribution] describes a mechanism by which links state and traffic engineering information can be collected from networks and shared with external components using the BGP routing protocol. This is achieved using a new BGP Network Layer Reachability Information (NLRI) encoding format. The mechanism is applicable to physical and virtual IGP links and can also include Traffic Engineering (TE) data. For instance, prefix data can be carried and originated in BGP, while TE data is originated and carried in an IGP. The mechanism described is subject to policy control. An ALTO Server can also use other mechanisms to get network data, for example, peering with multiple IGP and BGP speakers.
• The Interface to the Routing System (I2RS) is a solution for state transfer in and out of the Internet's routing system [I-D.ietf-i2rs-architecture]. An ALTO server could use an I2RS client to observe routing-related information.
• An ALTO server can also leverage a Network Management System (NMS) or an Operations Support System (OSS) as data sources. NMS or OSS solutions are used to control, operate, and manage a network, e.g., using the Simple Network Management Protocol (SNMP) or NETCONF. As explained for instance in [I-D.farrkingel-pce-abno-architecture], the NMS and OSS can be consumers of network events reported and can act on these reports as well as displaying them to users and raising alarms. The NMS and OSS can also access the Traffic Engineering Database (TED) and Label Switched Path Database (LSP-DB) to show the users the current state of the network. In addition, NMS and OSS systems may have access to IGP/BGP routing information, network inventory data (e.g., links, nodes, or link properties not visible to routing protocols, such as Shared Risk Link Groups), statistics collection system that provides traffic information, such as traffic demands or link utilizations obtained from IP Flow Information Export (IPFIX), as well as other Operations, Administration, and Maintenance (OAM) information (e.g., syslog). NMS or OSS systems also may have functions to correlate and orchestrate information originating from other data sources. For instance, it could be required to correlate IP prefixes with routers (Provider, Provider Edge, Customer Edge, etc.), IGP areas, VLAN IDs, or policies.

3.2.2. Privacy Requirements

Providing ALTO guidance results in a win-win situation both for network providers and users of the ALTO information. Applications possibly get a better performance, while the the network provider has means to optimize the traffic engineering and thus its costs.

Still, ISPs may have other important requirements when deploying ALTO. In particular, an ISP may not be willing to expose sensitive operational details of its network. The topology abstraction of ALTO enables an ISP to expose the network topology at a desired granularity only, determined by security policies.

With the ALTO Endpoint Cost Service, the ALTO client does not to have to implement any specific algorithm or mechanism in order to retrieve, maintain and process network topology information (of any kind). The complexity of the network topology (computation, maintenance and distribution) is kept in the ALTO server and ECS is delivered on demand. This allows the ALTO server to enhance and modify the way the topology information sources are used and combined. This simplifies the enforcement of privacy policies of the ISP.

The ALTO Network Map and Cost Map service expose an abstracted view on the ISP network topology. Therefore, in this case care is needed when constructing those maps, as further discussed in Section 3.2.3.

3.2.3. Partitioning and Grouping of IP Address Ranges

Host group descriptors are used in the ALTO client protocol to describe the location of a host in the network topology. These identifiers are called Partition ID (PID) and e.g. expand to a set of IP address ranges (CIDR). A PID is characterized by a string identifier. If an ALTO server offers the Map Service, corresponding identifiers have to be configured.

An automated ALTO implementation may use dynamic algorithms to aggregate network topology. However, it is often desirable to have a mechanism through which the network operator can control the level and details of network aggregation based on a set of requirements and constraints. This will typically be governed by policies that enforce a certain level of abstraction and prevent leakage of sensitive operational data.

For instance, an ALTO server may leverage BGP information that is available in a networks service provider network layer and compute the group of prefix. An example are BGP communities, which are used in MPLS/IP networks as a common mechanism to aggregate and group prefixes. A BGP community is an attribute used to tag a prefix to group prefixes based on mostly any criteria (as an example, most ISP networks originate BGP prefixes with communities identifying the Point of Presence (PoP) where the prefix has been originated). These BGP communities could be used to map IP address ranges to PIDs. By an additional policy, the ALTO server operator may decide an arbitrary cost defined between groups. Alternatively, there are algorithms that allow a dynamic computation of cost between groups. The ALTO protocol itself is independent of such algorithms and policies.

3.2.4. Rating Criteria and/or Cost Calculation

Rating criteria are used in the ALTO protocol to express topology- or connectivity-related properties, which are evaluated in order to generate the ALTO guidance. The ALTO protocol specification defines as basic set of rating criteria the "routingcost" metric, which has to be supported by all implementations. It is up to the ALTO server how that metric is calculated.

There is also an extension procedure for adding new criteria and metrics. The following list gives an overview on further rating criteria that have been proposed or which are in use by ALTO-related prototype implementations. This list is not intended as normative text; a formal definition of metrics can be found in [I-D.wu-alto-te-metrics]. Instead, the only purpose of the following list is to document the rating criteria that have been proposed so far. It can also depend on the use case of ALTO whether such rating criteria are useful, and whether the corresponding information would indeed be made available by ISPs.

Distance-related rating criteria:

• Relative topological distance: The term relative means that a larger numerical value means greater distance, but it is up to the ALTO service how to compute the values, and the ALTO client will not be informed about the nature of the information. One way of generating this kind of information may be counting AS hops, but when querying this parameter, the ALTO client must not assume that the numbers actually are AS hops. In addition to the AS path, a relative cost value could also be calculated taking into account other routing protocol parameters, such as BGP local preference or multi-exit discriminator (MED) attributes.
• Absolute topological distance, expressed in the number of traversed autonomous systems (AS).
• Absolute topological distance, expressed in the number of router hops (i.e., how much the TTL value of an IP packet will be decreased during transit).
• Absolute physical distance, based on knowledge of the approximate geolocation (e.g., continent, country) of an IP address.

Performance-related rating criteria:

• The minimum achievable throughput between the resource consumer and the candidate resource provider, which is considered useful by the application (only in ALTO queries).
• An arbitrary upper bound for the throughput from/to the candidate resource provider (only in ALTO responses). This may be, but is not necessarily the provisioned access bandwidth of the candidate resource provider.
• The maximum round-trip time (RTT) between resource consumer and the candidate resource provider, which is acceptable for the application for useful communication with the candidate resource provider (only in ALTO queries).
• An arbitrary lower bound for the RTT between resource consumer and the candidate resource provider (only in ALTO responses). This may be, for example, based on measurements of the propagation delay in a completely unloaded network.

Charging-related rating criteria:

• Traffic volume caps, in case the Internet access of the resource consumer is not charged by "flat rate". For each candidate resource provider, the ALTO service could indicate the amount of data that may be transferred from/to this resource provider until a given point in time, and how much of this amount has already been consumed. Furthermore, it would have to be indicated how excess traffic would be handled (e.g., blocked, throttled, or charged separately at an indicated price). The interaction of several applications running on a host, out of which some use this criterion while others don't, as well as the evaluation of this criterion in resource directories, which issue ALTO queries on behalf of other peers, are for further study.
• Other metrics representing an abstract cost, e.g., determined by policies that distinguish "cheap" from "expensive" IP subnet ranges, e.g., without detailing the cost function.

These rating criteria are subject to the remarks below:

The ALTO client must be aware that with high probability the actual performance values differs from whatever an ALTO server exposes. In particular, an ALTO client must not consider a throughput parameter as a permission to send data at the indicated rate without using congestion control mechanisms.

The discrepancies are due to various reasons, including, but not limited to the facts that

• the ALTO service is not an admission control system
• the ALTO service may not know the instantaneous congestion status of the network
• the ALTO service may not know all link bandwidths, i.e., where the bottleneck really is, and there may be shared bottlenecks
• the ALTO service may not have all information about the actual routing
• the ALTO service may not know whether the candidate peer itself is overloaded
• the ALTO service may not know whether the candidate peer throttles the bandwidth it devotes for the considered application
• the ALTO service may not know whether the candidate peer will throttle the data it sends to us (e.g., because of some fairness algorithm, such as tit-for-tat).

Because of these inaccuracies and the lack of complete, instantaneous state information, which are inherent to the ALTO service, the application must use other mechanisms (such as passive measurements on actual data transmissions) to assess the currently achievable throughput, and it must use appropriate congestion control mechanisms in order to avoid a congestion collapse. Nevertheless, these rating criteria may provide a useful shortcut for quickly excluding candidate resource providers from such probing, if it is known in advance that connectivity is in any case worse than what is considered the minimum useful value by the respective application.

Rating criteria that should not be defined for and used by the ALTO service include:

• Performance metrics that are closely related to the instantaneous congestion status. The definition of alternate approaches for congestion control is explicitly out of the scope of ALTO. Instead, other appropriate means, such as using TCP based transport, have to be used to avoid congestion.
• Performance metrics that raise privacy concerns. For instance, it has been questioned whether an ALTO service could publicly expose the provisioned access bandwidth, e.g. of cable / DSL customers, because this could enables identification of "premium" customers.

3.3. Known Limitations of ALTO

3.3.1. Limitations of Map-based Approaches

The specification of the Map Service in the ALTO protocol [I-D.ietf-alto-protocol] is based on the concept of network maps. The network map approach uses host group descriptors that group one or multiple subnetworks (i.e., IP prefixes) to a single aggregate. A set of IP prefixes is called partition and the associated Host Group Descriptor is called Partition ID (PID). The "costs" between the various partition IDs is stored in a second map, the cost map. Map-based approaches lower the signaling load on the server as maps have to be retrieved only if they change.

One main assumption for map-based approaches is that the information provided in these maps is static for a longer period of time. This assumption is fine as long as the network operator does not change any parameter, e.g., routing within the network and to the upstream peers, IP address assignment stays stable (and thus the mapping to the partitions). However, there are several cases where this assumption is not valid:

1. ISPs reallocate IP subnets from time to time;
2. ISPs reallocate IP subnets on short notice;
3. IP prefix blocks may be assigned to a router that serves a variety of access networks;
4. Network costs between IP prefixes may change depending on the ISP's routing and traffic engineering.

These effects can be explained as follows:

Case 1: ISPs may reallocate IPv4 subnets within their infrastructure from time to time, partly to ensure the efficient usage of IPv4 addresses (a scarce resource), and partly to enable efficient route tables within their network routers. The frequency of these "renumbering events" depend on the growth in number of subscribers and the availability of address space within the ISP. As a result, a subscriber's household device could retain an IPv4 address for as short as a few minutes, or for months at a time or even longer.

It has been suggested that ISPs providing ALTO services could sub-divide their subscribers' devices into different IPv4 subnets (or certain IPv4 address ranges) based on the purchased service tier, as well as based on the location in the network topology. The problem is that this sub-allocation of IPv4 subnets tends to decrease the efficiency of IPv4 address allocation. A growing ISP that needs to maintain high efficiency of IPv4 address utilization may be reluctant to jeopardize their future acquisition of IPv4 address space.

However, this is not an issue for map-based approaches if changes are applied in the order of days.

Case 2: ISPs can use techniques that allow the reallocation of IP prefixes on very short notice, i.e., within minutes. An IP prefix that has no IP address assignment to a host anymore can be reallocated to areas where there is currently a high demand for IP addresses.

Case 3: In residential access networks (e.g., DSL, cable), IP prefixes are assigned to broadband gateways, which are the first IP-hop in the access-network between the Customer Premises Equipment (CPE) and the Internet. The access-network between CPE and broadband gateway (called aggregation network) can have varying characteristics (and thus associated costs), but still using the same IP prefix. For instance one IP addresses IP11 out of a IP prefix IP1 can be assigned to a VDSL (e.g., 2 MBit/s uplink) access line while the subsequent IP address IP12 is assigned to a slow ADSL line (e.g., 128 kbit/s uplink). These IP addresses are assigned on a first come first served basis, i.e., a single IP address out of the same IP prefix can change its associated costs quite fast. This may not be an issue with respect to the used upstream provider (thus the cross ISP traffic) but depending on the capacity of the aggregation-network this may raise to an issue.

Case 4: The routing and traffic engineering inside an ISP network, as well as the peering with other autonomous systems, can change dynamically and affect the information exposed by an ALTO server. As a result, cost map and possibly also network maps can change.

3.3.2. Limitiations of Non-Map-based Approaches

The specification of the ALTO protocol [I-D.ietf-alto-protocol] also includes the Endpoint Cost Service (ECS) mechanism. ALTO clients can ask guidance for specific IP addresses to the ALTO server, thereby avoiding the need of processing maps. This can mitigate some of the problems mentioned in the previous section.

However, asking for IP addresses, asking with long lists of IP addresses, and asking quite frequently may overload the ALTO server. The server has to rank each received IP address, which causes load at the server. This may be amplified by the fact that not only a single ALTO client is asking for guidance, but a larger number of them. The results of the ECS are also more difficult to cache than ALTO maps. Therefore, the ALTO client may have to await the server response before starting a communication, which results in an additional delay.

Caching of IP addresses at the ALTO client or the usage of the H12 approach [I-D.kiesel-alto-h12] in conjunction with caching may lower the query load on the ALTO server.

When ALTO server receives an ECS request, it may not have the most appropriate topology information in order to accurately determine the ranking. [I-D.ietf-alto-protocol] generally assumes that a server can always offer some guidance. In such a case the ALTO server could adopt one of the following strategies:

• Reply with available information (best effort).
• Query another ALTO server presumed to have better topology information and return that response (cascaded servers).
• Redirect the request to another ALTO server presumed to have better topology information (redirection).

The protocol mechanisms and decision processes that would be used to determine if redirection is necessary and which mode to use is out of the scope of this document, since protocol extensions could be required.

3.4. Monitoring ALTO

3.4.1. Impact and Observation on Network Operation

ALTO presents a new opportunity for managing network traffic by providing additional information to clients. In particular, the deployment of an ALTO Server may shift network traffic patterns, and the potential impact to network operation can be large. An ISP providing ALTO may want to assess the benefits of ALTO as part of the management and operations (cf. [I-D.ietf-alto-protocol]). For instance, the ISP might be interested in understanding whether the provided ALTO maps are effective, and in order to decide whether an adjustment of the ALTO configuration would be useful. Such insight can be obtained from a monitoring infrastructure. An NSP offering ALTO could consider the impact on (or integration with) traffic engineering and the deployment of a monitoring service to observe the effects of ALTO operations. The measurement of impacts can be challenging because ALTO-enabled applications may not provide related information back to the ALTO Service Provider.

To construct an effective monitoring infrastructure, the ALTO Service Provider should decide how to monitor the performance of ALTO and identify and deploy data sources to collect data to compute the performance metrics. In certain trusted deployment environments, it may be possible to collect information directly from ALTO clients. It may also be possible to vary or selectively disable ALTO guidance for a portion of ALTO clients either by time, geographical region, or some other criteria to compare the network traffic characteristics with and without ALTO. Monitoring an ALTO service could also be realized by third parties. In this case, insight into ALTO data may require a trust relationship between the monitoring system operator and the network service provider offering an ALTO service.

The required monitoring depends on the network infrastructure and the use of ALTO, and an exhaustive description is outside the scope of this document.

3.4.2. Measurement of the Impact

ALTO realizes an interface between the network and applications. This implies that an effective monitoring infrastructure may have to deal with both network and application performance metrics. This document does not comprehensively list all performance metrics that could be relevant, nor does it formally specify metrics.

The impact of ALTO can be classified regarding a number of different criteria:

• Total amount and distribution of traffic: ALTO enables ISPs to influence and localize traffic of applications that use the ALTO service. An ISP may therefore be interested in analyzing the impact on the traffic, i.e., whether network traffic patterns are shifted. For instance, if ALTO shall be used to reduce the inter-domain P2P traffic, it makes sense to evaluate the total amount of inter-domain traffic of an ISP. Then, one possibility is to study how the introduction of ALTO reduces the total inter-domain traffic (inbound and/our outbound). If the ISPs intention is to localize the traffic inside his network, the network-internal traffic distribution will be of interest. Effectiveness of localization can be quantified in different ways, e.g., by the load on core routers and backbone links, or by considering more advanced effects, such as the average number of hops that traffic traverses inside a domain.
• Application performance: The objective of ALTO is improve application performance. ALTO can be used by very different types applications, with different communication characteristics and requirements. For instance, if ALTO guidance achieves traffic localization, one would expect that applications achieve a higher throughput and/or smaller delays to retrieve data. If application-specific performance characteristics (e.g., video or audio quality) can be monitored, such metrics related to user experience could also help to analyze the benefit of an ALTO deployment. If available, selected statistics from the TCP/IP stack in hosts could be leveraged, too.

Of potential interest can also be the share of applications or customers that actually use an offered ALTO service, i.e., the adoption of the service.

Monitoring statistics can be aggregated, averaged, and normalized in different ways. This document does not mandate specific ways how to calculate metrics.

3.4.3. System and Service Performance

A number of interesting parameters can be measured at the ALTO server. [I-D.ietf-alto-protocol] suggests certain ALTO-specific metrics to be monitored:

• Requests and responses for each service listed in a Information Directory (total counts and size in bytes).
• CPU and memory utilization
• ALTO map updates
• Number of PIDs
• ALTO map sizes (in-memory size, encoded size, number of entries)

This data characterizes the workload, the system performance as well as the map data. Obviously, such data will depend on the implementation and the actual deployment of the ALTO service. Logging is also recommended in [I-D.ietf-alto-protocol].

3.4.4. Monitoring Infrastructures

Understanding the impact of ALTO may require interaction between different systems, operating at different layers. Some information discussed in the preceding sections is only visible to an ISP, while application-level performance can hardly be measured inside the network. It is possible that not all information of potential interest can directly be measured, either because no corresponding monitoring infrastructure or measurement method exists, or because it is not easily accessible.

One way to quantify the benefit of deploying ALTO is to measure before and after enabling the ALTO service. In addition to passive monitoring, some data could also be obtained by active measurements, but due to the resulting overhead, the latter should be used with care. Yet, in all monitoring activities an ALTO service provider has to take into account that ALTO clients are not bound to ALTO server guidance as ALTO is only one source of information, and any measurement result may thus be biased.

Potential sources for monitoring the use of ALTO include:

• Network Operations, Administration, and Maintenance (OAM) systems: Many ISPs deploy OAM systems to monitor the network traffic, which may have insight into traffic volumes, network topology, and bandwidth information inside the management area. Data can be obtained by SNMP, NETCONF, IP Flow Information Export (IPFIX), syslog, etc.
• Applications/clients: Relevant data could be obtained by instrumentation of applications.
• ALTO server: If available, log files or other statistics data could be analyzed.
• Other application entities: In several use cases, there are other application entities that could provide data as well. For instance, there may be centralized log servers that collect data.

In many ALTO use cases some data sources are located within an ISP network while some other data is gathered at application level. Correlation of data could require a collaboration agreement between the ISP and an application owner, including agreements of data interchange formats, methods of delivery, etc. In practice, such a collaboration may not be possible in all use cases of ALTO, because the monitoring data can be sensitive, and because the interacting entities may have different priorities. Details of how to build an over-arching monitoring system for evaluating the benefits of ALTO are outside the scope of this memo.

3.5. Map Examples for Different Types of ISPs

3.5.1. Small ISP with Single Internet Uplink

The ALTO protocol does not mandate how to determine costs between endpoints and/or determine map data. In complex usage scenarios this can be a non-trivial problem. In order to show the basic principle, this and the following section explain for different deployment scenarios how ALTO maps could be structured.

For a small ISP, the inter-domain traffic optimizing problem is how to decrease the traffic exchanged with other ISPs, because of high settlement costs. By using the ALTO service to optimize traffic, a small ISP can define two "optimization areas": one is its own network; the other one consists of all other network destinations. The cost map can be defined as follows: the cost of link between clients of inner ISP's networks is lower than between clients of outer ISP's networks and clients of inner ISP's network. As a result, a host with ALTO client inside the network of this ISP will prefer retrieving data from hosts connected to the same ISP.

An example is given in Figure 10. It is assumed that ISP A is a small ISP only having one access network. As operator of the ALTO service, ISP A can define its network to be one optimization area, named as PID1, and define other networks to be the other optimization area, named as PID2. C1 is denoted as the cost inside the network of ISP A. C2 is denoted as the cost from PID2 to PID1, and C3 from PID1 to PID2. For the sake of simplifity, in the following C2=C3 is assumed. In order to keep traffic local inside ISP A, it makes sense to define: C1<C2

           -----------
       ////           \\\\
     //                   \\
   //                       \\                  /-----------\
  | +---------+               |             ////             \\\\
  | | ALTO    |  ISP A        |    C2      |    Other Networks   |
 |  | Service |  PID 1         <-----------     PID 2
  | +---------+  C1           |----------->|                     |
  |                           |  C3 (=C2)   \\\\             ////
   \\                       //                  \-----------/
     \\                   //
       \\\\           ////
           -----------

   HTTP/1.1 200 OK
   ...
   Content-Type: application/alto-networkmap+json

   {
    ...
     "network-map" : {
       "PID1" : {
         "ipv4" : [
           "192.0.2.0/24",
           "198.51.100.0/25"
         ]
       },
       "PID2" : {
         "ipv4" : [
           "0.0.0.0/0"
         ],
         "ipv6" : [
           "::/0"
         ]
       }
     }
   }

   HTTP/1.1 200 OK
   ...
   Content-Type: application/alto-costmap+json

   {
       ...
       "cost-type" : {"cost-mode"  : "numerical",
                      "cost-metric": "routingcost"
       }
     },
     "cost-map" : {
       "PID1": { "PID1": C1,  "PID2": C2 },
       "PID2": { "PID1": C2,  "PID2": 0 },
     }
   }

 +------------------------------------+         +----------------+
 | ISP A   +---------------+          |         |                |
 |         |    Backbone   |          |   C7    |      ISP B     |
 |     +---+    Network    +----+     |<--------+      PID 4     |
 |     |   +-------+-------+    |     |         |                |
 |     |           |            |     |         |                |
 |     |           |            |     |         +----------------+
 | +---+--+     +--+---+     +--+---+ |
 | |AN A  |  C4 |AN B  |  C5 |AN C  | |
 | |PID 1 +<--->|PID 2 |<--->+PID 3 | |
 | |C1    |     |C2    |     |C3    | |         +----------------+
 | +---+--+     +------+     +--+---+ |         |                |
 |     ^                        ^     |   C8    | Other Networks |
 |     |                        |     |<--------+ PID 5          |
 |     +------------------------+     |         |                |
 |                  C6                |         |                |
 +------------------------------------+         +----------------+

	

 +-----------------------------------------------------------------+
 |                                                                 |
 |  ISP A                 +-------------+                          |
 |               +--------+   ALTO      +---------+                |
 |               |        |   Service   |         |                |
 |               |        +------+------+         |                |
 |               |               |                |                |
 |               |               |                |                |
 |               |               |                |                |
 |       +-------+-------+       | C6    +--------+------+         |
 |       |     AN A      |<--------------|      AN B     |         |
 |       |     PID 2     |   C7  |       |      PID 3    |         |
 |       |     C2        |-------------->|      C3       |         |
 |       +---------------+       |       +---------------+         |
 |             ^    |            |              |     ^            |
 |             |    |            |              |     |            |
 |             |    |C4          |              |     |            |
 |          C5 |    |            |              |     |            |
 |             |    |   +--------+---------+    |     |            |
 |             |    +-->|  Mobile Network  |<---+     |            |
 |             |        |  PID 1           |          |            |
 |             +------- |  C1              |----------+            |
 |                      +------------------+                       |
 +-----------------------------------------------------------------+

	

                              ,-------.
       ,---.               ,-'         `-.   +-----------+
    ,-'     `-.           /     ISP 1     \  |   Peer 1  |*****
   /           \         / +-------------+ \ |           |    *
  /    ISP X    \   +=====>| ALTO Server |  )+-----------+    *
 /               \  =    \ +-------------+ / +-----------+    *
; +-----------+   : =     \               /  |   Peer 2  |    *
| |  Tracker  |<====+      `-.         ,-'   |           |*****
| |ALTO Client|<====+         `-------'      +-----------+   **
| +-----------+   | =         ,-------.                      **
:        *        ; =      ,-'         `-.   +-----------+   **
 \       *       /  =     /     ISP 2     \  |   Peer 3  |   **
  \      *      /   =    / +-------------+ \ |           |*****
   \     *     /    +=====>| ALTO Server |  )+-----------+  ***
    `-.  *  ,-'          \ +-------------+ / +-----------+  ***
       `-*-'              \               /  |   Peer 4  |*****
         *                 `-.         ,-'   |           | ****
         *                    `-------'      +-----------+ ****
         *                                                 ****
         *                                                 ****
         ***********************************************<******
    Legend:
    === ALTO client protocol
    *** Application protocol

                         ,-------.         +-----------+
       ,---.          ,-'         `-.  +==>|   Peer 1  |*****
    ,-'     `-.      /     ISP 1     \ =   |ALTO Client|    *
   /           \    / +-------------+<=+   +-----------+    *
  /    ISP X    \   | + ALTO Server |<=+   +-----------+    *
 /               \  \ +-------------+ /=   |   Peer 2  |    *
;   +---------+   :  \               / +==>|ALTO Client|*****
|   | Global  |   |   `-.         ,-'      +-----------+   **
|   | Tracker |   |      `-------'                         **
|   +---------+   |      ,-------.         +-----------+   **
:        *        ;   ,-'         `-.  +==>|   Peer 3  |   **
 \       *       /   /     ISP 2     \ =   |ALTO Client|*****
  \      *      /   / +-------------+<=+   +-----------+  ***
   \     *     /    | | ALTO Server |<=+   +-----------+  ***
    `-.  *  ,-'     \ +-------------+ /=   |   Peer 4  |*****
       `-*-'         \               / +==>|ALTO Client| ****
         *            `-.         ,-'      +-----------+ ****
         *               `-------'                       ****
         *                                               ****
         ***********************************************<****
    Legend:
    === ALTO client protocol
    *** Application protocol

                          ,-------.         +-----------+
        ,---.          ,-'  ISP 1  `-.  ***>|   Peer 1  |
     ,-'     `-.      /+-------------+\ *   |           |
    /           \    / +   Tracker   |<**   +-----------+
   /    ISP X    \   | +-----===-----+<**   +-----------+
  /               \  \ +-----===-----+ /*   |   Peer 2  |
 ;   +---------+   :  \+ ALTO Server |/ ***>|           |
 |   | Global  |   |   +-------------+      +-----------+
 |   | Tracker |   |      `-------'
 |   +---------+   |                        +-----------+
 :          ^      ;      ,-------.         |   Peer 3  |
  \         *     /    ,-'  ISP 2  `-.  ***>|           |
   \        *    /    /+-------------+\ *   +-----------+
    \       *   /    / +   Tracker   |<**   +-----------+
     `-.    *,-'     | +-----===-----+ |    |   Peer 4  |<*
        `---*        \ +-----===-----+ /    |           | *
            *         \+ ALTO Server |/     +-----------+ *
            *          +-------------+                    *
            *             `-------'                       *
            ***********************************************
     Legend:
     === ALTO client protocol
     *** Application protocol

Peer w. ALTO cli.            Tracker               ALTO Server 
--------+--------       --------+--------       --------+--------
        | F1 Tracker query      |                       |    
        |======================>|                       |
        | F2 Tracker reply      |                       |    
        |<======================|                       |    
        | F3 ALTO client protocol query                 |
        |---------------------------------------------->|    
        | F4 ALTO client protocol reply                 |
        |<----------------------------------------------|    
        |                       |                       |    

====  Application protocol (i.e., tracker-based P2P app protocol)
----  ALTO client protocol

      Peer               Tracker w. RDAC           ALTO Server 
--------+--------       --------+--------       --------+--------
        | F1 Tracker query      |                       |    
        |======================>|                       |
        |                       | F2 ALTO cli. p. query |
        |                       |---------------------->|    
        |                       | F3 ALTO cli. p. reply |
        |                       |<----------------------|
        | F4 Tracker reply      |                       |    
        |<======================|                       |    
        |                       |                       |    

====  Application protocol (i.e., tracker-based P2P app protocol)
----  ALTO client protocol

User Agent                  Request Router                 Surrogate
     |                             |                           |
     |     F1 Initial Request      |                           |
     +---------------------------->|                           |
     |                             +--+                        |
     |                             |  | F2 Surrogate Selection |
     |                             |<-+       (using ALTO)     |
     |   F3 Redirection Response   |                           |
     |<----------------------------+                           |
     |                             |                           |
     |     F4 Content Request      |                           |
     +-------------------------------------------------------->|
     |                             |                           |
     |                             |          F5 Content       |
     |<--------------------------------------------------------+
     |                             |                           |

                    +---------------+
                    |  Customer's   |
                    |   management  |
                    |  application  |.
                    | (ALTO client) |  .
                    +---------------+    .  VPN provisioning
                            ^              . (out-of-scope)
                            | ALTO           .
                            V                  .
                 +---------------------+       +----------------+
                 |     ALTO server     |       | VPN portal/OSS |
                 |   provided by NSP   |       | (out-of-scope) |
                 +---------------------+       +----------------+
                            ^ VPN network
                            * and cost maps
                            *
                  /---------*---------\ Network service provider
                  |         *         |
     +-------+   _______________________   +-------+
     | App a | ()_____. .________. .____() | App d |
     +-------+    |   | |        | |  |    +-------+
                  \---| |--------| |--/           
                      | |        | |
                      |^|        |^| Customer VPN
                       V          V
                   +-------+  +-------+
                   | App b |  | App c |
                   +-------+  +-------+
        

   +--------------+                +------+
   | ISP 1 network+----------------+Peer 1|
   +-----+--------+                +------+
   |
+--------+------------------------------------------------------+
|        |                                      ISP 2 network   |
|  +---------+                                                  |
|  |L1 Cache |                                                  |
|  +-----+---+                                                  |
|        +--------------------+----------------------+          |
|        |                    |                      |          |
| +------+------+      +------+-------+       +------+-------+  |
| | AN1         |      | AN2          |       | AN3          |  |
| | +---------+ |      | +----------+ |       |              |  |
| | |L2 Cache | |      | |L2 Cache  | |       |              |  |
| | +---------+ |      | +----------+ |       |              |  |
| +------+------+      +------+-------+       +------+-------+  |
|        |                                           |          |
|        +--------------------+                      |          |
|        |                    |                      |          |
| +------+------+      +------+-------+       +------+-------+  |
| | SUB-AN11    |      | SUB-AN12     |       | SUB-AN31     |  |
| | +---------+ |      |              |       |              |  |
| | |L3 Cache | |      |              |       |              |  |
| | +---------+ |      |              |       |              |  |
| +------+------+      +------+-------+       +------+-------+  |
|        |                    |                      |          |
+--------+--------------------+----------------------+----------+
         |                    |                      |
     +---+---+            +---+---+                  |
     |       |            |       |                  |
  +--+--+ +--+--+      +--+--+ +--+--+            +--+--+
  |Peer2| |Peer3|      |Peer4| |Peer5|            |Peer6|
  +-----+ +-----+      +-----+ +-----+            +-----+