| Network Working Group | E. McMurry |
| Internet-Draft | B. Campbell |
| Intended status: Standards Track | Tekelec |
| Expires: March 03, 2013 | September 2012 |
Diameter Overload Control Requirements
When a Diameter server or agent becomes overloaded, it needs to be able to gracefully reduce its load, typically by informing clients to reduce sending traffic for some period of time. Otherwise, it must continue to expend resources parsing and responding to Diameter messages, possibly resulting in congestion collapse. The existing mechanisms provided by Diameter are not sufficient for this purpose. This document describes the limitations of the existing mechanisms, and provides requirements for new overload management mechanisms.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http:/⁠/⁠datatracker.ietf.org/⁠drafts/⁠current/⁠.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 03, 2013.
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http:/⁠/⁠trustee.ietf.org/⁠license-⁠info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
When a Diameter [I-D.ietf-dime-rfc3588bis] server or agent becomes overloaded, it needs to be able to gracefully reduce its load, typically by informing clients to reduce sending traffic for some period of time. Otherwise, it must continue to expend resources parsing and responding to Diameter messages, possibly resulting in congestion collapse. The existing mechanisms provided by Diameter are not sufficient for this purpose. This document describes the limitations of the existing mechanisms, and provides requirements for new overload management mechanisms.
This document draws on [RFC5390] and the work done on SIP overload control as well as on overload practices in SS7 networks and studies done by 3GPP.
Diameter is not typically an end-user protocol; rather it is generally used as one component in support of some end-user activity. For example, a WiFi access point might use Diameter to authenticate and authorize user access via 802.11. Overload in a network that uses Diameter applications will likely spill over into the end-user application network. The impact of Diameter overload on the client application (a client application may use the Diameter protocol and other protocols to do its job) is beyond the scope of this document.
This document presents non-normative descriptions of causes of overload along with related scenarios and studies. Finally, it offers a set of normative requirements for an improved overload indication mechanism.
Overload occurs when an element, such as a Diameter server or agent, has insufficient resources to successfully process all of the traffic it is receiving. Resources include all of the capabilities of the element used to process a request, including CPU processing, memory, I/O, and disk resources. It can also include external resources such as a database or DNS server, in which case the CPU, processing, memory, I/O, and disk resources of those elements are effectively part of the logical element processing the request.
Overload can occur for many reasons, including:
Modern Diameter networks, comprised of application layer multi-node deployments of Diameter elements, may operate at very large transaction volumes. If a Diameter node becomes overloaded, or even worse, fails completely, a large number of messages may be lost very quickly. Even with redundant servers, many messages can be lost in the time it takes for failover to complete. While a Diameter client or agent should be able to retry such requests, an overloaded peer may cause a sudden large increase in the number of transaction transactions needing to be retried, rapidly filling local queues or otherwise contributing to local overload. Therefore Diameter devices need to be able to shed load before critical failures can occur.
This document uses the term "overload" to refer to application-layer overload at Diameter nodes. This is distinct from "network congestion", that is, congestion that occurs at the lower networking layers that may impact the delivery of Diameter messages between nodes. The authors recognize that element overload and network congestion are interrelated, and that overload can contribute to network congestion and vice versa.
Network congestion issues are better handled by the transport protocols. Diameter uses TCP and SCTP, both of which include congestion management features. Analysis of whether those features are sufficient for transport level congestion between Diameter nodes, and any work to further mitigate network congestion is out of scope both for this document, and for the work proposed by this document.
Most elements using Diameter applications do not use Diameter exclusively. It is important to realize that overload of an element can be caused by a number of factors that may be unrelated to the processing of Diameter or Diameter applications.
A element communicating via protocols other than Diameter that is also using a Diameter application needs to be able to signal to Diameter peers that it is experiencing overload regardless of the cause of the overload, since the overload will affect that element's ability to process Diameter transactions. The element may also need to signal this on other protocols depending on its function and the architecture of the network and application it is providing services for. Whether that is necessary can only be decided within the context of that architecture and application. A mechanism for signaling overload with Diameter, which this specification details the requirements for, provides applications the ability to signal their Diameter peers of overload, mitigating that part of the issue. Applications may need to use this, as well as other mechanisms, to solve their broader overload issues. Indicating overload on protocols other than Diameter is out of scope for this document, and for the work proposed by this document.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
The terms "client", "server", "agent", "node", "peer", "upstream", and "downstream" are used as defined in [I-D.ietf-dime-rfc3588bis].
Several Diameter deployment scenarios exist that may impact overload management. The following scenarios help motivate the requirements for an overload management mechanism.
These scenarios are by no means exhaustive, and are in general simplified for the sake of clarity. In particular, the authors assume for the sake of clarity that the client sends Diameter requests to the server, and the server sends responses to client, even though Diameter supports bidirectional applications. Each direction in such an application can be modeled separately.
In a large scale deployment, many of the nodes represented in these scenarios would be deployed as clusters of servers. The authors assume that such a cluster is responsible for managing its own internal load balancing and overload management so that it appears as a single Diameter node. That is, other Diameter nodes can treat it as single, monolithic node for the purposes of overload management.
These scenarios do not illustrate the client application. As mentioned in Section 1, Diameter is not typically an end-user protocol; rather it is generally used in support of some other client application. These scenarios do not consider the impact of Diameter overload on the client application.
This section describes Diameter peer-to-peer scenarios. That is, scenarios where a Diameter client talks directly with a Diameter server, without the use of a Diameter agent.
Figure 1 illustrates the simplest possible Diameter relationship. The client and server share a one-to-one peer-to-peer relationship. If the server becomes overloaded, either because the client exceeds the server's capacity, or because the server's capacity is reduced due to some resource dependency, the client needs to reduce the amount of Diameter traffic it sends to the server. Since the client cannot forward requests to another server, it must either queue requests until the server recovers, or itself become overloaded in the context of the client application and other protocols it may also use.
+------------------+
| |
| |
| Server |
| |
+--------+---------+
|
|
+--------+---------+
| |
| |
| Client |
| |
+------------------+
Figure 1: Basic Peer to Peer Scenario
Figure 2 shows a similar scenario, except in this case the client has multiple servers that can handle work for a specific realm and application. If server 1 becomes overloaded, the client can forward traffic to server 2. Assuming server 2 has sufficient reserve capacity to handle the forwarded traffic, the client should be able to continue serving client application protocol users. If server 1 is approaching overload, but can still handle some number of new request, it needs to be able to instruct the client to forward a subset of its traffic to server 2.
+------------------+ +------------------+
| | | |
| | | |
| Server 1 | | Server 2 |
| | | |
+--------+-`.------+ +------.'+---------+
`. .'
`. .'
`. .'
`. .'
+-------`.'--------+
| |
| |
| Client |
| |
+------------------+
Figure 2: Multiple Server Peer to Peer Scenario
Figure 3 illustrates a peer-to-peer scenario with multiple Diameter realm and application combinations. In this example, server 2 can handle work for both applications. Each application might have different resource dependencies. For example, a server might need to access one database for application A, and another for application B. This creates a possibility that Server 2 could become overloaded for application A but not for application B, in which case the client would need to divert some part of its application A requests to server 1, but should not divert any application B requests. This requires server 2 to be able to distinguish between applications when it indicates an overload condition to the client.
On the other hand, it's possible that the servers host many applications. If server 2 becomes overloaded for all applications, it would be undesirable for it to have to notify the client separately for each application. Therefore it also needs a way to indicate that it is overloaded for all possible applications.
+----------------------------------------------+
| Application A +------------------------+----------------------+
|+------------------+ | +------------------+ | +------------------+|
|| | | | | | | ||
|| | | | | | | ||
|| Server 1 | | | Server 2 | | | Server 3 ||
|| | | | | | | ||
|+--------+---------+ | +--------+---------+ | +-+----------------+|
| | | | | | |
+---------+-----------+-----------+------------+ | |
| | | | |
| | | | Application B |
| +-----------+-----------------+-----------------+
``-.._ | |
`-..__ | _.-''
`--._ | _.-''
``-.__ | _.-''
+------`-.-''------+
| |
| |
| Client |
| |
+------------------+
Figure 3: Multiple Application Peer to Peer Scenario
This section describes scenarios that include a Diameter agent, either in the form of a Diameter relay or Diameter proxy. These scenarios do not consider Diameter redirect agents, since they are more readily modeled as end-servers.
Figure 4 illustrates a simple Diameter agent scenario with a single client, agent, and server. In this case, overload can occur at the server, at the agent, or both. But in most cases, client behavior is the same whether overload occurs at the server or at the agent. From the client's perspective, server overload and agent overload is the same thing.
+------------------+
| |
| |
| Server |
| |
+--------+---------+
|
|
+--------+---------+
| |
| |
| Agent |
| |
+--------+---------+
|
|
+--------+---------+
| |
| |
| Client |
| |
+------------------+
Figure 4: Basic Agent Scenario
Figure 5 shows an agent scenario with multiple servers. If server 1 becomes overloaded, but server 2 has sufficient reserve capacity, the agent may be able to transparently divert some or all Diameter requests originally bound for server 1 to server 2.
In most cases, the client does not have detailed knowledge of the Diameter topology upstream of the agent. If the agent uses dynamic discovery to find eligible servers, the set of eligible servers may not be enumerable from the perspective of the client. Therefore, in most cases the agent needs to deal with any upstream overload issues in a way that is transparent to the client. If one server notifies the agent that it has become overloaded, the notification should not be passed back to the client in a way where the client could mistakenly perceive the agent itself as being overloaded. If the set of all possible destinations upstream of the agent no longer has sufficient capacity for incoming load, the agent itself becomes effectively overloaded.
On the other hand, there are cases where the client needs to be able to select a particular server from behind an agent. For example, if a Diameter request is part of a multiple-round-trip authentication, or is otherwise part of a Diameter "session", it may have a DestinationHost AVP that requires the request to be served by server 1. Therefore the agent may need to inform a client that a particular upstream server is overloaded or otherwise unavailable. Note that there can be many ways a server can be specified, which may have different implications (e.g. by IP address, by host name, etc).
+------------------+ +------------------+
| | | |
| | | |
| Server 1 | | Server 2 |
| | | |
+--------+-`.------+ +------.'+---------+
`. .'
`. .'
`. .'
`. .'
+-------`.'--------+
| |
| |
| Agent |
| |
+--------+---------+
|
|
|
+--------+---------+
| |
| |
| Client |
| |
+------------------+
Figure 5: Multiple Server Agent Scenario
Figure 6 shows a scenario where an agent routes requests to a set of servers for more than one Diameter realm and application. In this scenario, if server 1 becomes overloaded or unavailable, the agent may effectively operate at reduced capacity for application A, but at full capacity for application B. Therefore, the agent needs to be able to report that it is overloaded for one application, but not for another.
+----------------------------------------------+
| Application A +------------------------+----------------------+
|+------------------+ | +------------------+ | +------------------+|
|| | | | | | | ||
|| | | | | | | ||
|| Server 1 | | | Server 2 | | | Server 3 ||
|| | | | | | | ||
|+---------+--------+ | +--------+---------+ | +--+---------------+|
| | | | | | |
+----------+----------+-----------+------------+ | |
| | | | |
| | | | Application B |
| +-----------+------------------+----------------+
| | |
``--.__ | _.
``-.__ | __.--''
`--.._ | _..--'
+-----``-+.-''-----+
| |
| |
| Agent |
| |
+--------+---------+
|
|
+--------+---------+
| |
| |
| Client |
| |
+------------------+
Figure 6: Multiple Application Agent Scenario
Another scenario to consider when looking at Diameter overload is that of multiple network operators using Diameter components connected through an interconnect service, e.g. using IPX. Figure 7 shows two network operators with an interconnect network in-between. There could be any number of these networks between any two network operator's networks.
+-------------------------------------------+
| Interconnect |
| |
| +--------------+ +--------------+ |
| | Edge Agent 1 |------| Edge Agent 2 | |
| +--------------+ +--------------+ |
| .' `. |
+------.-'--------------------------`.------+
.' `.
.-' `.
------------.'-----+ +----`.---------------
+----------+ | | +----------+
| Server 1 | | | | Server 2 |
+----------+ | | +----------+
| |
Network Operator 1 | | Network Operator 2
-------------------+ +---------------------
Figure 7: Two Network Interconnect Scenario
The characteristics of the information that an operator would want to share over such a connection are different than the information shared between components within a network operator's network. Network operators may not want to convey topology or operational information, which limits how overload and loading information can be sent. For the interconnect scenario shown, Server 2 may want to signal overload to Server 1, to affect traffic coming from Network Operator 1.
This is different than internal to an network operator's network, where there may be many more elements in a more complicated topology. Also, the elements in the interconnect network may not support diameter overload control, and the network operators may not want the interconnect to use overload or loading information intended to pass through the interconnect even if the elements in the interconnect network do support diameter overload control.
Diameter offers both implicit and explicit mechanisms for a Diameter node to learn that a peer is overloaded or unreachable. The implicit mechanism is simply the lack of responses to requests. If a client fails to receive a response in a certain time period, it assumes the upstream peer is unavailable, or overloaded to the point of effective unavailability. The watchdog mechanism [RFC3539] ensures that a certain rate of transaction responses occur even when there is otherwise little or no other Diameter traffic.
The explicit mechanism involves specific protocol error responses, where an agent or server can tell a downstream peer that it is either too busy to handle a request (DIAMETER_TOO_BUSY) or unable to route a request to an upstream destination (DIAMETER_UNABLE_TO_DELIVER), perhaps because that destination itself is overloaded to the point of unavailability.
Once a Diameter node learns that an upstream peer has become overloaded via one of these mechanisms, it can then attempt to take action to reduce the load. This usually means forwarding traffic to an alternate destination, if available. If no alternate destination is available, the node must either reduce the number of messages it originates (in the case of a client) or inform the client to reduce traffic (in the case of an agent.)
Diameter requires the use of a congestion-managed transport layer, currently TCP or SCTP, to mitigate network congestion. It is expected that these transports manage network congestion and that issues with transport (e.g. congestion propagation and window management) are managed at that level. But even with a congestion-managed transport, a Diameter node can become overloaded at the Diameter protocol or application layers due to the causes described in Section 1.1 and congestion managed transports do not provide facilities (and are at the wrong level) to handle server overload. Transport level congestion management is also not sufficient to address overload in cases of multi-hop and multi-destination signaling.
The currently available Diameter mechanisms for indicating an overload condition are not adequate to avoid service outages due to overload. This may, in turn, contribute to broader congestion collapse due to unresponsive Diameter nodes causing application or transport layer retransmissions. In particular, they do not allow a Diameter agent or server to shed load as it approaches overload. At best, a node can only indicate that it needs to entirely stop receiving requests, i.e. that it has effectively failed. Even that is problematic due to the inability to indicate durational validity on the transient errors available in the base Diameter protocol. Diameter offers no mechanism to allow a node to indicate different overload states for different categories of messages, for example, if it is overloaded for one Diameter application but not another.
The implicit mechanism doesn't allow an agent or server to inform the client of a problem until it is effectively too late to do anything about it. The client does not know to take action until the upstream node has effectively failed. A Diameter node has no opportunity to shed load early to avoid collapse in the first place.
Additionally, the implicit mechanism cannot distinguish between overload of a Diameter node and network congestion. Diameter treats the failure to receive an answer as a transport failure.
The Diameter specification is ambiguous on how a client should handle receipt of a DIAMETER_TOO_BUSY response. The base specification [I-D.ietf-dime-rfc3588bis] indicates that the sending client should attempt to send the request to a different peer. It makes no suggestion that a the receipt of a DIAMETER_TOO_BUSY response should affect future Diameter messages in any way.
The Authentication, Authorization, and Accounting (AAA) Transport Profile [RFC3539] recommends that a AAA node that receives a "Busy" response failover all remaining requests to a different agent or server. But while the Diameter base specification explicitly depends on RFC3539 to define transport behavior, it does not refer to RFC3539 in the description of behavior on receipt of DIAMETER_TOO_BUSY. There's a strong likelihood that at least some implementations will continue to send Diameter requests to an upstream peer even after receiving a DIAMETER_TOO_BUSY error.
BCP 41 [RFC2914] describes, among other things, how end-to-end application behavior can help avoid congestion collapse. In particular, an application should avoid sending messages that will never be delivered or processed. The DIAMETER_TOO_BUSY behavior as described in the Diameter base specification fails at this, since if an upstream node becomes overloaded, a client attempts each request, and does not discover the need to failover the request until the initial attempt fails.
The situation is improved if implementations follow the [RFC3539] recommendation and keep state about upstream peer overload. But even then, the Diameter specification offers no guidance on how long a client should wait before retrying the overloaded destination. If an agent or server supports multiple realms and/or applications, DIAMETER_TOO_BUSY offers no way to indicate that it is overloaded for one application but not another. A DIAMETER_TOO_BUSY error can only indicate overload at a "whole server" scope.
Agent processing of a DIAMETER_TOO_BUSY response is also problematic as described in the base specification. DIAMETER_TOO_BUSY is defined as a protocol error. If an agent receives a protocol error, it may either handle it locally or it may forward the response back towards the downstream peer. (The Diameter specification is inconsistent about whether a protocol error MAY or SHOULD be handled by an agent, rather than forwarded downstream.) If a downstream peer receives the DIAMETER_TOO_BUSY response, it may stop sending all requests to the agent for some period of time, even though the agent may still be able to deliver requests to other upstream peers.
DIAMETER_UNABLE_TO_DELIVER also has no mechanisms for specifying the scope or cause of the failure, or the durational validity.
As the number of Third Generation (3G) and Long Term Evolution (LTE) enabled smartphone devices continue to expand in mobility networks, there have been situations where high signaling traffic load led to overload events at the Diameter-based Home Location Registries (HLR) and/or Home Subscriber Servers (HSS). The root causes of the HLR congestion events were manifold but included hardware failure and procedural errors. The result was high signaling traffic load on the HLR and HSS.
The 3GPP standards specification[need citation] for the end-to-end signaling call flows in 3G and LTE, from the end user device traversing through the radio and the core networks to the HLR/HSS, did not have an equivalent load control mechanism which is provided in the more traditional SS7 elements in GSM [need citation]. The capabilities specified in the 3GPP standards do not adequately address the abnormal condition where excessively high signaling traffic load situations are experienced.
Smartphones contribute much more heavily to the continuation of a registration surge due to their very aggressive registration algorithms. The aggressive smartphone logic is designed to:
Non-smartphones typically have logic to wait for a time period after registering successfully on voice and data.
The smartphone aggressive registration is problematic in two ways:
A study in 3GPP SA2 on core network overload has produced the technical report [TR23.843]. This enumerates several causes of overload in mobile core networks including portions that are signaled using Diameter. This document is a work in progress and is not complete. However, it is useful for pointing out scenarios and the general need for an overload control mechanism for Diameter.
It is common for mobile networks to employ more than one radio technology and to do so in an overlay fashion with multiple technologies present in the same location (such as GSM, UMTS or CDMA along with LTE). This presents opportunities for traffic storms when issues occur on one overlay and not another as all devices that had been on the overlay with issues switch. This causes a large amount of Diameter traffic as locations and policies are updated.
Another scenario called out by this study is a flood of registration and mobility management events caused by some element in the core network failing. This flood of traffic from end nodes falls under the network initiated traffic flood category. There is likely to also be traffic resulting directly from the component failure in this case.
Subscriber initiated traffic floods are also indicated in this study as an overload mechanism where a large number of mobile devices attempting to access services at the same time, such as in response to an entertainment event or a catastrophic event.
While this study is concerned with the broader effects of these scenarios on wireless networks and their elements, they have implications specifically for Diameter signaling. One of the goals of this document is to provide guidance for a core mechanism that can be used to mitigate the scenarios called out by this study.
This section proposes requirements for an improved mechanism to control Diameter overload, with the goals of improving the issues described in Section 4 and supporting the scenarios described in Section 2
This document makes no requests of IANA.
A Diameter overload control mechanism is primarily concerned with the load and overload related behavior of nodes in a Diameter network, and the information used to affect that behavior. Load and overload information is shared between nodes and directly affects the behavior and thus is potentially vulnerable to a number of methods of attack.
Load and overload information may also be sensitive from both business and network protection viewpoints. Operators of Diameter equipment want to control visibility to load and overload information to keep it from being used for competitive intelligence or for targeting attacks. It is also important that the Diameter overload control mechanism not introduce any way in which any other information carried by Diameter is sent inappropriately.
This document includes requirements intended to mitigate the effects of attacks and to protect the information used by the mechanism.
To control the visibility of load and overload information, sending should be subject to some form of authentication and authorization of the receiver. It is also important to the receivers that they are confident the load and overload information they receive is from a legitimate source. Note that this implies a certain amount of configurability on the nodes supporting the Diameter overload control mechanism.
An overload control mechanism provides a very attractive target for denial-of-service attacks. A small number of messages may affect a large service disruption by falsely reporting overload conditions. Alternately, attacking servers nearing, or in, overload may also be facilitated by disrupting their overload indications, potentially preventing them from mitigating their overload condition.
A design goal for the Diameter overload control mechanism is to minimize or eliminate the possibility of using the mechanism for this type of attack.
As the intent of some denial-of-service attacks is to induce overload conditions, an effective overload control mechanism should help to mitigate the effects of an such an attack.
An attacker that has managed to obtain some messages from the overload control mechanism may attempt to affect the behavior of nodes supporting the mechanism by sending those messages at potentially inopportune times. In addition to time shifting, replay attacks may send messages to other nodes as well (target shifting).
A design goal for the Diameter overload control mechanism is to minimize or eliminate the possibility of causing disruption by using a replay attack on the Diameter overload control mechanism.
By inserting themselves in between two nodes supporting the Diameter overload control mechanism, an attacker may potentially both access and alter the information sent between those nodes. This can be used for information gathering for business intelligence and attack targeting, as well as direct attacks.
A design goal for the Diameter overload control mechanism is to minimize or eliminate the possibility of causing disruption man-in-the-middle attacks on the Diameter overload control mechanism. A transport using TLS and/or IPSEC may be desirable for this.
A compromised host that supports the Diameter overload control mechanism could be used for information gathering as well as for sending malicious information to any Diameter node that would normally accept information from it. While is is beyond the scope of the Diameter overload control mechanism to mitigate any operational interruption to the compromised host, a reasonable design goal is to minimize the impact that a compromised host can have on other nodes through the use of the Diameter overload control mechanism. Of course, a compromised host could be used to cause damage in a number of other ways. This is out of scope for a Diameter overload control mechanism.
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| [I-D.ietf-dime-rfc3588bis] | Fajardo, V, Arkko, J, Loughney, J and G Zorn, "Diameter Base Protocol", Internet-Draft draft-ietf-dime-rfc3588bis-29, August 2011. |
| [RFC2914] | Floyd, S., "Congestion Control Principles", BCP 41, RFC 2914, September 2000. |
| [RFC3539] | Aboba, B. and J. Wood, "Authentication, Authorization and Accounting (AAA) Transport Profile", RFC 3539, June 2003. |
| [RFC5390] | Rosenberg, J., "Requirements for Management of Overload in the Session Initiation Protocol", RFC 5390, December 2008. |
| [TR23.843] | 3GPP , "Study on Core Network Overload Solutions", TR 23.843 0.4.0, April 2011. |
Significant contributions to this document were made by Adam Roach and Eric Noel.
Review of, and contributions to, this specification by Martin Dolly, Carolyn Johnson, Jianrong Wang, Imtiaz Shaikh, Jouni Korhonen, and Robert Sparks were most appreciated. We would like to thank them for their time and expertise.