| Individual | D. Otis |
| Internet-Draft | D. Rand |
| Intended status: Informational | Trend Micro |
| Expires: November 30, 2013 | May 29, 2013 |
Macros Nixed by Major Providers
draft-otis-spfbis-macros-nixed-01
SPF is a popular tool for managing email blocking issues. It has also helped reduce backscatter related with Non-Delivery Notifications as was intended. A practically unused feature is the SPF macro capability that SPFbis still in part retains. This feature greatly diminishes SPF's utility and actually threatens email interchange, especially when used in conjunction with secondary policy layers when also ignored by major providers. The macro mechanism is also unable to cope with internationalization, and is found in virtually none of the published resource records. It is counter productive to retain the largely unused and often unsupported macro feature, despite efforts expended to develop the macro language.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 30, 2013.
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Checking reverse DNS [RFC1034] entries per SMTP [RFC5321] connection requires queries against the entire IP address. SPF [I-D.ietf-spfbis-4408bis] macros may go a step further of searching for address matches with the SMTP client resolved from reverse DNS PTR record sets in a manner similar to address matches with the 'MX' mechanism hostname list. For IPv6, reverse DNS checks by SMTP servers may inadvertently induce DDoS effects against the DNS managed by the sender's network provider while at the same time consuming receiving SMTP server's connection resources held waiting for subsequent DNS responses. Prior IPv4 mapping of reverse DNS is often used to avoid the wasting of resources resulting from reverse DNS, but this strategy is not possible with IPv6. Use of reverse DNS is seldom practical simply due to network resource considerations. While use of the SPF 'PTR' mechanism and the {%p} macro now offers 'do not use' advice, it is unfortunate similar considerations for uninvolved DNS servers that can be similarly burdened by local-part macros or various macro expansions of SMTP client addresses were not offered the same protective advice.
At least reasonable limits were placed on the number of NXDOMAIN responses. Unfortunately even that strategy is undermined by a technique growing in popularity of using synthetic domains to act as an alternative for web cookies where the resulting responses actually increases the level of amplification. There is no justification to use macros that are able to impose queries constructed of odd elements of the email-address local-part or that of the SMTP client IP address.
Windows, OS X, iOS, Linux employ IPv6 privacy extensions [RFC4941] where any problematic IP address is likely to be reassigned different addresses regularly and remain valid beyond the reassignment interval. Since a major portion of spam originates from compromised residential systems, any attempt to handle or cache this highly diverse information should not be considered practical.
It should also be pointed out use of macros may not function when [RFC6531] permits international email addresses that includes both local-part and domain portions of the email address. Tests and comments by large email providers show an understandable reluctance to process SPF macros where often they simply do not implement the macro portion of SPF.
The SPF review documented in [RFC6686] failed to provide a breakdown on macro use. SPF's macros can prove hazardous, especially with IPv6. Rather than permitting receivers to decide their acceptance methods, an SPF resource record based on the Mail From parameter can induce 100 DNS transactions using labels constructed from random strings such as those found in the message local-part or the expansion of the SMTP client IP address. Recent changes made in the current version of SPF recommends against reverse DNS use where processing the reverse namespace may place an extreme burden on the sender's network provider's DNS and the SMTP server's connection resources. Too bad unrelated DNS providers where not afforded similar consideration.
SPF macros have not anticipated the use of internationalized email addresses and lack any negotiation with respect to this type of email. A fair amount of problematic repair regarding construction of DNS queries based on internationalized email-addresses can be avoided by simply acknowledging that currently SPF macros are seldom used and often not implemented by the larger providers.
[RFC5890] does not convert non-domain components permitted by [RFC6531]. As such, this may inject UTF8 strings into code expecting ASCII and potentially expose vulnerabilities that may permit SMTP servers to be compromised. Once again, this concern is easily avoided by deprecating implementation of macros, where such deprecation should also be assumed by senders to ensure email exchange.
Access to an outbound SMTP servers may or may not restrict use of the Mail From Parameter. An alternative mitigation strategy may rely upon rate limits and denial of access subsequent to abuse reports. Reasons for publishing SPF records might be to mitigate backscatter based on records that offer Non-Delivery-Notifications (NDNs) authorization by the Mail From parameter where the loss of some of these notifications is considered acceptable. When used to authorize bulk senders, or provide results that are independent of the SMTP client, the process will not offer a status safely associated with any particular message.
SPF failure rates of even a few percent are too high for it to offer a solid basis for either acceptance or rejection. There are some policy strategies for specific domains [I-D.kucherawy-dmarc-base] that attempt to combine SPF [I-D.ietf-spfbis-4408bis] and DKIM [RFC6376] where when either of their related domains manage to pass, only then is the message to be placed in the in-box.
Domains that reference SPF resource records should not be considered authenticated because the SPF process authorized the SMTP client. Often an SMTP client is shared by many domains. As such, it would be incorrect to assume SPFv1 records that authorize a provider by way of the Mail From Parameters is only permitted by authenticated accounts from that domain.
As a result, no domain information in SPF or DKIM, under current circumstances, can be used as a domain reference for either acceptance or reputation.
When IPv6 becomes more commonly used, some will attempt white-listing IP addresses as a practical albeit non-scalable method to deal with the highly increased address range confronted when dealing with IPv6. There are currently no practical solutions able to scale to the challenging range of IP addresses that might be controlled by malefactors.
Unlike most DNS resources that segregate IPv4 and IPv6 datasets, SPF consolidates both IPv4 and IPv6 addresses into a common list comprised of a chain of DNS transactions. In addition to explicit chaining, SPF expects receivers to process the first SPF resource record which may contain 10 SPF mechanisms. Each mechanism may then require an additional 10 DNS transactions to resolve the mechanism's status. According to [spf-all.com] out of 112,311,175 domains, 24,073,182 (21.4 %) publish SPF records where only 12,768 (0.053%) employ macros. Even the 0.053% figure overstates use since many represent questionable server farm deployments. These are questionable since many major providers do not process SPF records that contain macros. Non-implementation of macros is very good from a security and reliability perspective, but problematic for email's integrity when used.
When SPF was first introduced, AOL claimed they used SPF to create IP address white-lists based upon domains with whom they collaborate. White-listing would be less effective if macros were used in SPF records to modify results based upon variables such as the IP address of the client or the local-part of the email address. Ask a group of email providers about SPF, and you'll likely hear SPF is used to identify IP addresses used by a domain to send email. With this information, they can then check these addresses against popular reputation services to determine where problems are apparent. Silent discard or placement into spam folders reduces delivery status integrity and makes supporting unreliable delivery difficult.
The information returned in the Authentication-Results header does not always clarify whether the SPF records were actually processed. In speaking with some of these providers about potential network amplification concerns along with resource consumption, they made assurances they do not process SPF macros and they have not had any complaints. It seems these providers are also aware of path registration failure rates associated with SPF, and do not reject email on the basis of SPF failure.
[I-D.kucherawy-dmarc-base] changes the importance of SPF records being processed, as they will more likely come into play during message acceptance and not be limited to just deciding whether a NDN is safe to send. As such, both 'do not use' and 'do not process' macros offers improved interchange and security. SPF's intent is to constrain the number of invalid NDNs being returned when someone spoofs their domain. Evidence of this intent might be denoted by SPF record's ending or providers overriding them with "?all". This approach better supports DMARC's efforts at improving delivery rates with improved policy enforcement. Because DMARC may actually convince a provider to refuse messages based upon SPF failures, it is important that it is not used for messages that are sent to mailing-lists for example.
[I-D.ietf-spfbis-4408bis] purports to be documenting current use. It dropped the SPF resource record (99) due to sparse use and recommends against reverse PTR use. If a similar threshold were applied, all the SPF macro expansion aspects of this protocol would be removed as well. Several large providers have offered their assurance that their SPF libraries had these macros removed. Any effort hoping to define current use MUST carefully reconsider the inclusion of this ill-considered macro feature. Otherwise, it can not be suggested SPF-bis simply documents current use.
Danger imposed by the use of the local-part macro is inherent in the query required to support both an IPv6 expansion, in conjunction with the expansion of local-part macros induced by possibly cached SPF records. The local-part, along with a range of IP addresses made readily possible by IPv6, can be manipulated to induce a flurry of large DNS transactions directed toward any otherwise uninvolved domain, all directed by cached DNS records that contain active content.
It is never a good idea to allow free attacks enabled through foreign scripts. The SPF local-part macro would allow a cached DNS record to be repeatedly exploited by a spam campaign with an attacker consuming little of their resources beyond what they already use in their campaign. In this case, their recipients change the domains queried on behalf of the attackers based upon the same SPF resource record. High levels of amplification still apply if SPF specifications are not changed and resulting implementation are then assumed part of some greater security requirement and are no longer deemed experimental. [dns-response-rate-limit] represents a DNS defense strategy having limited benefit until most recursive DNS servers also include Access Control Lists (ACLs). The adoption of the use of ACLs is far more complex than blocking Directed Broadcasts in Routers [RFC2644] which remains a problem 14 years later. See: [smurf-attack]. In addition, even if [dns-response-rate-limit] was fully implemented, SPF macros offer an effective bypass method for rate limits imposed on common answers from IPv4 /24 or IPv6 /56 network blocks.
Even The SPF macro DOS overview demonstrated in 2006 [otis-spf-dos-exploit] did not consider IPv6 use which offers higher gain, where the gain achieved sending a single 1KB message to a single recipient offered a 1:156 reflected gain over IPv4 that can never be filtered or blocked by using BCP38 [RFC2827]. When made against a wildcard record, even higher gains can be achieved. It should also be noted that [I-D.kucherawy-dmarc-base] section 4.3 "For example, [DKIM] authenticates the domain that affixed a signature to the message, while [SPF] authenticates either the domain that appears in then RFC5321.MailFrom portion of [SMTP] or the RFC5321.EHLO/HELO domain if the RFC5321.MailFrom is null (in the case of Delivery Status Notifications). This conflicts with the assertion made by [I-D.ietf-spfbis-4408bis] section 2.1 that states "If a conclusive determination about the message can be made based on a check of HELO, then the use of DNS resources to process the typically more complex MAIL FROM can be avoided." Few bother to ensure verification of the EHLO domain, and may even expect the EHLO will not be validated by SPF. The result of this dichotomy in preferred results will likely necessitate repeated SPF processing. No cached results can be used whenever macros have been included and this can significantly increase network amplifications and not constrain SPF to a single process layer.
Logically, local-part macros would not safely provide positive results without the query being combined with an IP address list of SMTP clients in some form. A list structure would need to be repeated for each user. Rcpt To using schemes like BATV provides an expedient way where the purported sender determines whether a NDN can be returned without the risk associated with active content placed within DNS resource records. Several parsers ignore local-part macro expansion since they rarely play any role in providing positive results.
The [I-D.ietf-spfbis-4408bis] Addendum E indicates a possible use of SPF macros such as exists:{%l}._spf.{%d} or exists:{%i}_spf.{%d} can be used in "specialized" DNS servers able to understand encrypted local-parts or react to inordinate query rates which suggests use of resource records having extremely short TTLs. Such an approach can add a sizeable burden on recipients and targeted DNS servers. Such a scheme represents a poor alternative to providing authenticated domain identifiers. The general intent and purpose of SPF is to offer Non-Delivery Notification authorization where no macros are required and by most practical measure are not being used.
Problems related to various types of network abuse made possible by SPF macro extensions are not prevented by assuming corrective behavior is obtained through advice given to senders. In general, it is the sender not to be trusted. Only by giving the advice 'do not use and do not implement' is both security and interchange ensured.
Upon greater reflection regarding deprecation of the SPF type (99) record, this was a mistake. When overlaying the TXT resource record was originally proposed, many strongly objected. Studies then showed scant use of TXT records which convinced those in the then MARID WG they could adopt the TXT record at the domain apex as belonging to SPF as their means to support wildcard use. Although support of unknown RR types has improved, the vendor that made adoption of a different RR type problematic still persists with their related name service extensions imposing barriers. SPF Type (99) should be retained to better ensure extensions to DNS retain an ability to cope with unknown RR types.
A manageable basis for assessments can leverage a smaller number of related domains, compared to IPv6 or even IPv4 addresses. Although technically the domain name space can be larger than the massively large IPv6 address space, in practice it is not. One hundred thousand domains control 90% of Internet traffic out of approximately 100 million domains active each month. The top 150 domains control 50% of the traffic, and the top 2,500 domains control 75%. This level of domain consolidation permits effective fast-path white-listing. Improvements achieved using domains to consolidate the threat landscape can easily justify added cryptographic authentication burdens. Even APL resource records [RFC3123] can authenticate EHLO using a single DNS transaction, but this would not allow IPv6 email to be more easily managed when facing extensive use of transitional technologies such as ISATAP, Teredo, 6to4, NAT64, and DNS64, which cryptographic technology can offer.
Unfortunately, the complexity associated with SPF was never intended to authenticate the SMTP client. SPF instead has become a mechanism allowing bulk senders a means to shift accountability by offering an authorization scheme to their customers. Unfortunately, this authorization mechanism becomes unusable when navigating through various IPv6 transitional technologies.
This document requires no IANA consideration.
This draft intends to describe serious security and interchange concerns with use of SPF macros. The contained recommendations are expected to reduce these concerns. Greater caution is warranted before making assumptions about DNS activity. Due to high volumes of traffic, rarely is this traffic logged directly by the server. Some items might be triggered while passing through firewalls, but without being given the item to watch, discovering a well orchestrated attack is analogous to discovering a needle in the proverbial haystack.
Many administrators overlook a serious problem made much worse by chatty protocols that impose processing delays. Examining server logs will not reveal any problem either, because the limited resource being consumed is the number of outstanding connections TCP is able to support. Reaching this limit will prevent new connections from being instantiated but this is not logged as an event. Over time administrators may hear complaints email is not being delivered or just see an ever growing percentage of spam.
Add language clearly indicating BOTH the implementation or use of SPF macro features is hereby deprecated. Any such use provides a neutral result. In addition, restore SPF type 99 to better insure the future health of DNS.