Internet DRAFT - draft-kucherawy-spfbis-experiment
draft-kucherawy-spfbis-experiment
SPFBIS Working Group M. Kucherawy
Internet-Draft Cloudmark
Intended status: Informational April 4, 2012
Expires: October 6, 2012
Resolution of The SPF/Sender-ID Experiment
draft-kucherawy-spfbis-experiment-04
Abstract
In 2006 the IETF published a suite of protocol documents comprising
SPF and Sender-ID, two proposed email authentication protocols.
Because of interoperability concerns created by simultaneous use of
the two protocols by a receiver, and some concerns with Sender-ID and
compatibility with existing standards, the IESG required them to have
Experimental status and invited the community to observe their
deployments for a period of time, hoping convergence would be
possible later.
After six years, sufficient experience and evidence have been
collected that the experiment thus created can be considered
concluded, and a single protocol can be advanced. This memo presents
those findings.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on October 6, 2012.
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Kucherawy Expires October 6, 2012 [Page 1]
Internet-Draft SPF/Sender-ID Experiment April 2012
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. The Need For Consensus . . . . . . . . . . . . . . . . . . . . 3
3. Evidence of Deployment . . . . . . . . . . . . . . . . . . . . 4
4. Evidence of Differences . . . . . . . . . . . . . . . . . . . . 5
5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5.1. From the Evidence . . . . . . . . . . . . . . . . . . . . . 5
5.2. Recommendations to the IESG . . . . . . . . . . . . . . . . 6
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7
7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7
8. Informative References . . . . . . . . . . . . . . . . . . . . 7
Appendix A. Experiences Developing SPF . . . . . . . . . . . . . . 8
Appendix B. Acknowledgments . . . . . . . . . . . . . . . . . . . 9
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 9
Kucherawy Expires October 6, 2012 [Page 2]
Internet-Draft SPF/Sender-ID Experiment April 2012
1. Introduction
In April, 2006, the IETF published the [SPF] and Sender-ID email
authentication protocols, the latter consisting of three documents
([SUBMITTER], [SENDER-ID], and [PRA]). Both of these enable one to
publish via the Domain Name System a policy declaring which mail
servers were authorized to send email on behalf of a specific domain
name. The two protocols made use of this policy statement and some
specific (but different) logic to evaluate whether or not the email
client sending or relaying a message was authorized to do so.
Because Sender-ID could use the same policy statement as SPF, the
IESG at the time was concerned that an implementation of Sender-ID
might erroneously apply that statement to a message and, depending on
selected recipient actions, could improperly interfere with message
delivery. As a result, the IESG required the publication of all of
these documents as Experimental, and requested that the community
observe deployment and operation of the protocols over a period of
two years from publication in order to determine a reasonable path
forward. (For further details about the IESG's concern, see the IESG
Note prepended to all of those documents.)
Accordingly, this memo resolves the experiment by presenting evidence
regarding both deployment and efficacy of the two protocols, and
further discusses the increasing need for consensus. At the end it
presents conclusions based on the data collected.
2. The Need For Consensus
These two protocols fall into a family of protocols that provide
domain-level email authentication services. For reference, another
prominent one is [DKIM]. Various efforts exist that use these as
building blocks to increased abuse filtering capabilities, and indeed
this sort of work has spawned another working group in the
Applications area, with still more of these incubating in
associations and trade groups outside of the IETF.
There is thus some palpable interest in having a path authorization
scheme, as well as a domain-level signing scheme, on the Standards
Track so that these newer technologies can develop with confidence.
This is, in part, why the community has decided to expend the effort
to bring this experiment to a conclusion and document the results,
and then advance a single path authorization technology.
Kucherawy Expires October 6, 2012 [Page 3]
Internet-Draft SPF/Sender-ID Experiment April 2012
3. Evidence of Deployment
Two large-scale DNS surveys were run that looked for the two
supported kinds of resource records (RR) that can contain SPF policy
statements.
One data source for this report requested SPF records from one
million domains. Approximately 287,267 domains included non-error
and non-empty replies. Of these, 287,250 included type 16 (DNS RR
TXT) replies, 4,613 included type 99 (DNS RR SPF) replies, and 4,596
included both types.
Another source requested SPF records from 251,651 domains for which
there was a history of previous observed SPF evaluations. Of these,
136,018 returned type 16 answers, 2,605 returned type 99 answers,
2,439 returned both types, and 115,465 returned neither. Of those
answers retrieved, 6,479 included records that start with the string
"spf2.0/pra" which are specific requests for Sender-ID processing by
receivers.
During this second survey, some domains were observed to provide
immediate answers for type 16 queries, but would time out waiting for
replies to type 99 queries. For example, it was observed that 3,953
distinct domains in the survey returned a result of some kind (a
record or an error) for the TXT query in time N, while the SPF query
ultimately failed but only after at least time 4N.
It is likely impossible to determine from a survey which MTAs have
SPF and/or Sender-ID checking enabled at message ingress since it
does not appear, for example, in the reply to the EHLO command from
extended [SMTP]. We therefore rely on evidence found via web
searches, and observed the following:
o A web site [SID-IMPL] dedicated to highlighting Sender-ID
implementations last updated in late 2007 listed 13
implementations, which we assume means they implement the PRA
checks. At least one of them is known no longer to be supported
by its vendor.
o The [OPENSPF] web site maintains a list of known implementations
of SPF. At the time of this memo's writing it listed six
libraries, 22 MTAs with built-in SPF implementations, and numerous
patches for MTAs and mail clients.
In a survey of numerous MTAs in current or recent use, only two
(Santronics WinServer and McAfee MxLogic) were found to contain
implementations of the SMTP SUBMITTER extension as part of the MTA
service, which could act as an enabler to Sender-ID. An unknown
Kucherawy Expires October 6, 2012 [Page 4]
Internet-Draft SPF/Sender-ID Experiment April 2012
number of clients implement it; although there is substantial
activity showing its use in logs, it is unclear whether these are
separate implementations by legitimate senders, or merely instances
of distributed automated malware seeking to improve their odds of
reaching the end user.
A survey was done of queries to type 16 and type 99 records by
observing nameserver logs. Only a few queries were ever received for
type 99 records, and those almost exclusively came from one large
email service provider that queried for both types. The vast
majority of other querying agents only ever requested type 16.
[pending: SPF query results from Hotmail]
[other data TBD]
4. Evidence of Differences
It is plain from inspection of the two protocols that they have much
in common: For a single message, both require the same number of DNS
queries, and both require the same code to parse the result. The PRA
algorithm applied by Sender-ID is, however, more expensive than
simply extracting the domain name from the omnipresent
RFC5321.MailFrom. Thus, SPF is cheaper to apply to a message.
One set of specific data collected by a working group contributor
shows that in more than 95.5% of cases, Sender-ID and SPF reach the
same conclusion about a message, meaning either both protocols return
a "pass" result or both return a "fail" result. The data set
yielding this response could not further characterize the cases in
which the answers differed.
[pending: MAIL FROM/PRA comparison report from Hotmail]
[other data TBD]
5. Conclusions
It is standard procedure within the IETF to document as standard
those protocols and practices that have come into sufficient common
use as to become part of the basic infrastructure.
5.1. From the Evidence
Given the six years that have passed since the publication of the
experimental RFCs, and the evidence reported in the earlier sections
Kucherawy Expires October 6, 2012 [Page 5]
Internet-Draft SPF/Sender-ID Experiment April 2012
of this document, the following conclusions are supported:
1. There has not been substantial adoption of the type 99 (SPF) DNS
resource record. In both large-scale surveys performed for this
work, less than 2% of responding domains published type 99
records, and almost no clients requested them.
2. Of the records retrieved, fewer than 5% requested processing of
messages using the PRA algorithm that was an integral part of
Sender-ID.
3. No data collected showed any substantial operational benefit
(e.g., cheaper processing, improved accuracy) to using Sender-ID
over SPF.
4. A survey of implementations shows significant support for both
protocols, though there were more implementations in support of
SPF than of Sender-ID. Further, the SPF implementations showed
better upkeep and current interest than the Sender-ID
implemenations.
5. A survey of implementations shows no significant use of the
SUBMITTER extension by servers, but some by clients.
5.2. Recommendations to the IESG
In light of the above, the working group recommends to the IESG the
following:
1. that the experiment comprising the series of RFCs defining the
SUBMITTER SMTP extension, the Sender-ID mechanism, the Purported
Responsible address algorithm, and SPF, be considered concluded;
2. that [SPF] be advanced to the standards track after appropriate
technical review with respect to the deployed base;
3. that this revision to SPF deprecate the use of the type 99 DNS
resource record by both clients and servers;
4. that [SUBMITTER], [SENDER-ID], and [PRA] have failed to gain any
substantial adoption and are thus, de facto, obsolete (and thus
could be marked "Obsoleted" and "Historic" by some future
action).
Appendix A is offered as a cautionary review of problems that
affected the process of developing SPF and Sender-ID in terms of use
of the DNS.
Kucherawy Expires October 6, 2012 [Page 6]
Internet-Draft SPF/Sender-ID Experiment April 2012
6. IANA Considerations
This memo presents no actions for IANA. [RFC Editor: Please remove
this section prior to publication.]
7. Security Considerations
This memo contains information for the community only, akin to an
implementation report, and does not introduce any new security
concerns. Its implications could, in fact, resolve some.
8. Informative References
[DKIM] Crocker, D., Ed., Hansen, T., Ed., and M. Kucherawy, Ed.,
"DomainKeys Identified Mail (DKIM) Signatures", RFC 6376,
September 2011.
[OPENSPF] "Sender Policy Framework: Project Overview",
<http://www.openspf.net>.
[PRA] Lyon, J., "Purported Responsible Address in E-Mail
Messages", RFC 4407, April 2006.
[SENDER-ID]
Lyon, J. and M. Wong, "Sender ID: Authenticating E-Mail",
RFC 4406, April 2006.
[SID-IMPL]
"Sender ID Framework Industry Support and Solutions",
October 2007, <http://www.microsoft.com/mscorp/safety/
technologies/senderid/support.mspx>.
[SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321,
October 2008.
[SPF] Wong, M. and W. Schlitt, "Sender Policy Framework (SPF)
for Authorizing Use of Domains in E-Mail, Version 1",
RFC 4408, April 2006.
[SUBMITTER]
Allman, E. and H. Katz, "SMTP Service Extension for
Indicating the Responsible Submitter of an E-Mail
Message", RFC 4405, April 2006.
Kucherawy Expires October 6, 2012 [Page 7]
Internet-Draft SPF/Sender-ID Experiment April 2012
Appendix A. Experiences Developing SPF
SPF was originally developed by a community of interested developers
outside the IETF. It was brought to the IETF for standardization
only after the specification was relatively mature and ready for the
rigors of the IETF publication process.
At the time, the prospect of getting a DNS resource record (RR) type
allocated for SPF was not seriously considered, partly because it was
perceived to have high barriers to entry. As a result, by the time
the working group was formed, there was already a substantial and
growing installed base that had SPF running using TXT RRs.
Eventually the application was made for the new RR type as a result
of pressure from the DNS experts in the community, who encouraged
doing so as the preferred path toward using the DNS for storing such
things as policy data.
Later, after type 99 was assigned (long after IESG approval of the
document, in fact), a plan was put into place to effect a gradual
transition to using type 99 instead of using type 16. This plan
failed to take effect for four primary reasons:
1. there was hesitation to make the transition because of concerns
that nameservers (and, in fact, DNS-aware firewalls) would drop
or reject requests for unknown RR types (see Section 3 for
evidence of this), which means successful rollout of a new RR
type is contingent upon widespread adoption of updated
nameservers and resolver functions;
2. many DNS provisioning tools (e.g., web interfaces to controlling
DNS zone data) were, and still are, typically lethargic about
adding support for new RR types;
3. the substantial deployed base was already using type 16, and it
was working just fine, leading to inertia;
4. [SPF] itself included a faulty transition plan: It said a server
SHOULD publish both types and MUST publish at least one, while a
client can query either or both, which means both can claim to be
fully compliant while failing utterly to interoperate.
It is likely that this will happen again if the bar to creating new
RR types even for experimental development purposes is not lowered,
and handling of unknown RR types becomes generally more graceful.
There are DNS experts within the community that will undoubtedly
point to DNS servers and firewalls that mistreat queries for unknown
RR types, and claim they are broken, as a way of answering this
Kucherawy Expires October 6, 2012 [Page 8]
Internet-Draft SPF/Sender-ID Experiment April 2012
concern. This is undoubtedly correct, but the reality is that they
are among us and likely will be for some time, and this needs to be
considered as new protocols and IETF procedures are developed.
Appendix B. Acknowledgments
The following provided operational data that contributed to the
evidence presented above:
Cisco: contributed data about observed Sender-ID and SPF records in
the DNS for a large number of domains
Hotmail: contributed data about the difference between
RFC5321.MailFrom and RFC5322.From domains across large mail
volumes, and a survey of DNS queries observed in response to
outgoing mail traffic
John Levine: conducted a survey of DNS server logs to evaluate SPF-
related query traffic
Santronics: contributed data about the use of the SUBMITTER
extension in aggregate SMTP client traffic
The Trusted Domain Project: contributed data about the difference
between Sender-ID and SPF results, and conducted one of the two
detailed TXT/SPF record surveys including collecting timing data
The author would also like to thank the following for their
contributions to the development of the text in this memo: Dave
Crocker, and Scott Kitterman
Author's Address
Murray S. Kucherawy
Cloudmark
128 King St., 2nd Floor
San Francisco, CA 94107
USA
Phone: +1 415 946 3800
Email: msk@cloudmark.com
Kucherawy Expires October 6, 2012 [Page 9]
