Internet DRAFT - draft-ylonen-sshkeybcp

draft-ylonen-sshkeybcp






draft-ylonen-sshkeybcp-01.txt
Network Working Group                                          T. Ylonen
Internet-Draft                               SSH Communications Security
Expires: October 06, 2013                                        G. Kent
                                                                SecureIT
                                                             M. Souppaya
                                                                    NIST
                                                          April 04, 2013


 Managing SSH Keys for Automated Access - Current Recommended Practice

Abstract

   This document presents current recommended practice for managing SSH
   user keys for automated access.  It provides guidelines for
   discovering, remediating, and continuously managing SSH user keys and
   other authentication credentials.

   Various threats from poorly managed SSH keys are identified,
   including virus spread, unaudited backdoors, illegitimate access
   using leaked keys, lack of proper termination of access, use of
   legitimate access for unintended purposes, and accidental human
   errors.

   Hundreds of thousands, even over a million SSH keys authorizing
   access have been found from the IT environments of many large
   organizations.  This is many times more than they have interactive
   users.  These access-granting credentials have largely been ignored
   in identity and access management, and present a real risk to
   information security.

   A process is presented for discovering who has access to what,
   bringing an existing IT environment under control with respect to
   automated access and SSH keys.  The process includes moving
   authorized keys to protected locations, removing unused keys,
   associating authorized keys with a business process or application
   and removing keys for which no valid purpose can be found, rotating
   existing keys, restricting what can be done with each authorized key,
   and establishing an approval process for new authorized keys.  A
   process is also presented for continuous monitoring and controlled
   authorized key setup.

   Finally, recommendations are made for security policy makers for
   ensuring that automated access and SSH keys are properly addressed in
   an organization's security policy.

   Specific requirements are presented that address the security issues
   while keeping costs reasonable.



Ylonen, et al.          Expires October 06, 2013                [Page 1]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Guidance is also provided on how to reduce operational cost while
   addressing the threats and how to use tools to automate the
   management process.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on October 06, 2013.

Copyright Notice

   Copyright (c) 2013 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Purpose and Scope . . . . . . . . . . . . . . . . . . . .   4
     1.2.  Audience  . . . . . . . . . . . . . . . . . . . . . . . .   4
     1.3.  Structure of This Document  . . . . . . . . . . . . . . .   4
     1.4.  Words Signifying Level of Requirement . . . . . . . . . .   5
     1.5.  Impact Levels for Information Systems . . . . . . . . . .   5
   2.  The Basics of SSH Protocol and Implementations  . . . . . . .   8
     2.1.  The SSH Protocol  . . . . . . . . . . . . . . . . . . . .   8
     2.2.  User Authentication in SSH  . . . . . . . . . . . . . . .   8
       2.2.1.  Password Authentication . . . . . . . . . . . . . . .   9



Ylonen, et al.          Expires October 06, 2013                [Page 2]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


       2.2.2.  Public Key Authentication . . . . . . . . . . . . . .   9
       2.2.3.  Kerberos Authentication . . . . . . . . . . . . . . .   9
       2.2.4.  Host-Based Authentication . . . . . . . . . . . . . .  10
       2.2.5.  Comparison of the Authentication Methods  . . . . . .  10
       2.2.6.  Dangers of Unverified and Shared Host Keys  . . . . .  10
     2.3.  Common Use Cases  . . . . . . . . . . . . . . . . . . . .  11
       2.3.1.  Interactive Use . . . . . . . . . . . . . . . . . . .  11
       2.3.2.  Automated Access  . . . . . . . . . . . . . . . . . .  11
       2.3.3.  File Transfers  . . . . . . . . . . . . . . . . . . .  12
   3.  Threats Arising from Poorly Managed Automated Access and SSH
       Keys  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  12
     3.1.  Virus Spread Threat . . . . . . . . . . . . . . . . . . .  13
     3.2.  Unaudited Backdoor Threat . . . . . . . . . . . . . . . .  14
     3.3.  Leaked Keys May Provide Access for Extended Periods . . .  15
     3.4.  Lack of Proper Termination of Access  . . . . . . . . . .  16
     3.5.  Use for Unintended Purpose  . . . . . . . . . . . . . . .  17
     3.6.  Accidental Data Transfers and Human Errors  . . . . . . .  18
     3.7.  Problem Under Radar . . . . . . . . . . . . . . . . . . .  19
   4.  Assessing the SSH Key Management Situation and Risks  . . . .  20
   5.  Key Remediation Solution Planning and Deployment  . . . . . .  23
     5.1.  Discovering SSH Keys and Trust Relationships  . . . . . .  24
     5.2.  Moving Authorized Keys to Protected Locations . . . . . .  28
     5.3.  Monitoring Use of Trust Relationships and Keys  . . . . .  29
     5.4.  Removing Trust Relationships That Are No Longer Used or
           Otherwise Inappropriate . . . . . . . . . . . . . . . . .  31
     5.5.  Associating Trust Relationships with Application and/or
           Purpose . . . . . . . . . . . . . . . . . . . . . . . . .  32
     5.6.  Implementing Approval Process for Setting Up New Trust
           Relationships . . . . . . . . . . . . . . . . . . . . . .  33
     5.7.  Rotating Existing SSH User Keys . . . . . . . . . . . . .  35
     5.8.  Configuring Command Restrictions on Authorized Keys . . .  36
     5.9.  Configuring IP Address Restrictions on Authorized Keys  .  37
   6.  Continuous Monitoring and Management of SSH Keys and
       Automated Access  . . . . . . . . . . . . . . . . . . . . . .  38
     6.1.  Continuous Monitoring of Changes to Trust Relationships .  38
     6.2.  Removal of Trust Relationships  . . . . . . . . . . . . .  41
     6.3.  Periodic Rotation of Trust Relationships  . . . . . . . .  41
   7.  Policy Recommendations  . . . . . . . . . . . . . . . . . . .  41
   8.  Considerations for Software Tools . . . . . . . . . . . . . .  45
     8.1.  Reducing Cost and Improving Security by Automation  . . .  46
   9.  Security Considerations . . . . . . . . . . . . . . . . . . .  47
   10. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  49
   11. Glossary  . . . . . . . . . . . . . . . . . . . . . . . . . .  49
   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  57
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  58

1.  Introduction




Ylonen, et al.          Expires October 06, 2013                [Page 3]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


1.1.  Purpose and Scope

   This document focuses on risks related to poorly managed automated
   access in information systems and particularly SSH user keys, and how
   to reduce the risks.  It documents current best practice of managing
   SSH keys for cost-effectively minimizing the risks, and provides
   security policy recommendations and auditing guidelines relating to
   SSH keys and other automated access.

1.2.  Audience

   This document is intended for information security policy makers,
   auditors, security managers, IT operations managers, system
   administrators, and others who are responsible for specifying,
   acquiring, testing, implementing, maintaining, and auditing identity
   and access management and SSH solutions.  Portions of the document
   may be of interest to technically advanced end users and systems
   programmers involved with SSH and other automated access
   technologies.

1.3.  Structure of This Document

   Section 1.4 specifies what certain words indicating level of
   requirement for compliance with this standard mean.

   Section 1.5 defines impact levels for information systems, including
   some important definitions relating to information systems having low
   impact themselves but having automated access to higher-impact
   information systems.

   Section 2 summarizes the basics of the SSH protocol and
   implementations, with particular emphasis on authentication methods
   for automated access and typical use cases for automated access.

   Section 3 describes threats arising from poorly managed SSH user
   keys.  Most of the threats are also relevant for other kinds of
   automated access.  However, because of the ubiquity of SSH keys and
   the acuteness of addressing them the discussion focuses on SSH keys.

   Section 4 introduces simple metrics and questions that are useful in
   scoping the risks related to SSH user keys and gaining basic
   understanding of the size of the problem in an organization.  The
   approach is suitable for both IT auditors responsible for assessing
   compliance with security policies as well as government and other
   policy makers wanting to measure the overall state of compliance and
   security across agencies.





Ylonen, et al.          Expires October 06, 2013                [Page 4]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Section 5 introduces the process for detailed analysis of existing
   automated trust relationships and risks (with an emphasis on SSH user
   keys), as well as recommended steps for remediating the risks.  This
   section also discusses the specific threats addressed by each
   remediation step and risks involved in not implementing a particular
   step.

   Section 6 provides recommendations for continuous monitoring and
   management of SSH user keys and other automated trust relationships,
   as well as for auditing steps to be taken for ensuring that an
   organization keeps automated access under control after an initial
   remediation phase.

   Section 7 provides recommendations for an organization's security
   policy for properly addressing SSH user keys and automated access.

   Section 8 summarizes issues to consider when planning use of
   automated software tools for managing automated access with SSH and
   particularly SSH user keys.  It also illustrates how to achieve cost
   savings in existing operational processes.

   Section 9 summarizes security considerations.  Most of this document
   is about security and managing elements of security and access, and
   this section serves as a conclusion and summary of this document.

   Section 11 provides a glossary of the technical terms used in this
   document.

1.4.  Words Signifying Level of Requirement

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119.

1.5.  Impact Levels for Information Systems

   The appropriate level of security and effort expended on security
   often depends on the level of impact from a failure or compromise of
   an information system.  FIPS Publication 199 [FIPS199] provides
   designations for impact levels on organizational information systems
   and a process for categorizing information systems.

   This document makes reference to the impact levels described FIPS 199
   (please see original document for exact definitions, this is just a
   simplifying summary):






Ylonen, et al.          Expires October 06, 2013                [Page 5]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Low impact:  Unauthorized disclosure, modification, destruction, or
      disruption of access have limited adverse effect on organizational
      operations, organizational assets, or individuals.

   Moderate impact:  Unauthorized disclosure, modification, destruction,
      or disruption of access could be expected to have a serious
      adverse effect on organizational operations, organizational
      assets, or individuals.

   High impact:  Unauthorized disclosure, modification, destruction, or
      disruption of access could be expected to have a severe or
      catastrophic adverse effect on organizational operations,
      organizational assets, or individuals.

   FIPS Publication 199 analyzes impact levels separately for
   confidentiality, integrity, and availability.  For considerations
   around automated access, the impact level of access to an account on
   an information system is taken to be the highest level of these three
   principles for the information system, since this specification
   primarily relates to operating system level access to information
   systems, and operating system level access can often be used to
   breach all three objectives simultaneously.  Furthermore, experience
   has shown that once an attacker has operating-system level access to
   one user account on a computer, various attack vectors (including
   bugs in system software and misconfigurations) can often be utilized
   to escalate the access to high-level administrative access.  That
   definitely compromises all three principles of information security.

   Configured trust relationships for automated access (e.g., using SSH
   user keys) may permit access from low-impact information systems to
   high-impact information systems without providing a password or other
   authentication credential from a user.  This is particularly relevant
   if the authentication credential or authorized key permits access on
   the high-impact information system without restrictions on the
   commands that can be executed on the high-impact information system.
   In this case, access to the low-impact information system implies
   access to the high-impact information system.  The information system
   owner inherently accepted this risk by allowing a low-impact system
   access to a high-impact system with providing compensation controls.
   There may also be situations where the high-impact system owner may
   not know the key has been copied to a low-impact system, or from one
   low-impact system to another.

   Therefore, whenever a low-impact information system has a configured
   trust relationship permitting it to access a high-impact information
   system without a restriction on the commands that can be executed on
   the high-impact information system, the low-impact information system
   MUST be treated as having the impact level of the highest-impact



Ylonen, et al.          Expires October 06, 2013                [Page 6]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   information system that it can access using automated trust
   relationships.

   This implies that to avoid treating low-impact information systems as
   high-impact systems, there must be a well-defined boundary in the IT
   environment that trust relationships can only cross in the direction
   allowing access from higher-impact systems into lower-impact systems,
   but not vice versa.  If such boundary is relied on, it MUST be
   audited and continuously monitored to enforce its existence.  Such a
   boundary could exist, for example, between development and production
   systems.

   Sometimes otherwise low-impact systems are used for producing code,
   software distributions, or data sets that will be used on higher-
   impact systems.  Such systems SHOULD be treated as higher-impact
   systems in view of Advanced Persistent Threat (APT) scenarios where
   an attacker could insert a backdoor in software that eventually gets
   copied to production systems.

   It should be noted that several current SSH implementations
   (including OpenSSH) only permit configuring command restrictions for
   access based on SSH user keys.  It is currently not possible to
   configure command restrictions for Kerberos-based authentication,
   host-based authentication, hard-coded passwords, or passwords coming
   from password vaults, which has implications for the above
   requirement.

   Command restrictions are a compensation control that can be leveraged
   to minimize the exposure to the additional risks exposed to a high-
   impact system from allowing limited access to the hosted resources
   from a low-impact system.  Command restrictions used for this purpose
   MUST be designed to be effective in limiting what actually can be
   done with the access, and MUST prevent interactive access and port
   forwarding.  For example, a command restriction permitting arbitrary
   commands or interactive shell is not effective.

   Furthermore, if a trust relationship has a command restriction that
   limits its use to file transfers but the directories from which files
   can be read or modified using it have not been restricted, it exposes
   the server to a more limited risk.  The trust relationship may be
   used to read any file or directory on the server accessible to the
   user account used for file transfers, even those outside the intended
   directories.  It may also be used to write any file that is writable
   by the user account; it is fairly common to have configuration files
   on servers that are inappropriately world-writable, in which case
   these files could be modified.





Ylonen, et al.          Expires October 06, 2013                [Page 7]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   If a trust relation is restricted to file transfers but does not
   limit the directories that can be accessed, the originating
   information system SHOULD be considered as having at least the impact
   level of the highest-impact information system to which it has such
   access.

2.  The Basics of SSH Protocol and Implementations

   SSH (Secure Shell) is a protocol and software tool for logging into a
   remote machine, executing commands remotely, and transferring files
   with a remote machine over a computer network.  SSH can also be used
   for implementing a protected tunnel for delivering other services.

   SSH is very widely used for administering Linux and Unix systems,
   virtual machines, routers, firewalls, and other network devices.  It
   is also embedded in many leading file transfer solutions, systems
   management solutions, identity management solutions, and privileged
   access management solutions.  It is widely used for integrating IT
   systems and automating their operation.

2.1.  The SSH Protocol

   The SSH protocol is an IETF standards track protocol and is described
   in RFC 4251 [RFC4251], RFC 4252 [RFC4252], RFC 4253 [RFC4253], and
   RFC 4254 [RFC4254].

   Many independent commercial and open source implementations are
   available, including Tectia SSH, OpenSSH, and many others.  SSH is
   available for nearly all platforms, including Linux/Unix, Windows,
   mainframes, routers, telephone exchanges, mobile devices such as
   smartphones and tablets, various embedded devices, and many
   industrial automation systems.

   In the SSH protocol, an SSH client application initiates a TCP/IP
   connection over a network to a destination server, negotiates
   encryption, authenticates the remote server, and then sends a
   destination user name and authentication credentials to the server.
   Server authentication is done using host keys, and its primary
   purpose is to prevent man-in-the-middle attacks; however server
   authentication is beyond the scope of this document.

2.2.  User Authentication in SSH

   The SSH protocol supports several mechanisms for authenticating
   users, including passwords, public key authentication, Kerberos, and
   host-based authentication.





Ylonen, et al.          Expires October 06, 2013                [Page 8]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


2.2.1.  Password Authentication

   There are two kinds of password authentication mechanisms in SSH:
   basic password authentication and keyboard-interactive
   authentication.  Keyboard-interactive authentication can support
   various types of challenge-response systems and various other
   authentication mechanisms.

   Password authentication is commonly used for interactive users, but
   less commonly for automated access (through it is sometimes seen with
   hard-coded passwords in scripts and management systems, especially
   for managing routers and file transfers).

2.2.2.  Public Key Authentication

   Public key authentication in SSH uses user keys or certificates to
   authenticate/authorize a connection.  An SSH client has an identity
   key, typically an RSA or DSA private key, and the server must have
   the corresponding public key configured as an authorized key for the
   destination user.  The private key may be protected by a passphrase,
   in which case it is encrypted by a key derived from the passphrase
   (passphrases are common for interactive users but rarely used for
   automated access).

   Many widely used SSH implementations support configuring restrictions
   for SSH user keys.  These may be used for limiting what can be done
   on the server using the key (command restrictions) and for limiting
   the IP addresses from which the key can be used (source
   restrictions).

   Public key authentication is by far the most frequently used method
   of configuring automated access using SSH at the time of this writing
   and represents best current practice.

2.2.3.  Kerberos Authentication

   Many organizations use Kerberos or Active Directory authentication
   with SSH.  Kerberos (usually together with LDAP) implements single
   sign-on and allows user accounts to be stored in a centralized
   directory.

   In practice, Kerberos is rarely used for non-interactive accounts.
   While it can be configured to use keytab files or cached tickets for
   functional accounts, these approaches rely on having long-term
   credentials stored on the host or at least accessible to the process
   on the host that is obtaining tickets.  These credentials can be
   exploited by an attacker largely in the same way as SSH user keys,
   e.g., by using them to obtain a ticket granting ticket (TGT) for the



Ylonen, et al.          Expires October 06, 2013                [Page 9]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   functional account and using the ticket to gain access to other hosts
   or accounts that the functional account can access (see virus spread
   threat below).

   One problem with Kerberos for automated access is that the single
   sign-on feature implies that once access has been gained to one
   account using Kerberos, it is usually possible to log in to any other
   server that has the same account and is in the same domain without
   further authentication.  This can very easily create lots of unwanted
   implicit trust relationships.  Existing implementations also do not
   support command restrictions for Kerberos.

2.2.4.  Host-Based Authentication

   Host-based authentication uses the source host's host key to
   authenticate the source host and to vouch for the identity of the
   user on the client side.  It is rarely used and does not permit
   configuring command restrictions.  Therefore its use for automated
   access is NOT RECOMMENDED.

2.2.5.  Comparison of the Authentication Methods

   All these authentication methods fundamentally rely on some secret
   information, and when used for automated access, this secret
   information must be stored on or accessible to the source host.

   A major advantage of public key authentication over the other methods
   is that it allows configuring a command restriction.  The command
   restriction can be used for preventing virus spread and other
   attacks, as described in Section 3.  It also does not create any
   implicit trust relationships and the permitted access can be reliably
   determined by inspecting the destination host (except for OpenSSH's
   proprietary certificate authentication, which SHOULD NOT be used
   because it cannot be reliably audited).  For these reasons, public
   key authentication is the RECOMMENDED authentication mechanism for
   automated access with SSH.

   Password authentication SHOULD NOT be used for automated access,
   because hard-coded passwords may be obtained by attackers and
   password vaults are also not particularly secure against local
   attacks on the client software.  If password authentication is used
   for automated access, the passwords MUST be rotated every three
   months.

2.2.6.  Dangers of Unverified and Shared Host Keys

   Many file transfer applications, privileged access management
   systems, and systems management applications do not check host keys



Ylonen, et al.          Expires October 06, 2013               [Page 10]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   for hosts that they connect to.  This permits a man-in-the-middle
   attack to be performed in the network.  Many tools are available for
   this and any device connected to a network through which the
   connection goes can be used for the attack - including, e.g.,
   reprogrammed smart switches.

   Man-in-the-middle attacks are a risk regardless of the authentication
   method if hosts keys are not properly verified.  The attack permits
   injection of arbitrary commands into the session, and reading and
   modifying any transferred files (including injection of bogus file
   transfers).  A successful man-in-the-middle attack from the network
   gives the same power as being able to use a trust relation leading to
   the destination host.

   Such man-in-the-middle attacks are practical, and are exploited in
   freely available attack tools and malware, as well as security
   software from multiple vendors for co-operative auditing purposes.

   Besides applications that do not check host keys, there are also some
   large enterprise that share the same host key on thousands of
   machines (for example, one Fortune 500 company is known to use the
   same host key on 14000 computers at the time of this writing).  If
   any of the computers is compromised, they all become vulnerable to
   man-in-the-middle attacks.

   Therefore, while this document is not really about host keys, the
   destination host MUST be properly authenticated by the client for all
   automated access and a unique host key MUST be used for each host.
   An exception may be made for the very first connection to a server to
   simplify system administration.

2.3.  Common Use Cases

2.3.1.  Interactive Use

   SSH has become the standard used by system administrators for
   configuring and managing Unix and Linux computers, routers, and
   various other equipment remotely.  It is also widely used by software
   developers, and in some organizations by ordinary end users for
   running applications remotely (particularly text-based legacy
   applications).  Public key authentication is often used by advanced
   end users for single sign-in.  Sometimes it is also used on jump
   servers.

2.3.2.  Automated Access

   SSH is very frequently used for automated access between systems.
   Such automated access is necessary for cost-efficiently managing



Ylonen, et al.          Expires October 06, 2013               [Page 11]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   large IT environments, for integrating applications, and for cost-
   effectively provisioning virtual machines in cloud services.

   Automated access refers to accessing a computer from another computer
   in an automated fashion.  Automated access may be unrestricted,
   allowing any commands to be executed, or may be limited to specific
   commands or operations, such as file transfers (perhaps limited to a
   specific directory).

   Automated access is most commonly used with functional accounts,
   system accounts, service accounts, and other non-interactive user
   accounts (sometimes also called non-user accounts).  Such accounts
   are used by operating systems, middleware, databases, and
   applications for running processes.  System or service accounts are
   likely to have sensitive levels of access to system resources (in
   that case they are often called privileged accounts).

   Automated access using SSH is common also in Windows and mainframe
   environments, especially for file transfer applications.  There are
   also various native mechanisms on Windows that can be used for
   automated access, but such mechanisms are beyond the scope of this
   document.

   Automated access has been largely ignored in Identity and Access
   Management.  Hundreds of thousands to over a million authorized SSH
   keys have been found from the IT systems of several large
   enterprises.  This means that they have many times more entry points
   for automated access configured on their servers than they have
   interactive users in the organization!  It is clear that such entry
   points cannot be ignored.

2.3.3.  File Transfers

   SSH is frequently used as a file transfer tool in itself, using the
   "scp" and "sftp" tools.  The SFTP [SFTP] protocol is gaining
   popularity in commercial and open source file transfer products, and
   a substantial fraction of the world's file transfers now use the SFTP
   protocol.  Automated file transfers using SSH typically use public
   key authentication or hard-coded passwords, and they are often also
   used between organizations.

3.  Threats Arising from Poorly Managed Automated Access and SSH Keys

   This section outlines some of the threats that have been identified
   with poorly managed SSH keys.  The guidelines and recommendations of
   this document are intended to address these while minimizing the
   administrative burden from managing the keys.




Ylonen, et al.          Expires October 06, 2013               [Page 12]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Several of the problems described below are present with many
   technologies for automated access besides SSH keys.  The issues must
   be addressed regardless of technology.

3.1.  Virus Spread Threat

   Malware can be engineered to use SSH keys to spread to most servers
   within an organization once it has penetrated one server.  Experience
   has shown that viruses frequently manage to penetrate individual
   servers in an organization.  Malware often uses multiple attack
   vectors to penetrate an organization and could use SSH user keys (or
   other trust relationships such as Kerberos) to spread within the
   organization's server infrastructure in minutes after penetrating the
   first server, thereby defeating layered security defenses.

   The Morris worm in 1988 utilized automated access trust relationships
   to spread in a similar manner (at that time based on ".rhosts"
   authentication).  This attack vector can be very powerful, and its
   importance is increasing as systems management becomes more
   automated.  Many computer forensics experts are aware of cases were
   SSH keys have been used to spread an attack from one server to
   another, and several high profile incidents in the last year have
   used SSH keys as an attack vector.

   Experience has shown that most large organizations have 3 to 100+ SSH
   keys configured granting access to each Unix/Linux server.  Some keys
   grant high-level administrative access or access to sensitive
   accounts, such as those storing database files or critical software.
   In practice it has been found that in many organizations
   approximately 10% of SSH keys grant access to root accounts or other
   privileged accounts.

   The mesh of automated access relationships is so dense in many cases
   that it is likely that an attack can spread to most servers in an
   organization after penetrating the first few servers, especially if
   other attack vectors are used to escalate privileges.

   Implementing SSH keys as an attack vector in malware is quite easy,
   requiring only a few hundred lines of code.  Once the malware has
   penetrated a server, it may use the server to further the attack and/
   or, e.g., leave a backdoor, steal, alter, or corrupt data, subvert
   encryption systems or databases, or outright destroy the server.

   The virus spread threat can be reduced by combining several
   approaches:

      Mandating forced command restrictions for as many trust
      relationships as possible.



Ylonen, et al.          Expires October 06, 2013               [Page 13]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Removing trust relationships that are no longer needed.

      Minimizing the number of trust relationships leading to root
      accounts (directly or indirectly).

      Minimizing implicit trust relationships arising from privilege
      escalation (e.g., "sudo"), single-sign-on (e.g., Kerberos), and
      host equivalence.

3.2.  Unaudited Backdoor Threat

   Many large organizations mandate that all privileged access to their
   servers take place through a privileged access management system that
   records any actions performed.  Key-based access (and other automated
   trust relationships) can be used for creating backdoors that bypasses
   such privileged access management systems.

   System administrators and production support personnel regularly gain
   access to various accounts in the course of legitimate work.  An
   administrator or production support person may add a new authorized
   key to an account with a single command (e.g., "echo ...keydata...
   >>~/.ssh/authorized_keys").  As of this writing, most organizations
   never audit authorized keys for their user accounts, and thus the
   added key may remain unnoticed for years.  Such a key can then be
   used to log into the account using any SSH client, bypassing the
   privileged access management system.  It thus provides a relatively
   permanent unaudited backdoor.

   Key-based backdoors can also circumvent password vaults and systems
   that change root (or other privileged account) passwords regularly.

   Experience has shown that many organizations have no control or
   tracking of trust relationship creation.  Any system administrator or
   production support personnel can create and install a user key pair
   as needed without any reporting, logging, or authorization.  Such
   practice undermines traditional controls for privileges access.

   The unaudited backdoor threat can be reduced by the following:

      Prevent non-root/non-Administrator users from granting automated
      access to accounts without proper approval.  For example, move all
      authorized keys files to root-owned directories that are not
      writable by normal users.

      Continuously monitor the environment to detect unauthorized trust
      relationships configured by someone having root access.





Ylonen, et al.          Expires October 06, 2013               [Page 14]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Require proper explanation and valid purpose for the existence of
      every trust relationship and remove any unused trust
      relationships.

      Use privileged access management systems that capture SSH and
      other protocols using automated access on the network level or at
      the server (transparent access auditing).  Enforce privileged
      access management for connections using automated trust
      relationships.

3.3.  Leaked Keys May Provide Access for Extended Periods

   Most security policies and regulations mandate that all passwords
   must be changed regularly, e.g., every three months.  Some security
   standards mandate that encryption keys must also be changed
   regularly.  However, very few security policies at this time make it
   explicit that authentication/authorization keys should also be
   regularly changed.  In a sense, authentication keys are even more
   critical than encryption keys, because once access to a user account
   has been gained, it is generally possible to access and modify any
   data for that user account - including reading and modifying memory
   of processes running under that user account and/or modifying any
   executable programs owned by that user account, thus subverting
   encryption systems and other critical applications.

   At the time of this writing, most organizations do not track which
   SSH keys their users, administrators, backup operators, and janitors
   may have had access to and copied over the years.  In addition, they
   never change their SSH keys.  Most environments do not use source
   restrictions on authorized keys.  Therefore, a leaked key may be used
   from any computer or network within the organization (unless limited
   by internal firewalls).

   This means that anyone who may have obtained a copy of a key (e.g.,
   by copying it from a host, accessing a backup, or having acquired
   some decommissioned equipment that was not securely wiped) may gain
   access to production servers in the organization.

   No audit or discovery process can ever guarantee finding all copies
   of identity keys, as they are small files that could be hidden
   anywhere, and there could be copies on laptops, tablets, USB sticks,
   offline, and even on paper (they are small enough to be typed in).
   Identity keys can easily be taken out from an organization on a
   single piece of paper, or by taking a photograph of a screen using a
   cellphone.

   There have also been many instances where private keys have been
   uploaded to, e.g., "github" (a repository used by many open source



Ylonen, et al.          Expires October 06, 2013               [Page 15]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   software development projects).  In January 2013, hundreds of private
   keys and passwords were found from the repository, some of which were
   being used for attacks.  Obviously, identity keys MUST NOT be
   uploaded into any public repository.

   The problems created by leaked or unchanged keys can be reduced by:

      Rotating all keys regularly to guarantee the eventual termination
      of access.

      Configuring source restrictions for authorized keys, making it
      more difficult to exploit copied identity keys.

      Using certificate-based authentication, which can provide
      revocation and expiration, but is cumbersome to manage (and still
      does not protect from some of the other threats).

      Using Kerberos authentication, which allows terminating access to
      the account; however, Kerberos authentication does not in itself
      prevent leaked keys that have not been changed from being used.

   Besides rotating keys at regular intervals to avoid their leakage and
   to limit the duration of the exploitation window should they leak,
   periodic rotation also applies to credentials used for obtaining
   actual authentication credentials; for example, it is not enough to
   periodically obtain a new Kerberos ticket - one must also regularly
   change the authentication credentials used for obtaining an initial
   ticket.  It is also not enough to issue a new certificate for the
   same private key - the private key must also be replaced by a newly
   generated private key.

3.4.  Lack of Proper Termination of Access

   Most security standards mandate proper termination of access when an
   employee leaves or changes roles.  If the user remains in possession
   of identity keys that continue to have access to the organization's
   information system, access is not being properly terminated.

   Since administrators can quite easily copy identity keys (and may
   have legitimately configured key-based access from their personal
   account to various accounts they used in their previous role), the
   only practical way to guarantee proper termination of access is to
   remove or rotate any keys that the employee may have had access to.

   At the time of this writing, most organizations do not know what each
   key is used for.  Without this knowledge, they seldom remove or
   rotate keys, because something could break if they accidentally
   remove a key that is needed for some important business process or



Ylonen, et al.          Expires October 06, 2013               [Page 16]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   omit some authorized keys corresponding to a public key being
   rotated.

   Some organizations have used manual spreadsheets for tracking key
   usage.  However, it has turned out they are usually out of date,
   inaccurate, and have not been maintained throughout the organization.
   Many organizations have no monitoring whatsoever of automated access
   or new user key setups.

   Proper termination of access can be ensured by:

      Moving all authorized keys files to root-owned directories that
      are not writeable by non-privileged users.

      Regularly rotating keys (ensures termination of access by copied
      keys latest at next key change).

      Triggering immediate key rotation for private keys on accounts
      accessible by the person whose access is being terminated.

3.5.  Use for Unintended Purpose

   Firewalls commonly have rules that permit specific communications for
   file transfer purposes.  When the file transfer is using SSH (or
   SFTP), it is important that a forced command be used on the server to
   ensure that the access permitted through the firewall cannot be used
   for other purposes, such as executing shell commands on the server.

   Another related use case is employees creating their own backdoors
   into the enterprise to circumvent corporate policies against
   uncontrolled remote access by opening an SSH connection from the
   office to their home machine with a port forwarding from the home
   machine back to the office machine.  Such backdoors may provide
   hackers an entry point into the company intranet, especially if the
   home machine is compromised and the user's password is obtained
   using, e.g., key logger malware.

   Various commercial products are available for auditing SSH
   connections at a firewall to enforce that opened ports are not used
   for unintended purposes regardless of server configuration.

   Various SSH implementations permit port forwarding even when forced
   commands are used.  Therefore, a trust relationship that is intended
   only for file transfer may actually be used to obtain a connection to
   any port at any host on the internal network (or external network,
   for hiding the source of an attack).

   The threat of unintended use can be minimized by:



Ylonen, et al.          Expires October 06, 2013               [Page 17]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Allowing SSH/SFTP through a firewall only when required and
      restricting sources and destinations to fewest systems required.

      Configuring forced commands for authorized keys used by external
      parties.

      Implementing tools to audit SSH connections at the firewall and
      monitoring use to ensure that access was not misused.

      Avoiding trust relationships that cross security boundaries or
      allow connections from low-impact systems to higher-impact
      systems.

3.6.  Accidental Data Transfers and Human Errors

   Not all risks associated with unmanaged automated access arise from
   malicious behavior.  If there is automated access from non-production
   systems to production systems, data may accidentally be copied from
   non-production systems into production systems, where the incorrect
   data may cause substantial loss of money.  Alternatively, data may be
   inadvertently copied from production systems to non-production
   systems, where the data may be exposed due to looser security
   controls.

   People are also known to make human errors when manually setting up
   new trust relationships.  For example, it is fairly easy for a
   security administrator to accidentally copy an authorized key to the
   root account on a host instead of some other account that was
   intended.  Such errors can go undetected for years.

   Some key setups involve thousands of hosts.  It is easy to miss one
   or more hosts when copying an authorized key to so many hosts
   manually.  Debugging such errors can be very time consuming.  Also,
   when manually fixing such problems, security administrators are
   likely to just copy the missing keys to the proper accounts, without,
   for example, checking whether they have accidentally been copied to
   the root account.

   The threat of accidents and human errors can be minimized by:

      Automating key provisioning to implement the authorized keys
      exactly as they were requested and approved.

      Configuring source and command restrictions for authorized keys.

      Enforcing policies for preventing trust relationships between
      systems that cross security zone boundaries.




Ylonen, et al.          Expires October 06, 2013               [Page 18]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


3.7.  Problem Under Radar

   The SSH key management problem has been recognized in various circles
   for some time.  The scope of the problem, and its relation to
   automated access overall, however, has not been widely understood.

   The problems have remained under the radar because they are deeply
   technical and obscure, within the domain of system administrators.
   Each system administrator typically only sees a small corner of the
   IT environment and does not have a full picture.  Although
   administrators and their managers may recognize that there is a
   problem, they simply have not had time to analyze the scope or
   possible implications of the problem.  There have also been no
   guidelines or training materials on how to address it.

   Most IT auditors do not have SSH key management or automated access
   more generally on their checklists or audit programs, yet it is
   central to identity and access governance given the prevalence of
   automated access entry points to systems.

   SSH keys, or control of credentials for automated access more
   generally, has not been sufficiently highlighted in security control
   frameworks and auditing guidelines for FFIEC, SOX, PCI, FISMA, HIPAA,
   NERC, or COBIT.  Even many CISOs are only vaguely aware of the
   problem, and many CIOs and IT risk management professionals have
   never heard of it.

   Training, books, and systems in the identity and access management
   space have largely only dealt with actual human users and control of
   interactive access by people.  Automated access by machines has been
   largely ignored, despite many organizations now having many times
   more credentials for automated access to their systems than they have
   interactive accounts.

   Despite the risk, the problem will likely not be addressed until IT
   security auditors, IT operations managers, security architects,
   CISOs, and IT risk management professionals understand the issue.  It
   must be addressed in security regulations, guidelines, standards, and
   internal security policies.  Education and training will also be
   needed to ensure that the SSH key management problem and remediation
   actions are understood.  It must be evaluated during audits to ensure
   that action is taken.









Ylonen, et al.          Expires October 06, 2013               [Page 19]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


4.  Assessing the SSH Key Management Situation and Risks

   Addressing threats related to automated access and SSH keys begins
   with understanding the extent to which automated access and SSH keys
   are used in an organization, understanding how they are managed, and
   identifying areas likely to require further attention.

   Risks associated with SSH key management are generally relevant for
   organizations where at least one of the following is true (the list
   is not exhaustive, and other automated access technologies affect
   other organizations):

      significant number of Unix or Linux systems;

      significant use of SSH or SFTP on Windows or Mainframe;

      virtualization or cloud services that are internally managed using
      SSH (possibly inside automated scripts/tools/frameworks);

      web server farms that are managed over SSH;

      network devices (e.g., routers, switches, xDSL models, firewalls)
      configured and managed using SSH and/or automated management
      systems;

      significant number of automated file transfers using SFTP;

      password management or privileged access management tools using
      SSH to connect to end servers; or

      systems management tools using SSH to communicate with managed
      systems.

   Results of the scoping phase help estimate risk exposure and the
   probability of non-compliance with mandatory regulations.  This
   information also helps auditors and decision-makers determine whether
   a more detailed discovery and remediation project is warranted.

   Certain types of relatively easily obtainable information are useful
   in understanding the scope of the problem in an organization.  This
   information may easily be obtained as part of an audit or regular
   review.

   Some preliminary indicators about the level of risk can be obtained
   by reviewing the sshd_config file for a sample of SSH servers.  The
   AuthorizedKeyFile parameter indicates the location of files that
   store user's authorized key files.  If the AuthorizedKeyFile is
   located within the user's home directory (which is the default), then



Ylonen, et al.          Expires October 06, 2013               [Page 20]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   it is likely that a significant problem exists because users are able
   to provision new trust relationships.  On the other hand, if the
   AuthorizedKeyFile is defined within the /etc directory, for example,
   then the risk of inappropriate trust relationships is significantly
   lessened.  Another significant configuration parameter is
   PermitRootLogin.  A value of "yes" or "without-password" indicates
   that SSH keys can be used for root access, which significantly
   increases the potential impact of poorly managed keys.  However, if
   this parameter is set to "no" or "forced-commands-only", then the
   potential impact is substantially lessened since interactive root
   access is disallowed.

   Examining authorized keys provides a meaningful indication of the
   level of risk.  The following metrics generally give insight into
   whether an organization is affected by the issue:

      total number of authorized keys in the environment;

      total number of authorized keys in the environment that grant
      access to a root account (any account with user id 0);

      total number of authorized keys in the environment that grant
      access to a system account or service account;

      total number of authorized keys without a command restriction; and

      total number of non-root service accounts or system accounts that
      have access to add new authorized keys at will.

   There are also a number of scoping questions that can give insight
   into the severity of the problem.  These questions are well-suited
   for a questionnaire or interview of knowledgeable personnel.  An
   affirmative answer indicates that SSH keys are being managed; a
   ``no'' indicates that risk exists.  The questions are provided below:

      Does installing a new authorized key require approval from a
      system resource owner or authorized manager 1) for keys granting
      root access, 2) for keys granting access to non-root service
      accounts or system accounts, or 3) for keys granting access to
      interactive user accounts?

      Are such approvals enforced through the provisioning process?

      Are non-root users technically prevented from installing new
      authorized keys, e.g., by moving the authorized keys files to
      root-owned locations?





Ylonen, et al.          Expires October 06, 2013               [Page 21]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Has this configuration been verified to be the case 1) across all
      high-risk systems and 2) all moderate-impact systems?

      Is a continuous monitoring process in place for detecting
      authorized keys that are added outside of the provisioning process
      and without proper approvals 1) for root accounts, 2) for service
      and system accounts, and 3) for interactive user accounts?

      Is a policy in place for rotating SSH user keys?  Are monitoring
      procedures in place to verify that all user keys have actually
      been rotated in accordance with the policy (and the private keys
      actually changed)?

      For all authorized keys for the root account (and critical system
      and service accounts), can the organization easily identify who is
      using the keys to connect?

      Has the reason of existence for every authorized key, including
      the application or business process it relates to, been documented
      1) for high-impact systems, 2) moderate-impact systems, and 3)
      low-impact systems?

      Are SSH keys systematically removed when they are no longer
      needed?

      Do all authorized keys used for external SFTP/SCP file transfers
      with other organizations have a command restriction?

      Are command restrictions enforced for trust relationships leading
      to moderate-impact and high-impact systems?

   Preliminary scoping information can be obtained relatively easily and
   scoping questions can be answered without having to install new
   software on servers.  However, the answers are only approximate.
   Experience has shown that many organizations do not know clear or
   definitive answers to the questions, and sometimes management
   perceptions do not match reality.  Therefore the answers are best
   obtained and analyzed as part of a regular audit that actually
   verifies the answers, at least by representative sampling.

   Another interesting diagnostic exercise to gauge the level of risk is
   to obtain listings from a few servers of the public keys (or
   signatures) from the authorized keys file of root and other sensitive
   accounts (such a system or service accounts).  If the organization
   can readily identify who is using those keys (or could use the keys)
   to connect and why, then it is likely that the organization is
   effectively managing SSH users key for access control.  If the
   organization is unable to identify who can use keys to obtain



Ylonen, et al.          Expires October 06, 2013               [Page 22]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   sensitive access to systems, then the access control problem and risk
   is self-evident.

   Freely available scripts and tools for doing a proper scoping
   analysis are available at http://www.ssh.com/auditing [1].  The
   scripts and tools will be useful for internal security professionals,
   system administrators, and auditors working with customers.

5.  Key Remediation Solution Planning and Deployment

   Once it has been determined that further analysis of automated access
   and/or SSH keys in an organization is warranted, the organization
   typically engages in a multi-step process consisting of additional
   discovery and remediation of existing trust relationships,
   establishment of appropriate policies, and continuous monitoring to
   ensure that automated access is only enabled in accordance with
   policy.

   A typical key remediation process consists of:

      discovering all existing trust relationships based on SSH keys
      (and other trust relationships, if applicable);

      moving authorized keys files to protected locations to prevent
      non-root users from adding new authorized keys;

      monitoring use of trust relationships and authorized keys in the
      existing environment;

      removing trust relationships that are no longer in use;

      associating each trust relationship with an application or some
      other valid purpose;

      implementing an approval process for setting up new trust
      relationships;

      rotating existing SSH user keys;

      configuring forced command restrictions on authorized keys; and

      configuring IP address restrictions on authorized keys.

   While it is possible to perform all the remediation steps manually,
   in a larger environment the use of software tools to assist in the
   process can save a huge amount of work.  Parts of the process can be
   fairly labor-intensive, for example, associating each trust
   relationship with an application or valid purpose may require a



Ylonen, et al.          Expires October 06, 2013               [Page 23]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   substantial amount of manual work, and removing of unused trust
   relationships needs to be done with care to avoid any problems with
   critical business applications.  (See Section 8 for more
   information.)

   Automating management of SSH keys and other trust relationships can
   also bring substantial cost savings.  Many organizations spend a
   substantial amount of administrator time setting up and maintaining
   trust relationships, and the cost of such manual key management can
   often be eliminated by automating the process.  Ideally, new trust
   relationships are approved in the organization's normal security
   entitlement approval system and automatically implemented throughout
   the IT environment by software for managing SSH authorized keys and/
   or other trust relationships.  Automation can also reduce human
   errors and radically reduce the number of administrators requiring
   root access on servers.  (See Section 8.1.)

   In some environments it may be desirable to prohibit public key
   authentication for interactive logins to ordinary user accounts.
   This can help enforce ordinary interactive logins to go through a
   privileged access management system (unless some administrators have
   copied private keys, which is generally possible).

5.1.  Discovering SSH Keys and Trust Relationships

   The purpose of the discovery phase is to obtain reliable and
   reasonably complete information about configured SSH keys and trust
   relationships throughout an IT environment.  Discovery should ideally
   include all Unix/Linux systems, Windows systems (at least those
   running SSH servers, SSH clients, or file transfer solutions running
   SFTP), Mac servers, workstations, laptops, mainframes, and other
   systems using the SSH protocol, including file transfer solutions
   using SFTP, virtualization platforms, and privileged access
   management gateways.

   Since it is not possible to know what trust relationships exist in an
   IT environment without scanning all systems for SSH authorized keys
   and identity keys, all organizations SHOULD perform initial discovery
   of SSH user keys on all systems that use the SSH protocol.

   Although some organizations may want to focus only on high-impact
   systems, a limited scope discovery process provides only limited
   visibility into the security risk of the current environment.  It is
   important to identify which low-impact systems can access high-impact
   systems, and this can only be known by scanning low-impact systems
   too.  An identity key intended to allow access between two high-
   impact systems could be copied onto a low-impact system.  Unless
   source restrictions are defined for the authorized key, the identity



Ylonen, et al.          Expires October 06, 2013               [Page 24]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   key can be used from the low-impact system to access the high-impact
   system.  Therefore, the scope of the discovery process SHOULD include
   all systems in the network, including even low-impact systems.

   Ideally, routers, BIOS management ports, and other specialized
   computing devices should also be included, but at the time of this
   writing, software is not yet available for full SSH key discovery for
   these devices.  This is expected to change in the future.  It is also
   be possible to audit them manually.

   The following MUST be determined during discovery:

      Configured authorized keys for all user accounts on all servers of
      interest (accounts may be local, in LDAP, in Active Directory, in
      NIS, or any combination)

      Configured identity keys on all user accounts on all servers and
      clients (workstations, laptops, etc) of interest.  It should be
      understood, however, that one can never be certain that all
      identity keys have been found, because some could be copied into
      non-standard directories, stored offline, or even printed on
      paper.  Thus not finding an identity key on a particular system
      does not guarantee that the key will not eventually be used from
      that system later.  On a broader scale, even if the discovery
      process fails to find a particular identity key anywhere on the
      network, this does not necessarily mean that the key cannot be
      used later.

      Configured restrictions for each authorized key, such as command
      restrictions and source restrictions

      Established trust relationships (source host, source account,
      destination host, destination account, and restrictions)

   The following SHOULD be determined during discovery (these may become
   MUST in the future):

      Kerberos-based trust relationships for automated access,
      particularly access using keytab files, cached tickets, or service
      processes holding tickets.

      Implicit trust relationships arising from Kerberos single sign-on.

      Implicit trust relationships arising from sharing the same home
      directory across multiple accounts.  Many organizations share the
      same home directory for multiple accounts.  Adding an authorized
      key for any of the accounts implies adding it to all of the
      accounts.  Thus, any accounts that can write to the shared home



Ylonen, et al.          Expires October 06, 2013               [Page 25]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      directory effectively have access to all accounts sharing the home
      directory.

      Trust relationships configured using host-based authentication
      (".shosts", ".rhosts", "hosts.equiv", or "shosts.equiv").

   The following MAY be determined during discovery (some of these may
   become SHOULD or MUST in the future):

      Trust relationships configured using password authentication
      (whether hard-coded in scripts or in password vaults).  (In
      practice it may be difficult to do this reliably.  However, it may
      be possible to, e.g., recognize certain syntactic patterns and
      commands from scripts as using hard-coded password
      authentication.)

      Implicit trust relationships configured using "sudo" or some other
      privilege escalation tool.

      Implicit trust relationships arising from user accounts that have
      NFS-mounted home directories.  NFS is usually not configured to
      provide security against network-level attacks, and an attacker
      who gains access to a network segment may be able to read and
      modify the NFS traffic of any host on the network and impersonate
      any other host or user on the network (including reading and
      modifying any file on NFS file systems).  When home directories
      are stored in NFS, a sophisticated attacker with root-level access
      to any device on the network (e.g., any unconfigured smart switch
      where the attacker can replace the firmware) may add new
      authorized keys to any home directory in an NFS file system.

      Implicit trust relationships arising from user accounts that have
      home directories on a Windows share.  Windows file sharing (CIFS,
      Common Internet File System) may suffer from same issues as NFS,
      and thus the same considerations may also apply to it.

   It is important to understand that even though the discovery process
   finds keys, and in the short term most trust relationships are
   configured using SSH keys, the primary concern is not with the keys
   themselves or the underlying cryptography.  Rather, the primary
   purpose of discovery is to determine who (or what) can access what
   and how such access is restricted.  In terms of cryptography, all SSH
   keys created with default settings since version 1.0 use the
   equivalent of at least 1024 bit RSA key, which is still relatively
   safe, especially since servers never disclose authorized keys
   (cryptographic attacks on the key would first require access to an
   authorized keys file, which usually already means access to the
   host).  Thus verifying key sizes or algorithms is not critical



Ylonen, et al.          Expires October 06, 2013               [Page 26]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   (though may be required by policy); however, knowing who (or what)
   can access what is critical and addresses a real security concern.
   SSH user keys grant access and are fundamentally authentication
   credentials, rather than encryption keys.  The whole issue is
   fundamentally not an encryption key problem, but an identity and
   access management problem!

   Doing discovery properly is complicated.  At least the following
   aspects need to be properly considered when planning discovery:

      SSH user keys cannot be discovered by a network scan because the
      SSH protocol was designed not to reveal authorized keys.  It is
      possible to query whether an already known key is acceptable for a
      particular user, but an SSH server will not reveal an authorized
      key that is otherwise unknown.  This means that the discovery
      process will need to connect to each host and access the
      authorized keys through the file system.  (Host key discovery, on
      the other hand, is possible over a network, but host keys are
      beyond the scope of this document.)

      Different SSH versions are commonly deployed.  Many large
      organizations have some combination of OpenSSH, Tectia SSH,
      SunSSH, Reflection for Secure IT, and various other products.  Not
      all SSH implementations use the same key formats, store SSH keys
      in the same locations, or use the same key fingerprint format.
      Furthermore, OpenSSH comes in many flavors and patch set
      combinations, and some vendors pack a version of OpenSSH with
      another product - sometimes without providing a proper way of
      identifying the particular version.  The discovery process (and
      tools) should be able to properly analyze keys and trust
      relationships for any SSH version that is deployed in the
      environment.

      Many organizations use the "root_squash" option for NFS exports,
      which converts file system accesses by root to accesses by an
      unprivileged account "nobody".  A side effect of this is that a
      key discovery process running as root may not be able to read SSH
      keys in NFS home directories.

      Systems using SELinux may not allow reading SSH authorized key
      files or identity key files by ordinary processes.  Reading such
      files may require special authorization or the use of special
      programs, such as "ssh-keycat".

      In a large environment, some servers are down for maintenance at
      any time, and in a geographically dispersed organization some
      networks may not be reachable at the time the discovery is
      initially run.  Thus, discovery cannot be assumed to succeed on



Ylonen, et al.          Expires October 06, 2013               [Page 27]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      all servers the first time.  This problem is compounded for
      discovery of SSH clients since laptops may be disconnected from
      the network and therefore be unreachable for scanning.

5.2.  Moving Authorized Keys to Protected Locations

   Moving authorized keys to protected locations, or locking down
   authorized keys, MUST be performed on all moderate-impact and high-
   impact information systems (including systems having automated access
   to such systems).  It MAY be performed on low-impact information
   systems.

   Moving authorized keys to a protected location may be performed,
   e.g., by copying authorized keys for each user to a root-owned
   directory, and modifying the system-wide SSH server configuration
   file to specify the authorized keys file path (typically using a
   pattern that refers to a user name).

   Failure to move authorized keys to protected locations allows system
   administrators and other users with legitimate access to create new
   trust relationships as unaudited backdoors and makes ensuring
   termination of access very difficult.  Locking down authorized keys
   files helps to enforce the requirement that new trust relationships
   be properly approved.  (See Section 5.3 for discussion of using an
   approval process for setting up new trust relationships.)

   It is important to lock down authorized keys files early in the
   remediation process to create a stable environment for discovery.
   Otherwise, inappropriate authorized keys that continue to be added
   may not identified during the discovery process.

   Similarly, authorized keys MUST be moved away from home directories
   susceptible to active network-level attacks (e.g., unencrypted NFS
   and CIFS home directories - in practice this includes most NFS home
   directories today) on all moderate-impact and high-impact systems
   (including systems having unrestricted automated access to such
   systems).  It is RECOMMENDED that the same be performed on low-impact
   information systems.

   Failure to move authorized keys away from NFS and CIFS home
   directories may allow a network-level attacker (whether human or
   automated) to add new authorized keys to any account that accepts
   authorized keys from such a directory, permitting unrestricted access
   to such accounts.  Such attacks are known to have been performed by
   some penetration testers and are certainly within the capabilities of
   Advanced Persistent Threat (APT) groups.





Ylonen, et al.          Expires October 06, 2013               [Page 28]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Furthermore, identity key files SHOULD be moved away from home
   directories susceptible to network-level attacks (e.g., unencrypted
   NFS and CIFS home directories) on all moderate-impact and high-impact
   systems.  Otherwise, an attacker who gains privileged access one one
   host on the network may be able to read identity keys from user's
   home directories and use them for attack (this technique is known to
   have been used by attackers).  It is RECOMMENDED that the same be
   performed on low-impact systems.

   Failure to move identity keys away from NFS and CIFS home directories
   may allow a network-level attacker (whether human or automated) to
   obtain copies of identity keys for later use or for immediately
   furthering the attack to other systems.  It substantially increases
   the risks associated with leaked keys and substantially expands the
   group of people who may be able to obtain copies of identity keys.
   Some penetation testers are known to use this technique for attacks.

5.3.  Monitoring Use of Trust Relationships and Keys

   After the initial discovery phase, the environment SHOULD be
   monitored for some time (preferably several months) to collect data
   on how authorized keys are actually used.  While the process can be
   performed manually in a small environment, use of scripts or
   commercial tools is highly recommended in larger environments.
   Software tools can gather and correlate log data from many hosts to
   determine the following types of information: which keys are
   currently being used, which source hosts they are used from, which
   keys are external keys, and what commands they are used with.  This
   information about the use of trust relationships will help the
   organization in later phases of remediation, such as deciding which
   authorized keys will be removed for non-use (see Section 5.4) and
   which keys can be easily configured with command and source
   restrictions (see Section 5.8 and Section 5.9).  In addition,
   information gathered during monitoring will help the organization
   understand automated access patterns in the existing environment and
   further evaluate the concrete risks.

   Monitoring use of trust relationships may involve configuring SSH
   servers and clients to use a higher level of logging and collecting
   and analyzing log data.  To determine which authorized keys are still
   being used, for example, an organization can configure SSH servers
   with a log level that causes the fingerprints of keys used for public
   key authentication to be logged, collect such log data for an
   extended period (several weeks to an year), and analyze the data to
   determine which authorized keys were actually used during the log
   collection period.





Ylonen, et al.          Expires October 06, 2013               [Page 29]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   On SSH clients, identity keys that have not been accessed for a long
   time (according to file system's file access timestamps) are also
   good candidates for removal.  However, many programs and commands may
   access identity key files, and a recent access time does not
   necessarily mean that the key was actually recently used for
   authentication.  Furthermore, the identity key file timestamp
   provides no information regarding the destination host for the last
   connection.  An identity key may still be in use for some destination
   host where it is authorized but not for another.

   It should be noted that orphaned keys (authorized keys without a
   corresponding identity key) may be either unused or external keys.
   Thus they SHOULD NOT be removed without monitoring, as if they are
   external keys, trust relationships with hosts outside the managed
   environment could be inadvertently broken.

   From a project management perspective, the monitoring period can well
   be used for assigning impact levels to systems, defining internal
   boundaries, and defining host groupings.  In practice in large
   environments, different parts of the IT environment may be at
   different stages of remediation during the project.  However, some of
   the remediation steps require a reasonably complete picture from
   longer-term monitoring before they can be safely performed.

   Identifying trust relationships crossing certain boundaries, such as
   access from test and development systems into high-impact production
   systems, is of high interest to auditors and security managers.
   Detecting and controlling such unwanted access is an important audit
   objective.  This information generally becomes available with
   reasonable certainty during the monitoring phase, after internal
   boundaries have been configured and impact levels for information
   systems determined.

   Information collected during the monitoring stage will be helpful in
   later stages of a remediation project.  The information gathering
   takes some time to get a reliable picture.  It is RECOMMENDEED that a
   sufficient period of time be given for monitoring use of keys: at
   least 3-6 months or even a year.  The remediation project should not
   be rushed, as it increases risk of having incomplete information
   about existing trust relationships or external keys, which in turn
   increases the risk that remediation activities may disrupt
   operations.

   Failure to perform the monitoring step properly increases risk of
   disruption when unused keys are removed (see Section 5.4), and may
   even make removing unused keys impossible (one cannot remove unused
   keys without knowing with a high degree of certainty which keys are
   unused).  Relying solely on the knowledge of application teams and



Ylonen, et al.          Expires October 06, 2013               [Page 30]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   managers to identify unused keys is generally insufficient due to the
   large number of legacy trust relationships, personnel changes, and
   poor documentation of trust relationships.

   Failure to perform the monitoring step properly also risks missing
   some external trust relationships and external keys.  This may cause
   key rotation (see Section 5.7) to break external connections with
   systems outside the managed environment, such as data transfers with
   suppliers, contractors, distributors, or regional offices.

5.4.  Removing Trust Relationships That Are No Longer Used or Otherwise
      Inappropriate

   Various security standards and prudent information security require
   that access to information systems must be properly terminated when
   it is no longer needed.  If trust relationships for automated access
   are left enabled on systems when no longer needed, they accumulate.
   Real-world experience has shown they sometimes accumulate at the rate
   of dozens of incoming trust relationships per year per system, even
   in security-sensitive environments.

   Therefore, all organizations MUST remove trust relationships leading
   to moderate-impact or high-impact information systems (including low-
   impact systems having automated access to such systems) that are no
   longer needed as part of the initial remediation process.  Unused
   trust relations leading to low-impact information systems SHOULD be
   removed.

   Unused keys SHOULD NOT be removed until it is known with reasonable
   certainty which keys are really unused.  This is usually accomplished
   by monitoring key usage over a period of time (see Section 5.3).  It
   is further RECOMMENDED that unused trust relationships be reviewed by
   respective application owners to reduce the possibility of disruption
   from removal of a trust relationship that is actually needed.  It is
   important to test infrequently used functionality, such as disaster
   recovery systems, reasonably soon after removing unused trust
   relationships.

   It is also likely that the remediation process will identify many
   trust relationships that are still being used but that have no
   legitimate business purpose (see Section 5.5), that cross configured
   boundaries in inappropriate ways, or that lead to accounts (e.g.,
   root) that should not be accessible.  Such inappropriate trust
   relationships should also be removed (and alternate steps taken to
   implement any functionality they support in a more appropriate way,
   if required).  Some of such trust relationships may have been created
   by attackers and may warrant further forensics investigation, such as
   identifying when they were created and by whom.



Ylonen, et al.          Expires October 06, 2013               [Page 31]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Failure to remove authorized keys for unused trust relationships
   increases the risk that key-based attacks for unauthorized access may
   succeed and spread throughout the network, allows previously created
   unaudited backdoors (using keys that are not regularly used) to
   remain in existence, and allows leaked keys (that are not regularly
   used) to remain usable indefinitely (if not rotated).

5.5.  Associating Trust Relationships with Application and/or Purpose

   Because authorized keys provide access to systems, their existence
   should be understood, justified, and controlled just like any other
   form of access control.  After trust relationships that are not used
   have been removed, it is important to analyze the remaining active
   trust relationships to distinguish those authorized keys that support
   a valid application or business purpose and those keys that do not.
   Just because a trust relationship is being used does not actually
   guarantee that it is needed or legitimate.  It may be used, e.g., by
   an attacker, by a user who created an inappropriate authorized key as
   a backdoor for access, or by a stale cron job relating to a
   decommissioned application.  Determining the purpose of trust
   relationships is important for detecting such illegitimate trust
   relationships so that they can removed (see Section 5.4).

   After having removed unused authorized keys, the existence of every
   remaining incoming trust relationship MUST be justified for moderate-
   impact and high-impact information systems (including low-impact
   systems having access to such systems).

   This is an area where the justification can be more lax on low-impact
   systems.  However, many low-impact systems, such as those used for
   internal software development or packaging, may generate binaries or
   distributions that later get installed on production systems.  An
   attacker could use such systems to gain access to production servers,
   especially in an Advanced Persistent Threat (APT) scenario.  It is
   thus RECOMMENDED that even access to low-impact systems be prudently
   justified, or alternatively systems with data/code paths to
   production be treated as moderate-impact or high-impact systems.

   One practical approach for low-impact systems may be to review
   discovered incoming trust relationships in bulk (perhaps for a group
   of hosts belonging to the same business process), and label them as
   legacy trust relationships relating to the associated business
   process.  While not ideal, such an approach may make a reasonable
   compromise between cost and security in many environments.

   It is RECOMMENDED that each trust relationship be associated with the
   business process and/or application that it supports, and that the
   application/business purpose be documented for future reference.



Ylonen, et al.          Expires October 06, 2013               [Page 32]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   This will help with removing trust relationships when applications
   are replaced and business processes re-engineered, and serves to
   assign responsibility for each trust relationship to somebody (the
   business process or application owner).  Assigning trust
   relationships to applications or business processes effectively
   assigns ownership for the trust relationships (and related authorized
   keys) to the application or business process owner (or group).  This
   owner MAY be permitted to approve new trust relationships leading to
   the same account on the same host, MAY schedule rotation of keys, and
   MAY be asked to periodically validate the existence of each trust
   relationship relating to the application or business process.

   Failure to associate trust relationships with a purpose, business
   process, or application means that there remains access to
   information systems without reason or justification.  Illegitimate
   backdoors may remain unnoticed and unnecessary trust relationships
   may remain in place that can be used by attackers, especially if keys
   are leaked (and not rotated).

5.6.  Implementing Approval Process for Setting Up New Trust
      Relationships

   Real-world experience has shown that many enterprises do not have a
   well-defined process for approving new trust relationships for
   automated access, and almost no enterprise today systematically
   enforces or audits approvals for automated access.

   Organizations MUST implement an approval process for ensuring the
   validity of new trust relationships granting access to moderate-
   impact and high-impact information systems.  It is further
   RECOMMENDED that organizations implement an approval process for
   trust relationships granting access into low-impact systems.  This
   reduces risk of low-impact systems being used for staging attacks
   into high-impact systems.  See also Section 8.1 for ideas on how
   automated setup of approved trust relationships can reduce costs.

   New trust relationships SHOULD NOT be approved without a proper
   justification and association with a business process, application,
   or other valid purpose.  In addition, the approval for new trust
   relationships MUST specify any command or source restrictions that
   should be implemented to limit security exposure.  (See Section 5.8
   and Section 5.9)

   The approval process SHOULD carefully review whether trust
   relationships violate internal boundaries, such as allowing access
   from test or development systems into production or allowing access
   from low-impact systems into high-impact systems.  Documented
   justification for crossing boundaries and secondary approval by



Ylonen, et al.          Expires October 06, 2013               [Page 33]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   higher-level management SHOULD be required for such trust
   relationships.  Organizations MAY also use the approval process to
   enforce other internal boundaries, such as those between business
   units or functions, or "Chinese walls" between, e.g., retail banking
   and investment banking.

   Special approvals SHOULD also be required for trust relationships
   leading to root accounts or other highly privileged accounts on
   moderate-impact and high-impact systems.

   The approval for each such trust relationship MUST be documented and
   MUST be retained for later audit.  Approvals MUST be organized so
   that it is possible find the approval for each new authorized key.

   Required approvals for new authorized keys MUST be enforced so that
   users cannot bypass the approval process.  Enforcement typically
   involves both continuous monitoring and securing authorized keys
   files:

      Continuous monitoring (as discussed in Section 6) is needed to
      detect trust relationships that were implemented without approval.
      Existing trust relationships must be regularly audited against
      approved trust relationships.  This requires periodically re-
      performing discovery to find all existing trust relationships so
      that they can be compared against a database of approved trust
      relationships.  If software tools are used to perform this
      auditing, enforcement may be performed in real time or very
      frequently, e.g., once an hour or once per day.

      Another way to help enforcement is to move all authorized keys to
      protected locations (as discussed in Section 5.2) and tightly
      control access to root accounts using a privileged access
      management system (preferably one that also logs key-based access
      to root accounts for accountability).  However, regular audits
      should still be performed, e.g., annually, to catch any trust
      relationships that may have been missed in the normal process.

   Although the focus of this section has been on approving new trust
   relationships, existing legacy trust relationships should also be
   approved, or at least associated with a business process,
   application, or proper purpose.  This was discussed in Section 5.5.

   Failure to implement or enforce approvals means it is impossible to
   ensure that new trust relationships are valid and appropriately
   restricted.  Not knowing what each trust relationship is used for
   makes it very difficult to know which trust relationships can be
   removed without substantial risk of disruption to business processes.
   Lack of up-to-date documentation of trust relationships, including



Ylonen, et al.          Expires October 06, 2013               [Page 34]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   lack of knowledge of which application or business process they
   relate to, is one of the main causes of the current poor situation
   with SSH user keys in many organization.  Failure to implement and
   enforce approvals for trust relationships also implies that system
   administrators can continue to create unaudited backdoors to
   production systems, bypassing most existing privileged access
   management systems.

5.7.  Rotating Existing SSH User Keys

   SSH user keys are authentication credentials, like passwords.  They
   should be rotated (i.e., changed) regularly.

   Rotating an SSH user key for a trust relationship means generating a
   new identity key (key pair), storing (and configuring, if applicable)
   the identity key for the source account of the trust relationship,
   configuring the corresponding public key as an authorized key for the
   destination account (with the same restrictions as the old key for
   the trust relationship), and finally removing the old authorized key
   from the destination account and the old identity key from the source
   account.  If the same identity key can access more than one
   destination account (i.e., is used for more than one trust
   relationship), then it the authorized key must be copied to (and the
   old key removed from) all such destination accounts.

   Rotating external keys (i.e., keys used with hosts outside the
   managed environment) require special care and coordination between
   the organizations responsible for the respective hosts.  The basic
   principle is that the new key should be added as an authorized key on
   all destination hosts where the old identity key is used before the
   old identity key is removed.

   Authentication credentials for all trust relationships leading to
   moderate-impact and high-impact systems MUST be rotated every 12
   months, and it is RECOMMENDED that trust relationships leading to
   low-impact systems be rotated every 12 months.  It is recommended
   that all keys be rotated as part of a remediation process to ensure
   that any previously leaked keys cease to be usable.

   If two or more users have had access to a shared account that has
   access to an identity key, the identity key and any trust
   relationships using it and leading to moderate-impact or high-impact
   systems MUST be rotated during the remediation process and thereafter
   every three months.

   If an employee leaves or changes roles, immediate rotation for all
   identity keys the employee is known to have accesss to and leading to
   moderate-impact or high-impact systems SHOULD be triggered.



Ylonen, et al.          Expires October 06, 2013               [Page 35]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   If a security breach is suspected, all identity keys stored on
   affected servers SHOULD be immediately rotated.

   If certificates are used for access, such certificates MUST be
   renewed (with new private keys) annually if they can be used for
   accessing moderate-impact or high-impact systems.  If Kerberos is
   used for configuring trust relationships, then the Kerberos
   credentials used for authentication MUST be rotated annually if they
   can be used for accessing moderate-impact or high-impact systems.

   Failure to rotate keys allows leaked keys to continue working
   forever.

   Failure to rotate keys in response to an employee leaving or changing
   roles means that there is no proper termination of access.  Many
   industries must comply with mandatory regulations that require proper
   termination of access.

   Failure to rotate keys in response to a suspected breach means that
   keys copied by the attacker may be used to attack the systems again,
   and there can be no guarantee that the system has been properly
   cleaned up after the attack.

5.8.  Configuring Command Restrictions on Authorized Keys

   Command restrictions limit what can be done with a trust relationship
   on the destination host.  Typically, a command restriction (also
   called "forced command") specifies the only command that can be
   executed on the server using that key.  If any other command is
   attempted, the configured command will be executed instead or the
   attempt is rejected.

   A command restriction may further limit directories that can be used
   for file transfers (if supported by the SSH implementation) and
   whether writing files is allowed.

   On some implementations, it may be necessary to prevent pseudo-tty
   allocation for command restrictions to be effective.

   It is usually desirable to prevent TCP/IP forwarding for all
   authorized keys.  Otherwise such keys could be used to mask the
   source of attacks by redirecting them using port forwarding.

   All non-interactive trust relationships leading to moderate-impact or
   high-impact information systems MUST be configured with a command
   restriction, unless an exemption has been approved as specified in
   the organization's security process and based on a valid reason for
   not having a forced command restriction (the relatively small effort



Ylonen, et al.          Expires October 06, 2013               [Page 36]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   of configuring the command restriction not being a valid reason).
   The specific command MUST be part of the approval, and a new approval
   MUST be required if the command is later changed.

   Trust relationships that are used for interactive access SHOULD NOT
   have a command restriction (command restrictions that permit running
   a shell and then arbitrary commands SHOULD NOT be used, because they
   may be mistaken as real command restriction; if they are detected in
   an audit, they SHOULD be flagged).

   Regardless of impact level of the destination system, all trust
   relationships intended for use with the SFTP protocol by external
   parties or by lower-impact information systems MUST have a command
   restriction that limits the use of the trust relationship to SFTP and
   prevents interactive use.

   Failure to configure command restrictions for keys increases virus
   spread risk and can be used for other attacks.  It also increases
   risk from leaked keys.

   Failure to configure command restrictions for trust relationships
   used with external parties may allow a virus or attack to enter the
   organization.

5.9.  Configuring IP Address Restrictions on Authorized Keys

   Source restrictions (also called "from" option in authorized keys
   files) specify from which IP addresses an authorized key can be used.

   Trust relationships permitting interactive access to moderate-impact
   and high-impact systems SHOULD specify a source restriction to
   hardened jump servers (privileged access management systems) or a
   transparent access auditing solution SHOULD be used to ensure such
   access is properly controlled and audited.  If any such trust
   relationships have been approved, they MUST be listed in an annual
   audit report and their existence rejustified annually.

   Source restrictions SHOULD be used for all trust relationships
   leading to high-impact systems.  Otherwise, the use of source
   restrictions is OPTIONAL.  They are laborious to configure manually
   and make, e.g., IP renumbering and IPv6 transition painful.  It is
   also easy to make mistakes where, e.g., a secondary server for some
   critical service is not permitted by a source restriction, which
   could increase risk of outages under unusual operating conditions.
   On the other hand, they can significantly reduce the exploitation
   potential of leaked keys.





Ylonen, et al.          Expires October 06, 2013               [Page 37]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Failure to configure source restrictions has only mild security
   impact if other recommendations are followed.  It is in part
   compensated by regular key rotation that also reduces the potential
   for exploitation of leaked keys, and thus a reasonable balance may be
   to not implement source restrictions for most trust relationships.
   However, source restrictions can completely prevent the exploitation
   of leaked keys (without sophisticated active network-level IP
   spoofing attacks), and thus is warranted for high-impact systems.

6.  Continuous Monitoring and Management of SSH Keys and Automated
    Access

   The remediation process (as discussed in Section 5) addresses both
   the one-time analysis and clean-up of existing legacy SSH trust
   relationships and the implementation of an ongoing approval process
   for validating, documenting, and restricting new trust relationships
   that are added to the environment.  Following the approval process
   (discussed in Section 5.6) for all new authorized keys added to the
   environment serves as a preventive control.  Continuous monitoring of
   trust relationships is needed to provide ongoing detection of non-
   compliance, including instances where the approval process was too
   lenient or was bypassed altogether.  Continuous monitoring is also
   important for identifying trust relationships that violate policy,
   that can be removed because they have become unused or otherwise not
   needed, or that require keys to rotated.

6.1.  Continuous Monitoring of Changes to Trust Relationships

   Proper management of automated access requires continuous monitoring
   of the IT environment because system administrators operating as root
   may add new trust relationships for any user account.  Continuous
   monitoring is also prudent for detecting keys that are no longer
   used, identifying external keys, and identifying changes in the
   patterns of usage of automated access.

   The main rationale for the continuous monitoring of the environment
   and annual audits and requiring reporting and revalidation of certain
   aspects of automated access annually is to enforce proper policy
   (policies usually do not get implemented if their implementation is
   not enforced or if waivers are too easily available).  However, IT
   environments are complex and sometimes there is a need to have
   automated access relationships for special purposes that would not
   otherwise be advisable.  Special waivers and corresponding approvals
   can be used for implementing such special cases, but they MUST be
   revisited annually and MUST NOT be used to circumvent remediating the
   existing environment.





Ylonen, et al.          Expires October 06, 2013               [Page 38]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Ideally, continuous monitoring should be a real-time or near-realtime
   process.  For some areas, hourly or daily analysis would generally be
   perfectly sufficient.  Using automated tools allows monitoring to be
   performed more frequently, cost-effectively, and more thoroughly.  On
   the other hand, if implemented manually using audits, cost
   constraints may limit continuous monitoring to annual audits.  Even
   when continuous monitoring is performed using software tools,
   auditors SHOULD do some random sampling and testing annually to
   verify that the continuous monitoring tools are working properly.

   In some respects, continuous monitoring resembles re-performing
   discovery on an ongoing basis.  Configured SSH user keys and trust
   relationships throughout the environment need to be discovered, and
   checked for validity.  Alerts, audit findings, or reports may be
   produced based on the results of the checks.  As in the discovery
   phase, the continuous monitoring process MUST identify every trust
   relationship and authorized key throughout the managed IT environment
   so that they can be compared against a database of approved trust
   relationships.

   For each found authorized key, the trust relationship should be
   analyzed to identify possible instances of non-compliance or
   excessive security risk.  Trust relationships leading to moderate-
   impact or high-impact hosts with the following attributes MUST be
   reported for further investigation:

      Trust relationships without proper approval

      Trust relationships without proper justification and an associated
      application/business process

      Trust relationships that have no command restriction configured

      Trust relationships with command restrictions that do not match
      the command restrictions specified during approval

      Trust relationships from low-impact hosts with no command
      restrictions

      Trust relationships that cross defined internal boundaries

      Trust relationships that have not been used in the last 12 months
      (or other time period specified by policy)

      Trust relationships whose keys have not been rotated in the last
      12 months (or other time period specified by policy)





Ylonen, et al.          Expires October 06, 2013               [Page 39]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Trust relationships leading to low-impact hosts with the following
   attributes SHOULD be reported for further investigation:

      Trust relationships without proper approval

      Trust relationships without proper justification and an associated
      application/business process

      Trust relationships leading to privileged accounts that have no
      command restriction configured

      Trust relationships that have not been used in the last 12 months
      (or other time period specified by policy)

      Trust relationships whose keys have not been rotated in the last
      12 months (or other time period specified by policy)

   If trust relationships have existing waivers (e.g., for having no
   command restrictions, crossing boundaries, or not being used or
   rotated), then special approval of the waiver MUST be verified and
   waivers SHOULD be re-justified and approved annually.  Trust
   relationships that are flagged by continuous monitoring MUST be
   investigated and resolved.  Possible resolution activities consist of
   the following:

      Obtaining approvals and justifications (including the associated
      application/business process) for trust relationships that are
      valid, including getting secondary approval by higher-level
      management for trust relationships that cross boundaries.  This
      would retroactively apply the approval process described in
      Section 5.6.

      Adding command restrictions to the authorized keys file to limit
      access according to policy.  (See Section 5.8)

      Removing trust relationships that are unused, not needed, or
      otherwise invalid.  (See Section 5.4)

      Rotating private keys.  (See Section 5.7)

      Obtaining waivers with appropriate levels of approval.

   Even if waivers are obtained, the resulting risk needs to be
   considered.  For example, if the trust relationship from a low-impact
   host to a medium-impact or high-impact host has inadequate command
   restrictions, then the low-impact host MUST be reclassified as having
   the impact level of the higher-impact host, even if a waiver is
   obtained.



Ylonen, et al.          Expires October 06, 2013               [Page 40]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Failure to monitor SSH trust relationships prevents the organization
   from enforcing policies related to SSH user keys.  Policy enforcement
   and detection of non-compliant trust relationships is needed to
   prevent new keys from re-creating the same type of problems that
   existed in the legacy population of user keys.  Failure to enforce
   approvals for newly-added trust relationships allows users to create
   unaudited backdoors or trust relationships that cross boundaries or
   are unrestricted.  If there is no continuous monitoring for
   unapproved or inappropriate trust relationships, such trust
   relationships will be essentially permanent.

6.2.  Removal of Trust Relationships

   Trust relationships MUST be removed when they are no longer needed.
   Ideally, the business or application owner of a trust relationship
   SHOULD expressly request that it be removed as soon as it is no
   longer needed.  In addition, the owner MAY periodically recertify and
   validate the continuing need for each trust relationship.

   Sometimes a trust relationship may be removed by express request,
   e.g., when a business process is changed so that it is no longer
   needed.

   Sometimes a trust relationship may be removed because the application
   or business process it relates to is decommissioned or replaced by
   another application.

   Sometimes a trust relationship may be removed because continuous
   monitoring detects that it is no longer being used.  This basically
   implies that something changed in the environment, but the trust
   relationship was inadvertently not removed at that time.  (This
   scenario appears to be very common in practice).  In addition, some
   trust relationships may be removed because continuous monitoring
   detected an unapproved or otherwise invalid trust relationship.

   When trust relationships are removed, the associated authorized key
   (if it is key-based) MUST be removed from the authorized keys file of
   the destination server.

   When there are no trust relationships remaining using a particular
   identity key, the identity key SHOULD be removed.

6.3.  Periodic Rotation of Trust Relationships

   Keys must be regularly rotated as specified in Section 5.7.

7.  Policy Recommendations




Ylonen, et al.          Expires October 06, 2013               [Page 41]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Effective security policies are important for defining expectations
   for controls and acceptable user behavior.  Well-defined policies are
   no less important for governing SSH user keys than for other elements
   of an organization's security program.  In fact, because few people
   understand the problem and poor SSH user key management practices are
   so pervasive, policies are essential to the success of any SSH key
   management remediation process.

   To support the key remediation and continuous monitoring steps
   outlined elsewhere in this document, there is a common core set of
   policy statements that should be adopted by all organizations.  The
   following policy statements are RECOMMENDED (with limited
   organizational-specific customization and optionally limited to apply
   to moderate-impact and high-impact systems) to provide the governance
   framework for controlling SSH user keys:

      All SSH servers shall be configured to store authorized keys in a
      root-owned /etc directory (or other suitable directory not
      writable by normal users).

      Users shall not create new identity keys or authorized keys, shall
      not share identity keys with other users, and shall not copy or
      move identity keys to other SSH client systems.

      SSH identity keys and authorized keys shall be provisioned only by
      the access management group.

      SSH user key requests shall follow the standard provisioning
      process.  All requests for SSH authorized keys shall be
      provisioned only when required by a valid business need and
      approved by the destination account's owner.

      Trust relationships shall not cross security zone boundaries.  If
      this is a requirement for a trust relationship, then the new user
      key request shall provide a rationale and require a waiver
      approved by the server operations director and information
      security director.

      Trust relationships shall not allow access from low-impact systems
      to higher-impact systems.  If such trust relationships are
      required, then those low-impact systems shall be reclassified as
      higher-impact systems and shall be subject to the higher security
      requirements of higher-impact systems, unless command restrictions
      prevent obtaining an interactive shell and writing arbitrary files
      using such trust relationships.

      Trust relationships for non-interactive access shall be configured
      with command restrictions.  If commands cannot be restricted, then



Ylonen, et al.          Expires October 06, 2013               [Page 42]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      the new user key request shall provide a rationale and require a
      waiver approved by the server operations director and information
      security director.

      Trust relationships permitting interactive access (especially to
      privileged accounts) shall enforce source restrictions to
      authorized, hardened jump servers or transparent access auditing
      solutions are used that ensure such access is properly controlled.

      A registry of SSH user keys shall be maintained for tracking trust
      relationships (including their owner, purpose, approval,
      restrictions, and business purpose) throughout the environment.

      SSH user keys and corresponding trust relationships shall be
      removed when no longer needed or no longer used.

      Usage of SSH user keys shall be tracked so that unused authorized
      keys can be identified.

      All SSH user keys shall be rotated annually.

      When a user terminates employment or transfers to new job
      responsibilities, all keys assigned to that user shall be rotated,
      or the corresponding authorized key relationships shall be
      removed.

      If a key is compromised or shared by two or more users, then the
      key shall immediately be rotated, or the corresponding authorized
      key relationships shall be removed.

      SSH authorized keys shall be revalidated annually by the
      destination account owner to ensure that trust relationships
      continue to be valid and proper.

      Authorized keys for privileged accounts such as root shall be
      revalidated annually and approved by the server operations
      director and information security director.

      Trust relationships throughout the network shall be monitored at
      least annually to enforce compliance with this policy.  At a
      minimum, monitoring activities shall be in place to detect the
      following types of non-compliance for immediate resolution:

         SSH user key trust relationships that bypassed the formal
         provisioning process and were not authorized and configured by
         the access management group





Ylonen, et al.          Expires October 06, 2013               [Page 43]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


         SSH user key trust relationships that cross security zone
         boundaries

         SSH user keys that have been not rotated in over a year

         Dormant trust relationships that have not been used

   Other policy statements are highly dependent on the risk tolerance
   and context of each organization.  Depending on the unique
   circumstances of the organization, these policy statements may or may
   not be applicable.  During the remediation process, organizations
   often make risk-based decisions about how to cost-effectively control
   and manage their SSH keys in their own context.  It is critical that
   these decisions be properly reflected in security policy in order to
   influence user behavior and provide a framework for organization-
   specific controls.  Examples of these types of policy statements are
   provided below:

      All SSH user keys assigned to human users for interactive logins
      shall be assigned a passphrase that is at least 15 characters
      long.  (The reason for this policy is self-evident.)

      SSH trust relationships for human accounts shall be limited to
      other human accounts.  Human accounts shall never have trust
      relationships to system accounts or service accounts.  (This
      policy makes sense for organizations with lots of keys and
      transitive trust relationships that are too difficult to manage.
      Eliminating human-to-system account trust relationships can help
      simplify the mesh of trust and therefore minimize the risk of
      inadvertently allowing unneeded access.)

      SSH user keys shall be used only for automated access and shall
      not be used for interactive logins by human users.  (An
      organization may decide to do this to reduce the number of keys in
      the environment and lighten the load on the provisioning process,
      for example, if no automation is available and provisioning is
      done manually.)

      SSH servers shall be configured to deny connections to the root
      account.  (If key-based connections to root are not required, then
      setting "PermitRootLogin no" can significantly contain the damage
      that can be done through unauthorized use of keys).

      Unique SSH host keys shall be created for every system.  (This is
      essential when SSH host-based authentication is used and for
      protecting against man-in-the-middle attacks.)





Ylonen, et al.          Expires October 06, 2013               [Page 44]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


8.  Considerations for Software Tools

   All requirements specified in this document can be implemented
   manually and with regular audits, without using software tools.  Use
   of software tools is OPTIONAL.  However, automated software tools for
   managing SSH keys are commercially available from multiple vendors
   and their use is RECOMMENDED in large environments, as they can
   substantially reduce the time, cost, and effort needed for
   remediating existing SSH user keys and provide substantial ongoing
   cost savings for continuously managing and monitoring SSH keys in an
   organization.

   Here are certain key things to consider in planning an SSH key
   management remediation solution and its deployment:

      Does the solution support all required operating systems where SSH
      keys need to be managed (including mainframe, if applicable)?

      Does the solution support all SSH implementations and versions
      that are use in the environment, including their key formats and
      fingerprint formats?

      Does the solution support keys moved to protected, root-only-
      writable locations?  Can it help move keys to such locations?  How
      does it determine where the keys are stored on each host?

      Can trust relationships that are not actually used be
      automatically detected and proposed for removal (with selective
      approval)?

      Can the solution associate trust relationships and keys with an
      application, business process, or other purpose?  Can it enforce
      that all authorized keys have a documented purpose?  How is this
      implemented for legacy trust relationships (from time before
      deployment of the solution)?  Can it distinguish legacy keys from
      those that are set up afterwards?

      How does the solution implement approvals for new keys?  How does
      it integrate to existing workflows and tools?  Does it support an
      approval workflow which integrates into external systems?

      Can creation of new keys and trust relationships be automated
      based on approvals done in an existing IT change control system?
      If no existing IT change control system is in use in the
      organization, does the solution provide one to enforce approvals?

      Does the solution support grouping systems based on the impact of
      their disruption or compromise?



Ylonen, et al.          Expires October 06, 2013               [Page 45]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Does the solution support rotating SSH user keys?

      How is key rotation implemented for external trust relationships/
      external keys?  Can it automatically recognize external keys?

      Does the solution support configuring command restrictions for
      authorized keys/trust relationships?  Does it support requiring
      special approvals for trust relationships that do not have a
      command restriction?

      Does the solution support configuring source restrictions for
      authorized keys/trust relationships?

      Does the solution provide continuous monitoring capabilities as
      specified in Section 6?

      If the management system is unavailable for some reason, will
      normal operation of managed hosts be disrupted (other than not
      being able to create/modify trust relationships)?

      Will the solution run as root on managed hosts, or can it use a
      non-root account and "sudo" (or equivalent) to perform limited
      operations as root?

      Is the solution able to retry discovery, key setups, etc.  on
      hosts that are down or unreachable at the time of the initial
      attempt?  How does the solution cope with poor network
      connectivity?

      Does the solution support user accounts stored in LDAP or Active
      Directory?  How does it prevent crashing LDAP or Active Directory
      servers by reading directory contents from all servers
      simultaneously?

      Can the solution discover keys from directories that are not
      readable by root (e.g., NFS directories using the "root_squash"
      option)?

      Does the solution work with SELinux, if such support is needed?

      How can the solution save operational costs in SSH user key
      management in the organization?  Have existing user key management
      costs been estimated on an annual level?

8.1.  Reducing Cost and Improving Security by Automation

   Some large organizations are seeing over a hundred thousand new
   authorized keys being configured every year.  Some trust relationship



Ylonen, et al.          Expires October 06, 2013               [Page 46]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   setups may involve installing the same authorized key on thousands of
   servers.  Given that setting up and a manual trust relationship can
   easily take fifteen minutes or more, the cost can be millions of
   dollars per year.

   Some software tools allow integration into existing security
   entitlement approval systems, and can implement a suitably formatted
   trust relationship setup request automatically, without manual
   intervention.

   Such automation provides several benefits:

      Substantial cost savings by eliminating the manual work associated
      with trust relationship setups.

      Substantial reduction in outages due to errors in manual key
      setups.

      Need for root access is significantly reduced, as SSH user key
      setups no longer require root access.

      Substantial security improvements from eliminating root access (or
      the need for being able to install new trust relationships) from
      most system administrators (having five people with access to the
      software tool system is much more secure than having two hundred
      administrators able to manually install keys).

9.  Security Considerations

   This document is all about security, including how to evaluate the
   impact of disruption or compromise of information systems, how to
   reduce the risk to information system from automated access, how to
   remediate current unmanaged base of SSH user key based trust
   relationships for automated access, and how to manage and
   continuously monitor automated access as an ongoing process.

   Section 1.5 defined information system impact levels based on FIPS
   199, but expanding on it by considering information systems having
   automated access to higher-impact information systems as also having
   the impact level of the higher-impact information system.

   Section 2.2.6 briefly discussed unmanaged host keys and how they can
   be used to compromise authentication and integrity protection using
   active network-level man-in-the-middle attacks.

   Section 3 discussed various threats arising from poorly managed
   automated access and SSH user keys, including virus spread threat,
   unaudited backdoor threat, leaked keys granting near-permanent



Ylonen, et al.          Expires October 06, 2013               [Page 47]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   access, and lack of proper termination of access when an employee
   leaves or changes roles.  It also discussed how ports opened in
   firewalls may be used for unintended purposes, including command
   execution, access to internal services, or for hiding source of
   attacks, if not properly controlled.

   Section 4 discussed assessing the threats and exposure of an
   organization to them as a quick precheck during audit, before
   engaging in a full discovery and remediation project.

   Section 5 provided recommendations on how to bring existing trust
   relationships for automated access, particularly SSH user keys, under
   control.

   Section 6 provided recommendations for continuous monitoring and
   management of automated access and SSH user keys.

   Section 7 provided recommendations for organizational security
   policy.

   As a summary, automated access between systems MUST NOT be overlooked
   in identity and access management.  It has become so prevalent that
   many organizations have many times more credentials for automated
   access to their information systems that they have user accounts for
   employees.

   Management of SSH keys is about managing access, with strong ties to
   identity and access management, security architecture, privileged
   access management, IT change control, and security audits.
   Cryptographic properties of the keys are in practice of little
   importance, as all keys generated with default settings by most
   commonly used SSH implementations are still cryptographically
   reasonably strong.

   Virus spread threat using automated trust relationships may remove
   defense in depth against attacks and malware.  Automated access may
   provide pathways for bypassing existing privileged access management
   systems.  Rogue administrators may use SSH user keys to create near-
   permanent unaudited backdoors, and leaked keys may be used for
   breaking into production servers.  Even accidental access using
   poorly configured trust relationships has in the past caused
   substantial financial losses.









Ylonen, et al.          Expires October 06, 2013               [Page 48]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Risks of unmanaged, unaudited automated access are sufficiently high
   and the state of their management in some of the largest
   organizations in the world so appalling that all organizations should
   evaluate to what extent they use automated access within and between
   their information systems, how it is managed and audited, and whether
   they are exposed to the risks.

   IT security auditors, policy makers, and security architects are
   urged to take automated access and SSH keys on their agenda.

10.  Acknowledgements

   The authors thank and acknowledge the contribution of the following
   people to the development of this document and/or the underlying
   ideas: Bruno Canamasas, Roman Hernandez, Jan Hlinovski, Kalle
   Jaaskelainen, Mitch Klein, Sami Lehtinen, Sami Marttinen, Matthew
   McKenna, S.  Moonesamy, Tim Polk, Joe Scaff.  We also wish to thank
   anyone else who has helped by providing comments or input.

11.  Glossary

   account:  A user account on a computer.  An account may belong to an
      actual person (interactive user) or may be used internally in a
      system (in which case it is sometimes called a functional account,
      process account, system account, or non-user account).

   Active Directory:  A directory service created by Microsoft for
      Windows domain networks, providing a central repository for user
      information, user groups, and various other kinds of configuration
      information.  Active Directory makes use of the LDAP and Kerberos
      protocols, among others, and can serve as an LDAP directory and
      Kerberos Key Distribution Center (KDC).

   Advanced Persistent Threat (APT):  A group, such as a government,
      with the capability and intent to persistently target an entity
      using a variety of cyberwarfare techniques, such as espionage,
      social engineering, custom malware, and sophisticated hacking and
      evasion techniques.

   authorized key:  A public key that has been configured as authorizing
      access to an account by anyone capable of using the corresponding
      private key (identity key) in the SSH protocol.  An authorized key
      may be configured with certain restrictions, most notably a forced
      command and a source restriction.

   automated access:  Access to a computer without an interactive user,
      generally machine-to-machine access.  Automated access is often
      triggered from scripts or schedulers, e.g., by executing an SSH



Ylonen, et al.          Expires October 06, 2013               [Page 49]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      client or a file transfer application.  Many programs may also use
      automated access using SSH internally, including many privileged
      access management systems and systems management tools.

   automated trust relationship:  A trust relationship for automated
      access.

   command restriction:  See forced command.

   certificate:  A public key signed by a certification authority (CA)
      key, together with additional information about the public key.
      X.509 [RFC3280] is a widely used standard for certificates.
      OpenSSH also implements its own proprietary certificate format;
      however, use of the proprietary format is NOT RECOMMENDED (in part
      because OpenSSH's authorization model does not permit reliably
      determining which trust relationships exist granting access to a
      server).

   CIFS:  Common Internet File System, a protocol used on Windows for
      file sharing.  The protocol is unencrypted and may be read and
      subverted by a network-level attacker.  The protocol is extremely
      widely used in Windows environments, less frequently with Unix/
      Linux.

   CISO:  Chief Information Security Officer.  A person responsible for
      IT security in an organization.

   COBIT:  Control Objectives for Information and Related Technology, a
      framework created by ISACA (Information Systems Audit and Control
      Association) for information technology (IT) management and IT
      governance.

   CryptoAuditor:  A product from SSH Communications Security for
      controlling and auditing content of SSH sessions and other
      encrypted communications, including file transfers.  Can also be
      used for auditing use of SSH/SFTP connections at a firewall and
      for privileged access auditing for key-based access.

   destination account:  In an SSH connection or trust relationship, the
      user account for which authentication is provided and under which
      any commands or other operations performed by the connection are
      executed (acknowledging that some commands, such as "sudo", may
      further escalate privileges).

   destination host:  In an SSH connection or trust relationship, the
      destination host of the connection.  A destination host would
      typically run an SSH server.




Ylonen, et al.          Expires October 06, 2013               [Page 50]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   DSA:  Digital Signature Algorithm.  An algorithm for public-key
      cryptography, particularly digital signatures.  It is a United
      States government standard, specified in FIPS 186-3.

   external key:  An authorized key that is used from outside the
      organization (or outside the environment considered for SSH user
      key management purposes), or an identity key that is used for
      authenticating to outside the organization (or outside the
      environment considered for SSH user key management purposes).  Key
      rotation can break external keys, and therefore it must be ensured
      that the other side of trust relationships involving external keys
      is also properly updated as part of rotation.  Alternatively,
      rotation of external keys may be prevented, but that is not a
      sustainable solution long-term.

   fingerprint:  A hash value of a (public) key encoded into a string
      (e.g., into hexadecimal).  Several fingerprint formats are in use
      by different SSH implementations.

   FISMA:  Federal Information Security Management Act of 2002, a United
      States law that mandates how US government agencies must implement
      their it security.

   forced command:  A restriction configured for an authorized key that
      prevents executing commands other than the specified command when
      logging in using the key.  In Tectia SSH and OpenSSH, forced
      command can be configured by using a "command=" restriction in an
      authorized keys file.

   functional account:  An account used for running applications or
      other processes, as opposed to an interactive account normally
      used by a person.  Functional accounts are sometimes also called
      process accounts, system accounts, or non-user accounts (with
      slight nuances in meaning).

   host:  A computer or other device on a network.  A host may be a
      physical computer, a virtual machine, or any other logical or
      physical device that can communicate on a network, typically using
      one or more IP addresses.  Some hosts may be multi-homed, meaning
      that they have more than one IP address.

   host certificate:  A certificate for a host key for host
      authentication in the SSH protocol (typically an X.509v3
      certificate).  Host certificates can eliminate the need for
      distributing host keys to all communicating hosts, greatly
      simplifying management and rotation of host keys.  Widely used
      with Tectia SSH to avoid copying host keys and to make rotating
      them easier.



Ylonen, et al.          Expires October 06, 2013               [Page 51]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   host credential:  A Kerberos credential that is used for
      authenticating to a Kerberos KDC as a host principal.

   host key:  A public key used for authenticating a host in the SSH
      protocol to hosts that want to communicate with it (each host also
      generally has its own private host key).  Some hosts may have more
      than one host key (e.g., one for each algorithm).  Host keys are
      used for authenticating hosts (machines) themselves, not users or
      accounts, whereas identity keys and authorized keys relate to
      authenticating users/accounts and authorizing access to accounts
      on hosts.  See also Section 2.2.6.

   identity key:  A private key that is used for authentication in the
      SSH protocol; grants access to the accounts for which the
      corresponding public key has been configured as an authorized key.

   indirect trust relationship:  A sequence of trust relationships that
      indirectly leads to another account.  For example, account A may
      be able to log into account B, which may be able to log into
      account C; then, account C indirectly trusts account A (and B
      directly trusts A and C directly trusts B).  Indirect trust
      relationships may involve many kinds of trust relationships (e.g.,
      SSH keys, Kerberos and privilege escalation).

   interactive user:  A person (human) that uses a computer (and may
      type passwords or provide other authentication credentials as
      needed), as opposed to a computer that performs operations on
      another computer in an automated fashion.

   jump host:  A server that a user logs into for the purpose of logging
      infrom there to another server.  They are used for privileged
      access management, centralizing configuration of access to a large
      number of servers (e.g., at retail locations), and for accessing
      restricted subnets that do not have normal routing from the rest
      of the organization.

   KDC:  Key Distribution Center, a component of Kerberos.

   Kerberos:  A centralized authentication and single-sign on system.
      Also used as part of Active Directory.  See RFC 4120 [RFC4120].

   key:  A cryptographic key.  In this document, keys generally refer to
      public key cryptography key pairs used for authentication of users
      and/or machines (using digital signatures).  Examples include
      identity key and authorized keys.  The SSH protocol also uses host
      keys that are used for authenticating SSH servers to SSH clients
      connecting them.




Ylonen, et al.          Expires October 06, 2013               [Page 52]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   Key Distribution Center:  A component of Kerberos and Active
      Directory infrastructure that verifies credentials and issues
      tickets to principals (e.g., users and hosts).  An Active
      Directory server includes a KDC.  Frequently multiple KDCs
      synchronize information for redundancy.

   known host:  A host whose host key is known (to a particular SSH
      client).

   LDAP:  Lightweight Directory Access Protocol, a protocol for
      accessing and maintaining distributed directory information
      services.  See RFC 4511 [RFC4511].

   locking down keys:  This refers to moving authorized keys to root-
      owned (or otherwise protected) locations, so that non-root users
      cannot add new authorized keys to themselves.  This helps prevent
      system administrators and users from creating key-based backdoors
      that may survive the termination of their account and bypass
      privileged access management systems.  See Section 5.2 for more
      information.

   NERC:  North American Electric Reliability Corporation, an
      organization that, among other things, maintains the Critical
      Infrastructure Protection (CIP) standards that set minimum
      security requirements for protecting power generation and
      distribution infrastructure.

   NFS:  Network File System, a file sharing protocol widely used in
      Unix/Linux environments in enterprises and universities.  The
      protocol is unencrypted and may be subverted by a network-level
      attacker, permitting modification of any file.  (NFS4 adds some
      security but is rarely used at the time of this writing, or is
      used with the security features disabled.)

   OpenSSH:  An open source implementation of SSH based on Tatu Ylonen's
      original free version of SSH from 1995 and further developed by
      the OpenBSD group.

   orphaned key:  An authorized key for which no corresponding public
      key can be found.  An orphaned key may be currently unused, or the
      identity key might just be on a server that was not part of the
      discovery process (it could be an external key).  Therefore
      orphaned keys should not be removed without first monitoring
      whether they are actually used.

   password logger:  A software or hardware module for recording
      keystrokes, especially user names and passwords, typed by an
      interactive user.  Password loggers are nowadays commonly included



Ylonen, et al.          Expires October 06, 2013               [Page 53]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      in various malware and used as part of Advanced Persistent Threat
      (APT) attacks.  Hardware-based key loggers may used in conjunction
      with physical access to a desktop or laptop (perhaps using a
      social engineering attack, such as getting hired as a janitor) to
      obtain passwords for entry into information systems.

   PCI DSS:  A set of Data Security Standards defined by the Payment
      Card Industry Security Standards Council, an organization
      originally formed by major credit card companies.

   PKI:  See Public Key Infrastructure.

   privilege escalation mechanism:  A means for escalating a user's (or
      processes) privileges from those of one account to those of
      another account (particularly a root or Administrator account).
      Examples of privilege escalation mechanisms include intentional
      provilege escalation tools such as "sudo" and unintentional
      privilege escalation possibilities based on vulnerabilities and
      configuration errors (experience has shown that it is very often
      possible to find vulnerabilities or misconfigurations on that
      enable privilege escalation once inside a host).  An attacker
      having access to an account can generally change the configuration
      of the account to cause the user to unknowingly run the attacker's
      programs that may, e.g., steal the user's password and then use
      the password to spread the attack.  Also, having high-level access
      on one host on a network may effectively imply access to every
      user account on every host whose home directory is in networked
      storage accessible through the same network as the compromised
      host.  Against advanced attackers, even vulnerable embedded
      devices such as switches, printers and copiers can be used to
      perform network-level active attacks against other hosts.  Some
      limit will have to be put on what theoretical possiblities are
      considered, however.  Privilege escalation possibilities
      effectively imply additional trust relationships that may in turn
      imply a multitude of indirect trust relationships.

   Public Key Infrastructure:  An arrangement that binds public keys
      with respective user identities using digital signatures issued by
      a certificate authority (CA).  See RFC 3280 [RFC3280].

   Putty:  An Open Source SSH client for Windows.

   Reflection for Secure IT:  A commercial version of SSH from
      Attachmate.

   root account:  In Linux/Unix, a privileged account that is usually
      able to do anything in a computer (including reading any files and
      modifying any programs).  In Windows, Local Administrator and



Ylonen, et al.          Expires October 06, 2013               [Page 54]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


      Domain Administrator have similar or even broader power.  (This
      document mostly talks about root access as SSH is mostly used on
      Linux/Unix and embedded devices, but the same issues often also
      apply to other technologies and the Windows environment.)

   rotating a key:  Key rotation means changing the key, i.e., replacing
      it by a new key.  The places that use the key or keys derived from
      it (e.g., authorized keys derived from an identity key, legitimate
      copies of the identity key, or certificates granted for a key)
      typically need to be correspondingly updated.  With SSH user keys,
      it means replacing an identity key by a newly generated key and
      updating authorized keys correspondingly.  See also external key.

   RSA:  An algorithm for public-key cryptography based on the
      difficulty of factoring large integers, invented by Ron Rivest,
      Adi Shamir and Leonard Adleman.

   SELinux:  Security-Enhanced Linux, a Linux feature that provides
      mechanisms for supporting access control security policies.
      SELinux is enabled by default on several Linux distributions (at
      least in what is called "targeted" mode, where it protects
      selected services).

   SFTP:  SSH File Transfer Protocol, a file transfer and file sharing
      protocol typically used with the SSH protocol and originally
      developed by Tatu Ylonen for ssh-2.0.  The protocol is
      unofficially described in SFTP [SFTP]; there is no normative
      reference available at the time of this writing.

   source account:  In an SSH connection or trust relationship, a source
      account is the user account on the host initiating the connection,
      typically the user account under which an SSH client runs.

   source host:  In an SSH connection or trust relationship, a source
      host is the host initiating the connection (typically by running
      an SSH client).

   source restriction:  A restriction configured for an authorized key
      that limits the IP addresses or host names from which login using
      the key may take place.  In OpenSSH, source restrictions can be
      configured by using a "from=" restriction in an authorized keys
      file.

   SOX:  Sarbanes-Oxley Act of 2002, also known as the Public Company
      Accounting Reform and Investor Protection Act, a United States law
      that, among other things, sets requirements for protecting certain
      sensitive information in listed companies.




Ylonen, et al.          Expires October 06, 2013               [Page 55]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   SSH:  SSH (Secure Shell) is a protocol and tool for remote system
      administration, file transfers, and for tunneling TCP/IP
      communications securely, originally developed by Tatu Ylonen.

   SSH Communications Security:  A company founded by Tatu Ylonen, the
      inventor of SSH, with products improving security and operational
      efficiency of large IT environments, particularly for large SSH
      environments.  See http://www.ssh.com [2].

   sudo:  A privilege escalation mechanism/tool on Unix/Linux that can
      be used for executing commands as root from a non-root account.
      The operation of "sudo" depends on its configuration.  In some
      configurations, certain accounts may perform any command as root
      using "sudo".  In some other systems, certain users, such as
      members of a "wheel" group can perform commands as root by
      confirming the operation with the user's password.  Several
      commercial tools also exist for the same purpose.

   Tectia Manager:  A product for managing SSH host keys and
      configurations, from SSH Communications Security.

   Tectia SSH:  A commercial version of SSH servers and clients for
      Windows, z/OS (IBM mainframes), Unix, and Linux from SSH
      Communications Security.

   transparent access auditing:  A method of doing privileged access
      management and auditing on the network (using a co-operative man-
      in-the-middle attack to transparently gain access to the
      connection) or at an SSH server (by having auditing code built
      into the server).  See, e.g., the CryptoAuditor solution.

   trust relationship:  Something that permits a source account to log
      in to a destination account (possibly on a different computer).
      In a sense, the destination account trusts the source account, and
      if the source account is compromised, so is the destination
      account.  An example is an authorized key (and corresponding
      identity key) configured for public key authentication in SSH.
      See also indirect trust relationship and privilege escalation.

   Universal SSH Key Manager:  A product from SSH Communications
      Security for managing and monitoring SSH keys and other trust
      relationships for automated access.

   user key:  A key that is used for granting access to a user account
      in the SSH protocol (as opposed to a host key, which does not
      grant access to anything but serves to authenticate a host).  Both
      authorized keys and identity keys are user keys.




Ylonen, et al.          Expires October 06, 2013               [Page 56]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   X.509:  A standardized widely used certificate format for public key
      infrastructure (PKI).  See RFC 3280 [RFC3280].

12.  References

   [FIPS199]  Evans, D. L., Bond, P. J., and A. L. Bement, "Standards
              for Security Categorization of Federal Information and
              Information Systems", FIPS Publication 199, February 2004.

   [FIPS200]  Gutierrez, C. M. and W. Jeffrey, "Minimum Security
              Requirements for Federal Information and Information
              Systems", FIPS Publication 200, March 2006.

   [NIST800-53]
              Locke, G. and P. D. Gallagher, "Recommended Security
              Controls for Federal Information Systems and
              Organizations", NIST Special Publication 800-53 (revision
              3 with updates as of 05-01-2010), August 2009.

   [KENT]     Kent, G. and B. Shrestha, "Unsecured SSH - The Challenge
              of Managing SSH Keys and Associations", SecureIT White
              Paper, 2010.

   [RFC3280]  Housley, R., Polk, W., Ford, W., and D. Solo, "Internet
              X.509 Public Key Infrastructure Certificate and
              Certificate Revocation List (CRL) Profile", RFC 4251,
              April 2002.

   [RFC4120]  Neuman, C., Yu, T., Hartman, S., and K. Raeburn, "The
              Kerberos Network Authentication Service (V5)", RFC 4251,
              July 2005.

   [RFC4251]  Ylonen, T. and C. Lonvick, "The Secure Shell (SSH)
              Protocol Architecture", RFC 4251, January 2006.

   [RFC4252]  Ylonen, T. and C. Lonvick, "The Secure Shell (SSH)
              Authentication Protocol", RFC 4252, January 2006.

   [RFC4253]  Ylonen, T. and C. Lonvick, "The Secure Shell (SSH)
              Transport Layer Protocol", RFC 4253, January 2006.

   [RFC4254]  Ylonen, T. and C. Lonvick, "The Secure Shell (SSH)
              Connection Protocol", RFC 4254, January 2006.

   [RFC4511]  Sermersheim, J., "Lightweight Directory Access Protocol
              (LDAP): The Protocol", RFC 4511, June 2006.





Ylonen, et al.          Expires October 06, 2013               [Page 57]

Internet-Draft   Managing SSH Keys for Automated Access       April 2013


   [SFTP]     Galbraith, J. and O. Saarenmaa, "SSH File Transfer
              Protocol", draft-ietf-secsh-filexfer-13.txt (work in
              progress), June 2006.

Authors' Addresses

   Tatu Ylonen
   SSH Communications Security

   Email: ylo@ssh.com
   URI:   http://www.ssh.com


   Greg Kent
   SecureIT

   Email: gkent@secureit.com


   Murugiah Soyppaya
   NIST

   Email: soyppaya@nist.gov



























Ylonen, et al.          Expires October 06, 2013               [Page 58]