Internet DRAFT - draft-lee-teas-actn-pm-telemetry-autonomics

draft-lee-teas-actn-pm-telemetry-autonomics



TEAS Working Group                                      Y. Lee (Editor)
Internet Draft                                              Dhruv Dhody
Intended Status: Standard Track                      Satish Karunanithi
Expires: November 8, 2019                                        Huawei

                                                         Ricard Vilalta
                                                                   CTTC

                                                            Daniel King
                                                   Lancaster University

                                                     Daniele Ceccarelli
                                                               Ericsson




                                                            May 8, 2019


  YANG models for VN & TE Performance Monitoring Telemetry and Scaling
                           Intent Autonomics

              draft-lee-teas-actn-pm-telemetry-autonomics-17

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with
   the provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on November 8, 2019.

Copyright Notice


Lee, et al.             Expires November 2019                  [Page 1]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   Copyright (c) 2019 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document.  Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Abstract

   This document provides YANG data models that describe performance
   monitoring telemetry and scaling intent mechanism for TE-tunnels and
   Virtual Networks (VN).

   The models presented in this draft allow customers to subscribe to
   and monitor their key performance data of their interest on the
   level of TE-tunnel or VN. The models also provide customers with the
   ability to program autonomic scaling intent mechanism on the level
   of TE-tunnel as well as VN.



Table of Contents


   1. Introduction...................................................3
      1.1. Terminology...............................................4
      1.2. Tree diagram..............................................5
      1.3. Prefixes in Data Node Names...............................5
   2. Use-Cases......................................................5
   3. Design of the Data Models......................................7
      3.1. TE KPI Telemetry Model....................................7
      3.2. VN KPI Telemetry Model....................................8
   4. Autonomic Scaling Intent Mechanism.............................9
   5. Notification..................................................11
      5.1. YANG Push Subscription Examples..........................11
   6. YANG Data Tree................................................13
   7. Yang Data Model...............................................15
      7.1. ietf-te-kpi-telemetry model..............................15
      7.2. ietf-vn-kpi-telemetry model..............................21
   8. Security Considerations.......................................25
   9. IANA Considerations...........................................26


Lee, et al.             Expires November 2019                  [Page 2]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   10. Acknowledgements.............................................27
   11. References...................................................27
      11.1. Normative References....................................27
      11.2. Informative References..................................28
   12. Contributors.................................................29
   Authors' Addresses...............................................29

1. Introduction

   The YANG model discussed in [VN] is used to operate customer-driven
   Virtual Networks (VNs) during the VN instantiation, VN computation,
   and its life-cycle service management and operations. YANG model
   discussed in [TE-Tunnel] is used to operate TE-tunnels during the
   tunnel instantiation, and its life-cycle management and operations.

   The models presented in this draft allow the applications hosted by
   the customers to subscribe to and monitor their key performance data
   of their interest on the level of VN [VN] or TE-tunnel [TE-Tunnel].
   The key characteristic of the models presented in this document is a
   top-down programmability that allows the applications hosted by the
   customers to subscribe to and monitor key performance data of their
   interest and autonomic scaling intent mechanism on the level of VN
   as well as TE-tunnel.

   According to the classification of [RFC8309], the YANG data models
   presented in this document can be classified as customer service
   models, which is mapped to CMI (Customer Network Controller (CNC)-
   Multi-Domain Service Coordinator (MSDC) interface) of ACTN
   [RFC8453].

   [RFC8233] describes key network performance data to be considered
   for end-to-end path computation in TE networks. Key performance
   indicator (KPI) is a term that describes critical performance data
   that may affect VN/TE-tunnel service. The services provided can be
   optimized to meet the requirements (such as traffic patterns,
   quality, and reliability) of the applications hosted by the
   customers.

   This document provides YANG data models generically applicable to
   any VN/TE-Tunnel service clients to provide an ability to program
   their customized performance monitoring subscription and publication
   data models and automatic scaling in/out intent data models. These
   models can be utilized by a client network controller to initiate
   these capability to a transport network controller communicating
   with the client controller via a NETCONF [RFC8341] or a RESTCONF
   [RFC8040] interface.



Lee, et al.             Expires November 2019                  [Page 3]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   The term performance monitoring being used in this document is
   different from the term that has been used in transport networks for
   many years. Performance monitoring in this document refers to
   subscription and publication of streaming telemetry data.
   Subscription is initiated by the client (e.g., CNC) while
   publication is provided by the network (e.g., MDSC/PNC) based on the
   client's subscription. As the scope of performance monitoring in
   this document is telemetry data on the level of client's VN or TE-
   tunnel, the entity interfacing the client (e.g., MDSC) has to
   provide VN or TE-tunnel level information. This would require
   controller capability to derive VN or TE-tunnel level performance
   data based on lower-level data collected via PM counters in the
   Network Elements (NE). How the controller entity derives such
   customized level data (i.e., VN or TE-tunnel level) is out of the
   scope of this document.

   The data model includes configuration and state data according to
   the new Network Management Datastore Architecture [RFC8342].



1.1. Terminology

   Refer to [RFC8453], [RFC7926], and [RFC8309] for the key terms used
   in this document.

   Key Performance Data: This refers to a set of data the customer is
   interested in monitoring for their instantiated VNs or TE-tunnels.
   Key performance data and key performance indicators are inter-
   exchangeable in this draft.

   Scaling: This refers to the network ability to re-shape its own
   resources. Scale out refers to improve network performance by
   increasing the allocated resources, while scale in refers to
   decrease the allocated resources, typically because the existing
   resources are unnecessary.

   Scaling Intent: To declare scaling conditions, scaling intent is
   used. Specifically, scaling intent refers to the intent expressed by
   the client that allows the client to program/configure conditions of
   their key performance data either for scaling out or scaling in.
   Various conditions can be set for scaling intent on either VN or TE-
   tunnel level.

   Network Autonomics: This refers to the network automation capability
   that allows client to initiate scaling intent mechanisms and
   provides the client with the status of the adjusted network


Lee, et al.             Expires November 2019                  [Page 4]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   resources based on the client's scaling intent in an automated
   fashion.

1.2. Tree diagram

   A simplified graphical representation of the data model is used in
   Section 5 of this this document.  The meaning of the symbols in
   these diagrams is defined in [RFC8340].

1.3. Prefixes in Data Node Names

   In this document, names of data nodes and other data model objects
   are prefixed using the standard prefix associated with the
   corresponding YANG imported modules, as shown in Table 1.


      +---------+------------------------------+-----------------+
      | Prefix  | YANG module                  | Reference       |
      +---------+------------------------------+-----------------+
      | rt      | ietf-routing-types           | [RFC8294]       |
      | te      | ietf-te                      | [TE-Tunnel]     |
      | te-types| ietf-te-types                | [TE-Types]      |
      | te-tel  | ietf-te-kpi-telemetry        | [This I-D]      |
      | vn      | ietf-vn                      | [VN]            |
      | vn-tel  | ietf-vn-kpi-telemetry        | [This I-D]      |
      +---------+------------------------------+-----------------+

             Table 1: Prefixes and corresponding YANG modules

2. Use-Cases

   [PERF] describes use-cases relevant to this draft. It introduces the
   dynamic creation, modification and optimization of services based on
   the performance monitoring. Figure 1 shows a high-level workflows
   for dynamic service control based on traffic monitoring.














Lee, et al.             Expires November 2019                  [Page 5]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


      +----------------------------------------------+
      | Client   +-----------------------------+     |
      |          | Dynamic Service Control APP |     |
      |          +-----------------------------+     |
      +----------------------------------------------+
      1.Traffic|  /|\4.Traffic            | /|\
      Monitor& |   | Monitor              |  | 8.Traffic
      Optimize |   | Result     5.Service |  | modify &
      Policy   |   |              modify& |  | optimize
              \|/  |        optimize Req.\|/ | result
      +----------------------------------------------+
      | Orchestrator                                 |
      |    +-------------------------------+         |
      |    |Dynamic Service Control Agent  |         |
      |    +-------------------------------+         |
      |    +---------------+ +-------------------+   |
      |    | Flow Optimize | | vConnection Agent |   |
      |    +---------------+ +-------------------+   |
      +----------------------------------------------+
      2. Path |   /|\3.Traffic            | /|\
      Monitor |    | Monitor              |  |7.Path
      Request |    | Result      6.Path   |  | modify &
              |    |             modify&  |  | optimize
             \|/   |        optimize Req.\|/ | result
      +----------------------------------------------+
      | Network SDN Controller                       |
      |  +----------------------+ +-----------------+|
      |  | Network Provisioning | |Abstract Topology||
      |  +----------------------+ +-----------------+|
      |  +------------------+ +--------------------+ |
      |  |Network Monitoring| |Physical Topology DB| |
      |  +------------------+ +--------------------+ |
      +----------------------------------------------+

      Figure 1 Workflows for dynamic service control based on traffic
                                monitoring


   Some of the key points from [PERF] are as follows:

     . Network traffic monitoring is important to facilitate automatic
        discovery of the imbalance of network traffic, and initiate the
        network optimization, thus helping the network operator or the
        virtual network service provider to use the network more
        efficiently and save the Capital Expense (CAPEX) and the
        Operating Expense (OPEX).



Lee, et al.             Expires November 2019                  [Page 6]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


     . Customer services have various Service Level Agreement (SLA)
        requirements, such as service availability, latency, latency
        jitter, packet loss rate, Bit Error Rate (BER), etc. The
        transport network can satisfy service availability and BER
        requirements by providing different protection and restoration
        mechanisms. However, for other performance parameters, there
        are no such mechanisms. In order to provide high quality
        services according to customer SLA, one possible solution is to
        measure the SLA related performance parameters, and dynamically
        provision and optimize services based on the performance
        monitoring results.
     . Performance monitoring in a large scale network could generate
        a huge amount of performance information. Therefore, the
        appropriate way to deliver the information in the client and
        network interfaces should be carefully considered.



3. Design of the Data Models

   The YANG models developed in this document describe two models:

   (i)   TE KPI Telemetry Model which provides the TE-Tunnel level of
          performance monitoring mechanism and scaling intent mechanism
          that allows scale in/out programming by the customer. (See
          Section 3.1 & 7.1 for details).

   (ii)  VN KPI Telemetry Model which provides the VN level of the
          aggregated performance monitoring mechanism and scaling
          intent mechanism that allows scale in/out programming by the
          customer (See Section 3.2 & 7.2 for details).



3.1. TE KPI Telemetry Model

   This module describes performance telemetry for TE-tunnel model. The
   telemetry data is augmented to tunnel state.  This module also
   allows autonomic traffic engineering scaling intent configuration
   mechanism on the TE-tunnel level. Various conditions can be set for
   auto-scaling based on the telemetry data (See Section 5 for details)

   The TE KPI Telemetry Model augments the TE-Tunnel Model to enhance
   TE performance monitoring capability. This monitoring capability


Lee, et al.             Expires November 2019                  [Page 7]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   will facilitate proactive re-optimization and reconfiguration of TEs
   based on the performance monitoring data collected via the TE KPI
   Telemetry YANG model.



                +------------+          +--------------+
                |  TE-Tunnel |          |    TE KPI    |
                |   Model    |<---------|  Telemetry   |
                +------------+ augments |     Model    |
                                        +--------------+




3.2. VN KPI Telemetry Model

   This module describes performance telemetry for VN model. The
   telemetry data is augmented both at the VN Level as well as
   individual VN member level. This module also allows autonomic
   traffic engineering scaling intent configuration mechanism on the VN
   level. Scale in/out criteria might be used for network autonomics in
   order the controller to react to a certain set of variations in
   monitored parameters (See Section 4 for illustrations).

   Moreover, this module also provides mechanism to define aggregated
   telemetry parameters as a grouping of underlying VN level telemetry
   parameters. Grouping operation (such as maximum, mean) could be set
   at the time of configuration. For example, if maximum grouping
   operation is used for delay at the VN level, the VN telemetry data
   is reported as the maximum {delay_vn_member_1, delay_vn_member_2,..
   delay_vn_member_N}. Thus, this telemetry abstraction mechanism
   allows the grouping of a certain common set of telemetry values
   under a grouping operation. This can be done at the VN-member level
   to suggest how the E2E telemetry be inferred from the per domain
   tunnel created and monitored by PNCs. One proposed example is the
   following:












Lee, et al.             Expires November 2019                  [Page 8]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


      +------------------------------------------------------------+
      | Client                                                     |
      |                                                            |
      +------------------------------------------------------------+

     1.Client sets the      |   /|\   2. Orchestrator pushes:
     grouping op, and       |    |
     subscribes to the      |    |    VN level telemetry for
     VN level telemetry for |    |    - VN Utilized-bw-percentage
     Delay and              |    |       (Minimum across VN Members)
     Utilized-bw-pecentage  |    |    - VN Delay (Maximum across VN
                           \|/   |     Members)
      +------------------------------------------------------------+
      | Orchestrator                                               |
      |                                                            |
      +------------------------------------------------------------+


   The VN Telemetry Model augments the basic VN model to enhance VN
   monitoring capability. This monitoring capability will facilitate
   proactive re-optimization and reconfiguration of VNs based on the
   performance monitoring data collected via the VN Telemetry YANG
   model.


                +----------+          +--------------+
                |    VN    | augments |      VN      |
                |   Model  |<---------|   Telemetry  |
                +----------+          |     Model    |
                                      +--------------+


4. Autonomic Scaling Intent Mechanism

   Scaling intent configuration mechanism allows the client to
   configure automatic scale-in and scale-out mechanisms on both the
   TE-tunnel and the VN level. Various conditions can be set for auto-
   scaling based on the PM telemetry data.

   There are a number of parameters involved in the mechanism:

     . scale-out-intent or scale-in-intent: whether to scale-out or
        scale-in.
     . performance-type: performance metric type (e.g., one-way-delay,
        one-way-delay-min, one-way-delay-max, two-way-delay, two-way-
        delay-min, two-way-delay-max, utilized bandwidth, etc.)



Lee, et al.             Expires November 2019                  [Page 9]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019




     . threshold-value: the threshold value for a certain performance-
        type that triggers scale-in or scale-out.
     . scaling-operation-type: in case where scaling condition can be
        set with one or more performance types, then scaling-operation-
        type (AND, OR, MIN, MAX, etc.) is applied to these selected
        performance types and its threshold values.
     . Threshold-time: the duration for which the criteria must hold
        true.
     . Cooldown-time: the duration after a scaling action has been
        triggered, for which there will be no further operation.



   The following tree is a part of ietf-te-kpi-telemetry tree whose
   model is presented in full detail in Sections 6 & 7.

   module: ietf-te-kpi-telemetry
     augment /te:te/te:tunnels/te:tunnel:
       +-rw te-scaling-intent
       |  +-rw scale-in-intent
       |  |  +-rw threshold-time?            uint32
       |  |  +-rw cooldown-time?             uint32
       |  |  +-rw scale-in-operation-type?   scaling-criteria-operation
       |  |  +-rw scaling-condition* [performance-type]
       |  |     +-rw performance-type           identityref
       |  |     +-rw threshold-value?           string
       |  |     +-rw te-telemetry-tunnel-ref?
                   -> /te:te/tunnels/tunnel/name
       |  +-rw scale-out-intent
       |     +-rw threshold-time?             uint32
       |     +-rw cooldown-time?              uint32
       |     +-rw scale-out-operation-type?   scaling-criteria-operation
       |     +-rw scaling-condition* [performance-type]
       |        +-rw performance-type           identityref
       |        +-rw threshold-value?           string
       |        +-rw te-telemetry-tunnel-ref?
                   -> /te:te/tunnels/tunnel/name


   Let say the client wants to set the scaling out operation based on
   two performance-types (e.g., two-way-delay and utilized-bandwidth
   for a te-tunnel), it can be done as follows:

     . Set Threshold-time: x (sec)  (duration for which the criteria
        must hold true)





Lee, et al.             Expires November 2019                 [Page 10]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


     . Set Cooldown-time:  y (sec) (the duration after a scaling
        action has been triggered, for which there will be no further
        operation)
     . Set AND for the scale-out-operation-type

   In the scaling condition's list, the following two components can be
   set:

   List 1: Scaling Condition for Two-way-delay

     . performance type: Two-way-delay
     . threshold-value: z milli-seconds

   List 2: Scaling Condition for Utilized bandwidth

     . performance type: Utilized bandwidth
     . threshold-value: w megabytes



5. Notification

   This model does not define specific notifications.  To enable
   notifications, the mechanism defined in [YANG-PUSH]
   and [Event-Notification] can be used.  This mechanism currently
   allows the user to:

   .  Subscribe to notifications on a per client basis.

   .  Specify subtree filters or xpath filters so that only interested
      contents will be sent.

   .  Specify either periodic or on-demand notifications.



5.1. YANG Push Subscription Examples

   [YANG-PUSH] allows subscriber applications to request a continuous,
   customized stream of updates from a YANG datastore.

   Below example shows the way for a client to subscribe to the
   telemetry information for a particular tunnel (Tunnel1). The
   telemetry parameter that the client is interested in is one-way-
   delay.




Lee, et al.             Expires November 2019                 [Page 11]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019



   <netconf:rpc netconf:message-id="101"
       xmlns:netconf="urn:ietf:params:xml:ns:netconf:base:1.0">
       <establish-subscription
          xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push:1.0">
          <filter netconf:type="subtree">
             <te xmlns="urn:ietf:params:xml:ns:yang:ietf-te">
                <tunnels>
                   <tunnel>
                     <name>Tunnel1</name>
                     <identifier/>
                     <state>
                       <te-telemetry xmlns="urn:ietf:params:xml:ns:yang:
                                            ietf-te-kpi-telemetry">
                           <one-way-delay/>
                        </te-telemetry>
                     </state>
                    </tunnel>
                 </tunnels>
             </te>
          </filter>
          <period>500</period>
          <encoding>encode-xml</encoding>
       </establish-subscription>
    </netconf:rpc>

   This example shows the way for a client to subscribe to the
   telemetry information for all VNs. The telemetry parameter that the
   client is interested in is one-way-delay and one-way-utilized-
   bandwidth.

   <netconf:rpc netconf:message-id="101"
       xmlns:netconf="urn:ietf:params:xml:ns:netconf:base:1.0">
       <establish-subscription
          xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push:1.0">
          <filter netconf:type="subtree">
             <vn-state xmlns="urn:ietf:params:xml:ns:yang:ietf-vn">
                <vn>
                   <vn-list>
                     <vn-id/>
                     <vn-name/>
                     <vn-telemetry xmlns="urn:ietf:params:xml:ns:yang:
                                           ietf-vn-kpi-telemetry">
                         <one-way-delay/>
                         <one-way-utilized-bandwidth/>
                     </vn-telemetry >
                   </vn-list>
                 </vn>
             </vn-state>
          </filter>
          <period>500</period>
       </establish-subscription>
    </netconf:rpc>



Lee, et al.             Expires November 2019                 [Page 12]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019




6. YANG Data Tree


module: ietf-te-kpi-telemetry
  augment /te:te/te:tunnels/te:tunnel:
    +--rw te-scaling-intent
    |  +--rw scale-in-intent
    |  |  +--rw threshold-time?            uint32
    |  |  +--rw cooldown-time?             uint32
    |  |  +--rw scale-in-operation-type?   scaling-criteria-operation
    |  |  +--rw scaling-condition* [performance-type]
    |  |     +--rw performance-type           identityref
    |  |     +--rw threshold-value?           string
    |  |     +--rw te-telemetry-tunnel-ref?
    |  |             -> /te:te/tunnels/tunnel/name
    |  +--rw scale-out-intent
    |     +--rw threshold-time?             uint32
    |     +--rw cooldown-time?              uint32
    |     +--rw scale-out-operation-type?   scaling-criteria-operation
    |     +--rw scaling-condition* [performance-type]
    |        +--rw performance-type           identityref
    |        +--rw threshold-value?           string
    |        +--rw te-telemetry-tunnel-ref?
    |                -> /te:te/tunnels/tunnel/name
    +--ro te-telemetry
       +--ro id?                            string
       +--ro performance-metrics-one-way
       |  +--ro one-way-delay?                           uint32
       |  +--ro one-way-delay-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-residual-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-residual-bandwidth-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-available-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-available-bandwidth-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-utilized-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-utilized-bandwidth-normality?
       |          te-types:performance-metrics-normality
       +--ro performance-metrics-two-way
       |  +--ro two-way-delay?             uint32
       |  +--ro two-way-delay-normality?
       |          te-types:performance-metrics-normality
       +--ro te-ref?
               -> /te:te/tunnels/tunnel/name


module: ietf-vn-kpi-telemetry
  augment /vn:vn/vn:vn-list:


Lee, et al.             Expires November 2019                 [Page 13]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


    +--rw vn-scaling-intent
    |  +--rw scale-in-intent
    |  |  +--rw threshold-time?            uint32
    |  |  +--rw cooldown-time?             uint32
    |  |  +--rw scale-in-operation-type?   scaling-criteria-operation
    |  |  +--rw scaling-condition* [performance-type]
    |  |     +--rw performance-type           identityref
    |  |     +--rw threshold-value?           string
    |  |     +--rw te-telemetry-tunnel-ref?
    |  |             -> /te:te/tunnels/tunnel/name
    |  +--rw scale-out-intent
    |     +--rw threshold-time?             uint32
    |     +--rw cooldown-time?              uint32
    |     +--rw scale-out-operation-type?   scaling-criteria-operation
    |     +--rw scaling-condition* [performance-type]
    |        +--rw performance-type           identityref
    |        +--rw threshold-value?           string
    |        +--rw te-telemetry-tunnel-ref?
    |                -> /te:te/tunnels/tunnel/name
    +--ro vn-telemetry
       +--ro performance-metrics-one-way
       |  +--ro one-way-delay?                           uint32
       |  +--ro one-way-delay-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-residual-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-residual-bandwidth-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-available-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-available-bandwidth-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-utilized-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-utilized-bandwidth-normality?
       |          te-types:performance-metrics-normality
       +--ro performance-metrics-two-way
       |  +--ro two-way-delay?             uint32
       |  +--ro two-way-delay-normality?
       |          te-types:performance-metrics-normality
       +--ro grouping-operation?            grouping-operation
  augment /vn:vn/vn:vn-list/vn:vn-member-list:
    +--ro vn-member-telemetry
       +--ro performance-metrics-one-way
       |  +--ro one-way-delay?                           uint32
       |  +--ro one-way-delay-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-residual-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-residual-bandwidth-normality?
       |  |       te-types:performance-metrics-normality
       |  +--ro one-way-available-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-available-bandwidth-normality?
       |  |       te-types:performance-metrics-normality


Lee, et al.             Expires November 2019                 [Page 14]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


       |  +--ro one-way-utilized-bandwidth?
       |  |       rt-types:bandwidth-ieee-float32
       |  +--ro one-way-utilized-bandwidth-normality?
       |          te-types:performance-metrics-normality
       +--ro performance-metrics-two-way
       |  +--ro two-way-delay?             uint32
       |  +--ro two-way-delay-normality?
       |          te-types:performance-metrics-normality
       +--ro te-grouped-params*
       |       -> /te:te/tunnels/tunnel/te-kpi:te-telemetry/id
       +--ro grouping-operation?            grouping-operation

7. Yang Data Model

7.1. ietf-te-kpi-telemetry model



   The YANG code is as follows:

<CODE BEGINS> file "ietf-te-kpi-telemetry@2019-04-18.yang"

module ietf-te-kpi-telemetry {
  yang-version 1.1;
  namespace "urn:ietf:params:xml:ns:yang:ietf-te-kpi-telemetry";
  prefix te-tel;

  import ietf-te {
    prefix te;
   reference
     "RFC YYYY: A YANG Data Model for Traffic Engineering
      Tunnels and Interfaces";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-ietf-teas-yang-te
    becomes an RFC.*/

  import ietf-te-types {
    prefix te-types;
   reference
     "RFC YYYY: Traffic Engineering Common YANG Types";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-ietf-teas-yang-te-types
    becomes an RFC.*/



Lee, et al.             Expires November 2019                 [Page 15]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019



  organization
    "IETF Traffic Engineering Architecture and Signaling (TEAS)
     Working Group";
  contact
    "Editor: Young Lee <leeyoung@huawei.com>
     Editor: Dhruv Dhody <dhruv.ietf@gmail.com>
     Editor: Ricard Vilalta <ricard.vilalta@cttc.es>
     Editor: Satish Karunanithi <satish.karunanithi@gmail.com>";
  description
    "This module describes YANG data model for performance
    monitoring telemetry for te tunnels.

    Copyright (c) 2019 IETF Trust and the persons identified
     as authors of the code.  All rights reserved.

     Redistribution and use in source and binary forms, with
     or without modification, is permitted pursuant to, and
     subject to the license terms contained in, the Simplified
     BSD License set forth in Section 4.c of the IETF Trust's
     Legal Provisions Relating to IETF Documents
     (http://trustee.ietf.org/license-info).

    This version of this YANG module is part of RFC XXXX; see
     the RFC itself for full legal notices.";

  /* Note: The RFC Editor will replace XXXX with the number
     assigned to the RFC once draft-lee-teas-pm-telemetry-
     autonomics becomes an RFC.*/

  revision 2019-04-18 {
    description
      "Initial revision. This YANG file defines
       a YANG model for TE telemetry.";
    reference "Derived from earlier versions of base YANG files";
  }

  identity telemetry-param-type {
    description
      "Base identity for telemetry param types";
  }

  identity one-way-delay {
    base telemetry-param-type;
    description
      "To specify average Delay in one (forward)
       direction";


Lee, et al.             Expires November 2019                 [Page 16]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   reference
      "RFC7471: OSPF Traffic Engineering (TE) Metric Extensions.
       RFC8570: IS-IS Traffic Engineering (TE) Metric Extensions.
       RFC7823: Performance-Based Path Selection for Explicitly
       Routed Label Switched Paths (LSPs) Using TE Metric
       Extensions";
  }

  identity two-way-delay {
    base telemetry-param-type;
    description
      "To specify average Delay in both (forward and reverse)
       directions";
   reference
      "RFC7471: OSPF Traffic Engineering (TE) Metric Extensions.
       RFC8570: IS-IS Traffic Engineering (TE) Metric Extensions.
       RFC7823: Performance-Based Path Selection for Explicitly
       Routed Label Switched Paths (LSPs) Using TE Metric
       Extensions";
  }

  identity one-way-delay-variation {
    base telemetry-param-type;
    description
      "To specify average Delay Variation in one (forward) direction";
   reference
      "RFC7471: OSPF Traffic Engineering (TE) Metric Extensions.
       RFC8570: IS-IS Traffic Engineering (TE) Metric Extensions.
       RFC7823: Performance-Based Path Selection for Explicitly
       Routed Label Switched Paths (LSPs) Using TE Metric
       Extensions";
  }

  identity two-way-delay-variation {
    base telemetry-param-type;
    description
      "To specify average Delay Variation in both (forward and reverse)
       directions";
   reference
      "RFC7471: OSPF Traffic Engineering (TE) Metric Extensions.
       RFC8570: IS-IS Traffic Engineering (TE) Metric Extensions.
       RFC7823: Performance-Based Path Selection for Explicitly
       Routed Label Switched Paths (LSPs) Using TE Metric
       Extensions";
  }

  identity utilized-bandwidth {


Lee, et al.             Expires November 2019                 [Page 17]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


    base telemetry-param-type;
    description
      "To specify utilized bandwidth over the specified source
       and destination.";
   reference
      "RFC7471: OSPF Traffic Engineering (TE) Metric Extensions.
       RFC8570: IS-IS Traffic Engineering (TE) Metric Extensions.
       RFC7823: Performance-Based Path Selection for Explicitly
       Routed Label Switched Paths (LSPs) Using TE Metric
       Extensions";
  }

  identity utilized-percentage {
    base telemetry-param-type;
    description
      "To specify utilization percentage of the entity
       (e.g., tunnel, link, etc.)";
  }

  typedef scaling-criteria-operation {
    type enumeration {
      enum AND {
        description
          "AND operation";
      }
      enum OR {
        description
          "OR operation";
      }
    }
    description
      "Operations to analize list of scaling criterias";
  }

  grouping scaling-duration {
    description
      "Base scaling criteria durations";
    leaf threshold-time {
      type uint32;
      units "seconds";
      description
        "The duration for which the criteria must hold true";
    }
    leaf cooldown-time {
      type uint32;
      units "seconds";
      description


Lee, et al.             Expires November 2019                 [Page 18]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


        "The duration after a scaling-in/scaling-out action has been
         triggered, for which there will be no further operation";
    }
  }

  grouping scaling-criteria {
    description
      "Grouping for scaling criteria";
    leaf performance-type {
      type identityref {
        base telemetry-param-type;
      }
      description
        "Reference to the tunnel level telemetry type";
    }
    leaf threshold-value {
      type string;
      description
        "Scaling threshold for the telemetry parameter type";
    }
    leaf te-telemetry-tunnel-ref {
      type leafref {
        path "/te:te/te:tunnels/te:tunnel/te:name";
      }
      description
        "Reference to tunnel";
    }
  }

  grouping scaling-in-intent {
    description
      "Basic scaling in intent";
    uses scaling-duration;
    leaf scale-in-operation-type {
      type scaling-criteria-operation;
      default "AND";
      description
        "Operation to be applied to check between
         scaling criterias to check if the scale in
         threshold condition has been met.
         Defaults to AND";
    }
    list scaling-condition {
      key "performance-type";
      description
        "Scaling conditions";
      uses scaling-criteria;


Lee, et al.             Expires November 2019                 [Page 19]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


    }
  }

  grouping scaling-out-intent {
    description
      "Basic scaling out intent";
    uses scaling-duration;
    leaf scale-out-operation-type {
      type scaling-criteria-operation;
      default "OR";
      description
        "Operation to be applied to check between
         scaling criterias to check if the scale out
         threshold condition has been met.
         Defauls to OR";
    }
    list scaling-condition {
      key "performance-type";
      description
        "Scaling conditions";
      uses scaling-criteria;
    }
  }

  augment "/te:te/te:tunnels/te:tunnel" {
    description
      "Augmentation parameters for config scaling-criteria
       TE tunnel topologies. Scale in/out criteria might be used
       for network autonomics in order the controller
       to react to a certain set of monitored params.";
    container te-scaling-intent {
      description
        "scaling intent";
      container scale-in-intent {
        description
          "scale-in";
        uses scaling-in-intent;
      }
      container scale-out-intent {
        description
          "scale-out";
        uses scaling-out-intent;
      }
    }
    container te-telemetry {
      config false;
      description


Lee, et al.             Expires November 2019                 [Page 20]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


        "telemetry params";
      leaf id {
        type string;
        description
          "Id of telemetry param";
      }
      uses te-types:performance-metrics-attributes;
      leaf te-ref {
        type leafref {
          path "/te:te/te:tunnels/te:tunnel/te:name";
        }
        description
          "Reference to measured te tunnel";
      }
    }
  }
}
<CODE ENDS>


7.2. ietf-vn-kpi-telemetry model

   The YANG code is as follows:



<CODE BEGINS> file "ietf-vn-kpi-telemetry@2019-04-18.yang"

module ietf-vn-kpi-telemetry {
  yang-version 1.1;
  namespace "urn:ietf:params:xml:ns:yang:ietf-vn-kpi-telemetry";
  prefix vn-tel;

  import ietf-vn {
    prefix vn;
   reference
     "RFC YYYY: A YANG Data Model for VN Operation";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-ietf-teas-actn-vn-yang
    becomes an RFC.*/

  import ietf-te {
    prefix te;
   reference
     "RFC YYYY: A YANG Data Model for Traffic Engineering


Lee, et al.             Expires November 2019                 [Page 21]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


      Tunnels and Interfaces";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-ietf-teas-yang-te
    becomes an RFC.*/

  import ietf-te-types {
    prefix te-types;
   reference
     "RFC YYYY: Traffic Engineering Common YANG Types";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-ietf-teas-yang-te-types
    becomes an RFC.*/

  import ietf-te-kpi-telemetry {
    prefix te-kpi;
   reference
     "RFC YYYY: YANG models for VN & TE Performance Monitoring
      Telemetry and Scaling Intent Autonomics";
  }

  /* Note: The RFC Editor will replace YYYY with the number
     assigned to the RFC once draft-lee-teas-actn-pm-telemetry
    -autonomics becomes an RFC.*/

  organization
    "IETF Traffic Engineering Architecture and Signaling (TEAS)
     Working Group";
  contact
    "Editor: Young Lee <leeyoung@huawei.com>
     Editor: Dhruv Dhody <dhruv.ietf@gmail.com>
     Editor: Ricard Vilalta <ricard.vilalta@cttc.es>
     Editor: Satish Karunanithi <satish.karunanithi@gmail.com>";

  description
    "This module describes YANG data models for performance
    monitoring telemetry for vn.

    Copyright (c) 2019 IETF Trust and the persons identified
     as authors of the code.  All rights reserved.

     Redistribution and use in source and binary forms, with
     or without modification, is permitted pursuant to, and
     subject to the license terms contained in, the Simplified


Lee, et al.             Expires November 2019                 [Page 22]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


     BSD License set forth in Section 4.c of the IETF Trust's
     Legal Provisions Relating to IETF Documents
     (http://trustee.ietf.org/license-info).

    This version of this YANG module is part of RFC XXXX; see
     the RFC itself for full legal notices.";

  /* Note: The RFC Editor will replace XXXX with the number
     assigned to the RFC once draft-lee-teas-pm-telemetry-
     autonomics becomes an RFC.*/

  revision 2019-04-18 {
    description
      "Initial revision. This YANG file defines
       the VN telemetry.";
    reference "Derived from earlier versions of base YANG files";
  }

  typedef grouping-operation {
    type enumeration {
      enum MINIMUM {
        description
          "Select the minimum param";
      }
      enum MAXIMUM {
        description
          "Select the maximum param";
      }
      enum MEAN {
        description
          "Select the MEAN of the params";
      }
      enum STD_DEV {
        description
          "Select the standard deviation of the
           monitored params";
      }
      enum AND {
        description
          "Select the AND of the params";
      }
      enum OR {
        description
          "Select the OR of the params";
      }
    }
    description


Lee, et al.             Expires November 2019                 [Page 23]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


      "Operations to analize list of monitored params";
  }

  grouping vn-telemetry-param {
    description
      "augment of te-kpi:telemetry-param for VN specific params";
    leaf-list te-grouped-params {
      type leafref {
        path "/te:te/te:tunnels/te:tunnel/te-kpi:te-telemetry/te-kpi:id";
      }
      description
        "Allows the definition of a vn-telemetry param
         as a grouping of underlying TE params";
    }
    leaf grouping-operation {
      type grouping-operation;
      description
        "describes the operation to apply to
         te-grouped-params";
    }
  }

  augment "/vn:vn/vn:vn-list" {
    description
      "Augmentation parameters for state TE VN topologies.";
    container vn-scaling-intent {
      description
        "scaling intent";
      container scale-in-intent {
        description
          "VN scale-in";
        uses te-kpi:scaling-in-intent;
      }
      container scale-out-intent {
        description
          "VN scale-out";
        uses te-kpi:scaling-out-intent;
      }
    }
    container vn-telemetry {
      config false;
      description
        "VN telemetry params";
      uses te-types:performance-metrics-attributes;
      leaf grouping-operation {
        type grouping-operation;
        description


Lee, et al.             Expires November 2019                 [Page 24]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


          "describes the operation to apply to the VN-members";
      }
    }
  }
  augment "/vn:vn/vn:vn-list/vn:vn-member-list" {
    description
      "Augmentation parameters for state TE vn member topologies.";
    container vn-member-telemetry {
      config false;
      description
        "VN member telemetry params";
      uses te-types:performance-metrics-attributes;
      uses vn-telemetry-param;
    }
  }
}
<CODE ENDS>


8. Security Considerations

   The YANG module specified in this document defines a schema for data
   that is designed to be accessed via network management protocols
   such as NETCONF [RFC6241] or RESTCONF [RFC8040]. The lowest NETCONF
   layer is the secure transport layer, and the mandatory-to-implement
   secure transport is Secure Shell (SSH) [RFC6242]. The lowest
   RESTCONF layer is HTTPS, and the mandatory-to-implement secure
   transport is TLS [RFC8446].

   The NETCONF access control model [RFC8341] provides the means to
   restrict access for particular NETCONF users to a preconfigured
   subset of all available NETCONF protocol operations and content. The
   NETCONF Protocol over Secure Shell (SSH) [RFC6242] describes a
   method for invoking and running NETCONF within a Secure Shell (SSH)
   session as an SSH subsystem. The Network Configuration Access
   Control Model (NACM) [RFC8341] provides the means to restrict access
   for particular NETCONF or RESTCONF users to a preconfigured subset
   of all available NETCONF or RESTCONF protocol operations and
   content.

   A number of configuration data nodes defined in this document are
   writable/deletable (i.e., "config true"). These data nodes may be
   considered sensitive or vulnerable in some network environments.

   There are a number of data nodes defined in this YANG module that
   are writable/creatable/deletable (i.e., config true, which is the
   default).  These data nodes may be considered sensitive or


Lee, et al.             Expires November 2019                 [Page 25]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   vulnerable in some network environments.  Write operations (e.g.,
   edit-config) to these data nodes without proper protection can have
   a negative effect on network operations.  These are the subtrees and
   data nodes and their sensitivity/vulnerability:

    /te:te/te:tunnels/te:tunnel/te-scaling-intent/scale-in-intent
   /te:te/te:tunnels/te:tunnel/te-scaling-intent/scale-out-intent

   /vn:vn/vn:vn-list/vn-scaling-intent/scale-in-intent
   /vn:vn/vn:vn-list/vn-scaling-intent/scale-out-intent





9. IANA Considerations

   This document registers the following namespace URIs in the IETF XML
   registry [RFC3688]:

   --------------------------------------------------------------------
   URI: urn:ietf:params:xml:ns:yang:ietf-te-kpi-telemetry
   Registrant Contact: The IESG.
   XML: N/A, the requested URI is an XML namespace.
   --------------------------------------------------------------------

   --------------------------------------------------------------------
   URI: urn:ietf:params:xml:ns:yang:ietf-vn-kpi-telemetry
   Registrant Contact: The IESG.
   XML: N/A, the requested URI is an XML namespace.
   --------------------------------------------------------------------


   This document registers the following YANG modules in the YANG
   Module.

   Names registry [RFC7950]:

   --------------------------------------------------------------------
   name:         ietf-te-kpi-telemetry
   namespace:    urn:ietf:params:xml:ns:yang:ietf-te-kpi-telemetry
   prefix:       te-tel
   reference:    RFC XXXX (TDB)


Lee, et al.             Expires November 2019                 [Page 26]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   --------------------------------------------------------------------

   --------------------------------------------------------------------
   name:         ietf-vn-kpi-telemetry
   namespace:    urn:ietf:params:xml:ns:yang:ietf-vn-kpi-telemetry
   prefix:       vn-tel
   reference:    RFC XXXX (TDB)
   --------------------------------------------------------------------

10. Acknowledgements

   We thank Rakesh Gandhi, Tarek Saad and Igor Bryskin for useful
   discussions and their suggestions for this work.

11. References

11.1. Normative References

   [RFC6242] M. Wasserman, "Using the NETCONF Protocol over Secure
             Shell (SSH)", RFC 6242, June 2011.

   [RFC7926] A. Farrel (Ed.), "Problem Statement and Architecture for
             Information Exchange between Interconnected Traffic-
             Engineered Networks", RFC 7926, July 2016.

   [RFC7950] M. Bjorklund, Ed., "The YANG 1.1 Data Modeling Language",
             August 2016.

   [RFC8040] A. Bierman, M. Bjorklund, K. Watsen, "RESTCONF Protocol",
             RFC 8040, January 2017.

   [RFC8233] D. Dhody, et al., "Extensions to the Path Computation
             Element Communication Protocol (PCEP) to compute service
             aware Label Switched Path (LSP)", RFC 8233, September
             2017.

   [RFC8341] A. Bierman, M. Bjorklund, "Network Configuration Access
             Control Model", RFC 8341, March 2018.

   [RFC8342] M. Bjorklund, J. Schoenwaelder, P. Shafer, K. Watsen, R.
             Wilton, "Network Management Datastore Architecture
             (NMDA)", RFC 8342, March 2018.





Lee, et al.             Expires November 2019                 [Page 27]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   [RFC8446] E. Rescorla, "The Transport Layer Security (TLS) Protocol
             Version 1.3", RFC8446, August 2018.

   [TE-Tunnel] T. Saad (Editor), "A YANG Data Model for Traffic
             Engineering Tunnels and Interfaces", draft-ietf-teas-yang-
             te, work in progress.

   [TE-Types] T. Saad, et.al, "Traffic Engineering Common YANG Types",
             draft-ietf-teas-yang-te-types, work in progress.

   [VN] Y. Lee (Editor), "A Yang Data Model for ACTN VN Operation",
             draft-lee-teas-actn-vn-yang, work in progress.

11.2. Informative References

   [RFC3688] M. Mealling, "The IETF XML Registry", RFC 3688, January
             2004.

   [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
             and A. Bierman, Ed., "Network Configuration Protocol
             (NETCONF)", RFC 6241.

   [RFC7471]  Giacalone, S., Ward, D., Drake, J., Atlas, A., and S.
              Previdi, "OSPF Traffic Engineering (TE) Metric
              Extensions", RFC 7471, March 2015.

   [RFC7823]  Atlas, A., Drake, J., Giacalone, S., and S.
             Previdi,"Performance-Based Path Selection for Explicitly
             Routed Label Switched Paths (LSPs) Using TE Metric
             Extensions", RFC 7823, May 2016.

   [RFC8294] X. Liu, et al, "Routing Area Common YANG Data Types", RFC
             8294, December 2017.

   [RFC8340] M. Bjorklund and L. Berger (Editors), "YANG Tree
             Diagrams", RFC 8340, March 2018.

   [YANG-PUSH] A. Clemm, et al, "Subscription to YANG Datastores",
             draft-ietf-netconf-yang-push, work in progress.

   [Event-Notification] E. Voit, et al, "Dynamic subscription to YANG
             Events and Datastores over NETCONF", draft-ietf-netconf-
             netconf-event-notifications, work in progress.






Lee, et al.             Expires November 2019                 [Page 28]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019


   [PERF] Y. XU, et al., "Use Cases and Requirements of Dynamic Service
             Control based on Performance Monitoring in ACTN
             Architecture", draft-xu-actn-perf-dynamic-service-control,
             work in progress.

   [RFC8309] Q. Wu, W. Cheng, and A. Farrel. "Service Models
             Explained", RFC 8309, January 2018.

   [RFC8453] D. Ceccarelli and Y. Lee (Editors), "Framework for
             Abstraction and Control of Traffic Engineered Networks",
             RFC 8453, August 2018.

   [RFC8570]  Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward,
             D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE)
             Metric Extensions", RFC 8570, March 2019.



12. Contributors


Authors' Addresses

   Young Lee
   Huawei Technologies
   5340 Legacy Drive Suite 173
   Plano, TX 75024, USA

   Email: leeyoung@huawei.com

   Dhruv Dhody
   Huawei Technology
   Leela Palace
   Bangalore, Karnataka 560008
   India

   Email: dhruv.dhody@huawei.com

   Satish Karunanithi
   Huawei Technology
   Leela Palace
   Bangalore, Karnataka 560008
   India

   Email: satish.karunanithi@gmail.com


Lee, et al.             Expires November 2019                 [Page 29]

Internet-Draft VN & TE PM Telemetry & Network Autonomics       May 2019



   Ricard Vilalta
   Centre Tecnologic de Telecomunicacions de Catalunya (CTTC/CERCA)
   Av. Carl Friedrich Gauss 7
   08860 - Castelldefels
   Barcelona (Spain)
   Email: ricard.vilalta@cttc.es



   Daniel King
   Lancaster University

   Email: d.king@lancaster.ac.uk


   Daniele Ceccarelli
   Ericsson
   Torshamnsgatan,48
   Stockholm, Sweden

   Email: daniele.ceccarelli@ericsson.com

























Lee, et al.             Expires November 2019                 [Page 30]