Message Sections for Each Message Type

Section	Cap.	Spec.	Result	Receipt	Envelope
Verb	req	req	req.	req.
Content Type					req
version	req	req	req	req	req
registry	req	req	req	opt
label	opt	opt	opt	opt	opt
when	req	req	req	req
parameters	req/tok	req	req	opt/tok
metadata	opt/tok	opt	opt	opt/tok
results	req/tok	req	req	opt/tok
resultvalues			req
export	opt	opt	opt	opt
link	opt	opt
token	opt	opt	opt	opt	opt
contents					req

Withdrawals take the same sections as capabilities; and redemptions and interrupts take the same sections as receipts. Exceptions are not shown in this table.

4.3.1. Message Type and Verb

The verb is the action to be performed by the component. The following verbs are supported by the base mPlane protocol, but arbitrary verbs may be specified by applications:

• measure: Perform a measurement
• query: Query a database about a past measurement
• collect: Receive results via indirect export
• callback: Used for callback control in component-initiated workflows

In the JSON representation of mPlane messages, the verb is the value of the key corresponding to the message’s type, represented as a lowercase string (e.g. capability, specification, result and so on).

Roughly speaking, probes implement measure capabilities, and repositories implement query and collect capabilities. Of course, any single component can implement capabilities with any number of different verbs.

Within the SDK, the primary difference between measure and query is that the temporal scope of a measure specification is taken to refer to when the measurement should be scheduled, while the temporal scope of a query specification is taken to refer to the time window (in the past) of a query.

Envelopes have no verb; instead, the value of the envelope key is the kind of messages the envelope contains, or message if the envelope contains a mixture of different unspecified kinds of messages.

4.3.2. Version

The version section contains the version of the mPlane protocol to which the message conforms, as an integer serially incremented with each new protocol revision. This section is required in all messages. This document describes version 2 of the protocol.

4.3.3. Registry

The registry section contains the URL identifying the element registry used by this message, and from which the registry can be retrieved. This section is required in all messages containing element names (statements, and receipts/redemptions/interrupts not using tokens for identification; see the token section).

4.3.4. Label

The label section of a statement contains a human-readable label identifying it, intended solely for use when displaying information about messages in user interfaces. Results, receipts, redemptions, and interrupts inherit their label from the specification from which they follow; otherwise, client and component software can arbitrarily assign labels . The use of labels is optional in all messages, but as labels do greatly ease human-readability of arbitrary messages within user interfaces, their use is recommended.

mPlane clients and components should never use the label as a unique identifier for a message, or assume any semantic meaning in the label – the test of message equality and relatedness is always based upon the schema and values as in Section 4.4.

4.3.5. Temporal Scope (When)

The when section of a statement contains its temporal scope.

A temporal scope refers to when a measurement can be run (in a capability), when it should be run (in a specification), or when it was run (in a result). Temporal scopes can be either absolute or relative, and may have an optional period, referring to how often single measurements should be taken.

The general form of a temporal scope (in BNF-like syntax) is as follows:

when = <singleton> |                # A single point in time
           <range> |                # A range in time
         <range> ' / ' <duration>   # A range with a period

singleton = <iso8601> | # absolute singleton
            'now'       # relative singleton

range = <iso8601> ' ... ' <iso8601> | # absolute range
        <iso8601> ' + ' <duration> |  # relative range
        'now' ' ... ' <iso8061> |     # definite future
        'now' ' + ' <duration> |      # relative future
        <iso8601> ' ... ' 'now' |     # definite past
        'past ... now' |              # indefinite past
        'now ... future' |            # indefinite future
        <iso8601> ' ... ' 'future' |  # absolute indefinite future
        'past ... future' |           # forever

duration = [ <n> 'd' ] # days
           [ <n> 'h' ] # hours
           [ <n> 'm' ] # minute
           [ <n> 's' ] # seconds

iso8601 = <n> '-' <n> '-' <n> [' ' <n> ':' <n> ':' <n> [ '.' <n> ]]

All absolute times are always given in UTC and expressed in ISO8601 format with variable precision.

In capabilities, if a period is given it represents the minimum period supported by the measurement; this is done to allow rate limiting. If no period is given, the measurement is not periodic. A capability with a period can only be fulfilled by a specification with period greater than or equal to the period in the capability. Conversely, a capability without a period can only be fulfilled by a specification without a period.

Within a result, only absolute ranges are allowed within the temporal scope, and refers to the time range of the measurements contributing to the result. Note that the use of absolute times here implies that the components and clients within a domain should have relatively well-synchronized clocks, e.g., to be synchronized using the Network Time Protocol [RFC5905] in order for results to be temporally meaningful.

So, for example, an absolute range in time might be expressed as:

when: 2009-02-20 13:02:15 ... 2014-04-04 04:27:19

A relative range covering three and a half days might be:

when: 2009-04-04 04:00:00 + 3d12h

In a specification for running an immediate measurement for three hours every seven and a half minutes:

when: now + 3h / 7m30s

In a capability noting that a Repository can answer questions about the past:

when: past ... now.

In a specification requesting that a measurement run from a specified point in time until interrupted:

when: 2017-11-23 18:30:00 ... future

4.3.6. Parameters

The parameters section of a message contains an ordered list of the parameters for a given measurement: values which must be provided by a client to a component in a specification to convey the specifics of the measurement to perform. Each parameter in an mPlane message is a key-value pair, where the key is the name of an element from the element registry. In specifications and results, the value is the value of the parameter. In capabilities, the value is a constraint on the possible values the component will accept for the parameter in a subsequent specification.

Four kinds of constraints are currently supported for mPlane parameters:

• No constraint: all values are allowed. This is signified by the special constraint string ‘*’.
• Single value constraint: only a single value is allowed. This is intended for use for capabilities which are conceivably configurable, but for which a given component only supports a single value for a given parameter due to its own out-of-band configuration or the permissions of the client for which the capability is valid. For example, the source address of an active measurement of a single-homed probe might be given as ‘source.ip4: 192.0.2.19’.
• Set constraint: multiple values are allowed, and are explicitly listed, separated by the ‘,’ character. For example, a multi-homed probe allowing two potential source addresses on two different networks might be given as ‘source.ip4: 192.0.2.19, 192.0.3.21’.
• Range constraint: multiple values are allowed, between two ordered values, separated by the special string ‘...’. Range constraints are inclusive. A measurement allowing a restricted range of source ports might be expressed as ‘source.port: 32768 ... 65535’
• Prefix constraint: multiple values are allowed within a single network, as specified by a network address and a prefix. A prefix constraint may be satisfied by any network of host address completely contained within the prefix. An example allowing probing of any host within a given /24 might be ‘destination.ip4: 192.0.2.0/24’.

Parameter and constraint values must be a representation of an instance of the primitive type of the associated element.

4.3.7. Metadata

The metadata section contains measurement metadata: key-value pairs associated with a capability inherited by its specification and results. Metadata can also be thought of as immutable parameters. This is intended to represent information which can be used to make decisions at the client as to the applicability of a given capability (e.g. details of algorithms used or implementation-specific information) as well as to make adjustments at post-measurement analysis time when contained within results.

An example metadata element might be ‘measurement.identifier: qof’, which identifies the underlying tool taking measurements, such that later analysis can correct for known peculiarities in the implementation of the tool. Another example might be ‘location.longitude = 8.55272’, which while not particularly useful for analysis purposes, can be used to draw maps of measurements.

4.3.8. Result Columns and Values

Results are represented using two sections: results which identify the elements to be returned by the measurement, and resultvalues which contains the actual values. results appear in all statements, while resultvalues appear only in result messages.

The results section contains an ordered list of result columns for a given measurement: names of elements which will be returned by the measurement. The result columns are identified by the names of the elements from the element registry.

The resultvalues section contains an ordered list of ordered lists (or, rather, a two dimensional array) of values of results for a given measurement, in row-major order. The columns in the result values appear in the same order as the columns in the results section.

Values for each column must be a representation of an instance of the primitive type of the associated result column element.

4.3.9. Export

The export section contains a URL or partial URL for indirect export. Its meaning depends on the kind and verb of the message:

• For capabilities with the collect verb, the export section contains the URL of the collector which can accept indirect export for the schema defined by the parameters and results sections of the capability, using the protocol identified by the URL’s schema.
• For capabilities with any verb other than collect, the export section contains either the URL of a collector to which the component can indirectly export results, or a URL schema identifying a protocol over which the component can export to arbitrary collectors.
• For specifications with any verb other than collect, the export section contains the URL of a collector to which the component should indirectly export results. A receipt will be returned for such specifications.

If a component can indirectly export or indirectly collect using multiple protocols, each of those protocols must be identified by its own capability; capabilities with an export section can only be used by specifications with a matching export section.

4.3.10. Link

The link section contains the URL to which messages in the next step in the workflow (i.e. a specification for a capability, a result or receipt for a specification) can be sent, providing indirection. The link URL must currently have the schema wss, and refers to the URL to which to initiate a connection.

If present in a capability, the client must send specifications for the given capability to the component at the URL given in order to use the capability, connecting to the URL if no connection is currently established. If present in a specification, the component must send results for the given specification back to the client at the URL given, connecting to the URL if no connection is currently established.

4.3.11. Token

The token section contains an arbitrary string by which a message may be identified in subsequent communications in an abbreviated fashion. Unlike labels, tokens are not necessarily intended to be human-readable; instead, they provide a way to reduce redundancy on the wire by replacing the parameters, metadata, and results sections in messages within a workflow, at the expense of requiring more state at clients and components. Their use is optional.

Tokens are scoped to the association between the component and client in which they are first created; i.e., at a component, the token will be associated with the client’s identity, and vice-versa at a client. Tokens should be created with sufficient entropy to avoid collision from independent processes at the same client or token reuse in the case of client or component state loss at restart.

If a capability contains a token, it may be subsequently withdrawn by the same component using a withdrawal containing the token instead of the parameters, metadata, and results sections.

If a specification contains a token, it may be answered by the component with a receipt containing the token instead of the parameters, metadata, and results sections. A specification containing a token may likewise be interrupted by the client with an interrupt containing the token. A component must not answer a specification with a token with a receipt or result containing a different token, but the token may be omitted in subsequent receipts and results.

If a receipt contains a token, it may be redeemed by the same client using a redemption containing the token instead of the parameters, metadata, and results sections.

When grouping multiple results from a repeating specification into an envelope, the envelope may contain the token of the repeating specification.

4.3.12. Contents

The contents section appears only in envelopes, and is an ordered list of messages. If the envelope’s kind identifies a message kind, the contents may contain only messages of the specified kind, otherwise if the kind is message, the contents may contain a mix of any kind of message.

4.4. Message Uniqueness and Idempotence

Messages in the mPlane protocol are intended to support state distribution: capabilities, specifications, and results are meant to be complete declarations of the state of a given measurement. In order for this to hold, it must be possible for messages to be uniquely identifiable, such that duplicate messages can be recognized. With one important exception (i.e., specifications with relative temporal scopes), messages are idempotent: the receipt of a duplicate message at a client or component is a null operation.

4.4.1. Message Schema

The combination of elements in the parameters and results sections, together with the registry from which these elements are drawn, is referred to as a message’s schema. The schema of a measurement can be loosely thought of as the definition of the table, rows of which the message represents.

The schema contributes not only to the identity of a message, but also to the semantic interpretation of the parameter and result values. The meanings of element values in mPlane are dependent on the other elements present in the message; in other words, the key to interpreting an mPlane message is that the unit of semantic identity is a message. For example, the element ‘destination.ip4’ as a parameter means “the target of a given active measurement” when together with elements describing an active metric (e.g. ‘delay.twoway.icmp.us’), but “the destination of the packets in a flow” when together with other elements in result columns describing a passively-observed flow.

The interpretation of the semantics of an entire message is application-specific. The protocol does not forbid the transmission of messages representing semantically meaningless or ambiguous schemas.

4.4.2. Message Identity

A message’s identity is composed of its schema, together with its temporal scope, metadata, parameter values, and indirect export properties. Concretely, the full content of the registry, when, parameters, metadata, results, and export sections taken together comprise the message’s identity.

One convenience feature complicates this somewhat: when the temporal scope is not absolute, multiple specifications may have the same literal temporal scope but refer to different measurements. In this case, the current time at the client or component when a message is invoked must be taken as part of the message’s identity as well. Implementations may use hashes over the values of the message’s identity sections to uniquely identify messages; e.g. to generate message tokens.

4.5. Designing Measurement and Repository Schemas

As noted, the key to integrating a measurement tool into an mPlane infrastructure is properly defining the schemas for the measurements and queries it performs, then defining those schemas in terms of mPlane capabilities. Specifications and results follow naturally from capabilities, and the mPlane SDK allows Python methods to be bound to capabilities in order to execute them. A schema should be defined such that the set of parameters, the set of result columns, and the verb together naturally and uniquely define the measurement or the query being performed. For simple metrics, this is achieved by encoding the entire meaning of the metric in its name. For example, delay.twoway.icmp.us as a result column together with source.ip4 and destination.ip4 as parameters uniquely defines a single ping measurement, measured via ICMP, expressed in microseconds.

Aggregate measurements are defined by returning metrics with aggregations: delay.twoway.icmp.min.us, delay.twoway.icmp.max.us, delay.twoway.icmp.mean.us, and delay.twoway.icmp.count.us as result columns represent aggregate ping measurements with multiple samples.

Note that mPlane results may contain multiple rows. In this case, the parameter values in the result, taken from the specification, apply to all rows. In this case, the rows are generally differentiated by the values in one or more result columns; for example, the time element can be used to represent time series, or the hops.ip different elements along a path between source and destination, as in a traceroute measurement.

For measurements taken instantaneously, the verb measure should be used; for direct queries from repositories, the verb query should be used. Other actions that cannot be differentiated by schema alone should be differentiated by a custom verb.

When integrating a repository into an mPlane infrastructure, only a subset of the queries the repository can perform will generally be exposed via the mPlane interface. Consider a generic repository which provides an SQL interface for querying data; wrapping the entire set of possible queries in specific capabilities would be impossible, while providing direct access to the underlying SQL (for instance, by creating a custom registry with a query.sql string element to be used as a parameter) would make it impossible to differentiate capabilities by schema (thereby making the interoperability benefits of mPlane integration pointless). Instead, specific queries to be used by clients in concert with capabilities provided by other components are each wrapped within a separate capability, analogous to stored procedure programming in many database engines. Of course, clients which do speak the precise dialect of SQL necessary can integrate directly with the repository separate from the capabilities provided over mPlane.

5. Representations and Session Protocols

The mPlane protocol is built atop an abstract data model in order to support multiple representations and session protocols. The canonical representation supported by the present SDK involves JSON [RFC7159] objects transported via Websockets [RFC6455] over TLS [RFC5246] (known by the “wss” URL schema).

5.1. JSON representation

In the JSON representation, an mPlane message is a JSON object, mapping sections by name to their contents. The name of the message type is a special section key, which maps to the message’s verb, or to the message’s content type in the case of an envelope.

Each section name key in the object has a value represented in JSON as follows:

• version : an integer identifying the mPlane protocol version used by the message.
• registry : a URL identifying the registry from which element names are taken.
• label : an arbitrary string.
• when : a string containing a temporal scope, as described in Section 4.3.5.
• parameters : a JSON object mapping (non-qualified) element names, either to constraints or to parameter values, as appropriate, and as described in Section 4.3.6.
• metadata : a JSON object mapping (non-qualified) element names to metadata values.
• results : an array of element names.
• resultvalues : an array of arrays of element values in row major order, where each row array contains values in the same order as the element names in the results section.
• export : a URL for indirect export.
• link : a URL for message indirection.
• token : an arbitrary string.
• contents : an array of objects containing messages.

5.1.1. Textual representations of element values

Each primitive type is represented as a value in JSON as follows, following the Textual Representation of IPFIX Abstract Data Types defined in [RFC7373].

Natural and real values are represented in JSON using native JSON representation for numbers.

Booleans are represented by the reserved words true and false.

Strings and URLs are represented as JSON strings, subject to JSON escaping rules.

Addresses are represented as dotted quads for IPv4 addresses as they would be in URLs, and canonical IPv6 textual addresses as in section 2.2 of [RFC4291] as updated by section 4 of [RFC5952]. When representing networks, addresses may be suffixed as in CIDR notation, with a ‘/’ character followed by the mask length in bits n, provided that the least significant 32 − n or 128 − n bits of the address are zero, for IPv4 and IPv6 respectively.

Timestamps are represented in [RFC3339] and ISO 8601, with two important differences. First, all mPlane timestamps are are expressed in terms of UTC, so time zone offsets are neither required nor supported, and are always taken to be 0. Second, fractional seconds are represented with a variable number of digits after an optional decimal point after the fraction.

Objects are represented as JSON objects.

5.1.2. Example mPlane Capabilities and Specifications in JSON

To illustrate how mPlane messages are encoded, we consider first two capabilities for a very simple application – ping – as mPlane JSON capabilities. The following capability states that the component can measure ICMP two-way delay from 192.0.2.19 to anywhere on the IPv4 Internet, with a minimum delay between individual pings of 1 second, returning aggregate statistics:

{
  "capability": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "ping-aggregate",
  "when":       "now ... future / 1s",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "*"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"]
}

In contrast, the following capability would return timestamped singleton delay measurements given the same parameters:

{
  "capability": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "ping-singletons",
  "when":       "now ... future / 1s",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "*"},
  "results":    ["time",
                 "delay.twoway.icmp.us"]
}

A specification is merely a capability with filled-in parameters, e.g.:

{
  "specification":  "measure",
  "version":        0,
  "registry":       "https://example.com/mplane/registry/core",
  "label":          "ping-aggregate-three-thirtythree",
  "token":          "0f31c9033f8fce0c9be41d4942c276e4",
  "when":           "now + 30s / 1s",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "192.0.3.33"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"]
}

Results are merely specifications with result values filled in and an absolute temporal scope:

{
  "result":   "measure",
  "version":  0,
  "registry": "https://example.com/mplane/registry/core",
  "label":    "ping-aggregate-three-thirtythree",
  "token":    "0f31c9033f8fce0c9be41d4942c276e4",
  "when":     2014-08-25 14:51:02.623 ... 2014-08-25 14:51:32.701 / 1s",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "192.0.3.33"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"],
  "resultvalues": [ [ 23901,
                      29833,
                      27619,
                      66002,
                      30] ]
}

More complex measurements can be modeled by mapping them back to tables with multiple rows. For example, a traceroute capability would be defined as follows:

{
  "capability": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "traceroute",
  "when":       "now ... future / 1s",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "*",
                 "hops.ip.max": "0..32"},
  "results":    ["time",
                 "intermediate.ip4",
                 "hops.ip",
                 "delay.twoway.icmp.us"]
}

with a corresponding specification:

{
  "specification": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "traceroute-three-thirtythree",
  "token":      "2f4123588b276470b3641297ae85376a",
  "when":       "now",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "192.0.3.33",
                 "hops.ip.max": 32},
  "results":    ["time",
                 "intermediate.ip4",
                 "hops.ip",
                 "delay.twoway.icmp.us"]
}

and an example result:

{
  "result": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "traceroute-three-thirtythree",
  "token":      "2f4123588b276470b3641297ae85376a,
  "when":       "2014-08-25 14:53:11.019 ... 2014-08-25 14:53:12.765",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "192.0.3.33",
                 "hops.ip.max": 32},
  "results":    ["time",
                 "intermediate.ip4",
                 "hops.ip",
                 "delay.twoway.icmp.us"],
  "resultvalues": [ [ "2014-08-25 14:53:11.019", "192.0.2.1",
                      1, 162 ],
                    [ "2014-08-25 14:53:11.220", "217.147.223.101",
                      2, 15074 ],
                    [ "2014-08-25 14:53:11.570", "77.109.135.193",  
                      3, 30093 ],
                    [ "2014-08-25 14:53:12.091", "77.109.135.34",
                      4, 34979 ],
                    [ "2014-08-25 14:53:12.310", "192.0.3.1",
                      5, 36120 ],
                    [ "2014-08-25 14:53:12.765", "192.0.3.33",
                      6, 36202 ]
                  ]

}

Indirect export to a repository with subsequent query requires three capabilities: one in which the repository advertises its ability to accept data over a given external protocol, one in which the probe advertises its ability to export data of the same type using that protocol, and one in which the repository advertises its ability to answer queries about the stored data. Returning to the aggregate ping measurement, first let’s consider a repository which can accept these measurements via direct POST of mPlane result messages:

{
  "capability": "collect",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "ping-aggregate-collect",
  "when":       "past ... future",
  "export":     "wss://repository.example.com:4343/",
  "parameters": {"source.ip4":      "*",
                 "destination.ip4": "*"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"]
}

This capability states that the repository at wss://repository.example.com:4343/ will accept mPlane result messages matching the specified schema, without any limitations on time. Note that this schema matches that of the export capability provided by the probe:

{
  "capability": "measure",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "ping-aggregate-export",
  "when":       "now ... future / 1s",
  "export":     "wss",
  "parameters": {"source.ip4":      "192.0.2.19",
                 "destination.ip4": "*"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"]
}

which differs only from the previous probe capability in that it states that results can be exported via the wss protocol. Subsequent queries can be sent to the repository in response to the query capability:

{
  "capability": "query",
  "version":    0,
  "registry":   "https://example.com/mplane/registry/core",
  "label":      "ping-aggregate-query",
  "when":       "past ... future",
  "link":       "wss://repo.example.com:4343/",
  "parameters": {"source.ip4":      "*",
                 "destination.ip4": "*"},
  "results":    ["delay.twoway.icmp.us.min",
                 "delay.twoway.icmp.us.mean",
                 "delay.twoway.icmp.us.50pct",
                 "delay.twoway.icmp.us.max",
                 "delay.twoway.icmp.count"]
}

The examples in this section use the following registry:

   { "registry-format": "mplane-0",
     "registry-uri", "https://example.com/mplane/registry/core",
     "registry-revision": 0,
     "includes": [],
     "elements": [
         { "name": "time",
           "prim": "time",
           "desc": "Time at which a single observation was taken"
         },
         { "name": "source.ip4",
           "prim": "address",
           "desc": "Source (or probe) IPv4 address"
         },
         { "name": "destination.ip4",
           "prim": "address",
           "desc": "Destination (or target) IPv4 address"
         },
         { "name": "intermediate.ip4",
           "prim": "address",
           "desc": "IPv4 address of intermediate node on a path"
         },
         { "name": "hops.ip",
           "prim": "natural",
           "desc": "IP-layer hops to identified node"
         },
         { "name": "hops.ip.max",
           "prim": "natural",
           "desc": "Maximum number of IP-layer hops to measure"
         },
         { "name": "delay.twoway.icmp.us",
           "prim": "natural",
           "desc": "Singleton two-way delay as measured via ICMP Echo"
         },
         { "name": "delay.twoway.icmp.us.min",
           "prim": "natural",
           "desc": "Minimum two-way delay as measured via ICMP Echo"
         },
         { "name": "delay.twoway.icmp.us.50pct",
           "prim": "natural",
           "desc": "Median two-way delay as measured via ICMP Echo"
         },
         { "name": "delay.twoway.icmp.us.mean",
           "prim": "natural",
           "desc": "Mean two-way delay as measured via ICMP Echo"
         },
         { "name": "delay.twoway.icmp.us.max",
           "prim": "natural",
           "desc": "Maximum two-way delay as measured via ICMP Echo"
         },
         { "name": "delay.twoway.icmp.us.count",
           "prim": "natural",
           "desc": "Count of two-way delay samples ... "
         },
     ]
   }

5.2. mPlane over WebSockets over TLS

The default session protocol for mPlane is WebSockets [RFC6455]. Once a WebSockets handshake between client and component is complete, messages are simply exchanged as outlined in Section 4.2 as JSON objects in WebSockets text frames over the channel.

When WebSockets is used as a session protocol for mPlane, it MUST be used over TLS for mPlane message exchanges. Both TLS clients and servers MUST present certificates for TLS mutual authentication. mPlane components MUST use the certificate presented by the mPlane client to determine the client’s identity, and therefore the capabilities which it is authorized to invoke. mPlane clients MUST use the certificate presented by the mPlane component to authenticate the results received. mPlane clients and components MUST NOT use network-layer address or name (e.g. derived from DNS PTR queries) information to identify peers.

mPlane components may either act as WebSockets servers, for client-initiated connection establishment, or as WebSockets clients, for component-initiated connection establishment. In either case, it is the responsibility of the connection initiator to re-establish connection in case it is lost. In this case, the entity acting as WebSockets server SHOULD maintain a queue of pending mPlane messages to identified peers to be sent on reconnection.

5.2.1. mPlane PKI for WebSockets

The clients and components within an mPlane domain generally share a single certificate issuer, specific to a single mPlane domain. Issuing a certificate to a client or component then grants it membership within the domain. Any client or component within the domain can then communicate with components and clients within that domain. In a domain containing one or more supervisors, all clients and components within the domain can connect to a supervisor. This deployment pattern can be used to scale mPlane domains to large numbers of clients and components without needing to specifically configure each client and component identity at the supervisor.

In the case of interdomain federation, where supervisors connect to each other, each supervisor will have its own issuer. In this case, each supervisor must be configured to trust each remote domain’s issuer, but only to identify that domain’s supervisor. This compartmentalization is necessary to keep one domain from authorizing clients within another domain.

5.2.2. Capability Advertisement for WebSockets

When a connection is first established between a client and a component, regardless of whether the client or the component initiates the connection, the component sends an envelope containing all the capabilities it wishes to advertise to the client, based on the client’s identity.

5.2.3. mPlane Link and Export URLs for WebSockets

Components acting as WebSockets servers (for client-initiated connection establishment) are identified in the Link sections of capabilities and receipts by URLs with the wss: schema. Similarly, clients acting as WebSockets servers (for component-initated connection establishment) are identified in the Link sections of specifications by URLs with the wss: schema.

When the wss: schema appears in the export section of the capability, this represents the component’s willingness to establish a WebSockets connection over which mPlane result messages will be pushed. A wss: schema URL in a specification export section, similarly, directs the component to the WebSockets server to push results to.

Path information in WebSockets URLs is presently unused by the mPlane protocol, but path information MUST be conserved. mPlane clients and components acting as WebSockets servers can use path information as they see fit, for example to separate client and component workflows on the same server (as on a supervisor), to run mPlane and other protocols over WebSockets on the same server, and/or to pass cryptographic tokens for additional context separation or authorization purposes. Future versions of the mPlane protocol may use path information in WebSockets URLs, but this path information will be relative to this conserved “base” URL, as opposed to relative to the root.

6. Deployment Considerations

This section outlines considerations for building larger-scale infrastructures from the building blocks defined in this document.

6.1. The Role of the Supervisor

For simple infrastructures, a set of components may be controlled directly by a client. However, in more complex infrastructures providing support for multiple clients, a supervisor can mediate between clients and components. From the point of view of the mPlane protocol, a supervisor is merely a combined component and client. The logic binding client and component interfaces within the supervisor is application-specific, as it involves the following operations according to the semantics of each application:

• translating lower-level capabilities from subordinate components into higher-level (composed) capabilities, according to the application’s semantics
• translating higher-level specifications from subordinate components into lower-level (decomposed) specifications
• relaying or aggregating results from subordinate components to supervisor clients

The workflows on each side of the supervisor are independent; indeed, the supervisor itself will generally respond to client-initiated exchanges, and use both component-initiated and supervisor-initiated exchanges with subordinate components.

Supervisors can of course be nested in this arrangement, e.g. allowing high-level measurements to be aggregated from a set of subdomains, or federation of measurement across administrative domain boundaries.

In the general case, the component first registers with the supervisor, POSTing its capabilities. The supervisor creates composed capabilities derived from these component capabilities, and makes them available to its client, which GETs them when it connects.

The client then initiates a measurement by POSTing a specification to the supervisor, which decomposes it into a more-specific specification to pass to the component, and hands the client a receipt for a the measurement. When the component polls the supervisor – controlled, perhaps, by callback control as described above – the supervisor passes this derived specification to the component, which executes it and POSTs its results back to the supervisor. When the client redeems its receipt, the supervisor returns results composed from those received from the component.

This simple example illustrates the three main responsibilities of the supervisor, which are described in more detail below.

6.1.1. Component Registration

In order to be able to use components to perform measurements, the supervisor must register the components associated with it. For client-initiated workflows – large repositories and the address of the components is often a configuration parameter of the supervisor. Capabilities describing the available measurements and queries at large-scale components can even be part of the supervisor’s externally managed static configuration, or can be dynamically retrieved and updated from the components or from a capability discovery server.

For component-initiated workflows, components connect to the supervisor and POST capabilities and withdrawals, which requires the supervisor to maintain a set of capabilities associated with a set of components currently part of the mPlane infrastructure it supervises.

6.1.2. Client Authentication

For many components – probes and simple repositories – very simple authentication often suffices, such that any client with a certificate with an issuer recognized as valid is acceptable, and all capabilities are available to. Larger repositories often need finer grained control, mapping specific peer certificates to identities internal to the repository’s access control system (e.g. database users).

In an mPlane infrastructure, it is therefore the supervisor’s responsibility to map client identities to the set of capabilities each client is authorized to access. This mapping is part of the supervisor’s configuration.

6.1.3. Capability Composition and Specification Decomposition

The most dominant responsibility of the supervisor is composing capabilities from its subordinate components into aggregate capabilities, and decomposing specifications from clients to more-specific specifications to pass to each component. This operation is always application-specific, as the semantics of the composition and decomposition operations depend on the capabilities available from the components, the granularity of the capabilities to be provided to the clients. It is for this reason that the mPlane SDK does not provide a generic supervisor.

6.2. Indirect Export

Many common measurement infrastructures involve a large number of probes exporting large volumes of data to a (much) smaller number of repositories, where data is reduced and analyzed. Since (1) the mPlane protocol is not particularly well-suited to the bulk transfer of data and (2) fidelity is better ensured when minimizing translations between representations, the channel between the probes and the repositories is in this case external to mPlane. This indirect export channel runs either a standard export protocol such as IPFIX, or a proprietary protocol unique to the probe/repository pair. It coordinates an exporter which will produce and export data with a collector which will receive it. All that is necessary is that (1) the client, exporter, and collector agree on a schema to define the data to be transferred and (2) the exporter and collector share a common protocol for export.

Here, we consider a client speaking to an exporter and a collector. The client first receives an export capability from the exporter (with verb measure and with a protocol identified in the export section) and a collection capability from the collector (with the verb collect and with a URL in the export section describing where the exporter should export), either via a client-initiated workflow or a capability discovery server. The client then sends a specification to the exporter, which matches the schema and parameter constraints of both the export and collection capabilities, with the collector’s URL in the export section.

The exporter initiates export to the collector using the specified protocol, and replies with a receipt that can be used to interrupt the export, should it have an indefinite temporal scope. In the meantime, it sends data matching the capability’s schema directly to the collector.

This data, or data derived from the analysis thereof, can then be subsequently retrieved by a client using a client-initiated workflow to the collector.

6.3. Error Handling in mPlane Workflows

Any component may signal an error to its client or supervisor at any time by sending an exception message. While the taxonomy of error messages is at this time left up to each individual component, given the weakly imperative nature of the mPlane protocol, exceptions should be used sparingly, and only to notify components and clients of errors with the mPlane infrastructure itself.

It is generally presumed that diagnostic information about errors which may require external human intervention to correct will be logged at each component; the mPlane exception facility is not intended as a replacement for logging facilities (such as syslog).

Specifically, components using component-initiated connection establishment should not use the exception mechanism for common error conditions (e.g., device losing connectivity for small network-edge probes) – specifications sent to such components are expected to be best-effort. Exceptions should also not be returned for specifications which would normally not be delayed but are due to high load – receipts should be used in this case, instead. Likewise, specifications which cannot be fulfilled because they request the use of capabilities that were once available but are no longer should be answered with withdrawals.

Exceptions should always be sent in reply to messages sent to components or clients which cannot be handled due to a syntactic or semantic error in the message itself.

7. Security Considerations

The mPlane protocol allows the control of network measurement devices. The protocol itself uses WebSockets using TLS as a session layer. TLS mutual authentication must be used for the exchange of mPlane messages, as access control decisions about which clients and components are trusted for which capabilities take identity information from the certificates TLS clients and servers use to identify themselves. Current operational best security practices for the deployment of TLS-secured protocols must be followed for the deployment of mPlane.

Indirect export, as a design feature, presents a potential for information leakage, as indirectly exported data is necessarily related to measurement data and control transported with the mPlane protocol. Though out of scope for this document, indirect export protocols used within an mPlane domain must be secured at least as well as the mPlane protocol itself.

8. IANA Considerations

This document has no actions for IANA. Future versions of this document, should it become a standards-track specification, may specify the initial contents of a core mPlane registry to be managed by IANA.

9. Contributors

This document is based on Deliverable 1.4, the architecture and protocol specification document produced by the mPlane project [D14], which is the work of the mPlane consortium, specifically Brian Trammell and Mirja Kuehlewind (the editors of this document), as well as Marco Mellia, Alessandro Finamore, Stefano Pentassuglia, Gianni De Rosa, Fabrizio Invernizzi, Marco Milanesio, Dario Rossi, Saverio Niccolini, Ilias Leontiadis, Tivadar Szemethy, Balas Szabo, Rolf Winter, Michael Faath, Benoit Donnet, and Dimitri Papadimitriou.

10. Acknowledgments

Thanks to Lingli Deng and Robert Kisteleki for feedback leading to the improvement of this document and the protocol.

This work is supported by the European Commission under grant agreement FP7-318627 mPlane and H2020-688421 MAMI, and by the Swiss State Secretariat for Education, Research, and Innovation under contract no. 15.0268. This support does not imply endorsement of the contents of this document.

11. Informative References

[RFC3205]	Moore, K., "On the use of HTTP as a Substrate", BCP 56, RFC 3205, DOI 10.17487/RFC3205, February 2002.
[RFC3339]	Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002.
[RFC4291]	Hinden, R. and S. Deering, "IP Version 6 Addressing Architecture", RFC 4291, DOI 10.17487/RFC4291, February 2006.
[RFC5246]	Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, DOI 10.17487/RFC5246, August 2008.
[RFC5905]	Mills, D., Martin, J., Burbank, J. and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010.
[RFC5952]	Kawamura, S. and M. Kawashima, "A Recommendation for IPv6 Address Text Representation", RFC 5952, DOI 10.17487/RFC5952, August 2010.
[RFC6455]	Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, DOI 10.17487/RFC6455, December 2011.
[RFC7011]	Claise, B., Trammell, B. and P. Aitken, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information", STD 77, RFC 7011, DOI 10.17487/RFC7011, September 2013.
[RFC7159]	Bray, T., "The JavaScript Object Notation (JSON) Data Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2014.
[RFC7230]	Fielding, R. and J. Reschke, "Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing", RFC 7230, DOI 10.17487/RFC7230, June 2014.
[RFC7373]	Trammell, B., "Textual Representation of IP Flow Information Export (IPFIX) Abstract Data Types", RFC 7373, DOI 10.17487/RFC7373, September 2014.
[D14]	Trammell, B., "mPlane Architecture Specification", April 2015.

Authors' Addresses