Internet DRAFT - draft-richih-opsawg-openmetrics
draft-richih-opsawg-openmetrics
Ops WG R. Hartmann, Ed.
Internet-Draft Grafana Labs
Intended status: Standards Track B. Kochie
Expires: May 29, 2021 GitLab
B. Brazil
Robust Perception
R. Skillington
Chronosphere
November 25, 2020
OpenMetrics, a cloud-native, highly scalable metrics protocol
draft-richih-opsawg-openmetrics-00
Abstract
OpenMetrics specifies today's de-facto standard for transmitting
cloud-native metrics at scale, with support for both text
representation and Protocol Buffers and brings it into IETF. It
supports both pull and push-based data collection.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 29, 2021.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
Hartmann, et al. Expires May 29, 2021 [Page 1]
Internet-Draft OpenMetrics November 2020
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Metrics and Time Series . . . . . . . . . . . . . . . . . 4
3. Data Model . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1. Data Types . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.1. Values . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.2. Timestamps . . . . . . . . . . . . . . . . . . . . . 5
3.1.3. Strings . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.4. Label . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.5. LabelSet . . . . . . . . . . . . . . . . . . . . . . 5
3.1.6. MetricPoint . . . . . . . . . . . . . . . . . . . . . 5
3.1.7. Exemplars . . . . . . . . . . . . . . . . . . . . . . 6
3.1.8. Metric . . . . . . . . . . . . . . . . . . . . . . . 6
3.1.9. MetricFamily . . . . . . . . . . . . . . . . . . . . 6
3.2. Metric Types . . . . . . . . . . . . . . . . . . . . . . 8
3.2.1. Gauge . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2.2. Counter . . . . . . . . . . . . . . . . . . . . . . . 8
3.2.3. StateSet . . . . . . . . . . . . . . . . . . . . . . 9
3.2.4. Info . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.5. Histogram . . . . . . . . . . . . . . . . . . . . . . 10
3.2.6. GaugeHistogram . . . . . . . . . . . . . . . . . . . 11
3.2.7. Summary . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.8. Unknown . . . . . . . . . . . . . . . . . . . . . . . 12
4. Data transmission & wire formats . . . . . . . . . . . . . . 12
4.1. Protocol Negotiation . . . . . . . . . . . . . . . . . . 12
4.1.1. ABNF . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1.2. Overall Structure . . . . . . . . . . . . . . . . . . 14
4.1.3. MetricFamily . . . . . . . . . . . . . . . . . . . . 17
4.1.4. MetricPoint . . . . . . . . . . . . . . . . . . . . . 18
4.1.5. Metric types . . . . . . . . . . . . . . . . . . . . 19
4.2. Protobuf format . . . . . . . . . . . . . . . . . . . . . 23
4.2.1. Overall Structure . . . . . . . . . . . . . . . . . . 23
4.2.2. Protobuf schema . . . . . . . . . . . . . . . . . . . 24
5. Design Considerations . . . . . . . . . . . . . . . . . . . . 28
5.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.1.1. Out of scope . . . . . . . . . . . . . . . . . . . . 29
5.2. Extensions and Improvements . . . . . . . . . . . . . . . 29
5.3. Units and Base Units . . . . . . . . . . . . . . . . . . 29
5.4. Statelessness . . . . . . . . . . . . . . . . . . . . . . 31
5.5. Exposition Across Time and Metric Evolution . . . . . . . 31
Hartmann, et al. Expires May 29, 2021 [Page 2]
Internet-Draft OpenMetrics November 2020
5.6. NaN . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.7. Missing Data . . . . . . . . . . . . . . . . . . . . . . 32
5.8. Exposition Performance . . . . . . . . . . . . . . . . . 33
5.9. Concurrency . . . . . . . . . . . . . . . . . . . . . . . 33
5.10. Metric Naming and Namespaces . . . . . . . . . . . . . . 33
5.11. Label Namespacing . . . . . . . . . . . . . . . . . . . . 35
5.12. Metric Names versus Labels . . . . . . . . . . . . . . . 36
5.13. Types of Metadata . . . . . . . . . . . . . . . . . . . . 37
5.13.1. Supporting Target Metadata in both Push-based and
Pull-based Systems . . . . . . . . . . . . . . . . . 37
5.14. Client Calculations and Derived Metrics . . . . . . . . . 38
5.15. Number Types . . . . . . . . . . . . . . . . . . . . . . 39
5.16. Exposing Timestamps . . . . . . . . . . . . . . . . . . . 39
5.16.1. Tracking When Metrics Last Changed . . . . . . . . . 40
5.17. Thresholds . . . . . . . . . . . . . . . . . . . . . . . 41
5.18. Size Limits . . . . . . . . . . . . . . . . . . . . . . . 42
6. Security Considerations . . . . . . . . . . . . . . . . . . . 43
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 43
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 44
8.1. Normative References . . . . . . . . . . . . . . . . . . 44
8.2. Informative References . . . . . . . . . . . . . . . . . 44
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 45
1. Introduction
Created in 2012, Prometheus has been the default for cloud-native
observability since 2015. A central part of Prometheus' design is
its text metric exposition format, called the Prometheus exposition
format 0.0.4, stable since 2014. In this format, special care has
been taken to make it easy to generate, to ingest, and to understand
by humans. As of 2020, there are more than 700 publicly listed
exporters, an unknown number of unlisted exporters, and thousands of
native library integrations using this format. Dozens of ingestors
from various projects and companies support consuming it.
With OpenMetrics, we are cleaning up and tightening the specification
with the express purpose of bringing it into IETF. We are
documenting a working standard with wide and organic adoption while
introducing minimal, largely backwards-compatible, and well-
considered changes. As of 2020, dozens of exporters, integrations,
and ingestors use and preferentially negotiate OpenMetrics already.
Given the wide adoption and significant coordination requirements in
the ecosystem, sweeping changes to either the Prometheus exposition
format 0.0.4 or OpenMetrics 1.0 are considered out of scope.
Hartmann, et al. Expires May 29, 2021 [Page 3]
Internet-Draft OpenMetrics November 2020
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Overview
Metrics are a specific kind of telemetry data. They represent a
snapshot of the current state for a set of data. They are distinct
from logs or events, which focus on records or information about
individual events.
OpenMetrics is primarily a wire format, independent of any particular
transport for that format. The format is expected to be consumed on
a regular basis and to be meaningful over successive expositions.
Implementers MUST expose metrics in the OpenMetrics text format in
response to a simple HTTP GET request to a documented URL for a given
process or device. This endpoint SHOULD be called "/metrics".
Implementers MAY also expose OpenMetrics formatted metrics in other
ways, such as by regularly pushing metric sets to an operator-
configured endpoint over HTTP.
2.1. Metrics and Time Series
This standard expresses all system states as numerical values;
counts, current values, enumerations, and boolean states being common
examples. Contrary to metrics, singular events occur at a specific
time. Metrics tend to aggregate data temporally. While this can
lose information, the reduction in overhead is an engineering trade-
off commonly chosen in many modern monitoring systems.
Time series are a record of changing information over time. While
time series can support arbitrary strings or binary data, only
numeric data is in scope for this RFC.
Common examples of metric time series would be network interface
counters, device temperatures, BGP connection states, and alert
states.
3. Data Model
This section MUST be read together with the ABNF section. In case of
disagreements between the two, the ABNF's restrictions MUST take
Hartmann, et al. Expires May 29, 2021 [Page 4]
Internet-Draft OpenMetrics November 2020
precedence. This reduces repetition as the text wire format MUST be
supported.
3.1. Data Types
3.1.1. Values
Metric values in OpenMetrics MUST be either floating points or
integers. Note that ingestors of the format MAY only support
float64. The non-real values NaN, +Inf and -Inf MUST be supported.
NaN MUST NOT be considered a missing value, but it MAY be used to
signal a division by zero.
3.1.1.1. Booleans
Boolean values MUST follow 1==true, 0==false.
3.1.2. Timestamps
Timestamps MUST be Unix Epoch in seconds. Negative timestamps MAY be
used.
3.1.3. Strings
Strings MUST only consist of valid UTF-8 characters and MAY be zero
length. NULL (ASCII 0x0) MUST be supported.
3.1.4. Label
Labels are key-value pairs consisting of strings.
Label names beginning with underscores are RESERVED and MUST NOT be
used unless specified by this standard. Label names MUST follow the
restrictions in the ABNF section.
Empty label values SHOULD be treated as if the label was not present.
3.1.5. LabelSet
A LabelSet MUST consist of Labels and MAY be empty. Label names MUST
be unique within a LabelSet.
3.1.6. MetricPoint
Each MetricPoint consists of a set of values, depending on the
MetricFamily type.
Hartmann, et al. Expires May 29, 2021 [Page 5]
Internet-Draft OpenMetrics November 2020
3.1.7. Exemplars
Exemplars are references to data outside of the MetricSet. A common
use case are IDs of program traces.
Exemplars MUST consist of a LabelSet and a value, and MAY have a
timestamp. They MAY each be different from the MetricPoints'
LabelSet and timestamp.
The combined length of the label names and values of an Exemplar's
LabelSet MUST NOT exceed 128 UTF-8 characters. Other characters in
the text rendering of an exemplar such as ",= are not included in
this limit for implementation simplicity and for consistency between
the text and proto formats.
Ingestors MAY discard exemplars.
3.1.8. Metric
Metrics are defined by a unique LabelSet within a MetricFamily.
Metrics MUST contain a list of one or more MetricPoints. Metrics
with the same name for a given MetricFamily SHOULD have the same set
of label names in their LabelSet.
MetricPoints SHOULD NOT have explicit timestamps.
If more than one MetricPoint is exposed for a Metric, then its
MetricPoints MUST have monotonically increasing timestamps.
3.1.9. MetricFamily
A MetricFamily MAY have zero or more Metrics. A MetricFamily MUST
have a name, HELP, TYPE, and UNIT metadata. Every Metric within a
MetricFamily MUST have a unique LabelSet.
3.1.9.1. Name
MetricFamily names are a string and MUST be unique within a
MetricSet. Names SHOULD be in snake_case. Metric names MUST follow
the restrictions in the ABNF section.
Colons in MetricFamily names are RESERVED to signal that the
MetricFamily is the result of a calculation or aggregation of a
general purpose monitoring system.
MetricFamily names beginning with underscores are RESERVED and MUST
NOT be used unless specified by this standard.
Hartmann, et al. Expires May 29, 2021 [Page 6]
Internet-Draft OpenMetrics November 2020
3.1.9.1.1. Suffixes
The name of a MetricFamily MUST NOT result in a potential clash for
sample metric names as per the ABNF with another MetricFamily in the
Text Format within a MetricSet. An example would be a gauge called
"foo_created" as a counter called "foo" could create a "foo_created"
in the text format.
Exposers SHOULD avoid names that could be confused with the suffixes
that text format sample metric names use.
o Suffixes for the respective types are:
o Counter: '_total', '_created'
o Summary: '_count', '_sum', '_created', '' (empty)
o Histogram: '_count', '_sum', '_bucket', '_created'
o GaugeHistogram: '_gcount', '_gsum', '_bucket'
o Info: '_info'
o Gauge: '' (empty)
o StateSet: '' (empty)
o Unknown: '' (empty)
3.1.9.2. Type
Type specifies the MetricFamily type. Valid values are "unknown",
"gauge", "counter", "stateset", "info", "histogram",
"gaugehistogram", and "summary".
3.1.9.3. Unit
Unit specifies MetricFamily units. If non-empty, it MUST be a suffix
of the MetricFamily name separated by an underscore. Be aware that
further generation rules might make it an infix in the text format.
3.1.9.4. Help
Help is a string and SHOULD non-empty. It is used to give a brief
description of the MetricFamily for human consumption and SHOULD be
short enough to be used as a tooltip.
Hartmann, et al. Expires May 29, 2021 [Page 7]
Internet-Draft OpenMetrics November 2020
3.1.9.5. MetricSet
A MetricSet is the top level object exposed by OpenMetrics. It MUST
consist of MetricFamilies and MAY be empty.
Each MetricFamily name MUST be unique. The same label name and value
SHOULD NOT appear on every Metric within a MetricSet.
There is no specific ordering of MetricFamilies required within a
MetricSet. An exposer MAY make an exposition easier to read for
humans, for example sort alphabetically if the performance tradeoff
makes sense.
If present, an Info MetricFamily called "target" per the "Supporting
target metadata in both push-based and pull-based systems" section
below SHOULD be first.
3.2. Metric Types
3.2.1. Gauge
Gauges are current measurements, such as bytes of memory currently
used or the number of items in a queue. For gauges the absolute
value is what is of interest to a user.
A MetricPoint in a Metric with the type gauge MUST have a single
value.
Gauges MAY increase, decrease, or stay constant over time. Even if
they only ever go in one direction, they might still be gauges and
not counters. The size of a log file would usually only increase, a
resource might decrease, and the limit of a queue size may be
constant.
A gauge MAY be used to encode an enum where the enum has many states
and changes over time, it is the most efficient but least user
friendly.
3.2.2. Counter
Counters measure discrete events. Common examples are the number of
HTTP requests received, CPU seconds spent, or bytes sent. For
counters how quickly they are increasing over time is what is of
interest to a user.
A MetricPoint in a Metric with the type Counter MUST have one value
called Total. A Total is a non-NaN and MUST be monotonically non-
decreasing over time, starting from 0.
Hartmann, et al. Expires May 29, 2021 [Page 8]
Internet-Draft OpenMetrics November 2020
A MetricPoint in a Metric with the type Counter SHOULD have a
Timestamp value called Created. This can help ingestors discern
between new metrics and long-running ones it did not see before.
A MetricPoint in a Metric's Counter's Total MAY reset to 0. If
present, the corresponding Created time MUST also be set to the
timestamp of the reset.
A MetricPoint in a Metric's Counter's Total MAY have an exemplar.
3.2.3. StateSet
StateSets represent a series of related boolean values, also called a
bitset. If ENUMs need to be encoded this MAY be done via StateSet.
A point of a StateSet metric MAY contain multiple states and MUST
contain one boolean per State. States have a name which are Strings.
A StateSet Metric's LabelSet MUST NOT have a label name which is the
same as the name of its MetricFamily.
If encoded as a StateSet, ENUMs MUST have exactly one Boolean which
is true within a MetricPoint.
This is suitable where the enum value changes over time, and the
number of States isn't much more than a handful.
EDITOR'S NOTE: This might be better as Consideration
MetricFamilies of type StateSets MUST have an empty Unit string.
3.2.4. Info
Info metrics are used to expose textual information which SHOULD NOT
change during process lifetime. Common examples are an application's
version, revision control commit, and the version of a compiler.
A MetricPoint of an Info Metric contains a LabelSet. An Info
MetricPoint's LabelSet MUST NOT have a label name which is the same
as the name of a label of the LabelSet of its Metric.
Info MAY be used to encode ENUMs whose values do not change over
time, such as the type of a network interface.
MetricFamilies of type Info MUST have an empty Unit string.
Hartmann, et al. Expires May 29, 2021 [Page 9]
Internet-Draft OpenMetrics November 2020
3.2.5. Histogram
Histograms measure distributions of discrete events. Common examples
are the latency of HTTP requests, function runtimes, or I/O request
sizes.
A Histogram MetricPoint MUST contain at least one bucket, and SHOULD
contain Sum, and Created values. Every bucket MUST have a threshold
and a value.
Histogram MetricPoints MUST have at least a bucket with an +Inf
threshold. Buckets MUST be cumulative. As an example for a metric
representing request latency in seconds its values for buckets with
thresholds 1, 2, 3, and +Inf MUST follow value_1 <= value_2 <=
value_3 <= value_+Inf. If ten requests took 1 second each, the
values of the 1, 2, 3, and +Inf buckets MUST equal 10.
The +Inf bucket counts all requests. If present, the Sum value MUST
equal the Sum of all the measured event values. Bucket thresholds
within a MetricPoint MUST be unique.
Semantically, Sum, and buckets values are counters so MUST NOT be NaN
or negative. Negative threshold buckets MAY be used, but then the
Histogram MetricPoint MUST NOT contain a sum value as it would no
longer be a counter semantically. Bucket thresholds MUST NOT equal
NaN. Count and bucket values MUST be integers.
A Histogram MetricPoint SHOULD have a Timestamp value called Created.
This can help ingestors discern between new metrics and long-running
ones it did not see before.
A Histogram's Metric's LabelSet MUST NOT have a "le" label name.
Bucket values MAY have exemplars. Buckets are cumulative to allow
monitoring systems to drop any non-+Inf bucket for performance/anti-
denial-of-service reasons in a way that loses granularity but is
still a valid Histogram.
EDITOR'S NOTE: The second sentence is a consideration, it can be
moved if needed
Each bucket covers the values less and or equal to it, and the value
of the exemplar MUST be within this range. Exemplars SHOULD be put
into the bucket with the highest value. A bucket MUST NOT have more
than one exemplar.
Hartmann, et al. Expires May 29, 2021 [Page 10]
Internet-Draft OpenMetrics November 2020
3.2.6. GaugeHistogram
GaugeHistograms measure current distributions. Common examples are
how long items have been waiting in a queue, or size of the requests
in a queue.
A GaugeHistogram MetricPoint MUST have at least one bucket with an
+Inf threshold, and SHOULD contain a Gsum value. Every bucket MUST
have a threshold and a value.
The buckets for a GaugeHistogram follow all the same rules as for a
Histogram.
The bucket and Gsum of a GaugeHistogram are conceptually gauges,
however bucket values MUST NOT be negative or NaN. If negative
threshold buckets are present, then sum MAY be negative. Gsum MUST
NOT be NaN. Bucket values MUST be integers.
A GaugeHistogram's Metric's LabelSet MUST NOT have a "le" label name.
Bucket values can have exemplars.
Each bucket covers the values less and or equal to it, and the value
of the exemplar MUST be within this range. Exemplars SHOULD be put
into the bucket with the highest value. A bucket MUST NOT have more
than one exemplar.
3.2.7. Summary
Summaries also measure distributions of discrete events and MAY be
used when Histograms are too expensive and/or an average event size
is sufficient.
They MAY also be used for backwards compatibility, because some
existing instrumentation libraries expose precomputed quantiles and
do not support Histograms. Precomputed quantiles SHOULD NOT be used,
because quantiles are not aggregatable and the user often can not
deduce what timeframe they cover.
A Summary MetricPoint MAY consist of a Count, Sum, Created, and a set
of quantiles.
Semantically, Count and Sum values are counters so MUST NOT be NaN or
negative. Count MUST be an integer.
A MetricPoint in a Metric with the type Summary which contains Count
or Sum values SHOULD have a Timestamp value called Created. This can
help ingestors discern between new metrics and long-running ones it
Hartmann, et al. Expires May 29, 2021 [Page 11]
Internet-Draft OpenMetrics November 2020
did not see before. Created MUST NOT relate to the collection period
of quantile values.
Quantiles are a map from a quantile to a value. An example is a
quantile 0.95 with value 0.2 in a metric called
myapp_http_request_duration_seconds which means that the 95th
percentile latency is 200ms over an unknown timeframe. If there are
no events in the relevant timeframe, the value for a quantile MUST be
NaN. A Quantile's Metric's LabelSet MUST NOT have "quantile" label
name. Quantiles MUST be between 0 and 1 inclusive. Quantile values
MUST NOT be negative. Quantile values SHOULD represent the recent
values. Commonly this would be over the last 5-10 minutes.
3.2.8. Unknown
Unknown SHOULD NOT be used. Unknown MAY be used when it is
impossible to determine the types of individual metrics from 3rd
party systems.
A point in a metric with the unknown type MUST have a single value.
4. Data transmission & wire formats
The text wire format MUST be supported and is the default. The
protobuf wire format MAY be supported and MUST ONLY be used after
negotiation.
The OpenMetrics formats are Regular Chomsky Grammars, making writing
quick and small parsers possible. The text format compresses well,
and protobuf is already binary and efficiently encoded.
Partial or invalid expositions MUST be considered erroneous in their
entirety.
4.1. Protocol Negotiation
All ingestor implementations MUST be able to ingest data secured with
TLS 1.2 or later. All exposers SHOULD be able to emit data secured
with TLS 1.2 or later. ingestor implementations SHOULD be able to
ingest data from HTTP without TLS. All implementations SHOULD use
TLS to transmit data.
Negotiation of what version of the OpenMetrics format to use is out-
of-band. For example for pull-based exposition over HTTP standard
HTTP content type negotiation is used, and MUST default to the oldest
version of the standard (i.e. 1.0.0) if no newer version is
requested.
Hartmann, et al. Expires May 29, 2021 [Page 12]
Internet-Draft OpenMetrics November 2020
Push-based negotiation is inherently more complex, as the exposer
typically initiates the connection. Producers MUST use the oldest
version of the standard (i.e. 1.0.0) unless requested otherwise by
the ingestor. Text format
4.1.1. ABNF
ABNF as per RFC 5234
EDITOR'S NOTE: Should we update to RFC 7405, in particular the case
insensitive bits?
"exposition" is the top level token of the ABNF.
exposition = metricset HASH SP eof [ LF ]
metricset = *metricfamily
metricfamily = *metric-descriptor *metric
metric-descriptor = HASH SP type SP metricname SP metric-type LF
metric-descriptor =/ HASH SP help SP metricname SP escaped-string LF
metric-descriptor =/ HASH SP unit SP metricname SP 1*metricname-char LF
metric = *sample
metric-type = counter / gauge / histogram / gaugehistogram / stateset
metric-type =/ info / summary / unknown
sample = metricname [labels] SP number [SP timestamp] [exemplar] LF
exemplar = SP HASH SP labels SP number [SP timestamp]
labels = "{" [label *(COMMA label)] "}"
label = label-name EQ DQUOTE escaped-string DQUOTE
number = realnumber
; Case insensitive
number =/ [SIGN] ("inf" / "infinity")
number =/ "nan"
timestamp = realnumber
; Not 100% sure this captures all float corner cases.
; Leading 0s explicitly okay
realnumber = [SIGN] 1*DIGIT
realnumber =/ [SIGN] 1*DIGIT ["." *DIGIT] [ "e" [SIGN] 1*DIGIT ]
Hartmann, et al. Expires May 29, 2021 [Page 13]
Internet-Draft OpenMetrics November 2020
realnumber =/ [SIGN] *DIGIT "." 1*DIGIT [ "e" [SIGN] 1*DIGIT ]
; RFC 5234 is case insensitive.
; Uppercase
eof = %d69.79.70
type = %d84.89.80.69
help = %d72.69.76.80
unit = %d85.78.73.84
; Lowercase
counter = %d99.111.117.110.116.101.114
gauge = %d103.97.117.103.101
histogram = %d104.105.115.116.111.103.114.97.109
gaugehistogram = gauge histogram
stateset = %d115.116.97.116.101.115.101.116
info = %d105.110.102.111
summary = %d115.117.109.109.97.114.121
unknown = %d117.110.107.110.111.119.110
BS = "\"
EQ = "="
COMMA = ","
HASH = "#"
SIGN = "-" / "+"
metricname = metricname-initial-char 0*metricname-char
metricname-char = metricname-initial-char / DIGIT
metricname-initial-char = ALPHA / "_" / ":"
label-name = label-name-initial-char *label-name-char
label-name-char = label-name-initial-char / DIGIT
label-name-initial-char = ALPHA / "_"
escaped-string = *escaped-char
escaped-char = normal-char
escaped-char =/ BS ("n" / DQUOTE / BS)
; Any unicode character, except newline, double quote, and backslash
normal-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
4.1.2. Overall Structure
UTF-8 MUST be used. Byte order markers (BOMs) MUST NOT be used. As
an important reminder for implementers, byte 0 is valid UTF-8 while,
for example, byte 255 is not.
Hartmann, et al. Expires May 29, 2021 [Page 14]
Internet-Draft OpenMetrics November 2020
The content type MUST be: application/openmetrics-text;
version=1.0.0; charset=utf-8
Line endings MUST be signalled with line feed (\n) and MUST NOT
contain carriage returns (\r). Expositions MUST end with EOF and
SHOULD end with 'EOF\n'.
An example of a complete exposition: ~~~~ # TYPE
acme_http_router_request_seconds summary # UNIT
acme_http_router_request_seconds seconds # HELP
acme_http_router_request_seconds Latency though all of ACME's HTTP
request router. acme_http_router_request_seconds_sum{path="/api/
v1",method="GET"} 9036.32
acme_http_router_request_seconds_count{path="/api/v1",method="GET"}
807283.0 acme_http_router_request_seconds_created{path="/api/
v1",method="GET"} 1605281325.0
acme_http_router_request_seconds_sum{path="/api/v2",method="POST"}
479.3 acme_http_router_request_seconds_count{path="/api/
v2",method="POST"} 34.0
acme_http_router_request_seconds_created{path="/api/
v2",method="POST"} 1605281325.0 # TYPE go_goroutines gauge # HELP
go_goroutines Number of goroutines that currently exist.
go_goroutines 69 # TYPE process_cpu_seconds counter # UNIT
process_cpu_seconds seconds # HELP process_cpu_seconds Total user and
system CPU time spent in seconds. process_cpu_seconds_total
4.20072246e+06 # EOF ~~~~
4.1.2.1. Escaping
Where the ABNF notes escaping, the following escaping MUST be applied
Line feed, '\n' (0x0A) -> literally '\n' (Bytecode 0x5c 0x6e) Double
quotes -> '"' (Bytecode 0x5c 0x22) Backslash -> '\' (Bytecode 0x5c
0x5c)
4.1.2.2. Numbers
Integer numbers MUST NOT have a decimal point. Examples are "23",
"0042", and "1341298465647914".
Floating point numbers MUST be represented either with a decimal
point or using scientific notation. Examples are "8903.123421" and
"1.89e-7". Floating point numbers MUST fit within the range of a
64-bit floating point value as defined by IEEE 754, but MAY require
so many bits in the mantissa that results in lost precision. This
MAY be used to encode nanosecond resolution timestamps.
Hartmann, et al. Expires May 29, 2021 [Page 15]
Internet-Draft OpenMetrics November 2020
Arbitrary integer and floating point rendering of numbers MUST NOT be
used for "quantile" and "le" label values as in section "Canonical
Numbers". They MAY be used anywhere else numbers are used.
4.1.2.2.1. Considerations: Canonical Numbers
Numbers in the "le" label values of histograms and "quantile" label
values of summary metrics are special in that they're label values,
and label values are intended to be opaque. As end users will likely
directly interact with these string values, and as many monitoring
systems lack the ability to deal with them as first-class numbers, it
would be beneficial if a given number had the exact same text
representation.
Consistency is highly desirable, but real world implementations of
languages and their runtimes make mandating this impractical. The
most important common quantiles are 0.5, 0.95, 0.9, 0.99, 0.999 and
bucket values representing values from a millisecond up to 10.0
seconds, because those cover cases like latency SLAs and Apdex for
typical web services. Powers of ten are covered to try to ensure
that the switch between fixed point and exponential rendering is
consistent as this varies across runtimes. The target rendering is
equivalent to the default Go rendering of float64 values (i.e. %g),
with a .0 appended in case there is no decimal point or exponent to
make clear that they are floats.
Exposers MUST produce output for positive infinity as +Inf.
Exposers SHOULD produce output for the values 0.0 up to 10.0 in 0.001
increments in line with the following examples: 0.0 0.001 0.002 0.01
0.1 0.9 0.95 0.99 0.999 1.0 1.7 10.0
Exposers SHOULD produce output for the values 1e-10 up to 1e+10 in
powers of ten in line with the following examples: 1e-10 1e-09 1e-05
0.0001 0.1 1.0 100000.0 1e+06 1e+10
Parsers MUST NOT reject inputs which are outside of the canonical
values merely because they are not consistent with the canonical
values. For example 1.1e-4 must not be rejected, even though it is
not the consistent rendering of 0.00011.
Exposers SHOULD follow these patterns for non-canonical numbers, and
the intention is by adjusting the rendering algorithm to be
consistent for these values that the vast majority of other values
will also have consistent rendering. Exposers using only a few
particular le/quantile values could also hardcode. In languages such
as C where a minimal floating point rendering algorithm such as
Hartmann, et al. Expires May 29, 2021 [Page 16]
Internet-Draft OpenMetrics November 2020
Grisu3 such as Grisu3 is not readily available, exposers MAY use a
different rendering.
A warning to implementers in C and other languages that share its
printf implementation: The standard precision of %f, %e and %g is
only six significant digits. 17 significant digits are required for
full precision, e.g. "printf("%.17g", d)".
4.1.2.3. Timestamps
Timestamps SHOULD NOT use exponential float rendering for timestamps
if nanosecond precision is needed as rendering of a float64 does not
have sufficient precision, e.g. 1604676851.123456789.
4.1.3. MetricFamily
There MUST NOT be an explicit separator between MetricFamilies. The
next MetricFamily MUST be signalled with either metadata or a new
sample metric name which cannot be part of the previous MetricFamily.
MetricFamilies MUST NOT be interleaved.
4.1.3.1. MetricFamily metadata
There are four pieces of metadata: The MetricFamily name, TYPE, UNIT
and HELP. An example of the metadata for a counter Metric called foo
is:
# TYPE foo counter
If no TYPE is exposed, the MetricFamily MUST be of type Unknown.
If a unit is specified it MUST be provided in a UNIT metadata line.
In addition, an underscore and the unit MUST be the suffix of the
MetricFamily name.
A valid example for a foo_seconds metric with a unit of "seconds":
~~~~ # TYPE foo_seconds counter # UNIT foo_seconds seconds ~~~~
An invalid example, where the unit is not a suffix on the name: ~~~~
# TYPE foo counter # UNIT foo seconds ~~~~
It is also valid to have: ~~~~ # TYPE foo_seconds counter ~~~~
If the unit is known it SHOULD be provided.
The value of a UNIT or HELP line MAY be empty. This MUST be treated
as if no metadata line for the MetricFamily existed.
Hartmann, et al. Expires May 29, 2021 [Page 17]
Internet-Draft OpenMetrics November 2020
# TYPE foo_seconds counter
# UNIT foo_seconds seconds
# HELP foo_seconds Some text and \n some \" escaping
There MUST NOT be more than one of each type of metadata line for a
MetricFamily. The ordering SHOULD be TYPE, UNIT, HELP.
Aside from this metadata and the EOF line at the end of the message,
you MUST NOT expose lines beginning with a #.
4.1.3.2. Metric
Metrics MUST NOT be interleaved.
See the example in "Text format -> MetricPoint". Labels A sample
without labels or a timestamp and the value 0 MUST be rendered either
like:
bar_seconds_count 0
or like
bar_seconds_count{} 0
Label values MAY be any valid UTF-8 value, so escaping MUST be
applied as per the ABNF. A valid example with two labels: ~~~~
bar_seconds_count{a="x",b="escaping" example \n "} 0 ~~~~
The rendering of values for a MetricPoint can include additional
labels (e.g. the "le" label for a Histogram type), which MUST be
rendered in the same way as a Metric's own LabelSet.
4.1.4. MetricPoint
MetricPoints MUST NOT be interleaved.
A correct example where there were multiple MetricPoints and Samples
within a MetricFamily would be:
Hartmann, et al. Expires May 29, 2021 [Page 18]
Internet-Draft OpenMetrics November 2020
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_sum{a="bb"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_sum{a="bb"} 0 456
foo_seconds_count{a="ccc"} 0 123
foo_seconds_sum{a="ccc"} 0 123
foo_seconds_count{a="ccc"} 0 456
foo_seconds_sum{a="ccc"} 0 456
An incorrect example where Metrics are interleaved:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_count{a="ccc"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_count{a="ccc"} 0 456
An incorrect example where MetricPoints are interleaved:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_sum{a="bb"} 0 123
foo_seconds_sum{a="bb"} 0 456
4.1.5. Metric types
4.1.5.1. Gauge
The Sample MetricName for the value of a MetricPoint for a
MetricFamily of type Gauge MUST NOT have a suffix.
An example MetricFamily with a Metric with no labels and a
MetricPoint with no timestamp: ~~~~ # TYPE foo gauge foo 17.0 ~~~~
An example of a MetricFamily with two Metrics with a label and
MetricPoints with no timestamp: ~~~~ # TYPE foo gauge foo{a="bb"}
17.0 foo{a="ccc"} 17.0 ~~~~
An example of a MetricFamily with no Metrics: ~~~~ # TYPE foo gauge
~~~~
An example with a Metric with a label and a MetricPoint with a
timestamp: ~~~~ # TYPE foo gauge foo{a="b"} 17.0 1520879607.789 ~~~~
Hartmann, et al. Expires May 29, 2021 [Page 19]
Internet-Draft OpenMetrics November 2020
An example with a Metric with no labels and MetricPoint with a
timestamp: ~~~~ # TYPE foo gauge foo 17.0 1520879607.789 ~~~~
An example with a Metric with no labels and two MetricPoints with
timestamps: ~~~~ # TYPE foo gauge foo 17.0 123 foo 18.0 456 ~~~~
4.1.5.2. Counter
The MetricPoint's Total Value Sample MetricName MUST have the suffix
"_total". If present the MetricPoint's Created Value Sample
MetricName MUST have the suffix "_created".
An example with a Metric with no labels, and a MetricPoint with no
timestamp and no created: ~~~~ # TYPE foo counter foo_total 17.0 ~~~~
An example with a Metric with no labels, and a MetricPoint with a
timestamp and no created: ~~~~ # TYPE foo counter foo_total 17.0
1520879607.789 ~~~~
An example with a Metric with no labels, and a MetricPoint with no
timestamp and a created: ~~~~ # TYPE foo counter foo_total 17.0
foo_created 1520430000.123 ~~~~
An example with a Metric with no labels, and a MetricPoint with a
timestamp and a created: ~~~~ # TYPE foo counter foo_total 17.0
1520879607.789 foo_created 1520430000.123 1520879607.789 ~~~~
Exemplars MAY be attached to the MetricPoint's Total sample.
4.1.5.3. StateSet
The Sample MetricName for the value of a MetricPoint for a
MetricFamily of type StateSet MUST NOT have a suffix.
StateSets MUST have one sample per State in the MetricPoint. Each
State's sample MUST have a label with the MetricFamily name as the
label name and the State name as the label value. The State sample's
value MUST be 1 if the State is true and MUST be 0 if the State is
false.
An example with the states "a", "bb", and "ccc" in which only the
value b is enabled and the metric name is foo:
# TYPE foo stateset
foo{foo="a"} 0
foo{foo="bb"} 1
foo{foo="ccc"} 0
Hartmann, et al. Expires May 29, 2021 [Page 20]
Internet-Draft OpenMetrics November 2020
An example of an "entity" label on the Metric: ~~~~ # TYPE foo
stateset foo{entity="controller",foo="a"} 1.0
foo{entity="controller",foo="bb"} 0.0
foo{entity="controller",foo="ccc"} 0.0 foo{entity="replica",foo="a"}
1.0 foo{entity="replica",foo="bb"} 0.0
foo{entity="replica",foo="ccc"} 1.0 ~~~~
4.1.5.4. Info
The Sample MetricName for the value of a MetricPoint for a
MetricFamily of type Info MUST have the suffix "_info". The Sample
value MUST always be 1.
An example of a Metric with no labels, and one MetricPoint value with
"name" and "version" labels: ~~~~ # TYPE foo info
foo_info{name="pretty name",version="8.2.7"} 1 ~~~~
An example of a Metric with label "entity" and one MetricPoint value
with "name" and "version" labels: ~~~~ # TYPE foo info
foo_info{entity="controller",name="pretty name",version="8.2.7"} 1.0
foo_info{entity="replica",name="prettier name",version="8.1.9"} 1.0
~~~~
Metric labels and MetricPoint value labels MAY be in any order.
4.1.5.5. Summary
If present, the MetricPoint's Sum Value Sample MetricName MUST have
the suffix "_sum". If present, the MetricPoint's Count Value Sample
MetricName MUST have the suffix "_count". If present, the
MetricPoint's Created Value Sample MetricName MUST have the suffix
"_created". If present, the MetricPoint's Quantile Values MUST
specify the quantile measured using a label with a label name of
"quantile" and with a label value of the quantile measured.
An example of a Metric with no labels and a MetricPoint with Sum,
Count and Created values: ~~~~ # TYPE foo summary foo_count 17.0
foo_sum 324789.3 foo_created 1520430000.123 ~~~~
An example of a Metric with no labels and a MetricPoint with two
quantiles: ~~~~ # TYPE foo summary foo{quantile="0.95"} 123.7
foo{quantile="0.99"} 150.0 ~~~~
Quantiles MAY be in any order.
Hartmann, et al. Expires May 29, 2021 [Page 21]
Internet-Draft OpenMetrics November 2020
4.1.5.6. Histogram
The MetricPoint's Bucket Values Sample MetricNames MUST have the
suffix "_bucket". If present, the MetricPoint's Sum Value Sample
MetricName MUST have the suffix "_sum". If present, the
MetricPoint's Created Value Sample MetricName MUST have the suffix
"_created". If and only if a Sum Value is present in a MetricPoint,
then the MetricPoint's +Inf Bucket value MUST also appear in a Sample
with a MetricName with the suffix "_count".
Buckets MUST be sorted in number increasing order of "le", and the
value of the "le" label MUST follow the rules for Canonical Numbers.
An example of a Metric with no labels and a MetricPoint with Sum,
Count, and Created values, and with 12 buckets. A wide and atypical
but valid variety of "le" values is shown on purpose: ~~~~ # TYPE foo
histogram foo_bucket{le="0.0"} 0 foo_bucket{le="1e-05"} 0
foo_bucket{le="0.0001"} 5 foo_bucket{le="0.1"} 8 foo_bucket{le="1.0"}
10 foo_bucket{le="10.0"} 11 foo_bucket{le="100000.0"} 11
foo_bucket{le="1e+06"} 15 foo_bucket{le="1e+23"} 16
foo_bucket{le="1.1e+23"} 17 foo_bucket{le="+Inf"} 17 foo_count 17
foo_sum 324789.3 foo_created 1520430000.123 ~~~~
4.1.5.6.1. Exemplars
Exemplars without Labels MUST represent an empty LabelSet as {}.
An example of Exemplars showcasing several valid cases: The "0.01"
bucket has no Exemplar. The 0.1 bucket has an Exemplar with no
Labels. The 1 bucket has an Exemplar with one Label. The 10 bucket
has an Exemplar with a Label and a timestamp. In practice all
buckets SHOULD have the same style of Exemplars. ~~~~ # TYPE foo
histogram foo_bucket{le="0.01"} 0 foo_bucket{le="0.1"} 8 # {} 0.054
foo_bucket{le="1"} 11 # {trace_id="KOO5S4vxi0o"} 0.67
foo_bucket{le="10"} 17 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
foo_bucket{le="+Inf"} 17 foo_count 17 foo_sum 324789.3 foo_created
1520430000.123 ~~~~
4.1.5.7. GaugeHistogram
The MetricPoint's Bucket Values Sample MetricNames MUST have the
suffix "_bucket". If present, the MetricPoint's Sum Value Sample
MetricName MUST have the suffix "_gsum". If present, the
MetricPoint's Created Value Sample MetricName MUST have the suffix
"_created". If and only if a Sum Value is present in a MetricPoint,
then the MetricPoint's +Inf Bucket value MUST also appear in a Sample
with a MetricName with the suffix "_gcount".
Hartmann, et al. Expires May 29, 2021 [Page 22]
Internet-Draft OpenMetrics November 2020
Buckets MUST be sorted in number increasing order of "le", and the
value of the "le" label MUST follow the rules for Canonical Numbers.
An example of a Metric with no labels, and one MetricPoint value with
no Exemplar with no Exemplars in the buckets: ~~~~ # TYPE foo
gaugehistogram foo_bucket{le="0.01"} 20.0 foo_bucket{le="0.1"} 25.0
foo_bucket{le="1"} 34.0 foo_bucket{le="10"} 34.0
foo_bucket{le="+Inf"} 42.0 foo_gcount 42.0 foo_gsum 3289.3
foo_created 1520430000.123 ~~~~
4.1.5.8. Unknown
The sample metric name for the value of the MetricPoint for a
MetricFamily of type Unknown MUST NOT have a suffix.
An example with a Metric with no labels and a MetricPoint with no
timestamp: ~~~~ # TYPE foo unknown foo 42.23 ~~~~
4.2. Protobuf format
4.2.1. Overall Structure
Protobuf messages MUST be encoded in binary and MUST have
"application/openmetrics-protobuf; version=1.0.0" as their content
type.
All payloads MUST be a single binary encoded MetricSet message, as
defined by the OpenMetrics protobuf schema.
4.2.1.1. Version
The protobuf format MUST follow the proto3 version of the protocol
buffer language.
4.2.1.2. Strings
All string fields MUST be UTF-8 encoded.
4.2.1.3. Timestamps
Timestamp representations in the OpenMetrics protobuf schema MUST
follow the published google.protobuf.Timestamp [timestamp] message.
The timestamp message MUST be in Unix epoch seconds as an int64 and a
non-negative fraction of a second at nanosecond resolution as an
int32 that counts forward from the seconds timestamp component. It
MUST be within 0 to 999,999,999 inclusive.
Hartmann, et al. Expires May 29, 2021 [Page 23]
Internet-Draft OpenMetrics November 2020
4.2.2. Protobuf schema
syntax = "proto3";
// The OpenMetrics protobuf schema which defines the protobuf wire
// format.
// Ensure to interpret "required" as semantically required for a valid
// message.
// All string fields MUST be UTF-8 encoded strings.
package openmetrics;
import "google/protobuf/timestamp.proto";
// The top-level container type that is encoded and sent over the wire.
message MetricSet {
// Each MetricFamily has one or more MetricPoints for a single Metric.
repeated MetricFamily metric_families = 1;
}
// One or more Metrics for a single MetricFamily, where each Metric
// has one or more MetricPoints.
message MetricFamily {
// Required.
string name = 1;
// Optional.
MetricType type = 2;
// Optional.
string unit = 3;
// Optional.
string help = 4;
// Optional.
repeated Metric metrics = 5;
}
// The type of a Metric.
enum MetricType {
// Unknown must use unknown MetricPoint values.
UNKNOWN = 0;
// Gauge must use gauge MetricPoint values.
GAUGE = 1;
// Counter must use counter MetricPoint values.
COUNTER = 2;
// State set must use state set MetricPoint values.
STATE_SET = 3;
Hartmann, et al. Expires May 29, 2021 [Page 24]
Internet-Draft OpenMetrics November 2020
// Info must use info MetricPoint values.
INFO = 4;
// Histogram must use histogram value MetricPoint values.
HISTOGRAM = 5;
// Gauge histogram must use histogram value MetricPoint values.
GAUGE_HISTOGRAM = 6;
// Summary quantiles must use summary value MetricPoint values.
SUMMARY = 7;
}
// A single metric with a unique set of labels within a metric family.
message Metric {
// Optional.
repeated Label labels = 1;
// Optional.
repeated MetricPoint metric_points = 2;
}
// A name-value pair. These are used in multiple places: identifying
// timeseries, value of INFO metrics, and exemplars in Histograms.
message Label {
// Required.
string name = 1;
// Required.
string value = 2;
}
// A MetricPoint in a Metric.
message MetricPoint {
// Required.
oneof value {
UnknownValue unknown_value = 1;
GaugeValue gauge_value = 2;
CounterValue counter_value = 3;
HistogramValue histogram_value = 4;
StateSetValue state_set_value = 5;
InfoValue info_value = 6;
SummaryValue summary_value = 7;
}
// Optional.
google.protobuf.Timestamp timestamp = 8;
}
// Value for UNKNOWN MetricPoint.
message UnknownValue {
Hartmann, et al. Expires May 29, 2021 [Page 25]
Internet-Draft OpenMetrics November 2020
// Required.
oneof value {
double double_value = 1;
int64 int_value = 2;
}
}
// Value for GAUGE MetricPoint.
message GaugeValue {
// Required.
oneof value {
double double_value = 1;
int64 int_value = 2;
}
}
// Value for COUNTER MetricPoint.
message CounterValue {
// Required.
oneof total {
double double_value = 1;
uint64 int_value = 2;
}
// The time values began being collected for this counter.
// Optional.
google.protobuf.Timestamp created = 3;
// Optional.
Exemplar exemplar = 4;
}
// Value for HISTOGRAM or GAUGE_HISTOGRAM MetricPoint.
message HistogramValue {
// Optional.
oneof sum {
double double_value = 1;
int64 int_value = 2;
}
// Optional.
uint64 count = 3;
// The time values began being collected for this histogram.
// Optional.
google.protobuf.Timestamp created = 4;
// Optional.
Hartmann, et al. Expires May 29, 2021 [Page 26]
Internet-Draft OpenMetrics November 2020
repeated Bucket buckets = 5;
// Bucket is the number of values for a bucket in the histogram
// with an optional exemplar.
message Bucket {
// Required.
uint64 count = 1;
// Optional.
double upper_bound = 2;
// Optional.
Exemplar exemplar = 3;
}
}
message Exemplar {
// Required.
double value = 1;
// Optional.
google.protobuf.Timestamp timestamp = 2;
// Labels are additional information about the exemplar value
// (e.g. trace id).
// Optional.
repeated Label label = 3;
}
// Value for STATE_SET MetricPoint.
message StateSetValue {
// Optional.
repeated State states = 1;
message State {
// Required.
bool enabled = 1;
// Required.
string name = 2;
}
}
// Value for INFO MetricPoint.
message InfoValue {
// Optional.
repeated Label info = 1;
}
Hartmann, et al. Expires May 29, 2021 [Page 27]
Internet-Draft OpenMetrics November 2020
// Value for SUMMARY MetricPoint.
message SummaryValue {
// Optional.
oneof sum {
double double_value = 1;
int64 int_value = 2;
}
// Optional.
uint64 count = 2;
// The time sum and count values began being collected for this
// summary.
// Optional.
google.protobuf.Timestamp created = 3;
// Optional.
repeated Quantile quantile = 4;
message Quantile {
// Required.
double quantile = 1;
// Required.
double value = 2;
}
}
5. Design Considerations
5.1. Scope
OpenMetrics is intended to provide telemetry for online systems. It
runs over protocols which do not provide hard or soft real time
guarantees, so it can not make any real time guarantees itself.
Latency and jitter properties of OpenMetrics are as imprecise as the
underlying network, operating systems, CPUs, and the like. It is
sufficiently accurate for aggregations to be used as a basis for
decision-making, but not to reflect individual events.
Systems of all sizes should be supported, from applications that
receive a few requests an hour up to monitoring bandwidth usage on a
400Gb network port. Aggregation and analysis of transmitted
telemetry should be possible over arbitrary time periods.
It is intended to transport snapshots of state at the time of data
transmission at a regular cadence.
Hartmann, et al. Expires May 29, 2021 [Page 28]
Internet-Draft OpenMetrics November 2020
5.1.1. Out of scope
How ingestors discover which exposers exist, and vice-versa, is out
of scope for and thus not defined in this standard.
5.2. Extensions and Improvements
This first version of OpenMetrics is based upon well established and
de facto standard Prometheus text format 0.0.4, deliberately without
adding major syntactic or semantic extensions, or optimisations on
top of it. For example no attempt has been made to make the text
representation of Histogram buckets more compact, relying on
compression in the underlying stack to deal with their repetitive
nature.
This is a deliberate choice, so that the standard can take advantage
of the adoption and momentum of the existing user base. This ensures
a relatively easy transition from the Prometheus text format 0.0.4.
It also ensures that there is a basic standard which is easy to
implement. This can be built upon in future versions of the
standard. The intention is that future versions of the standard will
always require support for this 1.0 version, both syntactically and
semantically.
We want to allow monitoring systems to get usable information from an
OpenMetrics exposition without undue burden. If one were to strip
away all metadata and structure and just look at an OpenMetrics
exposition as an unordered set of samples that should be usable on
its own. As such, there are also no opaque binary types, such as
sketches or t-digests which could not be expressed as a mix of gauges
and counters as they would require custom parsing and handling.
This principle is applied consistently throughout the standard. For
example a MetricFamily's unit is duplicated in the name so that the
unit is available for systems that don't understand the unit
metadata. The "le" label is a normal label value, rather than
getting its own special syntax, so that ingestors don't have to add
special histogram handling code to ingest them. As a further
example, there are no composite data types. For example, there is no
geolocation type for latitude/longitude as this can be done with
separate gauge metrics.
5.3. Units and Base Units
For consistency across systems and to avoid confusion, units are
largely based on SI base units. Base units include seconds, bytes,
Hartmann, et al. Expires May 29, 2021 [Page 29]
Internet-Draft OpenMetrics November 2020
joules, grams, meters, ratios, volts, amperes, and celsius. Units
should be provided where they are applicable.
For example, having all duration metrics in seconds, there is no risk
of having to guess whether a given metric is nanoseconds,
microseconds, milliseconds, seconds, minutes, hours, days or weeks
nor having to deal with mixed units. By choosing unprefixed units,
we avoid situations like ones in which kilomilliseconds were the
result of emergent behaviour of complex systems.
As values can be floating point, sub-base-unit precision is built
into the standard.
Similarly, mixing bits and bytes is confusing, so bytes are chosen as
the base. While Kelvin is a better base unit in theory, in practice
most existing hardware exposes Celsius. Kilograms are the SI base
unit, however the kilo prefix is problematic so grams are chosen as
the base unit.
While base units SHOULD be used in all possible cases, Kelvin is a
well-established unit which MAY be used instead of Celsius for use
cases such as color or black body temperatures where a comparison
between a Celsius and Kelvin metric are unlikely.
Ratios are the base unit, not percentages. Where possible, raw data
in the form of gauges or counters for the given numerator and
denominator should be exposed. This has better mathematical
properties for analysis and aggregation in the ingestors.
Decibels are not a base unit as firstly, deci is a SI prefix and
secondly, bels are logarithmic. To expose signal/energy/power ratios
exposing the ratio directly would be better, or better still the raw
power/energy if possible. Floating point exponents are more than
sufficient to cover even extreme scientific uses. An electron volt
(~1e-19 J) all the way up to the energy emitted by a supernova (~1e44
J) is 63 orders of magnitude, and a 64-bit floating point number can
cover over 2000 orders of magnitude.
If non-base units can not be avoided and conversion is not feasible,
the actual unit should still be included in the metric name for
clarity. For example, joule is the base unit for both energy and
power, as watts can be expressed as a counter with a joule unit. In
practice a given 3rd party system may only expose watts, so a gauge
expressed in watts would be the only realistic choice in that case.
Not all MetricFamilies have units. For example a count of HTTP
requests wouldn't have a unit. Technically the unit would be HTTP
requests, but in that sense the entire MetricFamily name is the unit.
Hartmann, et al. Expires May 29, 2021 [Page 30]
Internet-Draft OpenMetrics November 2020
Going to that extreme would not be useful. The possibility of having
good axes on graphs in downstream systems for human consumption
should always be kept in mind.
5.4. Statelessness
The wire format defined by OpenMetrics is stateless across
expositions. What information has been exposed before MUST have no
impact on future expositions. Each exposition is a self-contained
snapshot of the current state of the exposer.
The same self-contained exposition MUST be provided to existing and
new ingestors.
A core design choice is that exposers MUST NOT exclude a metric
merely because it has had no recent changes, or observations. An
exposer must not make any assumptions about how often ingestors are
consuming expositions.
5.5. Exposition Across Time and Metric Evolution
Metrics are most useful when their evolution over time can be
analysed, so accordingly expositions must make sense over time.
Thus, it is not sufficient for one single exposition on its own to be
useful and valid. Some changes to metric semantics can also break
downstream users.
Parsers commonly optimize by caching previous results. Thus,
changing the order in which labels are exposed across expositions
SHOULD be avoided even though it is technically not breaking This
also tends to make writing unit tests for exposition easier.
Metrics and samples SHOULD NOT appear and disappear from exposition
to exposition, for example a counter is only useful if it has
history. In principle, a given Metric should be present in
exposition from when the process starts until the process terminates.
It is often not possible to know in advance what Metrics a
MetricFamily will have over the lifetime of a given process (e.g. a
label value of a latency histogram is a HTTP path, which is provided
by an end user at runtime), but once a counter-like Metric is exposed
it should continue to be exposed until the process terminates. That
a counter is not getting increments doesn't invalidate that it still
has its current value. There are cases where it may make sense to
stop exposing a given Metric; see the section on Missing Data.
In general changing a MetricFamily's type, or adding or removing a
label from its Metrics will be breaking to ingestors.
Hartmann, et al. Expires May 29, 2021 [Page 31]
Internet-Draft OpenMetrics November 2020
A notable exception is that adding a label to the value of an Info
MetricPoints is not breaking. This is so that you can add additional
information to an existing Info MetricFamily where it makes sense to
be, rather than being forced to create a brand new info metric with
an additional label value. ingestor systems should ensure that they
are resilient to such additions.
Changing a MetricFamily's Help is not breaking. For values where it
is possible, switching between floats and ints is not breaking.
Adding a new state to a stateset is not breaking. Adding unit
metadata where it doesn't change the metric name is not breaking.
Histogram buckets SHOULD NOT change from exposition to exposition, as
this is likely to both cause performance issues and break ingestors
and cause. Similarly all expositions from any consistent binary and
environment of an application SHOULD have the same buckets for a
given Histogram MetricFamily, so that they can be aggregated by all
ingestors without ingestors having to implement histogram merging
logic for heterogeneous buckets. An exception might be occasional
manual changes to buckets which are considered breaking, but may be a
valid tradeoff when performance characteristics change due to a new
software release.
Even if changes are not technically breaking, they still carry a
cost. For example frequent changes may cause performance issues for
ingestors. A Help string that varies from exposition to exposition
may cause each Help value to be stored. Frequently switching between
int and float values could prevent efficient compression.
5.6. NaN
NaN is a number like any other in OpenMetrics, usually resulting from
a division by zero such as for a summary quantile if there have been
no observations recently. NaN does not have any special meaning in
OpenMetrics, and in particular MUST NOT be used as a marker for
missing or otherwise bad data.
5.7. Missing Data
There are valid cases when data stops being present. For example a
filesystem can be unmounted and thus its Gauge Metric for free disk
space no longer exists. There is no special marker or signal for
this situation. Subsequent expositions simply do not include this
Metric.
Hartmann, et al. Expires May 29, 2021 [Page 32]
Internet-Draft OpenMetrics November 2020
5.8. Exposition Performance
Metrics are only useful if they can be collected in reasonable time
frames. Metrics that take minutes to expose are not considered
useful.
As a rule of thumb, exposition SHOULD take no more than a second.
Metrics from legacy systems serialized through OpenMetrics may take
longer. For this reason, no hard performance assumptions can be
made.
Exposition SHOULD be of the most recent state. For example, a thread
serving the exposition request SHOULD NOT rely on cached values, to
the extent it is able to bypass any such caching
5.9. Concurrency
For high availability and ad-hoc access a common approach is to have
multiple ingestors. To support this, concurrent expositions MUST be
supported. All BCPs for concurrent systems SHOULD be followed,
common pitfalls include deadlocks, race conditions, and overly-coarse
grained locking preventing expositions progressing concurrently.
5.10. Metric Naming and Namespaces
EDITOR'S NOTE: This section might be good for a BCP paper.
We aim for a balance between understandability, avoiding clashes, and
succinctness in the naming of metrics and label names. Names are
separated through underscores, so metric names end up being in
"snake_case".
To take an example "http_request_seconds" is succinct but would clash
between large numbers of applications, and it's also unclear exactly
what this metric is measuring. For example, it might be before or
after auth middleware in a complex system.
Metric names should indicate what piece of code they come from. So a
company called A Company Manufacturing Everything might prefix all
metrics in their code with "acme_", and if they had a HTTP router
library measuring latency it might have a metric such as
"acme_http_router_request_seconds" with a Help string indicating that
it is the overall latency.
It is not the aim to prevent all potential clashes across all
applications, as that would require heavy handed solutions such as a
global registry of metric namespaces or very long namespaces based on
Hartmann, et al. Expires May 29, 2021 [Page 33]
Internet-Draft OpenMetrics November 2020
DNS. Rather the aim is to keep to a lightweight informal approach,
so that for a given application that it is very unlikely that there
is clash across its constituent libraries.
Across a given deployment of a monitoring system as a whole the aim
is that clashes where the same metric name means different things are
uncommon. For example acme_http_router_request_seconds might end up
in hundreds of different applications developed by A Company
Manufacturing Everything, which is normal. If Another Corporation
Making Entities also used the metric name
acme_http_router_request_seconds in their HTTP router that's also
fine. If applications from both companies were being monitored by
the same monitoring system the clash is undesirable, but acceptable
as no application is trying to expose both names and no one target is
trying to (incorrectly) expose the same metric name twice. If an
application wished to contain both My Example Company's and Mega
Exciting Company's HTTP router libraries that would be a problem, and
one of the metric names would need to be changed somehow.
As a corollary, the more public a library is the better namespaced
its metric names should be to reduce the risk of such scenarios
arising. acme_ is not a bad choice for internal use within a company,
but these companies might for example choose the prefixes
acmeverything_ or acorpme_ for code shared outside their company.
After namespacing by company or organisation, namespacing and naming
should continue by library/subsystem/application fractally as needed
such as the http_router library above. The goal is that if you are
familiar with the overall structure of a codebase, you could make a
good guess at where the instrumentation for a given metric is given
its metric name.
For a common very well known existing piece of software, the name of
the software itself may be sufficiently distinguishing. For example
bind_ is probably sufficient for the DNS software, even though
isc_bind_ would be the more usual naming.
Metric names prefixed by scrape_ are used by ingestors to attach
information related to individual expositions, so should not be
exposed by applications directly. Metrics that have already been
consumed and passed through a general purpose monitoring system may
include such metric names on subsequent expositions. If an exposer
wishes to provide information about an individual exposition, a
metric prefix such as myexposer_scrape_ may be used. A common
example is a gauge myexposer_scrape_duration_seconds for how long
that exposition took from the exposer's standpoint.
Hartmann, et al. Expires May 29, 2021 [Page 34]
Internet-Draft OpenMetrics November 2020
Within the Prometheus ecosystem a set of per-process metrics has
emerged that are consistent across all implementations, prefixed with
process_. For example for open file ulimits the MetricFamiles
process_open_fds and process_max_fds gauges provide both the current
and maximum value. (These names are legacy, if such metrics were
defined today they would be more likely called process_fds_open and
process_fds_limit). In general it is very challengings to get names
with identical semantics like this, which is why different
instrumentation should use different names.
Avoid redundancy in metric names. Avoid substrings like "metric",
"timer", "stats", "counter", "total", "float64" and so on - by virtue
of being a metric with a given type (and possibly unit) exposed via
OpenMetrics information like this is already implied so should not be
included explicitly. You should not include label names of a metric
in the metric name for the same reasons, and in addition subsequent
aggregation of the metric by a monitoring system could make such
information incorrect.
Avoid including implementation details from other layers of your
monitoring system in the metric names contained in your
instrumentation. For example a MetricFamily name should not contain
the string "openmetrics" merely because it happens to be currently
exposed via OpenMetrics somewhere, or "prometheus" merely because
your current monitoring system is Prometheus.
5.11. Label Namespacing
For label names no explicit namespacing by company or library is
recommended, namespacing from the metric name is sufficient for this
when considered against the length increase of the label name.
However some minimal care to avoid common clashes is recommended.
There are label names such as region, zone, cluster,
availability_zone, az, datacenter, dc, owner, customer, stage,
service, team, job, instance, environment, and env which are highly
likely to clash with labels used to identify targets which a general
purpose monitoring system may add. Try to avoid them, adding minimal
namespacing may be appropriate in these cases.
The label name "type" is highly generic and should be avoided. For
example for HTTP-related metrics "method" would be a better label
name if you were distinguishing between GET, POST, and PUT requests.
While there is metadata about metric names such as HELP, TYPE and
UNIT there is no metadata for label names. This is as it would be
bloating the format for little gain. Out-of-band documentation is
one way for exposers could present this their ingestors.
Hartmann, et al. Expires May 29, 2021 [Page 35]
Internet-Draft OpenMetrics November 2020
5.12. Metric Names versus Labels
There are situations in which both using multiple Metrics within a
MetricFamily or multiple MetricFamilies seem to make sense. Summing
or averaging aMetricFamily should be meaningful even if it's not
always useful. For example, mixing voltage and fan speed is not
meaningful.
As a reminder, OpenMetrics is built with the assumption that
ingestors can process and perform aggregations on data.
Exposing a total sum alongside other metrics is wrong, as this would
result in double-counting upon aggregation in downstream ingestors.
~~~~ wrong_metric{label="a"} 1 wrong_metric{label="b"} 6
wrong_metric{label="total"} 7 ~~~~
Labels of a Metric should be to the minimum needed to ensure
uniqueness as every extra label is one more that users need to
consider when determining what Labels to work with downstream.
Labels which could be applied many MetricFamilies are candidates for
being moved into _info metrics similar to database [normalization].
If virtually all users of a Metric could be expected to want the
additional label, it may be a better trade-off to add it to all
MetricFamilies. For example if you had a MetricFamily relating to
different SQL statements where uniqueness was provided by a label
containing a hash of the full SQL statements, it would be okay to
have another label with the first 500 characters of the SQL statement
for human readability.
Experience has shown that downstream ingestors find it easier to work
with separate total and failure MetricFamiles rather than using
{result="success"} and {result="failure"} Labels within one
MetricFamily. Also it is usually better to expose separate read &
write and send & receive MetricFamiles as full duplex systems are
common and downstream ingestors are more likely to care about those
values separately than in aggregate.
All of this is not as easy as it may sound. It's an area where
experience and engineering trade-offs by domain-specific experts in
both exposition and the exposed system are required to find a good
balance. Metric and Label Name Characters
OpenMetrics builds on the existing widely adopted Prometheus text
exposition format and the ecosystem which formed around it.
Backwards compatibility is a core design goal. Expanding or
contracting the set of characters that are supported by the
Prometheus text format would work against that goal. Breaking
backwards compatibility would have wider implications than just the
Hartmann, et al. Expires May 29, 2021 [Page 36]
Internet-Draft OpenMetrics November 2020
wire format. In particular, the query languages created or adopted
to work with data transmitted within the Prometheus ecosystem rely on
these precise character sets. Label values support full UTF-8, so
the format can represent multi-lingual metrics.
5.13. Types of Metadata
Metadata can come from different sources. Over the years, two main
sources have emerged. While they are often functionally the same, it
helps in understanding to talk about their conceptual differences.
"Target metadata" is metadata commonly external to an exposer.
Common examples would be data coming from service discovery, a CMDB,
or similar, like information about a datacenter region, if a service
is part of a particular deployment, or production or testing. This
can be achieved by either the exposer or the ingestor adding labels
to all Metrics that capture this metadata. Doing this through the
ingestor is preferred as it is more flexible and carries less
overhead. On flexibility, the hardware maintenance team might care
about which server rack a machine is located in, whereas the database
team using that same machine might care that it contains replica
number 2 of the production database. On overhead, hardcoding or
configuring this information needs an additional distribution path.
"Exposer metadata" is coming from within an exposer. Common examples
would be software version, compiler version, or Git commit SHA.
5.13.1. Supporting Target Metadata in both Push-based and Pull-based
Systems
In push-based consumption, it is typical for the exposer to provide
the relevant target metadata to the ingestor. In pull-based
consumption the push-based approach could be taken, but more
typically the ingestor already knows the metadata of the target
a-priori such as from a machine database or service discovery system,
and associates it with the metrics as it consumes the exposition.
OpenMetrics is stateless and provides the same exposition to all
ingestors, which is in conflict with the push-style approach. In
addition the push-style approach would break pull-style ingestors, as
unwanted metadata would be exposed.
One approach would be for push-style ingestors to provide target
metadata based on operator configuration out-of-band, for example as
a HTTP header. While this would transport target metadata for push-
style ingestors, and is not precluded by this standard, it has the
disadvantage that even though pull-style ingestors should use their
Hartmann, et al. Expires May 29, 2021 [Page 37]
Internet-Draft OpenMetrics November 2020
own target metadata, it is still often useful to have access to the
metadata the exposer itself is aware of.
The preferred solution is to provide this target metadata as part of
the exposition, but in a way that does not impact on the exposition
as a whole. Info MetricFamilies are designed for this. An exposer
may include an Info MetricFamily called "target" with a single Metric
with no labels with the metadata. An example in the text format
might be: ~~~~ # TYPE target info # HELP target Target metadata targe
t_info{env="prod",hostname="myhost",datacenter="sdc",region="europe",
owner="frontend"} 1 ~~~~
When an exposer is providing this metric for this purpose it SHOULD
be first in the exposition. This is for efficiency, so that
ingestors relying on it for target metadata don't have to buffer up
the rest of the exposition before applying business logic based on
its content.
Exposers MUST NOT add target metadata labels to all Metrics from an
exposition, unless explicitly configured for a specific ingestor.
Exposers MUST NOT prefix MetricFamily names or otherwise vary
MetricFamily names based on target metadata. Generally, the same
Label should not appear on every Metric of an exposition, but there
are rare cases where this can be the result of emergent behaviour.
Similarly all MetricFamily names from an exposer may happen to share
a prefix in very small expositions. For example an application
written in the Go language by A Company Manufacturing Everything
would likely include metrics with prefixes of acme_, go_, process_,
and metric prefixes from any 3rd party libraries in use.
Exposers can expose exposer metadata as Info MetricFamilies.
The above discussion is in the context of individual exposers. An
exposition from a general purpose monitoring system may contain
metrics from many individual targets, and thus may expose multiple
target info Metrics. The metrics may already have had target
metadata added to them as labels as part of ingestion. The metric
names MUST NOT be varied based on target metadata. For example it
would be incorrect for all metrics to end up being prefixed with
staging_ even if they all originated from targets in a staging
environment).
5.14. Client Calculations and Derived Metrics
Exposers should leave any math or calculation up to ingestors. A
notable exception is the Summary quantile which is unfortunately
required for backwards compatibility. Exposition should be of raw
values which are useful over arbitrary time periods.
Hartmann, et al. Expires May 29, 2021 [Page 38]
Internet-Draft OpenMetrics November 2020
As an example, you should not expose a gauge with the average rate of
increase of a counter over the last 5 minutes. Letting the ingestor
calculate the increase over the data points they have consumed across
expositions has better mathematical properties and is more resilient
to scrape failures.
Another example is the average event size of a histogram/summary.
Exposing the average rate of increase of a counter since an
application started or since a Metric was created has the problems
from the earlier example and it also prevents aggregation.
Standard deviation also falls into this category. Exposing a sum of
squares as a counter would be the correct approach. It was not
included in this standard as a Histogram value because 64bit floating
point precision is not sufficient for this to work in practice. Due
to the squaring only half the 53bit mantissa would be available in
terms of precision. As an example a histogram observing 10k events
per second would lose precision within 2 hours. Using 64bit integers
would be no better due to the loss of the floating decimal point
because a nanosecond resolution integer typically tracking events of
a second in length would overflow after 19 observations. This design
decision can be revisited when 128bit floating point numbers become
common.
Another example is to avoid exposing a request failure ratio,
exposing separate counters for failed requests and total requests
instead.
5.15. Number Types
For a counter that was incremented a million times per second it
would take over a century to begin to lose precision with a float64
as it has a 53 bit mantissa. Yet a 100 Gbps network interface's
octet throughput precision could begin to be lost with a float64
within around 20 hours. While losing 1KB of precision over the
course of years for a 100Gbps network interface is unlikely to be a
problem in practice, int64s are an option for integral data with such
a high throughput.
Summary quantiles must be float64, as they are estimates and thus
fundamentally inaccurate.
5.16. Exposing Timestamps
One of the core assumptions of OpenMetrics is that exposers expose
the most up to date snapshot of what they're exposing.
Hartmann, et al. Expires May 29, 2021 [Page 39]
Internet-Draft OpenMetrics November 2020
While there are limited use cases for attaching timestamps to exposed
data, these are very uncommon. Data which had timestamps previously
attached, in particular data which has been ingested into a general
purpose monitoring system may carry timestamps. Live or raw data
should not carry timestamps. It is valid to expose the same metric
MetricPoint value with the same timestamp across expositions, however
it is invalid to do so if the underlying metric is now missing.
Time synchronization is a hard problem and data should be internally
consistent in each system. As such, ingestors should be able to
attach the current timestamp from their perspective to data rather
than based on the system time of the exposer device.
With timestamped metrics it is not generally possible to detect the
time when a Metric went missing across expositions. However with
non-timestamped metrics the ingestor can use its own timestamp from
the exposition where the Metric is no longer present.
All of this is to say that, in general, MetricPoint timestamps should
not be exposed, as it should be up to the ingestor to apply their own
timestamps to samples they ingest.
5.16.1. Tracking When Metrics Last Changed
Presume you had a counter my_counter which was initialized, and then
later incremented by 1 at time 123. This would be a correct way to
expose it in the text format: ~~~~ # HELP my_counter Good increment
example # TYPE my_counter counter my_counter_total 1 ~~~~ As per the
parent section, ingestors should be free to attach their own
timestamps, so this would be incorrect: ~~~~ # HELP my_counter Bad
increment example # TYPE my_counter counter my_counter_total 1 123
~~~~
In case the specific time of the last change of a counter matters,
this would be the correct way: ~~~~ # HELP my_counter Good increment
example # TYPE my_counter counter my_counter_total 1 # HELP
my_counter_last_increment_timestamp_seconds When my_counter was last
incremented # TYPE my_counter_last_increment_timestamp_seconds gauge
# UNIT my_counter_last_increment_timestamp_seconds seconds
my_counter_last_increment_timestamp_seconds 123 ~~~~
By putting the timestamp of last change into its own Gauge as a
value, ingestors are free to attach their own timestamp to both
Metrics.
Experience has shown that exposing absolute timestamps (epoch is
considered absolute here) is more robust than time elapsed, seconds
since, or similar. In either case, they would be gauges. For
Hartmann, et al. Expires May 29, 2021 [Page 40]
Internet-Draft OpenMetrics November 2020
example ~~~~ # TYPE my_boot_time_seconds gauge # HELP
my_boot_time_seconds Boot time of the machine # UNIT
my_boot_time_seconds seconds my_boot_time_seconds 1256060124 ~~~~
Is better than ~~~~ # TYPE my_time_since_boot_seconds gauge # HELP
my_time_since_boot_seconds Time elapsed since machine booted # UNIT
my_time_since_boot_seconds seconds my_time_since_boot_seconds 123
~~~~
Conversely, there are no best practice restrictions on exemplars
timestamps. Keep in mind that due to race conditions or time not
being perfectly synced across devices, that an exemplar timestamp may
appear to be slightly in the future relative to a ingestor's system
clock or other metrics from the same exposition. Similarly it is
possible that a "_created" for a MetricPoint could appear to be
slightly after an exemplar or sample timestamp for that same
MetricPoint.
Keep in mind that there are monitoring systems in common use which
support everything from nanosecond to second resolution, so having
two MetricPoints that have the same timestamp when truncated to
second resolution may cause an apparent duplicate in the ingestor.
In this case the MetricPoint with the earliest timestamp MUST be
used.
5.17. Thresholds
Exposing desired bounds for a system can make sense, but proper care
needs to be taken. For values which are universally true, it can
make sense to emit Gauge metrics for such thresholds. For example, a
data center HVAC system knows the current measurements, the
setpoints, and the alert setpoints. It has a globally valid and
correct view of the desired system state. As a counter example, some
thresholds can change with scale, deployment model, or over time. A
certain amount of CPU usage may be acceptable in one setting and
undesirable in another. Aggregation of values can further change
acceptable values. In such a system, exposing bounds could be
counter-productive.
For example a the maximum size of a queue may be exposed alongside
the number of items currently in the queue like: ~~~~ # HELP
acme_notifications_queue_capacity The capacity of the notifications
queue. # TYPE acme_notifications_queue_capacity gauge
acme_notifications_queue_capacity 10000 # HELP
acme_notifications_queue_length The number of notifications in the
queue. # TYPE acme_notifications_queue_length gauge
acme_notifications_queue_length 42 ~~~~
Hartmann, et al. Expires May 29, 2021 [Page 41]
Internet-Draft OpenMetrics November 2020
5.18. Size Limits
This standard does not prescribe any particular limits on the number
of samples exposed by a single exposition, the number of labels that
may be present, the number of states a stateset may have, the number
of labels in an info value, or metric name/label name/label value/
help character limits.
Specific limits run the risk of preventing reasonable use cases, for
example while a given exposition may have an appropriate number of
labels after passing through a general purpose monitoring system a
few target labels may have been added that would push it over the
limit. Specific limits on numbers such as these would also not
capture where the real costs are for general purpose monitoring
systems. These guidelines are thus both to aid exposers and
ingestors in understanding what is reasonable.
On the other hand, an exposition which is too large in some dimension
could cause significant performance problems compared to the benefit
of the metrics exposed. Thus some guidelines on the size of any
single exposition would be useful.
ingestors may choose to impose limits themselves, for in particular
to prevent attacks or outages. Still, ingestors need to consider
reasonable use cases and try not to disproportionately impact them.
If any single value/metric/exposition exceeds such limits then the
whole exposition must be rejected.
In general there are three things which impact the performance of a
general purpose monitoring system ingestion time series data: the
number of unique time series, the number of samples over time in
those series, and the number of unique strings such as metric names,
label names, label values, and HELP. ingestors can control how often
they ingest, so that aspect does not need further consideration.
The number of unique time series is roughly equivalent to the number
of non-comment lines in the text format. As of 2020, 10 million time
series in total is considered a large amount and is commonly the
order of magnitude of the upper bound of any single-instance
ingestor. Any single exposition should not go above 10k time series
without due diligence. One common consideration is horizontal
scaling: What happens if you scale your instance count by 1-2 orders
of magnitude? Having a thousand top-of-rack switches in a single
deployment would have been hard to imagine 30 years ago. If a target
was a singleton (e.g. exposing metrics relating to an entire cluster)
then several hundred thousand time series may be reasonable. It is
not the number of unique MetricFamilies or the cardinality of
individual labels/buckets/statesets that matters, it is the total
Hartmann, et al. Expires May 29, 2021 [Page 42]
Internet-Draft OpenMetrics November 2020
order of magnitude of the time series. 1,000 gauges with one Metric
each are as costly as a single gauge with 1,000 Metrics.
If all targets of a particular type are exposing the same set of time
series, then each additional targets' strings poses no incremental
cost to most reasonably modern monitoring systems. If however each
target has unique strings, there is such a cost. As an extreme
example, a single 10k character metric name used by many targets is
on its own very unlikely to be a problem in practice. To the
contrary, a thousand targets each exposing a unique 36 character UUID
is over three times as expensive as that single 10k character metric
name in terms of strings to be stored assuming modern approaches. In
addition, if these strings change over time older strings will still
need to be stored for at least some time, incurring extra cost.
Assuming the 10 million times series from the last paragraph, 100MB
of unique strings per hour might indicate a use case for then the use
case may be more like event logging, not metric time series.
There is a hard 128 UTF-8 character limit on exemplar length, to
prevent misuse of the feature for tracing span data and other event
logging.
6. Security Considerations
Implementors MAY choose to offer authentication, authorization, and
accounting; if they so choose, this SHOULD be handled outside of
OpenMetrics.
All exposer implementations SHOULD be able to secure their HTTP
traffic with TLS 1.2 or later. If an exposer implementation does not
support encryption, operators SHOULD use reverse proxies,
firewalling, and/or ACLs where feasible.
Metric exposition should be independent of production services
exposed to end users; as such, having a /metrics endpoint on ports
like TCP/80, TCP/443, TCP/8080, and TCP/8443 is generally discouraged
for publicly exposed services using OpenMetrics.
7. IANA Considerations
While currently most implementations of the Prometheus exposition
format are using non-IANA-registered ports from an informal registry
at [PrometheusPorts], OpenMetrics can be found on a well-defined
port.
The port assigned by IANA for clients exposing data is <9099
requested for historical consistency>.
Hartmann, et al. Expires May 29, 2021 [Page 43]
Internet-Draft OpenMetrics November 2020
If more than one metric endpoint needs to be reachable at a common IP
address and port, operators might consider using a reverse proxy that
communicates with exposers over localhost addresses. To ease
multiplexing, endpoints SHOULD carry their own name in their path,
i.e. "/node_exporter/metrics". Expositions SHOULD NOT be combined
into one exposition, for the reasons covered under "Supporting target
metadata in both push-based and pull-based systems" and to allow for
independent ingestion without a single point of failure.
OpenMetrics would like to register two MIME types, "application/
openmetrics-text" and "application/openmetrics-proto".
EDITOR'S NOTE: "application/openmetrics-text" is in active use since
2018, "application/openmetrics-proto" is not yet in active use.
EDITOR'S NOTE: We would like to thank Sumeer Bhola, but kramdown 2.x
does not support "Contributor:" any more so we will add this by hand
once consensus has been achieved.
8. References
8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234,
DOI 10.17487/RFC5234, January 2008,
<https://www.rfc-editor.org/info/rfc5234>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
8.2. Informative References
[normalization]
"Database Normalization", n.d.,
<https://en.wikipedia.org/wiki/Database_normalization>.
[PrometheusPorts]
"Prometheus informal port allocation", n.d.,
<https://github.com/prometheus/prometheus/wiki/Default-
port-allocations>.
Hartmann, et al. Expires May 29, 2021 [Page 44]
Internet-Draft OpenMetrics November 2020
[timestamp]
"Go Timestamp ProtoBuf", n.d., <https://github.com/protoco
lbuffers/protobuf/blob/2f6a7546e4539499bc08abc6900dc929782
f5dcd/src/google/protobuf/timestamp.proto>.
Authors' Addresses
Richard Hartmann (editor)
Grafana Labs
Email: richih@richih.org
Ben Kochie
GitLab
Email: superq@gmail.com
Brian Brazil
Robust Perception
Email: brian.brazil@gmail.com
Rob Skillington
Chronosphere
Email: rob.skillington@gmail.com
Hartmann, et al. Expires May 29, 2021 [Page 45]
